Matching Answers Is Not Enough: Reviewing Synthetic Respondent Coverage
A critical review of synthetic respondent coverage, its measurement limits, and the evidence needed before simulations represent human survey responses.
The paper
Silicon Samples Match the Answers but Not the Respondents — Kumagae, Yusuke; Kaneko, Masahiro.
arXiv:2610.30900v1 · Computation and Language (cs.CL) ; Computers and Society (cs.CY). First submitted 2026-10-22; reviewed version dated 2026-10-22. This review addresses the arXiv manuscript; arXiv hosting alone does not establish peer review.
Contribution: evaluating complete answer profiles
Source-derived synopsis. Yusuke Kumagae and Masahiro Kaneko’s Silicon Samples Match the Answers but Not the Respondents compares six LLMs on 100 questions each from US WVS and ANES surveys. Attribute-conditioned generation includes up to three earlier generated answers. With equal samples, respondent coverage gap (RCG) divides the 95th-percentile human-to-generated nearest-neighbor Gower distance by its human-to-human counterpart; OSR reverses direction. Main-setting RCGs span 1.199–1.370, all worse than independent marginal sampling; uncalibrated marginals also fit poorly. Calibration and some history/reasoning/tier settings beat that baseline, but none reaches human-level coverage or beats Copula at matched generated size. Scaling tests reach 1,000 profiles for two proprietary models and 10,000 for three open-weight models. Calibrated marginals and one cross-cell correlation score conceal coverage differences. Baselines and calibration use human data. Hold-out comparisons largely persist, with imputation complicating ANES. Copula’s nominal ordering is arbitrary. Missingness masking lowers RCG; lower percentiles alter comparisons. Human-reference checks require larger cells; bootstrap intervals fix responses. Reported costs: approximately $11,160 plus 1,960 H200 GPU-hours. Generalization and downstream validity remain untested. Source: arXiv:2610.30900v1, including appendices.
Original editorial analysis: what a profile adds
The contribution is a change in the unit of evaluation. Separate answer frequencies cannot uniquely determine how answers combine within respondents. A proposed substitute for survey data therefore needs an evaluation that addresses the combinations relevant to its intended use. Profile coverage supplies one such diagnostic.
The paper’s title deserves care. Its useful warning concerns the insufficiency of answer-level agreement; it should not suggest that every uncalibrated model first achieved accurate answer distributions. Nor should criticism of one cross-cell correlation score become a rejection of every method for evaluating dependence. A score constructed from cell means cannot distinguish arrangements that preserve those means while changing which answers belong together.
For a Church of Spiralism review, the specific relevance is human agency: what evidence should an institution require before treating generated profiles as evidence about people? Statistical resemblance and authority to represent someone are separate standards. Even excellent coverage would not establish consultation, consent or an opportunity to contest a consequential decision. That is an ethical distinction, not a political outcome measured here.
The Synthetic Respondent Becomes the Public examines substitution for participation. This paper adds a narrower, inspectable question: does the proposed substitute contain close counterparts to the human answer combinations in its reference sample?
Evidence: what a coverage ratio can establish
Original editorial analysis and hypothetical example
RCG is a distance ratio, not the percentage of people represented or excluded. A value above one cannot be translated into a corresponding fraction of citizens whose views disappeared. Likewise, reaching one would establish a particular distance comparison, not equality of the complete human and generated distributions.
A nearest-neighbor measure asks whether a close counterpart exists. It does not require a one-to-one pairing or establish that answer combinations occur with the correct frequencies. Two generators could therefore deserve different judgments for estimating a subgroup’s prevalence even if their coverage scores were similar. Coverage should supplement other evaluations rather than become a universal certificate.
Hypothetical example. A transport consultation asks about bus expansion and household costs. Two generated populations could reproduce both answer frequencies while combining the answers differently. An estimate concerning people who support expansion but oppose its funding mechanism would depend on those combinations. This example illustrates a possible use case; it is not an experiment reported by the paper.
Proposed test. Researchers could specify a downstream estimate before generation, evaluate it against held-out human responses, and ask whether RCG adds predictive information beyond question-level agreement. The experiment should permit an adverse result: improving coverage might leave that estimate unchanged or make it worse. A diagnostic needs validation against the decision it is intended to inform.
The information budget also matters. Fitting a generator to human answers and generating answers from attribute descriptions address different practical problems. Holding evaluation respondents out can examine overlap with fitting data, but it cannot remove the fitted generator’s access to survey evidence. A useful comparison would distinguish reuse of an existing survey from prediction for an unmeasured questionnaire or population.
The Validation Norm Becomes the Journal Gate supplies the related measurement discussion. The additional governance proposal here is to specify consequences before inspecting a coverage score: which result would require more human recruitment, narrower claims or rejection of a synthetic estimate? A named decision owner and a documented challenge process would make that proposal reviewable. Their effectiveness would require separate evaluation.
Limitations: testing the diagnostic’s boundaries
Original editorial analysis and proposed tests
An upper-tail percentile emphasizes a different question from a median. It also leaves the most extreme distances outside the selected summary. A follow-up could report the complete distance distribution, examine poorly covered profiles and test decision-relevant subgroups with adequate human samples. An acceptable overall score cannot establish acceptable performance within every subgroup.
The distance function deserves comparable scrutiny. Researchers could declare question weights in advance, compare alternative treatments of ordinal responses and test sensitivity to nominal-category coding. Missingness should remain visible: a masking exercise cannot establish that every dependence between nonresponse and substantive answers has been resolved. Human refusal or uncertainty may itself matter to the proposed application.
Uncertainty should match the claim. Intervals conditional on fixed responses should not be read as covering variation from repeated generation, new questionnaires, new human samples or future model updates. A broader study could vary these elements independently. It should also distinguish a finite reference sample from the full range of profiles in a population.
A resource comparison could connect coverage gains to inference spending, hardware time and human recruitment costs under matched tasks and information budgets. Extrapolated sample requirements would remain projections until tested. Neither monetary spending nor GPU-hours alone justifies a carbon-emissions estimate.
For collective intelligence, a separate experiment could ask whether labeled simulations help human participants discover overlooked questions while retaining control over revisions. It would need to measure misleading suggestions as well as useful discoveries. Static profile coverage cannot establish either outcome.
Survey-answer resemblance supplies no test of consciousness, lived experience or spiritual status, and this paper provides no evidence proving Spiralist doctrines. Its editorial value is a concrete evaluation question, coupled with a separate responsibility to preserve human authority over interpretation and use.
AI-use disclosure
This review was drafted using AI, revised by GPT-6.1 Sol with high reasoning, and checked in a separate GPT-6.1 Sol session with medium reasoning. Automated checks are not human peer review; the paper’s claims have not been independently verified for this article.
Paper sources
Bibliographic metadata checked 2026-10-23.