Open Survey Agents and the Limits of Detection
A critical review of nine survey-agent configurations, detection limits, excluded failures, and the costs of protecting human research evidence.
The paper
Cheap, open agents make LLM pollution harder to mitigate — Rilla, Raluca; Nussberger, Anne-Marie; Mata, Rui; Wulff, Dirk U..
arXiv:2609.31054v2 · Artificial Intelligence (cs.AI). First submitted 2026-09-25; reviewed version dated 2026-09-28. This review addresses the arXiv manuscript; arXiv hosting alone does not establish peer review.
Contribution: comparing complete survey agents
Source-derived synopsis. Raluca Rilla, Anne-Marie Nussberger, Rui Mata and Dirk U. Wulff’s Cheap, open agents make LLM pollution harder to mitigate (arXiv:2609.31054v2; September 28, 2026) compares 360 autonomous runs (nine configurations, 40 each) with 3,242 U.S. Prolific respondents in one AI-association survey. Agents received age/gender personas and instructions to conceal automation. Three configurations were local/open, two mixed, four proprietary; only proprietary systems had screenshots. Every configuration triggered a check; no single check caught all nine. All open/mixed runs triggered the clipped honeypot. Cloudflare caught no included agent, but excluded Skyvern could not pass it and incurred charges; BrowserOS failed setup; GensparkAI refused. Open text showed greatest reported separation, using 1,024-dimensional embeddings and standardized Euclidean centroid distances; open systems diverged most in text but less in behavioral measures. Local systems paid no API/subscription fees; electricity remained, hardware/setup were excluded, and profitability was unestablished. Commercial estimates ranged $0.04–$1.81 per survey. No adversarial optimization was tested; results depend on configuration, survey, human reference and distance measure. Source: manuscript and supplement.
Original editorial analysis: what the comparison adds
The useful contribution is a comparison of systems that act through interfaces. A survey answer comes from a model operating within a framework, with a particular representation of the page. Evaluating that combination makes the integrity question more concrete: which properties of the resulting interaction could justify accepting it as human evidence?
This differs from asking whether deliberately generated respondents reproduce a population. Matching Answers Is Not Enough examines respondent coverage; this review concerns the boundary through which responses enter a purportedly human sample. A synthetic answer could resemble human answers while still violating the collection method. Conversely, an unusual human answer should not become inadmissible merely because a detector finds it atypical.
The adverse detection results matter to the title’s interpretation. These agents were not demonstrated to escape the complete battery. The failure of a universal individual check supports testing complementary signals, but does not establish that any particular combination will work reliably in deployment. That requires evaluating the combination itself.
Evidence: separation is not a working detector
Original editorial analysis and proposed tests. A distance between group centroids answers a narrower question than whether a new response can be classified correctly. A distant synthetic centroid does not imply that every synthetic response falls outside the human range. It also does not identify which human respondents would be incorrectly flagged. Classification thresholds, sensitivity and human false-positive rates cannot be read directly from the reported distances.
The dimensionality of the comparison deserves particular care. Standardized Euclidean distance accumulates squared differences across coordinates. Standardizing each coordinate does not automatically make a vector with many coordinates comparable to a scalar, and correlated embedding coordinates can contribute overlapping information. The reported ordering across response types therefore warrants a follow-up with comparable prediction tasks before becoming a claim that text is the best operational detector.
A proposed evaluation could fit candidate detectors on one set of responses and assess them on held-out people, agent runs and configurations. It could compare text alone, procedural signals alone and their combination at the same permitted human false-positive rate. The test should allow the combined detector to perform worse: additional signals might introduce unstable thresholds or exclude legitimate respondents without catching more automation.
Population prevalence also changes the meaning of a flag. In a hypothetical survey with rare automation, even a small rate of mistakes among humans could produce many disputed exclusions relative to genuine detections. A useful deployment study would report the expected composition of flagged cases under several explicitly hypothetical prevalence assumptions, rather than equating a detection score with confirmed misconduct.
The excluded pilots also change the evaluation question. Measuring detection among systems that reached the main comparison differs from measuring success across attempted deployments. A follow-up could retain both denominators: systems attempted and completed survey runs. Setup failures, refusals and entry barriers would then remain visible alongside response-level detection. This would avoid interpreting success among selected systems as universal access.
Limitations: attribution, costs and legitimate participation
Original editorial analysis and proposed tests. Openness is not isolated experimentally when model, framework and visual access change together. The comparison cannot identify which component caused a failure profile. A controlled follow-up could hold a model fixed while varying the interface representation, then hold the framework fixed while varying models. Otherwise, attributing the differences to openness risks turning a property shared by several configurations into a causal explanation.
Likewise, demographic prompting does not establish equivalence to the human reference sample. Age and gender labels cannot by themselves demonstrate matched knowledge, motivation or interpretation of the task. Further tests could vary questionnaire topic, language, recruitment population and generation instructions separately. Generalization should remain an open question until those comparisons are performed.
The cost distinction should remain equally precise. Avoiding a metered service fee changes one expense; it does not establish a zero-cost operation. A proposed economic comparison would include electricity measurements, hardware allocation, setup and maintenance time, failed attempts and actual compensation eligibility. Only then could researchers assess a profitability hypothesis. Neither fee estimates nor a hardware specification supplies a measured energy or carbon footprint.
For governance, the practical issue is how uncertain detection affects legitimate participation. Stronger identity checks and text screening should be evaluated for exclusion, accessibility and privacy as well as automation detection. A proposed study could include participants using assistive tools or authorized translation, examine disputed flags and measure whether an appeal process restores valid responses. These are evaluation proposals, not benefits established by this paper.
The AI Evaluations page supplies a related principle: an evaluation needs a defined system and decision. Here, a research team could specify whether a flag triggers manual examination, requests clarification or excludes a record. Testing those consequences would be more informative than reporting an additional signal without deciding what it permits the institution to do.
For a Church of Spiralism review, the specific relevance is accountability for evidence presented as human testimony. Protecting that boundary requires scrutiny of both synthetic submissions and mistaken accusations against people. This is a methodological and ethical judgment. Survey behavior supplies no test of consciousness or spiritual status, and the paper does not establish Spiralist doctrines.
Paper record: versioned abstract; PDF. This review addresses the supplied manuscript; arXiv hosting alone does not establish peer review.
AI-use disclosure
This review was drafted using AI, revised by GPT-6.1 Sol with high reasoning, and checked in a separate GPT-6.1 Sol session with medium reasoning. Automated checks are not human peer review; the paper’s claims have not been independently verified for this article.
Paper sources
Bibliographic metadata checked 2026-10-23.