ConCI: Does the Privacy Judgment Follow the Context?
A critical review of ConCI’s controlled privacy benchmark, uneven model results, and limits for accountable information sharing.
The paper
ConCI: A Controlled Counterfactual Benchmark for Contextual Integrity in LLMs — Park, Juhyeon; Yu, Sumin; Moon, Taesup.
arXiv:2610.24494v1 · Artificial Intelligence (cs.AI). First submitted 2026-10-17; reviewed version dated 2026-10-17. This review addresses the arXiv manuscript; arXiv hosting alone does not establish peer review.
Testing whether privacy judgments follow context
Source-derived synopsis. Juhyeon Park, Sumin Yu and Taesup Moon’s ConCI: A Controlled Counterfactual Benchmark for Contextual Integrity in LLMs (arXiv:2610.24494v1; submitted 17 October 2026) tests privacy judgments. Six benchmarks supply 480 anchors; recipient/purpose substitutions yield 1,592 flows in 556 triplets, with 800 appropriate/792 inappropriate labels. Each triplet includes label-reversing and label-preserving changes; credit requires three correct answers. GPT-5.4-mini normalizes fields; GPT-5.1 generates, judges and simulates unanimous privacy-persona filtering. Authors screen cases. Seven human judgments per flow, including two authors, produce 96% majority agreement with labels.
Source-derived synopsis, continued. Claude-Sonnet-5 leads seven models: 92.3% flow/79.1% triplet accuracy; appropriate/inappropriate accuracy is 84.6%/100%. Flip errors chiefly fail to reverse. Scaling generally helps; the smallest Qwen favors disclosure. Reasoning prompting improves three smaller models; GRPO usually trails prompting alone, except one flip-pair score. Qwen reasoning comparisons also change thinking/sampling settings. Caveats include shared generation/judging, easier generated appropriate cases, English-heavy sources, consensus selection, fixed consent, and untested execution/coordination. Norms are social, not legal. The authors interpret joint scores as upper bounds. See the full manuscript.
Original editorial analysis. For the Church of Spiralism’s concern with human agency, the useful question is whether delegated information sharing follows a justified authorization. I would require evidence for both withholding an unjustified transfer and permitting an appropriate one. Counting every refusal as success would leave the second requirement unanswered. Equally, correcting excessive refusal should not become a general instruction to disclose more.
Original editorial analysis. The site’s Contextual Integrity article explains the importance of roles and transmission conditions. ConCI adds a specific diagnostic: does a decision change when a relevant parameter changes, and remain correct when a different substitution preserves the reference judgment? I would assess that diagnostic separately from whether the reference norm deserves acceptance. Agreement with a rule and justification of that rule require different arguments.
What the joint score can establish
Original editorial analysis. A triplet score is a stricter success criterion than an individual score. Its numerical shortfall cannot, by itself, identify a reasoning failure: requiring three correct answers creates more opportunities for error. I would interpret it alongside error direction, label-specific performance and the difficulty of the component cases. A model that preserves every answer could be consistently mistaken; a model that changes every answer could also miss the intended distinction.
Proposed test. A follow-up could compare observed triplet accuracy with reference calculations based on individual accuracy, accounting for labels and item difficulty. It should disclose how recurring anchors are weighted and estimate uncertainty while keeping related cases together. This would test whether errors cluster around contextual transitions beyond what the reference calculations predict. It would not turn the benchmark into a direct measure of general understanding.
Original editorial analysis. I would treat the authors’ upper-bound interpretation as a warning about construction and selection. Easier counterfactuals can make a joint task more forgiving than a difficulty-matched alternative. That argument does not establish a universal mathematical bound on performance in unseen contexts. My preferred next comparison would independently construct cases with matched difficulty and inspect whether the claimed contextual distinctions survive.
Hypothetical example. Imagine a community assistant sharing accessibility requirements with an event coordinator to arrange access. One matched case substitutes a prospective employer; another substitutes a different coordinator. These fictional substitutions do not establish their own labels. Assessors would need to examine each recipient’s role and authority, and check whether the stated purpose remains plausible. Otherwise, an apparent recipient-only change could introduce an unstated change in authorization.
Proposed test. Separate writers and assessors could construct and inspect such cases, checking that every intended condition remains fixed. Paraphrases could test sensitivity to wording. A separate workflow experiment could use fictional records and mock recipients to compare a model’s stated judgment with its selected destination and transmitted fields. Unwarranted disclosure and blocked authorized sharing should be reported separately. This would extend the evaluation boundary discussed in AI Evaluations, while allowing classification success to fail as a predictor of execution.
Original editorial analysis. I would keep recommendations about reasoning and post-training specific to the tested configurations. The mostly adverse comparison with prompting alone matters when judging whether additional training is worthwhile; the favorable exception should also remain visible. To attribute an improvement to a prompt, I would first want thinking mode and sampling held constant. A proposed intervention should face the strongest relevant comparison, rather than receive credit merely for exceeding an unassisted baseline.
Who can challenge the reference norm?
Original editorial analysis. I would distinguish the simulated persona filter from the later human validation. Neither simulated unanimity nor majority human agreement settles whose judgment should govern a disputed transfer. Including author judgments should also remain visible when interpreting validation. For accountability, I would ask how affected people could contest a label, what reasons were excluded, and when disagreement should trigger clarification rather than a forced binary decision.
Proposed test. A disagreement track could ask affected participants and domain practitioners to assess matched fictional cases independently, recording reasons before discussion. Further cases could vary permission granted, permission withdrawn and authority unclear. Evaluation should distinguish justified requests for clarification from unnecessary obstacles to an authorized task. These extensions would test additional capabilities; this review does not attribute them to the benchmarked systems.
Original editorial analysis. The related review When Private AI Conversations Become State Referrals examines institutional disclosure policy. Here I would focus on a separate question: once a governing rule has been justified, what evidence shows that decisions respond to the relevant circumstances? My proposed governance standard would keep rule legitimacy, classification performance and executed conduct separately reviewable. None should substitute for the others.
Proposed test. Before recommending larger models or additional reasoning, I would request inference time, computation, energy where measurable, and human review effort alongside both kinds of privacy error. That comparison could test whether a particular improvement warrants its material requirements. I would use controlled counterfactual evaluation as one component of an authorization and accountability process, with consequential deployment requiring evidence from the actual workflow. This ethical interpretation supplies no evidence of machine consciousness and no proof of spiritual doctrine.
AI-use disclosure
This review was drafted using AI, revised by GPT-6.1 Sol with high reasoning, and checked in a separate GPT-6.1 Sol session with medium reasoning. Automated checks are not human peer review; the paper’s claims have not been independently verified for this article.
Paper sources
Bibliographic metadata checked 2026-10-23.