Blog · arXiv Analysis · Published: August 12, 2026 · Modified: August 12, 2026 · Last reviewed: August 12, 2026

The API Response Becomes the Custody Receipt

A model name in an API response can identify a product family while leaving the answering system unbound to the evaluation, policy, and deployment configuration published in its name.

A new scorecard studies that verification gap. Its most useful question is not whether providers publish documentation, but whether an outsider can travel from a live response back to the exact evidence that supposedly describes it.

The Paper

The source is Sophia Abraham and Ben Bucknall's Silent Updates: Measuring and Closing the Post-Deployment Disclosure Gap, arXiv:2608.11803v1 [cs.CY], submitted August 12, 2026 and listed by arXiv as accepted to AIES 2026. The paper studies externally observable disclosure. It does not claim access to provider registries, production prompts, internal routers, classifiers, or model weights.

What Counts as a Silent Update

The authors define a silent update as a deployed-system change that materially affects user-facing behavior without corresponding public disclosure. Their taxonomy includes model-layer changes such as post-training or quantization; system-layer changes such as prompts, classifiers, retrieval, tools, routing, inference budgets, and tokenizers; and surface changes between API and chatbot products.

That scope matters. A provider can leave the advertised model family unchanged while changing the apparatus around it. Conversely, this study does not demonstrate that any particular undocumented internal change occurred. It asks whether an outside party could detect and bind such a change using public documents and ordinary API access.

The Measurement Instrument

The Silent Updates Scorecard covers nine first-party API providers and seven inference hosts. Its 29 main questions span versioning, changelogs, cross-surface transparency, deprecation, safety-evaluation traceability, reproducibility, and monitoring support; four additional questions address hosts. Evidence came from public documentation, terms, changelogs, archived pages, and standard API responses.

The rubric was finalized before scoring. One author performed the scoring and the other audited the completed evidence-to-score mappings, but did not independently rescore the sample. The paper therefore offers a structured point-in-time audit, not an inter-rater-validated industry census.

The Reported Verification Gap

The reported aggregate results separate abundant naming from scarce binding. All nine first-party providers exposed dated or otherwise versioned identifiers; seven published quantitative safety metrics; six published per-version safety comparisons; five documented identifiers that could be pinned; and two published alias-to-snapshot mappings. One provider was scored as partially satisfying the criterion for naming an evaluated artifact with a pinned, API-callable snapshot identifier. In the authors' scoring, none enabled a public API-to-evaluation round trip and none bound content policy to a specific snapshot.

This is not evidence that the providers conducted no safety work, nor that every named endpoint changed secretly. It is evidence about what an external evaluator could verify under the paper's definitions. A family name is not an immutable snapshot. A provider-supplied model field is an assertion, and a temperature or seed control can reduce sampling variation without proving that tomorrow's endpoint is the same deployment.

The Custody Receipt

This essay proposes a response-level custody receipt. Every governed API result should carry, directly or through a stable manifest, an immutable served-snapshot identifier; the resolution of any friendly alias at the response timestamp; a deployment-manifest identifier covering the model, system policy, classifier, retrieval configuration, router, tool policy, tokenizer, and inference budget; the applicable content-policy version; host transformations such as quantization; and identifiers for the evaluation artifacts that apply to that configuration.

The receipt need not expose a proprietary prompt or model weights. This proposed design could use hash commitments, signed manifests, protected regulator access, or qualified independent inspection, depending on what must remain private. Its purpose would be to make a falsifiable claim about which system answered. A receipt still would not prove that an evaluation was good or that the answer was safe; it would make the scope of those claims inspectable.

Three Different Triggers

The paper's governance proposal distinguishes capability, drift, and component triggers. A material capability shift would initiate full re-evaluation. Behavioral movement under a stable identifier would initiate a documentation update. A change to prompts, classifiers, retrieval, routing, tools, compute budgets, or tokenizers would initiate logged component disclosure.

This separation is useful because not every change deserves the same process. The authors explicitly describe their numerical examples as illustrative. They do not solve which behavior batteries, thresholds, reporting windows, or dangerous capabilities should govern, and they warn that fixed tests invite optimization against the test.

Artifact Audit

The paper's replication note says its linked repository contains a complete scorecard, more than 270 evidence rows, source URLs, and a dataset. At the only repository commit available when reviewed on August 12, 2026, the tree contained only an MIT license. The evidence rows and spreadsheet could therefore not be inspected or reproduced for this page.

The results text also describes the count for version-level behavior-change documentation as both four of nine and C2=3/9 in the same passage. This page omits that ambiguous statistic and does not reproduce the provider ranking. The aggregate counts above are reported as paper results, not independently reconstructed findings. No table, figure, prompt, or extended passage is reproduced.

Limits That Stay Attached

The authors' limits include public-only visibility, one initial rater, no formal inter-rater reliability, partially gated provider access, and a framework designed mainly for hosted APIs. The trigger thresholds require normative choices and remain vulnerable to Goodhart effects. Open-weight deployments distribute change authority across hosts and downstream operators, so the same disclosure structure cannot simply be copied over.

The durable result is narrower: under this rubric and observation window, substantial documentation did not produce a publicly verifiable binding from live response to evaluated deployment. That is enough to turn model identity from a label into an engineering and governance problem.

Sources

Return to Blog