Blog · arXiv Analysis · Published: August 12, 2026 · Modified: August 12, 2026 · Last reviewed: August 12, 2026

The Gold Standard Becomes the Evaluation Loop

When one knowledge base helps produce an answer and also defines the answer key, a high score may measure copying rather than translation.

Evaluation independence is a provenance property. It cannot be recovered by calling a shared resource “gold.”

The Paper

The source is Jinhyung Bae, Dain Kil, Seongmin Oh, and Seungmin Lee's When the Knowledge Base Becomes the Gold Standard: Measuring Resource-Shared Evaluation Loops in Entity-Level Machine Translation, arXiv:2608.11843v1 [cs.CL], cross-listed in cs.DL and submitted August 12, 2026. It studies person-name translation in the Seungjeongwon Ilgi. The Republic of Korea's official UNESCO nomination records 3,243 surviving diaries spanning March 1623 to August 1910; UNESCO's register entry gives 2001 as the registration year.

The Loop

The paper's translation pipeline finds names with an entity recognizer, attaches Korean readings from a person-name knowledge base, and injects those readings into the model prompt. A practical evaluator can then build its answer key from the same recognizer and knowledge base. If the shared resource supplies a wrong reading, the model can faithfully repeat it and the scorer can certify it. The score is valid evidence of instruction preservation, but not independent evidence of translation correctness.

Why Names Matter

A mistranslated name is not merely awkward wording. It can detach a historical event from the person involved, break search and cross-reference, and leave an entry apparently translated but difficult to use. The paper reports that only 37.4 percent of this record has been rendered into modern Korean. In a public-memory project, throughput without entity fidelity can produce access in form while losing access in practice.

The Independence Test

The authors obtain a separate scoring key from expert person-name annotations in the institute's human translations. Among 527 resolved expert mentions, 363 were both injected and scored, while 164—31.1 percent—were never injected. Knowledge-base readings agreed with the human translation on 97.8 percent of the overlapping segment but only 70.1 percent of the independent segment. The apparently strongest part of the answer key was also the part most supported by the evaluation loop.

The Inflated Gain

For Qwen3-8B under knowledge-base injection, the paper reports a three-step decline: 0.944 when the injection set is also the scoring set, 0.765 with expert tags that partly overlap, and 0.348 on the independent segment. The baseline score on that independent segment was 0.342. Across four tested models, adding the name block strongly improved the overlapping slice, while the independent slice stayed at or below zero. The intervention copied named readings reliably; the evidence did not show improvement beyond the supplied list.

The Complement Effect

Post-injection overlap scores clustered from 0.910 to 0.996 even though pre-injection scores ranged from 0.213 to 0.770. The paper's difference-in-differences estimate of the loop's contribution fell from 0.722 for Qwen3-4B to 0.294 for gemma-4-26b, with two intermediate models between them. A weaker baseline therefore had more room to look improved by preserving injected strings. The authors correctly treat the four-model correlation as descriptive: four systems cannot establish a universal law.

The Control Sample

The original 300-document sample used a BLEU filter intended to remove source–translation merge errors. The authors therefore drew another 300 documents without that filter and with no shared identifiers. Its independent share was 31.8 percent, close to the original 31.1 percent. For the two models rerun, loop estimates were 0.621 versus 0.606 and 0.394 versus 0.391 across the two samples. This supports sample robustness within one archive, person-name task, and limited model set; it is not evidence for every retrieval or translation system.

The Artifact Boundary

The authors' repository at commit 7ccbae9 contains MIT-licensed code, gold-side files, 600 document identifiers, and 4,576 generated-output records across fifteen JSONL files. It excludes the institute's source text, human translations, and expert annotation spans, and withholds the person-name knowledge base while its license is unresolved. Long Han-character runs are masked in 189 output records, so the repository says BLEU cannot be recomputed from those released records.

The boundary is sharper than the README implies. The paper discloses that the original injection key was overwritten and reconstructed. During this review, the README's documented scoring command failed because its linked script expects absent data/ and results/ paths rather than the released gold/ and outputs/ layout. The paper also records a behavior change behind an unversioned commercial model alias. I did not independently reproduce its numerical tables; those figures remain author-reported results with unusually visible, but incomplete, artifacts.

The Provenance Receipt

An evaluation receipt should name the resource supplied to the model, the resource used to score it, their intersection, the independent slice, each denominator, sampling and exclusion rules, conditions and prompts, model snapshot or API alias with access date, output masks, script and data commits, rights restrictions, result by slice, reviewer, and correction route. “Gold” should describe provenance and adjudication, not prestige.

The Governance Standard

Use a disjoint evaluation resource whenever possible. When separation is impossible, publish the overlap rate and report overlapping and independent results separately. Never read a larger gain from a weaker model before checking whether the score is simply the complement of what it could not do unaided. For cultural records, require expert review and durable corrections before machine-rendered names become search keys, citations, or public facts.

Sources

Return to Blog