Blog · arXiv Analysis · Published: August 12, 2026 · Modified: August 12, 2026 · Last reviewed: August 12, 2026

The Tokenizer Becomes the Corpus Trace

Qingjie Zhang and colleagues ask whether a released tokenizer vocabulary preserves enough statistical residue to estimate ratios inside an undisclosed corpus.

A tokenizer provenance receipt keeps that inference useful without mistaking an estimate for a document inventory, a consent record, or proof about any person.

The Paper

The paper is Qingjie Zhang, Xingzhang Ren, Zixuan Chen, Jinfeng Li, Yuefeng Chen, Yitong Yang, Hui Xue, Dayiheng Liu, and Han Qiu's Can Released LLM Vocabularies Support Token-Level Estimation of Hidden Corpora?, arXiv:2608.10690v1 [cs.CL], submitted August 11, 2026. Its 13-page PDF lists Tsinghua University, the Qwen Team at Alibaba Group, and Alibaba Group. The authors also link a public QGDE reproduction repository.

The question is narrower than training-data extraction. The method does not prompt a model for memorized text. It studies an artifact that may ship beside the weights: the tokenizer vocabulary, including token strings and their assigned IDs.

A Vocabulary Is a Learned Residue

Byte-pair encoding builds a fixed vocabulary of variable-length units by repeatedly merging frequent adjacent units. Because those merge decisions depend on corpus statistics, the resulting vocabulary can retain information about the material used to train the tokenizer. Zhang and colleagues focus on the relationship between a token's ID and its ratio in that tokenizer-training corpus.

This is a consequential qualification. A tokenizer can be trained on a different corpus from the language model, reused from an older model, or modified during development. The vocabulary is therefore not a transparent window into all model training. It is a statistical trace of the process that produced the tokenizer and, only where the two corpora align, an indirect trace of pretraining composition.

From Token IDs to Ratio Estimates

The authors compare BPE tokenizers trained on four language slices—English, French, Japanese, and Chinese—and four domains: web text, Wikipedia, code, and mathematics. They report a broadly stable global shape when token ID and corpus ratio are plotted in log-log space. That motivates transferring a distribution learned from known corpora to a target vocabulary whose corpus ratios are hidden.

The proposed Quantile-Guided Density Estimation method, or QGDE, fits several quantile trends rather than one median curve. For a target token ID, those trends supply candidate ratios; nearby known ID-ratio points then weight the candidates through local density. The output is a point estimate for an arbitrary target token. Token estimates can also be aggregated into language or domain mixtures.

What the Tests Establish

For controlled language tests, mC4 supplies known-corpus statistics and OSCAR supplies held-out target corpora. For domain tests, the source and target datasets differ while retaining broad web, encyclopedia, code, and mathematics categories. QGDE is compared with direct position-wise transfer and with PoCTrace; category aggregation is also compared with Data Mixture Inference.

The paper reports mean relative error as low as 3.00 percent for token-level estimation and 3.08 percent after category aggregation. Those are best observed results, not universal error guarantees. Domain transfer is harder than language transfer when source and target collection pipelines differ, and performance improves when known data cover multiple relevant components. The exact mixture of those known components matters less than covering them at all.

The SmolLM Check

Broader released-model validation is blocked by missing ground truth, so the paper uses SmolLM as its realistic case. SmolLM's published corpus contains FineWeb-Edu, Cosmopedia-v2, and Python-Edu; the paper expresses their proportions as 87.30, 11.11, and 1.59 percent. The estimators are fitted on proxy corpora rather than those exact components.

Across three proxy-mixture settings, QGDE's token-level mean relative error ranges from 5.72 to 5.78 percent, compared with 7.71 to 10.53 for direct transfer and 13.84 to 20.33 for PoCTrace. At category level, QGDE ranges from 5.90 to 6.08 percent against 9.11 for Data Mixture Inference. This is a useful artifact-level check, but it is one tokenizer whose corpus is unusually available.

The Governance Reading

Open weights and an open vocabulary do not equal open data provenance. A tokenizer provenance receipt should name the model revision, tokenizer revision and artifact hash, tokenizer algorithm, vocabulary and merge files, initial-symbol and special-token handling, token-ID convention, tokenizer-training corpus or its documented proxy, whether the tokenizer was reused or retrained, known differences from the model's pretraining corpus, and any published composition evidence.

An external estimate should travel with its target artifact, known reference corpora, category definitions, token-selection rule, estimator and parameters, source-coverage analysis, error metric, validation setting, uncertainty or sensitivity analysis, and explicit non-claims. That record makes the estimate contestable. It also prevents a low error in one controlled mixture from becoming a confident accusation about copyrighted works, private records, language exclusion, or regulatory compliance.

What the Trace Cannot Prove

Version 1 cannot directly evaluate QGDE on the opaque corpora behind ChatGPT, Qwen, or DeepSeek. It tests controlled tokenizers and one realistic released tokenizer. The estimator requires relevant known corpora, and broad domain labels can hide substantial collection differences. Its ratios do not identify documents, establish that a specific record was present, recover licenses or consent, or show which data trained the model weights.

The careful conclusion is neither that vocabularies are harmless metadata nor that they reveal a secret archive. They are learned artifacts with measurable provenance signals. Release policy should disclose how they were made; audit policy should preserve the gap between statistical inference and documentary proof.

Source Discipline

Primary sources were the arXiv abstract, complete version 1 HTML and PDF, the authors' QGDE repository, the original BPE paper, and Hugging Face's SmolLM corpus account. They were checked for metadata, affiliations, method, datasets, comparison baselines, reported errors, code scope, corpus proportions, limitations, and the tokenizer-versus-pretraining-corpus boundary. No figures, formulas, data tables, or long passages are reproduced here.

Sources

Return to Blog