The Training Corpus Becomes the Editable Surface
A July 2026 arXiv paper treats pre-training data as something that can be edited by executable programs, not merely filtered or rewritten. The governance issue is not only data quality. It is whether each corpus edit leaves a record good enough to inspect before the next model is trained on it.
The Paper
The paper is Xinlong Zhao, Dongsheng Liu, Hengyu Zhao, Zixuan Fu, Zheng Wang, Jie Cai, Jie Zhou, Qiang Ma, Xuanhe Zhou, Xu Han, Yudong Wang, and Zhiyuan Liu's UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing, arXiv:2607.08646 [cs.CL, cs.AI]. The arXiv API lists version 1 as submitted on July 9, 2026. The PDF metadata reports 35 pages.
This page extends the site's work on pretraining, training-set feedback loops, and training-data consent. Its fresh angle is executable corpus editing: the training set becomes an object transformed by a program, not only text selected by filters.
What UltraX Does
The authors frame the problem as a shift from data expansion to data utilization. They argue that rule-based corpus cleaning is cheap but brittle, while full LLM rewriting can be stronger but too costly and unreliable for pre-training-scale corpora.
UltraX uses a function-calling refinement framework. Instead of generating a replacement document, a lightweight refiner predicts edit operations. The function space includes deletion, modification, and insertion. UltraX builds supervision by asking an expert LLM for refined texts, then converting original-refined text pairs into structured programs through Line Alignment Mapping and Dynamic Context Replacement. It also filters low-confidence examples and uses ratio-controlled sampling by operation combination.
At execution time, UltraX uses sliding-window prediction, global operation aggregation, validation, and post-processing. A deterministic executor applies the operations to the original text, making the path more traceable than end-to-end rewriting.
The Experiment
The evaluation trains approximately 1B-parameter MiniCPM models from scratch on five corpora: FineWeb, RedPajama-v2, AICC, Ultra-FineWeb, and FineWeb-ProX-Doc. For each corpus, the authors construct roughly 20B-token training sets for Raw, ProX-C, and UltraX, then evaluate with LightEval on ten downstream tasks.
The reported result is that UltraX has the highest average performance across all five corpora. The paper reports average relative improvements of about 2.00% over Raw and 1.53% over ProX-C, and says UltraX wins 34 of 50 task-corpus pairs. On FineWeb, UltraX reaches an average score of 45.49 with 16B training tokens, above the final Raw and ProX-C scores at 20B tokens, which are 45.08 and 45.05. At 20B tokens, UltraX reaches 46.14.
The authors also run an LLM-based quality evaluation on 80K sampled FineWeb documents using DeepSeek-V3.2 as judge. UltraX receives an average score of 9.6042 versus 9.1737 for ProX-C, with lower low-score share: 0.38% versus 2.59%. A paired comparison reports UltraX better on 22.90% of samples, tied on 65.30%, and worse on 11.80%.
The Edit Record
The most interesting part is not the small benchmark lift by itself. It is the claim that corpus refinement can become an auditable edit surface. A document-level filter says keep or discard. A full rewrite says trust the replacement. A programmatic editor can say: remove these lines, replace this string, add this line, keep this document, or remove all of it.
That does not make the edits automatically correct. It makes the unit of review more concrete. A reviewer can ask whether the operation removed boilerplate, destroyed a table, preserved a code block, repaired crawler damage, or over-edited useful language. The corpus changes through operations that can be counted, sampled, replayed, and challenged.
Why It Matters
Training-data work is often discussed as discovery: find more data, filter bad data, deduplicate, remove obvious junk. UltraX points toward industrial-scale editing of existing corpora. Model behavior may depend on millions of pretraining transformations.
The governance burden grows with that capability. A corpus editor can improve data density, but it can also normalize dialect, erase marginal formats, alter quoted material, preserve private data, or remove auditor context. Corpus editing needs receipts, not just downstream scores.
What It Does Not Prove
The paper's limitations are important. The authors say their pre-training experiments use a limited token budget and do not yet cover larger model scales, longer schedules, or substantially larger corpora. They also do not include a direct empirical comparison with RefineX because, according to the paper, it had not been open-sourced to date. The main focus is English web corpora, with multilingual extension left for future work.
This is also not a consent solution. UltraX can make the corpus cleaner and denser, but the method does not settle whether the underlying web text should have been collected, licensed, retained, or used for model training.
Governance Reading
The Spiralist reading is that data quality is becoming an active policy layer. A corpus editor decides which fragments count as noise, which structures deserve preservation, which broken pages can be repaired, and which documents should disappear. Those are not neutral clerical acts. They are choices about what the model will later treat as language, knowledge, style, and evidence.
The right response is not to reject corpus editing. It is to make the editing legible. If a lab says a model was trained on a refined corpus, the refinement policy should travel with the model card, the dataset card, and the evaluation record.
The Receipt
A pre-training corpus-edit receipt should include source corpus, sampling rule, language scope, deduplication status, expert model, prompt-optimization path, edit function space, supervision-construction method, low-confidence filtering rule, operation distribution, execution validator, token counts before and after editing, sampled before-after examples, over-editing rate, benchmarks, judge model, and artifact access status.
The practical rule: a refined training corpus is not just cleaner data. It is a processed artifact, and the processing record is part of the model's provenance.
Sources
- Xinlong Zhao, Dongsheng Liu, Hengyu Zhao, Zixuan Fu, Zheng Wang, Jie Cai, Jie Zhou, Qiang Ma, Xuanhe Zhou, Xu Han, Yudong Wang, and Zhiyuan Liu, UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing, arXiv:2607.08646 [cs.CL, cs.AI], submitted July 9, 2026.
- arXiv API record for arXiv:2607.08646, checked for title, authors, subject classes, submission date, update date, and abstract.
- arXiv PDF for arXiv:2607.08646, checked for page count, affiliations, method, training corpora, experimental setup, reported results, quality evaluation, and limitations.