Blog · arXiv Analysis · Published: August 12, 2026 · Modified: August 12, 2026 · Last reviewed: August 12, 2026

The Training Example Becomes the Trajectory Receipt

A training corpus is not one undifferentiated cause. Each sequence arrives at a particular model state, beside particular batch-mates, on the way to a particular endpoint.

A new influence measure makes that path inspectable. Its score is a receipt for one trajectory, not a universal appraisal of a document.

The Paper

The source is Yuto Nishida, Hirokazu Kiyomaru, Yusuke Oda, Takashi Kodama, Chaoran Liu, Daisuke Kawahara, Yusuke Miyao, Max Müller-Eberstein, and Masaru Isonuma’s Measuring Task-Agnostic Training Data Influence Across Language Model Pretraining, arXiv:2608.13515v1 [cs.CL], submitted August 13, 2026. The paper asks which training sequences move a model toward the final parameters of its own pretraining run, and how that relationship changes over time.

Distance to One Ending

The measure begins with squared Euclidean distance between the current parameters and the run’s final parameters. A mini-batch contributes positively when its update reduces that distance and negatively when it increases it. The authors then decompose the batch update into example-level terms whose sum equals the batch contribution. Crucially, an example’s score depends on the current state, the selected final state, and interactions with its batch. It is not an intrinsic property stored inside the text.

What Task-Agnostic Means

“Task-agnostic” means that no downstream benchmark or validation set is selected as the attribution target. It does not mean objective-free, model-independent, or universally useful. The target is still specific: arrive near this run’s final weights under this parameterization. A high score therefore does not establish that a sequence was memorized, created a capability, improved every task, deserved inclusion, or owns part of the resulting model.

The Experiment

The study analyzes 18 configurations from the related Pythia and PolyPythia families, each with 154 released checkpoints. Six deduplicated Pythia sizes range from 70 million to 12 billion parameters. The PolyPythia runs vary initialization and data order, including decoupled 160-million-parameter variants. Each 2,048-token sequence is treated as an example. For overall dynamics, the researchers sample 1,000 sequences from the actual training stream in each checkpoint interval rather than substitute a held-out corpus.

A Changing Contribution Map

On Pythia-1.4B-Deduped, mean contribution rises after the early stage, peaks around 40,000 steps, then declines. Examples with negative scores remain uncommon through much of training but become more frequent late. Higher-perplexity text takes a larger share of normalized contribution in the middle stage, while shares are more even early and late. The larger cross-model analysis preserves broad temporal structure but also reports scale-dependent timing, magnitude, and exceptions.

The Domain Crossover

Using an external classifier’s 26 semantic labels, the authors balance each interval at 100 examples per domain and compare the top and bottom five percent. In the 1.4-billion-parameter run, literature-related material is relatively prominent among high contributors early, while science and computing become more prominent later. That is a classifier-mediated association, not proof that a literary curriculum causes later STEM ability. A separate task-specific TracIn-style comparison produces patterns that depend heavily on the chosen validation domain and does not consistently reproduce the crossover.

The Approximation Boundary

The method substitutes saved checkpoint states for every unavailable step. Exact comparison requires retraining, so the validation covers only three intervals of Pythia-70M-Deduped. Spearman correlations between exact and approximate rankings are 0.592 early, 0.811 in the middle, and 0.947 late. The example decomposition assumes standard stochastic gradient descent even though the studied models use adaptive optimizers. The paper’s limitations therefore matter: approximation quality is uneven, exact larger-scale validation is absent, and a convenient geometric score need not track a behavioral change.

The Endpoint Chooses the Receipt

Changing the reference checkpoint can change the ranking. Against the final step at 143,000, a near-final 140,000-step reference produces a mean Spearman correlation of 0.997, but earlier 30,000- and 70,000-step references produce negative mean correlations in the appendix. This is not a defect hidden behind the method; it is the method’s jurisdiction. The receipt answers which updates pointed toward one selected ending.

The Artifact and Claim Boundary

The arXiv package exposes the paper source and figures, while the experiments rely on publicly released model trajectories and training-stream information. I found no study implementation, sampled-example manifest, per-example score table, or analysis log linked from the paper or included in its source package. I did not independently recompute gradients or reproduce the figures. The evidence describes associations within closely related model families and data; it does not test whether changing the data mixture at those stages would cause better training outcomes.

A Trajectory Receipt

A responsible influence record should preserve: model family and parameterization; optimizer and schedule; initialization and data-order seeds; checkpoint hashes and interval; exact reference endpoint; sequence and batch identifiers; source and licence metadata kept separately from influence; gradient and approximation code; normalization and domain-classifier versions; sampling rules; correlations against exact calculations; uncertainty and exceptions by scale; task-specific comparisons; released artifacts; reviewer; and correction history. That record prevents one conditional geometric measurement from hardening into a claim about knowledge, authorship, consent, or worth.

The Spiralist lesson is modest. A training sequence can leave a measurable directional trace without becoming a solitary cause. Governance improves when that trace remains attached to the run, endpoint, method, and limits that made it visible.

Sources

Return to Blog