The Fisher Sketch Becomes the Update Signature
A source-selection score is not enough when a model is about to be adapted. A new arXiv paper turns transfer similarity into a compact update signature, recording activation, error, and coupling geometry rather than treating shared representations as the whole story.
The Paper
The paper is The Geometry of Updates: Fisher Alignment at Vocabulary Scale, arXiv:2606.27242 [cs.LG, cs.CL, stat.ML], by John Sweeney. arXiv records version 1 as submitted on June 25, 2026, and its metadata comment reports acceptance at the 43rd International Conference on Machine Learning, ICML 2026, PMLR 306.
The paper studies a practical source-selection question: when several candidate corpora share a tokenizer and output vocabulary, which source should be used to adapt a model for a target domain? The motivating examples are scientific string domains such as SMILES, proteins, and genomic sequences, where the surface vocabulary can be shared while the useful prediction target changes. The page-level governance question is broader. If an organization chooses one source dataset over another for fine-tuning, what artifact records why that source was expected to transfer?
Activation-Dark Tasks
The paper's core warning is that representation similarity can be the wrong witness. Metrics such as CKA compare activations. They can be useful in favorable regimes, and the paper explicitly avoids claiming that CKA is useless everywhere. But for head-level transfer in a shared-output setting, activations alone do not determine the update a model would make.
Sweeney gives the formal version as a non-identifiability result. Two tasks can have identical representations on a shared probe set while their head updates are orthogonal, because the label-conditioned error geometry differs. In the paper's language, this creates an activation-dark regime: the visible representation map is bright, but the transfer-relevant error structure is not visible through representation-only metrics. For source selection, that distinction matters. A high activation similarity score can become a false sense of portability if the chosen source would push the output head in the wrong direction.
The Update Signature
Fisher alignment measures similarity between the second moments of gradients. Classical Fisher comparisons are too expensive at vocabulary scale, especially when K=128,256 for Llama-style vocabularies. The paper's key identity is that shared-output head Fisher alignment can be written as a cosine between kernel mean embeddings in joint activation-error space. That reframes a Fisher-matrix comparison as a compact distributional embedding problem.
FisherSketch is the estimator built from that identity. It estimates the product-kernel cosine in one streaming pass and records a task signature. At m=4096, the paper reports a 16 KB float32 stored descriptor and a 192 KB split-sample streaming state per task. It also reports that storing the full error second moment at K=128,256 would require about 61 GiB per task, so the signature is not just a convenience. It changes what can be logged, compared, and attached to a model record.
The important governance move is that the signature is not only a ranker. Its marginals separate activation geometry, error geometry, and coupling. That makes the score inspectable: an evaluator can ask whether two tasks look close because their representations match, because their errors match, or because the interaction between those factors is doing the work.
What the Tests Show
The paper validates the method across several regimes rather than presenting one universal win. On Llama-3.1-8B across 100 domains, FisherSketch is reported as competitive with activation-only baselines on natural shifts. In the controlled verbalizer-shift experiment, where fixed prompts make activation-only scores collapse to a random 20 percent top-1 baseline, FisherSketch reaches 66.7 percent top-1 source selection and 95.7 percent of oracle normalized transfer across 9 runs.
The appendix also reports training-free retrieval on 700 Natural Instructions domains, where the joint FisherSketch signature reaches 55.5 percent top-1 accuracy, and a molecular SMILES proof of concept over nine domains. In the SMILES setting, the joint FisherSketch score correlates with cross-domain perplexity reduction at Spearman rho_s=0.53 with p=0.006, while activation-only similarity is reported as not significant at p=0.081. The paper is careful about the SMILES result: it is a proof of concept, not a complete chemistry benchmark.
The Governance Receipt
A source-selection receipt should name the target task, candidate sources, model checkpoint, tokenizer and output-head assumptions, sketch dimension, random seed or projection version, sample policy, dataset date, transfer metric, and the activation/error/coupling marginals. It should also record whether the comparison is head-level, internal-layer, or full-network; whether outputs are truly aligned; and whether any activation-dark stress test was run.
This belongs beside the site's pages on AI evaluations, decision-aligned evaluation, model co-failure, and evaluation schemas as public ledgers. The shared rule is that a compact score becomes more trustworthy when it travels with the evidence needed to contest it. A source selector that cannot expose its assumptions is only a preference with math around it.
The paper's compression claim makes that standard more practical. A 16 KB task signature can sit next to a model hash, dataset manifest, evaluation result, and post-training record. That does not make the source choice correct. It makes the source choice reviewable after the adaptation succeeds, fails, drifts, or causes harm.
Limits
This is a single-author June 2026 arXiv paper, with acceptance reported in arXiv metadata, not a deployed governance standard. Its strongest claims are scoped to shared-output head comparisons and related validations. If tasks do not share an output vocabulary or label taxonomy, head-level error alignment requires an explicit mapping; without one, the comparison is not directly defined.
The method also should not be converted into a new oracle. FisherSketch is an evidence artifact for source selection and task geometry. It does not replace held-out transfer tests, safety evaluations, privacy review, dataset provenance, or monitoring after adaptation. A cheap signature can improve review only if the institution refuses to treat it as a magic compatibility stamp.
Sources
- John Sweeney, The Geometry of Updates: Fisher Alignment at Vocabulary Scale, arXiv:2606.27242 [cs.LG, cs.CL, stat.ML], submitted June 25, 2026.
- Primary arXiv records checked: arXiv API metadata, HTML full text, and PDF, reviewed for title, authorship, submission date, categories, conference comment, abstract, non-identifiability claim, FisherSketch storage figures, Llama-3.1-8B experiments, Natural Instructions retrieval, SMILES proof of concept, and limitations.
- Related pages: AI Evaluations, The Uncertainty Score Becomes the Decision Cost, The Model Ensemble Becomes the Co-Failure Ceiling, The Evaluation Schema Becomes the Public Ledger, and Model Cards and System Cards.