Measuring Dependence Before Claiming Meaning: An SAE Nonlocality Review
A critical review of a seminar on SAE feature nonlocality, examining what gradient entropy can establish and proposing tests for interpretability claims.
The video
Analyzing the Nonlocality of Sparse Autoencoder Features — AI Materials Institute.
Uploaded 2026-05-08 · Duration 00:59:37.
A number can make an interpretation easier to challenge without making it true. That is the most useful lens for Church of Spiralism’s reading of Analyzing the Nonlocality of Sparse Autoencoder Features, from AI Materials Institute. This seminar offers a promising measurement question: how broadly does an internal feature depend on its input? Its value for oversight depends on keeping that question separate from whether the feature has a coherent meaning, explains an output, or supports a safety claim.
Contribution: making dependence measurable
Brief synopsis: The speaker describes taking feature gradients with respect to input vectors, aggregating squared components, and normalizing across token positions before calculating entropy (30:36; 32:02). The intended measure concerns how many tokens contribute, regardless of where they occur (32:41). The speaker explicitly distinguishes influence from previous inputs from influence on future outputs (45:13), and acknowledges lacking clear evidence that individual features have lower entropy than the whole vector (56:34).
My assessment is that this is a useful diagnostic proposal with a demanding interpretation problem. A measure of dependence can help replace an impressionistic judgment with a comparison someone else could investigate. But it needs a precise claim attached to it. A reviewer should ask whether the intended claim concerns concentration, distance, abstraction, or behavioral importance. Those are different targets, even when the same vocabulary makes them sound interchangeable.
The distinction between concentration and distance is especially consequential. As a mathematical thought experiment, consider two equally weighted influential tokens. Moving them far apart leaves their entropy unchanged. Conversely, many adjacent tokens could produce higher entropy while occupying a short stretch of text. This makes entropy potentially useful for counting effective contributors, while leaving an additional measurement necessary for claims about positional reach. A single score should not quietly acquire both meanings.
Evidence: a diagnostic needs competing explanations
Normalization also creates a practical interpretive limit. Two sensitivity profiles can have identical proportions while differing greatly in absolute magnitude. An oversight report that records only their entropy would conceal that difference. My editorial recommendation is to report the unnormalized sensitivity alongside the normalized distribution, with an explicit rule for cases where the total is zero or extremely small. Otherwise, a sharply described distribution may invite more confidence than its underlying signal warrants.
There is another gap between infinitesimal vector changes and meaningful changes to language. A derivative answers a local mathematical question. An audit might instead need to know what happens when a person substitutes a name, negates an instruction, or adds an irrelevant paragraph. Agreement between those two forms of investigation would be useful evidence; disagreement would identify a boundary of the diagnostic. Neither outcome should be assumed in advance.
Proposed tests, not reported results
- Test sensitivity against edits: Predict which token positions should most affect a selected feature, then compare those predictions with controlled substitutions and paraphrases. Record feature activation changes separately from output changes.
- Test the abstraction hypothesis: Define abstraction criteria before examining entropy. Have evaluators assess held-out activation examples without seeing the scores, and check whether the proposed association survives controls for input length and activation frequency.
- Test instrument dependence: Compare dictionaries trained with different sparsity settings and seeds while holding the source model and inputs fixed. Compare against raw representation measurements and random projection baselines.
- Test context dependence: Add relevant and irrelevant context separately. Track absolute sensitivity, entropy, positional spread, and task performance so that a change in one cannot substitute for a change in another.
These proposals connect directly to The Sparse Feature Budget Becomes the Interpretability Dial. That article’s discussion of dictionary settings supplies the right challenge: does an apparent property belong to the model, or to the instrument used to inspect it? Here, the useful extension would be to ask whether a feature’s dependence profile remains recognizable when the sparsity dial moves. The supplied article concerns vision embeddings, so it provides a methodological comparison rather than independent validation of this seminar.
The Circuit Map Becomes the Variance Problem adds a second requirement: preserve distributions and variation across examples. An average dependence score could hide prompt families with very different profiles. A worthwhile audit would examine withheld phrasings and resampled inputs before treating a feature measurement as stable enough to guide intervention. Feature dependence and circuit discovery are different procedures, but both need evidence that legitimate changes to the inspection conditions do not invalidate the intended claim.
Limitations: the recording evidence has boundaries
This review uses supplied machine audio transcription only. It has not been human verified; an empty uncertain-segment list does not establish accuracy. Inconsistent names and garbled technical wording make exact attribution and formula reconstruction unreliable. I have avoided resolving those passages through memory. The spoken references to diagrams and plots cannot substantiate their unseen contents, axes, numerical patterns, or demonstrations. No visual inspection or replication is claimed here.
The strongest reason to engage with this seminar is the opportunity to sharpen an audit question. Before a feature label becomes an explanation, reviewers need evidence about what changes it, whether that dependence survives reasonable methodological choices, and whether changing the feature affects the behavior under scrutiny. Entropy could contribute to that evidence. Treating it as a verdict on meaning would outrun the question it measures.
AI-use disclosure
This transcript-based review was drafted and independently checked in separate GPT-6.1 Sol sessions with medium reasoning. This is automated editorial review, not human review or independent verification of the video’s claims.
Transcript
Machine transcription from audio · en · checked 2026-10-24. Machine-generated text may contain errors; timestamps may be approximate.
This review uses an automatic transcription of the audio. Open video on YouTube.