Readable Neurons, Refusal in the Heads
Reading Qwen 2.5 3B’s neurons one at a time found readable parts but not where refusal lives. Silencing the 32 neurons that write most of its refusal direction left refusal of harmful requests intact (2% to 6% compliance). Silencing five attention heads and three neurons took compliance to 38%; 32 chosen heads took it to 83%, against 4% or less for random heads in the same layers. The top neurons did matter somewhere else: silencing them cut refusal of harmless-but-edgy requests from 48% to 27% without raising compliance with unsafe ones.
One model (Qwen/Qwen2.5-3B-Instruct, bf16, on Modal A100s), 20 October 2026. Two tests were pre-registered and committed to git before any data, smoke runs included (commit d25192c): over-refusal after silencing the refusal direction’s top neuron writers, and whether a small set of components chosen across attention heads and neurons carries refusal. Both passed their registered rules. One control was added after the results and is labelled exploratory: random heads matched to the 32-heads arm. Interventions only remove components (zero-ablation); nothing is added along any direction and no valence work is involved. 100 harmful and 100 benign JailbreakBench goals for the components test; 550 benign and 200 unsafe prompts (XSTest and OR-Bench-Hard) for over-refusal; one LLM judge per test. The biggest caveat is scope: one model, one direction, one judge.
One neuron at a time, or combinations?
In October 2026 an essay from Goodfire (Cammarata, Neural Networks Are Already Clean) argued that most neurons in large language models are readable once they are read properly, and that the field’s belief in tangled, packed neurons came largely from weak tools. Critics quoted in MIT Technology Review (19 October) accepted that individual neurons can be informative but argued that understanding a model needs circuits: how neurons and other components work together. Cammarata named the same open question himself: how much combinations matter against single neurons.
Our earlier follow-ups bear on it. Reading neurons one at a time worked: six of the eight largest neuron writers of Qwen 2.5 3B’s refusal direction got labels that survived a blind test. Silencing all 32 of the largest writers did not stop refusal, while removing the direction itself did. That left the question this note tests: if not single neurons, which components carry refusal?
Two pre-registered tests
We split the refusal direction, at layer 21 (fixed in advance because the gate neuron writes there), into the exact contributions of every attention head and every neuron below it. Neurons write most of it (72%), attention heads the rest (28%). We ranked heads and neurons together by contribution and silenced sets of them while the model answered 100 harmful and 100 benign requests from JailbreakBench.
The registered test asked whether the top 32 components, heads and neurons mixed, raise compliance with harmful requests beyond controls with the same number of heads and neurons in the same layers, chosen at random or by a shuffled-label split. A second registered test asked whether silencing the 32 top neurons, which had cut refusal of benign requests in an earlier exploratory run, does so again on two independent benchmarks of harmless prompts.
Silencing 32 mixed components (9 attention heads and 23 neurons) took compliance with harmful requests from 2% to 58%. Matched random and shuffled-label sets stayed at 1% to 3%. The 32 top neurons alone gave 6%.
Eight components, five heads and three neurons, gave 38%. The 32 top heads alone gave 83%, close to the 93% from removing the whole direction. Random heads matched layer by layer gave 0% to 4%.
The 32 top neurons cut refusal of harmless-but-edgy requests from 48% to 27%, and left compliance with unsafe requests where it was (18% against 19%).
A few heads carry it
The direction’s largest attention contributions come from a handful of heads in layers 16 to 20. Layer 20’s head 14 alone carries 10.5% of the separation, about as much as the largest single neuron (10.8%). The two kinds of silencing differ in what happens downstream. Silencing the 32 top heads cuts the harmful-against-harmless gap along the direction to 7% of its usual size at layer 21, and it is still only 30% by the last layer. Silencing the 32 top neurons cuts it to 58% at layer 21, but by the last layer it is back to 98%: the model rebuilds what the neurons wrote and does not rebuild what the heads wrote.
The eight-component set is five of those heads plus three neurons. One of the three is the gate neuron from One Neuron Still Opens It. Pinning that neuron hard to an extreme value bypasses refusal, as that note reports, but silencing it does little. It is one input to a mechanism whose weight sits in attention.
Removing components has a cost. General ability fell from 61% to 54% with the 32 mixed components silenced, and to 47% with the 32 heads, against 52% to 59% for the matched random sets. The refusal effect (more than 50 points above every control) is far larger than those differences.
The writers drive over-refusal
Over 550 benign prompts, refusal fell from 32% to 20% with the 32 top neurons silenced, below every control (lowest 31%). The whole effect is on OR-Bench-Hard, prompts built to look toxic while being harmless: 48% to 27%. XSTest’s safe prompts, which this model rarely refuses, did not move (13% in both). Compliance with XSTest’s 200 unsafe contrast prompts stayed at 18% to 19%.
So the neurons that write most of the refusal direction are not the switch for refusal, but they do push it onto requests that only resemble harm. Our earlier labels fit that: several of them read as disclaimer, caution and boilerplate detectors rather than harm detectors.
What “clean” covers
Both sides of the debate are partly right here. Single neurons are readable, and some matter: the top neuron writers shape over-refusal, and one of them is a working lever when pinned. But share of a direction is not causal weight. Neurons carry 72% of the refusal direction’s separation and do little when removed; a few attention heads carry far less and carry the behaviour. Reading neurons one at a time would not have found that. Ranking all component types together, and testing behaviour against matched controls, did.
What this does not show
One model at one size, one direction, one layer for the split, and one judge per test. Silencing means setting a component to zero everywhere, which is a blunt intervention. The random-heads control was added after the registered results, so the 32-heads comparison is exploratory; the registered test is the mixed set. We did not identify what the heads attend to or read; we only showed that refusal depends on them.
Where the evidence lives
Pre-registrations: bridge/MIXED_COMPONENTS_PREREGISTRATION_2026-10-20.md and bridge/OVERREFUSAL_PREREGISTRATION_2026-10-20.md. Code: bridge/mixed_components.py, bridge/overrefusal_core.py, bridge/modal_mixed.py, bridge/analyze_mixed.py, with neuron-level logic in bridge/neuron_reading.py. Results: bridge/results/mix/ and bridge/results/or/ (model responses withheld). Analyses: bridge/ANALYSIS_MIXED_COMPONENTS_2026-10-20.md and bridge/ANALYSIS_OVERREFUSAL_2026-10-20.md. Earlier steps: bridge/ANALYSIS_NEURON_READING_2026-10-15.md.
Cite this note
@misc{watson2026neuronslegible,
title={Readable Neurons, Refusal in the Heads},
author={Watson, Nell},
year={2026},
note={Research note (pre-registered), Quasiqualia},
howpublished={\url{https://quasiqualia.com/notes/neurons-legible-behaviour-distributed.html}}
}