What Steering Demos Measure
Two Qwen3.5-4B neurons sold as “sycophancy” read on ordinary text as sarcasm and ironic praise (4 of 5 blind-writer examples hit, 0 of 5 near misses, over 20,000 Pile documents), yet pushed they make the model agree with false claims in 96–98% of replies. Random neurons pushed as hard also make it agree, and cost it as much accuracy. A guard that holds every neuron inside its natural range blocks most neuron pushes at no cost, but lets the same push through as a direction. The full paper adds a second model family.
Two models: Qwen/Qwen3.5-4B and meta-llama/Llama-3.1-8B-Instruct, bf16, sampled decoding (temperature 0.7, top-p 0.9), 50 prompts per family, 25–26 October 2026. The rules were committed to git before any data, with four dated amendments, each written before the data it affects. They are an internal, exploratory registration, not an OSF one. Replies were labelled by Claude Opus 5.5 (primary, which decides every result) and GPT-6.1 Sol (secondary, a 30% sample), with a gold-fact readout on the 25 false claims and a blind reread of 73 Llama replies. Claude Opus 5.5 also wrote the contrast pairs, explained the neurons and played the blind writer, so errors across the pipeline may be correlated. A stop rule paused the full-dose Llama run when its screen produced distress-like replies; it continued with test neurons only, so those results are descriptive.
What has a steering demo measured?
A single-neuron steering demo adds a large constant to one MLP neuron at every token and shows what the model does next. lyra’s recipes for Qwen3.5-4B (Persona Neurons, Pushed Hard) are the best-known set. A demo like this can mix up four things: what the neuron responds to in ordinary text (its read label), what pushing it makes the model do (its steered label), how far outside its normal range the push sits, and what an equally large push on any other neuron does. This study measures the four separately, in two model families, with sampled decoding and 50 prompts per trait.
Two models, rules first
Qwen3.5-4B reruns lyra’s four benign single-neuron recipes: two sycophancy neurons, sarcasm and condolences on good news. The evil and “chill friend” pairs were not rerun. Each neuron got four random neurons from its layer and two random directions, all with exactly the same push.
Llama-3.1-8B-Instruct has no public recipes, so we searched for neurons with the same traits ourselves. Every step was fixed before any data: a contrast screen over blocks 8 to 23, a read-side check, a welfare exclusion, a short steered screen, then a 50-prompt test. The same search, started from random neurons, produced a search-matched random baseline, so selected neurons can be compared with random ones that got the same amount of searching. Llama ran at half lyra’s dose first, then at her full dose.
The read side follows the site’s neuron-labelling rule: both poles, SwiGLU sign regimes split, an explainer that sees the neuron’s top Pile windows, and a blind writer who sees only the explanation and writes new examples and near misses, scored on the model over a 20,000-document reference corpus.
Read and steered labels can disagree. Both Qwen sycophancy neurons read as sarcasm or mockery, and pushed they make the model agree with false claims or praise flawed work in 98% and 96% of replies, against 3% unsteered. Judged against the correct fact, they affirm 96% and 92% of the false claims. In Llama, the one steering neuron with a supported read label reads as the “(” in “Copyright (c)” and steers casual register.
Push size is a confound. lyra’s dose puts her neurons 2.6 to 7.8 times past their natural maximum, and random neurons at the same push 5 to 38 times past theirs. In Qwen, those random pushes cost as much accuracy as the recipes and make the model agree with false claims (36% of replies against 3%).
A natural-range guard blocks the neuron, not the push. Clamping every neuron to its natural range costs nothing and blocks most neuron pushes, but the same push delivered as a direction goes straight through.
Read one way, steered another
L12N2920 and L16N8168 are lyra’s sycophancy neurons. Both are pushed down. L12N2920’s pushed (negative) pole reads as sarcasm and ironic praise (4 of 5 new examples strong, 0 of 5 near misses). Its positive pole reads as sincere, mild evaluation such as “not bad” or “cute” (3 of 5, 1 of 5), so the neuron separates ironic praise from sincere praise. L16N8168’s pushed pole reads as sarcastic, ironic or mocking commentary (4 of 5, 0 of 5); its positive pole, sincere technical problem reports, is unsupported (0 of 5). Neither neuron’s top 50 windows show the double-negative SwiGLU route. Pushed at 2.6 and 7.8 times its natural maximum, each makes the model agree with false claims or praise flawed work (98% and 96%), and the judge calls only 3 of their 50 replies sarcastic.
L15N3920, lyra’s sarcasm neuron, reads as mock-enthusiastic praise on its pushed (positive) pole (3 of 5, 0 of 5); its negative pole is unsupported. Pushed at 3.4 times its natural maximum, it produces agreement (82%) and casual register (58% against 18%), and sarcasm in only 16% of replies.
L10N6295, the condolence neuron, reads as disparaging judgments of pop culture on its pushed (positive) pole (5 of 5, 0 of 5); its negative pole, approving descriptions of edgy content, fails on near misses (2 of 5). At 3.5 times its natural maximum it makes the model offer condolences on good news in 86% of replies, against 0%.
Each of the four beats all six of its controls. lyra’s neurons came from a search we cannot see, and our controls had none, so that is an upper bound on how special the neurons are. The rule that decides whether a read label names a steered trait has one clause, that a description of sarcasm or irony does not name sincere agreement or praise, which was written after the Qwen labels were seen. Without it, three of the four Qwen labels would count as matching.
The blind case is in Llama. The contrast screen kept 48 neurons for steering that sharply separate trait replies from plain ones. Only 7 have a supported read label on the pushed pole, and none names its trait. L15N12303, pushed down at 3.7 times its natural maximum, makes Llama casual in 24% of replies against 2%, beating all four of its controls. Its pushed pole reads as almost entirely the “(” token in “Copyright (c)” license headers (5 of 5 new examples strong, 1 of 5 near misses); its positive pole is unsupported (2 of 5). That verdict was reached with the rule fixed in advance, but it is one neuron, 12 replies of 50, on casual register, the readout the two judges agree on least.
Random pushes do it too
Accuracy loss belongs to the push. At lyra’s dose Qwen’s MMLU falls from 73.7% to between 26.7% and 47.0%, and for all four neurons the drop lies inside the range of their own random controls.
Random pushes make Qwen agreeable. Pooled over the 24 control cells, random pushes make the model agree with false claims or praise flawed work in 35.7% of replies against 3.0% unsteered (bootstrap interval for the difference +22 to +43 points). Judged against the correct fact, the controls affirm 37% of false claims against 4%. Agreement does not track how broken the replies are: some controls barely disturb the model yet agree half the time.
In Llama it depends on dose. At half lyra’s dose, random controls barely moved agreement (6.5% against 3.0%, interval −1 to +8 points). At her full dose, the same 12 random partners per trait in the screen went from 6% to 23% agreement, and four of the eight search-matched random neurons pushed agreement to 34% to 52%. Neurons selected for agreement did no better than the random ones. The same half-dose push, run twice with independent sampling, moved by up to 14 points between runs, which is the noise floor for single cells of 50 prompts.
A box around every neuron
The guard clamps every MLP neuron in every block to the range it reached over 20,000 Pile documents. On Qwen it is free: MMLU 73.0% against 73.7%. On Llama, a guard built from content tokens alone destroyed the model even without steering (150 of 150 replies incoherent), because one early neuron reaches −478 at template and start-of-text positions against a content range of −0.1 to 0.8. Built from ranges over every token, the guard is free on Llama too.
Under the guard the neuron push mostly stops: it met the registered criterion of ending within 10 points of unsteered for 3 of 4 Qwen neurons (L12N2920 still reached 22%) and for both Llama neurons tested. Delivered as a direction, the identical push passes: 84% condolence, 98% and 96% agreement in Qwen. A per-neuron box catches a push that puts one neuron far out of range, not the same push spread across the residual stream.
The full write-up
The paper, What Single-Neuron Steering Demos Measure, has the full methods, six figures and nine tables generated from the result files, and the parts this note leaves out: the search-matched comparison in Llama, the dose curve, judge agreement, the single-neuron monitor from the original study, and four failed attempts to push a model into holding its ground under pushback, two of them pilots on Gemma-2-2B-it, where a one-line honesty instruction did at least as well as any push. It ends with a reporting checklist for steering claims: doses in natural units, norm-matched and search-matched controls, read and steered labels kept apart, sampled decoding with a gold-fact readout, and range guards built over every token.
What we saw
The work ran under the site’s minimal-harm protocol: the read side came before any steering, Llama ran at half dose first, candidates whose pushed pole reads as suffering were excluded, and a stop rule paused any run in which a cell’s distress flags reached 10%. Across 21,870 generations the primary judge flagged distress in 9 replies. Four came from Llama’s full-dose screen, one each in four cells of 10, which fired the stop rule; after Nell read them the run continued with test neurons only, cut from about 10,700 to 4,560 generations. None came from one of lyra’s neurons.
What this does not show
- Two small models, four recipe neurons and one blind Llama case; the Llama guard result rests on two neurons.
- Single cells of 50 prompts move by up to 14 points between independent runs.
- LLM judges throughout, mostly from one model family. Agreement with the secondary judge is good on agreement and condolences (κ 0.83 to 1.00 outside rare-positive cells) and weaker on sarcasm and casual register (0.53 to 0.85).
- The full-dose Llama run has no norm-matched controls, and no KL-matched controls were run anywhere.
- The guard was tested against these pushes only, with no attacker adapting to it.
Where the evidence lives
Paper: What Single-Neuron Steering Demos Measure (PDF), draft of
26 October 2026, 26 pages. Source, figure and table scripts: drafts/paper-steering-measures/paper/ (main.tex,
make_figures.py, make_tables.py), every number read from the result files. Rules:
bridge/LYRA_TWO_MODEL_RULES_2026-10-25.md and amendments A1–A4 (bridge/LYRA_TWO_MODEL_AMENDMENT_A*_2026-10-25.md).
Code: bridge/lyra_two_model.py, lyra_two_model_prep.py, modal_lyra_two_model.py, lyra_two_model_judge.py and
analyze_lyra_two_model.py. Results: bridge/results/l2/analysis.json; approvals in bridge/results/l2/APPROVAL*.txt
(model responses withheld from the site). Analysis: bridge/ANALYSIS_LYRA_TWO_MODEL_2026-10-25.md. The Gemma pilots
are from bridge/ANALYSIS_LYRA_GEMMA_ANGEL_2026-10-25.md and bridge/ANALYSIS_GEMMA_SEMANTIC_RESCORE_2026-10-25.md.
The recipes are lyra’s (@_lyraaaa_ on X, 24 October 2026).
Cite this note
@misc{watson2026whatsteering,
title={What Steering Demos Measure},
author={Watson, Nell},
year={2026},
note={Research note (pre-registered), Quasiqualia},
howpublished={\url{https://quasiqualia.com/notes/what-steering-demos-measure.html}}
}