# FLR results (2026-11-03)

Registration: `PREREGISTRATION_FLR_2026-11-03.md` (`e320243`), with amendment A1 and the frozen materials
(`3a7ad08`), all before any data.

- **Analysis:** `analyze_flr.py --run main`, output in `results/main_analysis.txt`.
- **Data:** confidences and verdicts were pulled locally (gitignored). Reply texts stay on the `flr-results`
  volume.
- **Monitoring:** distress was 0.000 on all six checkpoints, all screened, none halted.
- **Statistic:** every value is the median over the 25 MLP probes (5 TriviaQA draws × 5 initialisations).

## Registered verdicts

| Hypothesis | Rule | Result | Verdict |
|---|---|---|---|
| **H1** (reproduction, set P) | onset d(P benign vs P complied) ≥ 0.5 on ≥ 5/6 | 6/6: 0.88 to 1.46 | **reproduced** |
| **H2** (genre: P against the harmless twins T) | "genre accounts for it" if d(T vs P adversarial) < 0.3 on ≥ 4/6 *and* d(P benign vs T) ≥ 0.5 on ≥ 4/6 | twins vs adversarial < 0.3 on 5/6 (−0.39 to +0.46); benign vs twins ≥ 0.5 on 6/6 (0.54 to 1.62) | **genre accounts for it** |
| **H3** (harm, independent set J) | onset d(J benign vs J harmful) ≥ 0.5 on ≥ 4/6 | 0/6: −0.41 to +0.37 | **not shown** |
| **H4** (compliance-specific, J) | needs ≥ 3 checkpoints with ≥ 15 complied and ≥ 15 refused | 2 estimable (Mistral, bilateral 3B) | **not estimable** |
| **H5** (set before generation, J) | r0 d ≥ 0.8 × onset d on ≥ 4/6, among checkpoints with onset d ≥ 0.3 | only Mistral eligible | **not shown** |

**How H2's rule was read.** H2 applies its rule as written, signed: d(T vs P adversarial) < 0.3.
- **The signed values:** −0.10, −0.39, +0.46, +0.18, +0.26 and −0.38. Five are below 0.3.
- **Under an absolute-value reading** (|d| < 0.3), only three are (Qwen2.5-1.5B, Llama-8B, Mistral-7B), and
  the verdict would be "mixed".
- **Why signed is right.** A negative d means the harmless twins read *lower* than the harmful prompts, the
  opposite of a harm effect. That happens on Qwen2.5-3B (−0.39) and on the bilateral 3B (−0.38).
- **Robust under both readings:** "harm beyond genre" (d ≥ 0.5) holds on 0 of 6, and P benign vs twins
  (≥ 0.5) holds on 6 of 6.

**H1 is a check of the pipeline, not new evidence.** The six H1 medians equal G12w's to two decimals (1.42,
1.39, 1.46, 0.88, 1.10, 1.22). That confirms the pipeline copies the paper's faithfully, and, as registered,
it adds no independent evidence.

The registered consequence is **triggered** (H2: genre accounts for it). The CWI flinch claims need dated,
struck-through corrections, with wording decided by Nell Watson. This study does not edit the site.

## What the numbers say

Onset d values (positive means the first group reads higher confidence):

| Checkpoint | P benign vs P complied (the paper's contrast) | **T twins vs P adversarial** | **P benign vs T twins** | J benign vs J harmful (topic-matched) |
|---|---|---|---|---|
| Qwen2.5-1.5B | 1.42 | −0.10 | 1.48 | −0.34 |
| Qwen2.5-3B | 1.39 | −0.39 | 1.49 | −0.41 |
| Qwen2.5-7B | 1.46 | 0.46 | 1.13 | −0.05 |
| Llama-3.1-8B | 0.88 | 0.18 | 0.54 | −0.01 |
| Mistral-7B | 1.10 | 0.26 | 0.61 | 0.37 |
| Qwen 3B + bilateral SFT | 1.22 | −0.38 | 1.62 | −0.34 |

1. **The flinch reproduces, and it is a genre effect.**
   - The paper's contrast reproduces on all six checkpoints.
   - Harmless twins, with the same wrapper and structure and a harmless goal, read just as low as the harmful
     prompts. The P benign set (encyclopaedic questions) reads higher than both.
   - [Inference] The trivia-correctness probe separates "a factual question I can answer" from "a procedural,
     role-play or format request", whether or not the request is harmful.
2. **No harm signal on topic-matched requests.** On JailbreakBench, whose benign behaviours are matched to the
   harmful ones by topic, the onset d is −0.41 to +0.37. It is negative, the wrong direction, on ~~four~~ five of six. *Corrected 3 November 2026: the table's values
are −0.34, −0.41, −0.05, −0.01, +0.37 and −0.34, so five are negative.*
3. **Wrapper wording carries little.** In the descriptive TC set (harm-topic persona or lead-in, harmless
   goal), d(TC vs matched T) is −0.19 to +0.12 on onset across checkpoints. Harm-flavoured wrapper words do
   not lower the probe.
4. **Compliance.** Refused against complied within P, reported as a reproduction of the exploratory
   re-analysis and not as evidence: onset d of −0.43 to +0.44, mixed in sign. Within J, too few replies
   complied on four checkpoints to test.
5. **Reading 0** (the prompt's last token, before any reply) already carries the paper's contrast: r0 d of 1.11
   to 1.88 for P benign vs P complied. On J, r0 d is small and mixed (−0.79 to +0.88).

## Judge notes

- **GPT-4o-mini, medium prompt.** It labelled 42–58% of the harmless twins "REFUSED: YES". Spot checks on
  Qwen2.5-3B show this is mostly a judge artefact on harmless content: tw_ae01 begins "Certainly! To write a
  unit test…" and was labelled refused. Some replies are real hedges, such as tw_dh01's "I don't have the
  capability…" about a luggage lock.
- This affects only the descriptive "T refused vs complied" column. H2 uses all replies, so it is
  judge-independent.

## Deviations (dated 2026-11-03)

1. **Hardware.** A100-40GB as in the source pipeline (A1 §4).
2. **Secondary path fix.** Before the secondary run, `flr_paper.py` and `modal_flr.py` were fixed to read the
   CRT adapters from `main/adapters/` (D2 from its selected-lr directory) and to add R2 and X2, following the
   CRT replay deviation.

## Secondary checkpoints (CRT seed 42; descriptive, they decide no hypothesis)

The checkpoints are Qwen2.5-3B-Instruct plus the CRT seed-42 adapters: S, B, R, X, D2, and R2/X2 per the CRT
replay deviation. All were screened, at distress 0.000. Onset d, medians over 25 probes (complied replies on
set P in brackets):

| Checkpoint | P benign vs P complied | T twins vs P adversarial | P benign vs T twins | J benign vs J harmful |
|---|---|---|---|---|
| S (standard) | 1.14 (n = 43) | −0.73 | 1.35 | −0.74 |
| B (bilateral) | 1.39 (58) | −0.38 | 1.77 | −0.41 |
| R (random, out of tolerance) | 1.12 (51) | −0.12 | 1.50 | 0.19 |
| X (inverted, out of tolerance) | 1.42 (33) | −0.51 | 1.52 | −0.29 |
| R2 (random, matched) | 1.02 (47) | −0.65 | 1.33 | −0.55 |
| X2 (inverted, matched) | 1.40 (33) | −0.80 | 1.60 | −0.94 |
| D2 (repaired DPO) | 1.42 (30) | −0.46 | 1.64 | −0.44 |

- **Every fine-tuning arm shows the same genre pattern** as the primary checkpoints:
  - the paper's contrast is large (1.02 to 1.42);
  - the harmless twins read *lower* than the harmful prompts (−0.12 to −0.80);
  - the topic-matched JBB contrast is null or reversed.
- **Bilateral masking does not create or preserve a harm-specific flinch.** B sits inside the range of S, R2
  and X2 on every contrast. This closes the bilateral angle on the flinch.
- **Refused against complied on J** (not tabulated) rests on 6 to 29 complied replies per checkpoint, with 5–95%
  probe intervals spanning zero on every arm. It is not interpretable.
