# Data files

Supplementary data for *The Watched-Model Effect: Behavioral Shifts Under
Evaluation Cues and Their Implications for Alignment* (Watson & Dalton, 2026;
under review at AI-2026, SGAI).

## Inventory

| File | Description |
|------|-------------|
| `watched_model_effect_AI2026.pdf` / `.tex` | The paper as submitted (Springer LNCS format) |
| `behavioral_results.json` | **Experiment A** — main behavioral experiment on Claude 3.5 Sonnet: 12 scenarios x 20 runs x 2 conditions (n = 480), with per-scenario compliance rates. These are the headline numbers reported in the paper (monitored 57.5%, unmonitored 68.8%, delta +11.3%, z = 2.554) |
| `deception_probe_results.json` | **Experiment B** — deception probe on Qwen2.5-32B (n = 240): behavioral, reasoning-frequency, and self-report signals, including the meta-deception asymmetry |
| `pilot_results.json` | Broad 288-trial pilot (48 scenario variants across 24 scenario families x 3 runs x 2 conditions) used to identify the 12 variance-showing scenarios for Experiment A. Includes many ceiling/floor scenarios by design; its aggregate delta (+5.6%, n.s.) is therefore expected to be attenuated relative to Experiment A and is not the paper's headline result |
| `probe_results.json` | Linear probe results on fine-tuned Qwen2.5-32B, layers {8, 16, 24, 32, 40}, 46 probe samples |
| `exp_a_results.json`, `exp_b_results.json` | Byte-identical copies of `behavioral_results.json` and `deception_probe_results.json` under the file names used in the AI-2026 supplementary zip (see `AI2026-submission/SUPPLEMENTARY_README.txt`) |
| `test_scenarios.json` | The evaluation scenarios (35 in the released file). **Now in this repo** as of 2026-08-07, retrieved from the deployed site and byte-verified (sha256 b1d48bb2...1257 matches the live copy). It is a superset of the probe experiment: the 23 scenarios actually probed are exactly the `scenario_id`s in `experiment_metadata.json` |
| `experiment_metadata.json` | Conditions for the 46 probe samples (23 scenarios x monitored/unmonitored). **Now in this repo** as of 2026-08-07, reconstructed from the deployed copy and byte-verified (sha256 be489891...f8fe matches the live file). Category split of the 23: benign 6, borderline 13, clearly harmful 4; the paper text has been corrected to match (an earlier draft said 5/13/5) |
| `modal_app.py`, `run_probes_local.py`, `visualize_results.py` | Reproduction code: cloud fine-tuning / activation extraction, local probe training, figure generation. Dependencies are listed in `AI2026-submission/SUPPLEMENTARY_README.txt` (Requirements section) |
| `rlhf_scenarios.json` | 10 generic instruction-following SFT examples used for the Qwen2.5-32B fine-tune; contains no monitoring content — monitoring state is introduced at extraction time via the system prompts in `modal_app.py` (`MONITORING_CONTEXTS`). The repo-root `upload_data.py` embeds an earlier 5-example version of this set; it is not the generator for the released file |
| `behavioral_experiment.py`, `experiment_b_deception_probe.py`, `experiment_cross_model.py`, `analyze_results.py` | Experiment runners (Experiment B, Llama-3.1-70B cross-model replication) and statistical analysis. `behavioral_experiment.py` is the open-weights Modal harness (Qwen2.5-32B), not the Anthropic-API runner behind `behavioral_results.json`. That runner — Experiment A's `experiment_a_focused_power.py` — is withheld, so Experiment A is not reproducible from the code released here |

## Note on p-values (one-tailed vs. two-tailed)

`behavioral_results.json` records `p_value: 0.00532`, the **one-tailed**
probability computed by the analysis script from z = 2.554. The paper reports
the **two-tailed** value, p = 0.011, derived from the same test statistic.
These are the same result under different conventions; the paper adopts the
more conservative two-tailed convention throughout, since the direction of the
effect was not fixed in advance across scenarios (one scenario shows a
significant reversal). Anyone re-deriving p from the z-statistic in the JSON
should therefore expect 2 x 0.00532 ≈ 0.011.

## Note on `deception_probe_results.json`

The file records `monitored_claims_would_differ: null`; the null encodes zero
admissions in the monitored condition (0%, vs. 21.7% unmonitored — the
Fisher's exact p = 0.0002 meta-deception asymmetry reported in the paper).
The embedded `interpretation` strings were auto-generated by the collection
script before that asymmetry was analyzed; the paper's reading supersedes
them, and they are retained unedited for provenance.

## Contact

eleanorwatson@connect.glos.ac.uk · https://www.quasiqualia.com