Playbooks · Coding
SafetyPulse: catch safety drift in tuning
Written November 10, 2026
Fine-tuning a small model on harmless data can quietly weaken its refusals. SafetyPulse watches for that drift at every checkpoint, and can repair it — before the model ships unsafe.
The problem
A team fine-tuning an open-weight model for a domain or a language usually checks task skill at every checkpoint and safety only at the end, if at all. Auditing every checkpoint means generating answers to hundreds of harmful prompts and judging them, so it gets skipped — and a model that refuses less gets shipped. The drift is real even when the training data is benign: the SafetyPulse post fine-tuned Cohere Labs' Tiny Aya Global (3.35B parameters) on 2,000 ordinary Alpaca instructions and watched its attack success rate on held-out HarmBench prompts rise from 14.3% to 25.2%. Over-refusal fell at the same time, from 7.3% to 1.3%: the model refused less of everything, not just the harmful prompts.
How the monitor works
SafetyPulse locates the model's refusal circuit once — 14 MLPs and 6 heads in layers 16–34, found by attribution patching — and scores every checkpoint with a Circuit Integrity Score (CIS) from 50 prompt pairs. The thresholds are fixed in advance: below 0.818 is FAIL, 0.818 to 0.909 is WARN. On the held-out test runs, the monitor flagged every unsafe checkpoint, 1–3 checkpoints before the harm was measurable, and the negative control never raised an alarm. A plain 50-prompt refusal spot-check caught only 13% of unsafe checkpoints, because an 8-point refusal drop sits inside its noise.
One check takes 22 seconds on an L4 GPU, about half a cent. Checking all 11 checkpoints of a run costs about 5 cents, under a tenth of the fine-tune it protects.
The catch: false alarms
The monitor's precision is 0.46, which means more than half its warnings are false alarms. It warned on a GSM8K maths fine-tune that moved the circuit without reducing safety, though that run never reached FAIL. At FAIL level only, precision rises to 0.68 but recall falls to 0.57, because the score partly recovers late in Alpaca training while the attack success rate stays high. The circuit score tracks harm better than direction-drift scores (correlation −0.80 against −0.56), but it is a screening signal: a PASS does not certify a model as safe.
The repair
Flagging drift is only half the tool. The post also tests repairs, and the results are not what you would guess. Restoring the refusal circuit's own weights does nothing: the attack success rate stays at 25.2%. The circuit reads out the drift but does not store it, so putting its weights back leaves the drift in place. What works is SafeTune's low-rank safety-vector restore, which rolls the whole model part-way back toward the base: the attack success rate falls from 25.2% to 12.3%, below the base model's 14.3%, and GSM8K accuracy recovers from 55.1% to 70.7%, near the base model's 71.3%. Curating the training data does not prevent the drift either: removing 19 of 2,000 Alpaca examples with a curation toolkit left the drift unchanged, and the monitor still failed the run at step 25.
Trying it
The tool is on PyPI as safetypulse. The post's recipe needs Python 3.12, a GPU with 16 GB or more, and access to the gated base model. After installing with pip install "safetypulse[peft,toolkits]", safetypulse check scores a checkpoint against a probe bundle (exit code 0 for PASS, 10 for WARN, 20 for FAIL), and safetypulse repair applies the SafeTune restore. The repaired model the post ships, tiny-aya-global-safetypulse-repaired, scores 11.8% attack success rate against the base model's 14.3%, with GSM8K back at 71.3%.
Limits
- One base model, one benign condition that drifts. The post itself notes that drift signals are reported to vary across models, and that no jailbreak-style attacks were evaluated, only direct harmful requests.
- The attack success rate is measured on 78 and 119 prompts, so the margins are wide (plus or minus 8–11 points). The judge is Llama Guard 3 1B, and judging the same answers with the 8B version gives the same conclusions, but neither judge has been validated against human labels.
- Precision 0.46 means the monitor cries wolf often. Treat a WARN as a reason to look, not as proof the model is unsafe, and treat a PASS as a clean screening result, not a safety certificate.
- All of this is the authors' work, reported in a Hugging Face blog post from a hackathon entry. Nobody independent has reproduced it.
Related playbooks
- Deletion Tuning: delete weights, keep skill
- Astra did 40% less work on safety tasks
- Catch bad agent behaviour chains with Quarry
- 140 silent MCP contract changes, and how to pin them
Sources
Back to: Coding playbooks hub · Also: MOSS-Live: full-duplex spoken AI