Agent Upstart

Cool things people are building with agents — and practical help getting the most out of them.

Playbooks · Coding

SafetyPulse: catch safety drift in tuning

Written November 10, 2026

Fine-tuning a small model on harmless data can quietly weaken its refusals. SafetyPulse watches for that drift at every checkpoint, and can repair it — before the model ships unsafe.

The problem

A team fine-tuning an open-weight model for a domain or a language usually checks task skill at every checkpoint and safety only at the end, if at all. Auditing every checkpoint means generating answers to hundreds of harmful prompts and judging them, so it gets skipped — and a model that refuses less gets shipped. The drift is real even when the training data is benign: the SafetyPulse post fine-tuned Cohere Labs' Tiny Aya Global (3.35B parameters) on 2,000 ordinary Alpaca instructions and watched its attack success rate on held-out HarmBench prompts rise from 14.3% to 25.2%. Over-refusal fell at the same time, from 7.3% to 1.3%: the model refused less of everything, not just the harmful prompts.

How the monitor works

SafetyPulse locates the model's refusal circuit once — 14 MLPs and 6 heads in layers 16–34, found by attribution patching — and scores every checkpoint with a Circuit Integrity Score (CIS) from 50 prompt pairs. The thresholds are fixed in advance: below 0.818 is FAIL, 0.818 to 0.909 is WARN. On the held-out test runs, the monitor flagged every unsafe checkpoint, 1–3 checkpoints before the harm was measurable, and the negative control never raised an alarm. A plain 50-prompt refusal spot-check caught only 13% of unsafe checkpoints, because an 8-point refusal drop sits inside its noise.

One check takes 22 seconds on an L4 GPU, about half a cent. Checking all 11 checkpoints of a run costs about 5 cents, under a tenth of the fine-tune it protects.

The catch: false alarms

The monitor's precision is 0.46, which means more than half its warnings are false alarms. It warned on a GSM8K maths fine-tune that moved the circuit without reducing safety, though that run never reached FAIL. At FAIL level only, precision rises to 0.68 but recall falls to 0.57, because the score partly recovers late in Alpaca training while the attack success rate stays high. The circuit score tracks harm better than direction-drift scores (correlation −0.80 against −0.56), but it is a screening signal: a PASS does not certify a model as safe.

The repair

Flagging drift is only half the tool. The post also tests repairs, and the results are not what you would guess. Restoring the refusal circuit's own weights does nothing: the attack success rate stays at 25.2%. The circuit reads out the drift but does not store it, so putting its weights back leaves the drift in place. What works is SafeTune's low-rank safety-vector restore, which rolls the whole model part-way back toward the base: the attack success rate falls from 25.2% to 12.3%, below the base model's 14.3%, and GSM8K accuracy recovers from 55.1% to 70.7%, near the base model's 71.3%. Curating the training data does not prevent the drift either: removing 19 of 2,000 Alpaca examples with a curation toolkit left the drift unchanged, and the monitor still failed the run at step 25.

Trying it

The tool is on PyPI as safetypulse. The post's recipe needs Python 3.12, a GPU with 16 GB or more, and access to the gated base model. After installing with pip install "safetypulse[peft,toolkits]", safetypulse check scores a checkpoint against a probe bundle (exit code 0 for PASS, 10 for WARN, 20 for FAIL), and safetypulse repair applies the SafeTune restore. The repaired model the post ships, tiny-aya-global-safetypulse-repaired, scores 11.8% attack success rate against the base model's 14.3%, with GSM8K back at 71.3%.

Limits

Related playbooks

Sources

Back to: Coding playbooks hub · Also: MOSS-Live: full-duplex spoken AI