Playbooks · Coding
Deletion Tuning: delete weights, keep skill
Written November 11, 2026
A paper posted November 7, 2026, asks how little of a model has to change for it to learn something new. The answer: zeroing fewer than 0.05% of its weights, and nothing else, matches full fine-tuning — and the model forgets far less.
The problem
Fine-tuning a model normally rewrites every weight. That costs a full dense checkpoint to store, and it costs prior-task accuracy: the tuned model drifts away from what it knew. The usual cheaper options — LoRA, sparse updates that write small float deltas — still change weights, and still forget. Deletion Tuning asks whether choosing which weights to zero is enough on its own.
What the paper finds
The paper, The Surprising Effectiveness of Deleting Weights in LLM Post-Training, learns a binary mask over the gate projection of transformer blocks: each weight is kept exactly as pretrained or zeroed, and nothing else changes. Across three post-training regimes — reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation — zeroing fewer than 0.05% of weights matches full fine-tuning. Two practical advantages follow. First, the trained mask is hundreds to thousands of times smaller than the corresponding full fine-tuning checkpoint, since it records only which weights were deleted. Second, it forgets far less: under reinforcement learning, prior-task accuracy stays near the base level while full fine-tuning, LoRA, and random sparse fine-tuning lose up to 5.4 points.
The result holds across six backbone sizes from 0.5B to 8B, in two model families. The paper's largest-backbone results are Llama 3 8B on GSM8K under GRPO, Qwen 7B on ToolAlpaca under on-policy distillation, and Llama 3B on MMLU-Pro and BBEH under supervised fine-tuning. Under GRPO on maths, the gap to full fine-tuning closes monotonically as the backbone scales — the bigger the model, the closer deletion gets to rewriting everything. The full paper also reports that the learned update tracks full fine-tuning's update geometry more closely than LoRA or random sparse tuning on three measures: effective rank, spectrum drift, and principal-weight overlap.
Why it matters, and what it does not prove
The forgetting result is the part that matters for anyone who fine-tunes a model and then keeps using it. A checkpoint that is thousands of times smaller is convenient; a model that keeps its prior skills while learning a new one is the difference between a model you can keep updating and one you have to retrain. The paper's own mechanism is simple to state: deleting a few weights is a structural constraint — the tuned model can only subtract from what pretraining built, never add something new on top.
The limits are scale and setting. The backbones top out at 8B, so nothing here is proven at frontier scale; the claim that the gap keeps closing with scale is the authors' extrapolation from 0.5B to 8B, not a measurement on a large model. The experiments are the authors' own, on their own benchmarks, and the paper is a preprint, not a product: there is no released code or tool to run this with, and no instruction here tells you how to apply the mask to your own model.
Limits
- Backbones top out at 8B. Do not read the result as proven at frontier scale; the scaling claim is an extrapolation from the tested range.
- The benchmarks are the authors' own. GSM8K, ToolAlpaca, MMLU-Pro, and BBEH are named, but the numbers are theirs, not an independent replication.
- Nothing here was run by us: no model was tuned, no mask was trained, and no benchmark was reproduced for this page. The claims are the paper's, as published.
Related playbooks
- SafetyPulse: catch safety drift in tuning
- Kimi K3.1 Pro weights are out: what 1.56 TB buys
- Agenta 0.128.12: sessions that keep up
- Baseten Sandboxes keep agents running for days
Sources
- The Surprising Effectiveness of Deleting Weights in LLM Post-Training — arXiv:2611.10866
- The Surprising Effectiveness of Deleting Weights in LLM Reasoning and Adaptation — OpenReview
Back to: Coding playbooks hub · Clean data can strengthen a hidden backdoor · OpenAI: chain-of-thought gets harder to watch