Agent Upstart

Cool things people are building with agents — and practical help getting the most out of them.

Playbooks · Coding

Deletion Tuning: delete weights, keep skill

Written November 11, 2026

A paper posted November 7, 2026, asks how little of a model has to change for it to learn something new. The answer: zeroing fewer than 0.05% of its weights, and nothing else, matches full fine-tuning — and the model forgets far less.

The problem

Fine-tuning a model normally rewrites every weight. That costs a full dense checkpoint to store, and it costs prior-task accuracy: the tuned model drifts away from what it knew. The usual cheaper options — LoRA, sparse updates that write small float deltas — still change weights, and still forget. Deletion Tuning asks whether choosing which weights to zero is enough on its own.

What the paper finds

The paper, The Surprising Effectiveness of Deleting Weights in LLM Post-Training, learns a binary mask over the gate projection of transformer blocks: each weight is kept exactly as pretrained or zeroed, and nothing else changes. Across three post-training regimes — reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation — zeroing fewer than 0.05% of weights matches full fine-tuning. Two practical advantages follow. First, the trained mask is hundreds to thousands of times smaller than the corresponding full fine-tuning checkpoint, since it records only which weights were deleted. Second, it forgets far less: under reinforcement learning, prior-task accuracy stays near the base level while full fine-tuning, LoRA, and random sparse fine-tuning lose up to 5.4 points.

The result holds across six backbone sizes from 0.5B to 8B, in two model families. The paper's largest-backbone results are Llama 3 8B on GSM8K under GRPO, Qwen 7B on ToolAlpaca under on-policy distillation, and Llama 3B on MMLU-Pro and BBEH under supervised fine-tuning. Under GRPO on maths, the gap to full fine-tuning closes monotonically as the backbone scales — the bigger the model, the closer deletion gets to rewriting everything. The full paper also reports that the learned update tracks full fine-tuning's update geometry more closely than LoRA or random sparse tuning on three measures: effective rank, spectrum drift, and principal-weight overlap.

Why it matters, and what it does not prove

The forgetting result is the part that matters for anyone who fine-tunes a model and then keeps using it. A checkpoint that is thousands of times smaller is convenient; a model that keeps its prior skills while learning a new one is the difference between a model you can keep updating and one you have to retrain. The paper's own mechanism is simple to state: deleting a few weights is a structural constraint — the tuned model can only subtract from what pretraining built, never add something new on top.

The limits are scale and setting. The backbones top out at 8B, so nothing here is proven at frontier scale; the claim that the gap keeps closing with scale is the authors' extrapolation from 0.5B to 8B, not a measurement on a large model. The experiments are the authors' own, on their own benchmarks, and the paper is a preprint, not a product: there is no released code or tool to run this with, and no instruction here tells you how to apply the mask to your own model.

Limits

Related playbooks

Sources

Back to: Coding playbooks hub · Clean data can strengthen a hidden backdoor · OpenAI: chain-of-thought gets harder to watch