Agent Upstart

Cool things people are building with agents — and practical help getting the most out of them.

Playbooks · Coding tasks

Check your agent still passes its known tasks after its model changes under the same name

Written October 30, 2026

For anyone running an agent on an API model name — a fixed name that can start serving new weights, with no code change on your side. Re-run a small fixed set of tasks you already know the answers to, and compare the outputs and the cost. Old scores describe weights that no longer answer, and shorter outputs can mean dropped steps, not savings.

What happened with DeepSeek Flash

DeepSeek's API changelog for 17 October 2026 says DeepSeek V4.1 Flash was upgraded to DeepSeek-V4.1-Flash-1017, and that the API calling method and pricing remain unchanged: callers keep using deepseek-flash to reach the latest model. The retired names deepseek-v4-flash and deepseek-v4-flash-vision-exp are routed to the new version. The same changelog entry says the new version reduces task runtime and output length. That is the risk in one line: results you measured in September describe a model that no longer answers your requests, and shorter outputs can mean dropped steps, not just saved tokens.

On the same day, DeepSeek opened its in-house Search API. Search and Fetch calls are a separate line from model tokens, so a re-run after a swap should count them separately. Budget DeepSeek Search and Fetch calls walks through counting them, and the Work out what DeepSeek Search and Fetch calls will cost before billing starts on 1 November 2026 estimates what a given call volume costs.

Before you start

Pick a regression set before you re-run anything: five to ten tasks your agent already does, with the exact prompt for each, the expected output or the test that passes, and the output length and token count from the last run you trusted. Save the prompts and the expected outputs in a file you do not edit during the re-run. If the set changes mid-test, the comparison is gone.

Steps

  1. Record the starting point. For each task in the set, write down the prompt, the expected output, and the previous run's output-token count and output length. Note the date and time of each run, so peak and off-peak runs can be told apart later.
  2. Run the whole set on the current endpoint, unchanged. Use the same prompts, the same settings, and the same model name you use in production. Save every output exactly as returned, together with the model name returned with the response and the token counts for that run.
  3. Split the cost comparison across peak and off-peak hours. Run the set once in a peak window and once in an off-peak window, or compare the two windows from your billing page if your provider prices them differently. Record the cost per task in each window. A swap that shortens outputs can still cost more if your runs land in peak hours.
  4. Compare output by output. For each task, note three things: did the result still pass, did the output get longer or shorter, and did the format change in a way a downstream step would trip on. A shorter output that still passes is a saving. A shorter output that drops a step, a heading, or a required field is a behaviour change.
  5. Decide, per task, not per model. Keep tasks that pass unchanged. For tasks that changed, adjust the prompt or the downstream step, then re-run that task alone until it passes again. Where your provider offers a pinned checkpoint name, pin it for the tasks that need stable behaviour, and keep the unpinned name only for tasks you re-run often. On DeepSeek's own API there is no pinned name for the September checkpoint — deepseek-flash and the retired Flash names all serve DeepSeek-V4.1-Flash-1017 — so pinning is not an option there; for a task that needs the old behaviour, change the prompt or move that task to a service that serves a dated checkpoint ID.
  6. Write the result down. One short note: the model name, the date, which tasks passed, which changed, and what you changed in response. The next swap starts from this note, not from memory.

What can go wrong

Limits

Related playbooks

Sources

Back to: Coding playbooks hub · Also: Budget DeepSeek Search and Fetch calls · Work out what DeepSeek Search and Fetch calls will cost before billing starts on 1 November 2026