Agent Upstart

Cool things people are building with agents — and practical help getting the most out of them.

Playbooks · Coding tasks

Together Compiler: agents that speed inference

Written November 6, 2026

The Together compiler loop uses coding agents to rewrite model forward passes, claiming up to 2.6x throughput. The claim, the scope, the catch.

Inference providers sell speed, and Together now claims coding agents rewriting forward passes deliver up to 2.6 times the throughput of the best published configurations. The system is called Together Compiler, and it is already running on several Together serverless endpoints. The number is a Together measurement against a named baseline, on a named set of models and hardware — and the per-setting gains range from about 4% to 2.6 times. A builder choosing an inference endpoint needs the scope before the headline.

What Together Compiler is

Together Compiler (TKCC) is a continual compilation loop: coding agents rewrite a model's forward pass, a persistent engine measures each candidate against real traffic, and the winning code ships into the serving engine without a full restart. As production traffic changes — longer prompts in one region, a burst from one customer, a new model release shifting usage toward agentic workloads — the loop re-optimizes for the new shape. That is the Together blog post's framing: tuning should never stop, because the traffic it tunes for never stops changing.

Three ideas make the loop practical, according to the post:

One finding in the paper is worth noting for anyone building agent systems: agents given raw engine source, with a single instruction to make the forward pass as fast as possible, beat agents handed an expert-curated knowledge base of optimization strategies by 25 to 31 percent less speedup after 12 hours — the naive agents won. The experts' guidance biased the search.

The claim, with its scope

The headline is up to 2.6 times higher tokens per second than the best published InferenceX configurations, on SemiAnalysis' InferenceX AgentX benchmark, reached in under 24 hours per setting on at most two nodes per model. The models in scope are Kimi K3, GLM 5.3, MiniMax M3.1 Flash, and DeepSeek v4.1 Flash, plus Llama 1B for the megakernel comparison. The hardware in scope is NVIDIA B200, B300, GB300, and H100, AMD MI355X, and AWS Trainium2.

Setting (batch size 1, P90 TPS per user)Together CompilerInferenceX best
MiniMax M3.1 Flash, B2001,111633
DeepSeek v4.1 Flash, GB3001,205943
GLM 5.3, B200417329
Kimi K3, GB300450423
GLM 5.3, MI355X480463

Read the table before the headline: the strongest result is 1.8 times on MiniMax, and two settings improve by about 4 to 6 percent. A secondary write-up reports the spread across all settings as roughly 4% to 2.6x, with a 1.38x geometric mean and wins on 149 of 235 settings. On Llama 1B, the megakernel workhorse, TKCC reports up to 1.7 times higher TPS than hand-tuned megakernels. Beyond language models, the system reports speedups on eight scientific models — weather modelling, PDEs, time series, biology — with bit-exact outputs.

The catch

What a builder takes from it

If you choose an inference endpoint by throughput, ask what the provider's number covers: which models, which hardware, which batch size, and against which baseline. Together names all four, which is more than most launch posts do — and the naming is what lets you see that the 2.6x sits on one setting while most settings gain far less. If you run your own stack, the transferable lesson is the architecture: decouple engine startup from forward-pass measurement so agents can iterate fast, measure on real traffic shapes rather than random inputs, and be careful with expert guidance — the paper's own ablation says naive agents with raw source beat curated expert strategies.

Limits

Related playbooks

Sources

Back to: Coding playbooks hub · Also: Kimi K3.1 Pro weights are out: what 1.56 TB buys · Ripple-65B: the $32M open model topping coding tests · Model prices: what the major models cost