Playbooks · Coding tasks
Together Compiler: agents that speed inference
Written November 6, 2026
The Together compiler loop uses coding agents to rewrite model forward passes, claiming up to 2.6x throughput. The claim, the scope, the catch.
Inference providers sell speed, and Together now claims coding agents rewriting forward passes deliver up to 2.6 times the throughput of the best published configurations. The system is called Together Compiler, and it is already running on several Together serverless endpoints. The number is a Together measurement against a named baseline, on a named set of models and hardware — and the per-setting gains range from about 4% to 2.6 times. A builder choosing an inference endpoint needs the scope before the headline.
What Together Compiler is
Together Compiler (TKCC) is a continual compilation loop: coding agents rewrite a model's forward pass, a persistent engine measures each candidate against real traffic, and the winning code ships into the serving engine without a full restart. As production traffic changes — longer prompts in one region, a burst from one customer, a new model release shifting usage toward agentic workloads — the loop re-optimizes for the new shape. That is the Together blog post's framing: tuning should never stop, because the traffic it tunes for never stops changing.
Three ideas make the loop practical, according to the post:
- Persistent engines. Existing agent systems relaunch the whole inference engine to test each candidate, and engine startup takes minutes. TKCC starts the engine once and hot-reloads candidate forward passes, cutting time-to-serving from 467 seconds to 9.0 seconds in the Together table — and reporting 252 times higher agent experiment throughput than a restart-based baseline.
- Tiered measurement. Fast engines load a few layers for rapid iteration; slower engines load many layers to check that an optimization holds on real data distributions. The split matters because modern architectures use data-dependent primitives — mixture-of-experts routing, sparse attention — where the input distribution decides which optimization wins.
- A cost model for parallelism. A tiered measurement strategy ranks parallelism configurations (tensor, expert, and other splits) before agents spend time optimizing them, so the search starts from a layout the measurements already favour.
One finding in the paper is worth noting for anyone building agent systems: agents given raw engine source, with a single instruction to make the forward pass as fast as possible, beat agents handed an expert-curated knowledge base of optimization strategies by 25 to 31 percent less speedup after 12 hours — the naive agents won. The experts' guidance biased the search.
The claim, with its scope
The headline is up to 2.6 times higher tokens per second than the best published InferenceX configurations, on SemiAnalysis' InferenceX AgentX benchmark, reached in under 24 hours per setting on at most two nodes per model. The models in scope are Kimi K3, GLM 5.3, MiniMax M3.1 Flash, and DeepSeek v4.1 Flash, plus Llama 1B for the megakernel comparison. The hardware in scope is NVIDIA B200, B300, GB300, and H100, AMD MI355X, and AWS Trainium2.
| Setting (batch size 1, P90 TPS per user) | Together Compiler | InferenceX best |
|---|---|---|
| MiniMax M3.1 Flash, B200 | 1,111 | 633 |
| DeepSeek v4.1 Flash, GB300 | 1,205 | 943 |
| GLM 5.3, B200 | 417 | 329 |
| Kimi K3, GB300 | 450 | 423 |
| GLM 5.3, MI355X | 480 | 463 |
Read the table before the headline: the strongest result is 1.8 times on MiniMax, and two settings improve by about 4 to 6 percent. A secondary write-up reports the spread across all settings as roughly 4% to 2.6x, with a 1.38x geometric mean and wins on 149 of 235 settings. On Llama 1B, the megakernel workhorse, TKCC reports up to 1.7 times higher TPS than hand-tuned megakernels. Beyond language models, the system reports speedups on eight scientific models — weather modelling, PDEs, time series, biology — with bit-exact outputs.
The catch
- It is a vendor benchmark. Together measured its own system against a named baseline and published the result. No independent reproduction exists. The 2.6x is a maximum across settings, not a typical gain — the geometric mean is 1.38 times.
- One setting ran on synthetic data. The Trainium2 result used a synthetic multi-turn dataset instead of the AgentX workload, because the Together implementation does not yet support higher concurrencies on that hardware's limited memory. The footnote in the blog post says so plainly.
- It is per-setting, not fleet-wide. Each model-and-hardware pair was optimized separately, in under 24 hours each. There is no single number for "Together is now 2.6x faster" — there are 235 settings, and 86 of them did not beat the baseline.
- You cannot download it. Together Compiler is an internal loop at Together, running on its own endpoints. If you buy inference from Together, the compiled endpoints are the product. If you run your own inference stack, the loop is an idea to borrow, not a tool to install.
What a builder takes from it
If you choose an inference endpoint by throughput, ask what the provider's number covers: which models, which hardware, which batch size, and against which baseline. Together names all four, which is more than most launch posts do — and the naming is what lets you see that the 2.6x sits on one setting while most settings gain far less. If you run your own stack, the transferable lesson is the architecture: decouple engine startup from forward-pass measurement so agents can iterate fast, measure on real traffic shapes rather than random inputs, and be careful with expert guidance — the paper's own ablation says naive agents with raw source beat curated expert strategies.
Limits
- This page is written from the Together blog post and paper, with a secondary write-up for the per-setting spread. No inference run was made here, and no throughput figure was measured independently.
- All performance figures are Together-reported. Treat them as the vendor's claim until an independent party reproduces them.
- The per-setting table above carries the settings the blog post names. The full 235-setting spread and the full hardware enumeration (B200, B300, GB300, H100, MI355X, Trainium2) come from the secondary write-up; The Together post names H100 and B200 in its figures, GB300 in its low-latency results, and Trainium2 in its footnote.
Related playbooks
- Claude agents now dream: memory that consolidates
- Qwen Code 0.27: agents that ask questions
- Addy Osmani's 25 agent skills for coding agents
- Fingerprinting AI coding agents from 33,580 PRs
Sources
- Together Compiler: Continual Compilation for AI inference — Together blog post: the loop, the 2.6x claim, the baseline, the per-setting results
- Together Compiler: Continual Compilation for AI Inference — Together paper: method, ablations, full evaluation
- Together AI Releases Together Compiler: Agents Make Inference Up to 2.6x Faster — MarkTechPost, secondary: per-setting spread, geometric mean, hardware list
- Kimi K3.1 Pro weights are out: what 1.56 TB buys — Agent Upstart: Kimi, one of the models in the evaluation
- Ripple-65B: the $32M open model topping coding tests — Agent Upstart: open models and inference choices
- Model prices: what the major models cost — Agent Upstart: what inference costs, alongside how fast it runs
Back to: Coding playbooks hub · Also: Kimi K3.1 Pro weights are out: what 1.56 TB buys · Ripple-65B: the $32M open model topping coding tests · Model prices: what the major models cost