and its system card. The decisions that made it good stay in-house: how it was trained, what its reinforcement learning ran against, how they got it cheap enough to serve. That’s the normal deal with a frontier lab, and most outside writing about these systems has to work from the outputs alone.
Kimi K3 is the exception worth reading. It’s open, it sits close to the best closed models on most benchmarks, and Moonshot published a forty-seven technical report that walks through the parts other labs keep private. I read it over an afternoon. What stayed with me is how small a share of the work is the model itself.
One caveat up front. Everything specific below is Moonshot’s. I assume the closed labs do their own versions of the same categories of work, but I’m inferring that from the outside, so read the wider “this is what the frontier looks like” claims as a guess.
The architecture is a stack of small changes
The architecture is the obvious place to start, and it holds the fewest surprises. K3 is a 2.8-trillion-parameter mixture-of-experts model, and what lifts it over the last Kimi is three fairly ordinary engineering changes stacked together.
The attention keeps a fixed-size running state instead of a cache that grows with the input, which is what makes a million-token context affordable; most layers use that cheap version, and every fourth does full attention. Positions aren’t encoded explicitly, so the recurrence has to carry them, which lets the model stretch to a million tokens without the usual rescaling hacks. Each layer can also look back at every layer beneath it, not only the one directly below, so early signal doesn’t wash out on the way up. And each token is routed to 16 of 896 experts, sparser than before, which takes real care to keep stable.
Moonshot reports a 2.5× gain in scaling efficiency over Kimi K2, roughly the same quality for under half the training compute. It credits the architecture together with refined data and training recipes, without apportioning the gain between them, so don’t read the 2.5× as three architecture changes multiplying out. Each piece is an ordinary, well-tested idea rather than a single breakthrough.
The three changes in plain English
Each of the three is easy to hold in your head once you drop the notation, and the plain version is where the intuition lives.
Attention is a lookup table, and that’s why long context is expensive. A transformer handles each token by looking back over every earlier token and pulling a weighted blend of them, a soft lookup. To do that it keeps a small key-and-value record for every token so far, the KV cache. Think of a filing cabinet that never throws a card away: each new word files a card, then flicks through every card already in the drawer to decide what to attend to. Fine at a few thousand words. At a million, the drawer is enormous and every new word riffles the entire thing, so the cost climbs with the square of the length.
K3’s main attention swaps the cabinet for a single running summary, a fixed-size notepad it writes over as it goes. The notepad never grows, and that fixed size is what makes a million-token context affordable. What you give up is that a summary can’t keep everything, so it has to forget, and K3 hands it a per-feature dial for how fast old detail fades. A summary also can’t recall an exact earlier token on demand, so K3 keeps one true cabinet layer for every three notepad layers and buys the precise recall back where it counts.
Figure 1: Two ways to mix over tokens. Full attention stores a key/value entry per token and compares each new token against all of them (cost ∝ n²); Kimi Delta Attention folds tokens into one fixed-size state through a forget gate α (cost ∝ n). K3 stacks three linear layers per global layer. 📖 Source: image by author.
The fixed-state layers already track order as they update, which lets K3 drop explicit positional encodings entirely. Most models bolt position onto attention with something like RoPE, and stretching the context window later means rescaling those frequencies or interpolating them, which is fiddly and lossy. K3’s recurrence carries position for free, so the same weights run at 8K during early training and at 1M after the long-context stage with no positional surgery in between.
The same lookup trick, one level up. Stack a lot of layers and each one normally adds its output to a shared running total that flows up the network. Picture that total as a notepad passed up a line of people, each scribbling a line. By the top the first few lines are buried, and the network burns capacity just keeping early information alive. Attention Residuals let a layer reach back and read earlier layers’ outputs directly, weighting them with a softmax, the same soft-lookup machinery as attention but pointed across depth instead of across the sequence. A layer pulls a blend from source instead of playing telephone up the stack. The cost is memory rather than new machinery: every layer’s output has to stay live for the ones above it, so K3 attends over a handful of block summaries instead of all of its ninety-odd layers to keep that bill down.
Figure 2: Passing information up through the layers. A standard residual stream carries a single running total, so early-layer signal is buried by the top; Attention Residuals instead let each layer read earlier layers’ outputs directly, weighting them with a softmax (the α on the arrows): the same soft-lookup idea applied across depth instead of across tokens. 📖 Source: image by author.
Mixture of experts routes each token to a handful of specialists. Rather than push every token through one giant feed-forward network, K3 keeps 896 smaller expert networks and a router that hands each token to just 16 of them. Picture a hospital with 896 specialists and a triage desk. No patient sees everyone; the desk picks the few who fit. That’s how the model can hold 2.8 trillion parameters and still only run 104 billion for any given token. Capacity and per-token cost come apart, so you can add experts to widen the model’s range without paying to run them all. The saving is in compute, not memory: all 2.8 trillion parameters still have to sit in fast memory to be reachable, which is part of why serving a model this size is a cluster problem. The snag is that routers play favourites. Leave it alone and it learns to funnel most tokens to a handful of star experts, which jam up while the rest sit idle and undertrained. So a working MoE has to force the load flat, the way a maître d’ spreads diners across all the waiters instead of seating every table in one section. K3 uses a rule that nudges each expert toward an equal share, and wrestling with that balance is a good part of why big sparse models are twitchy to train.
Figure 3: A big panel and a selective router. Each token is routed to 16 of 896 experts, so a 2.8T-parameter model runs only 104B per token. Left unmanaged, the router overloads a few experts and starves the rest; a balancing rule spreads the load evenly. 📖 Source: image by author.
It comes down to three ideas: soft lookup, which turns up over tokens and again over layers; a fixed-size running state that stands in for that lookup wherever it got too expensive; and sparsity, which lets the model grow wide without growing its per-token cost.
Making a serial recurrence run on parallel hardware
The running state that makes Kimi Delta Attention (KDA) cheap creates its own problem, and how Moonshot handles it is one of the more interesting parts of the report. A recurrence is serial by nature: the state after token t depends on the state after token t−1, so the naive implementation walks the sequence one token at a time. A GPU is the opposite kind of machine, built to run thousands of operations at once, and a strict left-to-right loop leaves nearly all of it idle.
The way out is a chunkwise formulation. Split the sequence into chunks of a few hundred tokens. Inside a chunk, the recurrence can be rewritten as a couple of dense matrix multiplications, which is exactly what the GPU’s tensor cores are fast at, so everything within a chunk runs in parallel. Only the small, fixed-size state has to pass from one chunk to the next, and that hand-off is the sole serial step left. Moonshot’s kernel then overlaps the hand-off with the next chunk’s matrix work, so the cores rarely stall waiting on it. Almost all of the cost turns into parallel matmul, and the serial part shrinks to a sliver of the runtime.
Spreading a single very long sequence across several GPUs needs one more idea. Ordinary attention has to ship a growing block of keys and values between GPUs as the context lengthens; linear attention only passes its fixed-size state, which is far cheaper. The complication is that KDA’s gated update won’t let you simply add up each GPU’s locally computed state, because its delta rule multiplies the incoming state by a token-dependent matrix, so what a segment does depends on the state that entered it. Moonshot’s answer splits each segment into two things a GPU can compute on its own, blind to the incoming state: a transition matrix that captures what the segment does to any state fed in, and a separate state built as if it had started from zero. Those pieces combine in order, so every GPU’s true starting state is rebuilt with a single fixed-size exchange and a scan down the ranks. The attention side of a million-token training step then needs only that small, fixed exchange per chunk, not traffic that grows with the sequence. The experts still pay their usual per-token shuffle between GPUs, but that was never the part that scaled with context; this fixes the part that did.
Most of the work is building environments
The closed labs sum up this stage in one line. “We trained it with reinforcement learning on agentic tasks.” Moonshot spends twenty pages on what that involves, and it’s the clearest view I’ve had of it.
Nearly all of K3’s reinforcement learning runs against environments the team had to build by hand, each one able to check its own answers. For coding there’s a GPU-kernel suite that scores a solution on correctness and speed against an expert version, wired up with a hacking detector that docks the model for faking the win through tricks like CUDA graph replay or dropping precision. For assistant work there are mock Gmail, Notion, Slack and Canvas that keep state across simulated days, where one task can run to thousands of tool calls.
A couple are worth a closer look.
The autonomous-execution tasks are the hardest of the set. Each one hands the agent a starting state, a goal, a set of tools, a budget, and a verifier, and nothing else, no reference solution and no procedure to copy. The agent has to decompose the problem, plan, act, recover from its own mistakes and decide when it’s done, and it only scores on what the verifier finds in the final state, not on its own claim to have finished. The report’s examples include reconstructing a hidden system from black-box queries, discovering quantitative factors and tax auditing. To stop the model gaming the grader, they split the verifier in two. A public one gives diagnostic feedback the agent can learn from; a hidden one scores held-out cases the agent never sees.
The kernel-optimisation suite runs the same arms race. Rewards there mix correctness and speed against an expert implementation, so a lazy model reaches for a shortcut, replaying a cached CUDA graph, caching inputs, quietly dropping numerical precision. Each of those scores well without doing the work, and each had to be detected and penalised, with new guards added as the model found fresh cheats during training. That back-and-forth, a grader and a model probing it for holes, is most of what building an environment comes to.
Where the training tasks come from
An environment only helps if you can feed it enough good tasks, and at this scale nobody is writing them by hand. Moonshot builds them from a knowledge graph the agents grow themselves. It starts from a handful of broad seed topics; an agent takes each node, searches the web to understand the concept, and adds finer sub-concepts beneath it, checking what already exists so it reuses nodes instead of duplicating them. A branch stops growing once its concept is specific enough to be atomic. What you end up with is a hierarchy running from wide domains down to narrow, specialised corners.
Tasks are then drawn from that graph by sampling nodes, sometimes a single fine concept, sometimes a cluster of related ones, and folding their keywords together with context from their parent nodes into web queries. The real articles, code and documents those queries return are handed to a synthesis agent that writes the actual task. Sampling deeper or shallower sets how specialised the task is; sampling across different branches sets the coverage. That gives the team a dial on the training distribution, so they can aim it at thin spots instead of generating more of what the model already handles.
A second trick targets a subtler failure. An agent trained inside one fixed harness, with one tool schema and one way of managing context, learns that harness as much as the task underneath it. So Moonshot makes the harness itself configurable, a kit of swappable parts for tools, prompts, memory and subagents, and assembles it into mainstream setups like Claude Code, Codex and its own Kimi Code, or into new ones. Across training the model meets many of these arrangements, so it generalises across scaffolds instead of memorising one.
From nine experts into one model
The pipeline runs in three stages. A supervised fine-tuning pass gives the model a competent starting policy. Then reinforcement learning, and not as a single run. Moonshot trains across three broad domains (general tasks, general agents and coding agents) at three separate levels of reasoning effort, low, high and max. Three domains times three effort levels gives nine specialist models, each good at its own slice. Training them separately likely sidesteps the interference you get when one policy has to serve coding, general work and three effort regimes at once; each specialist climbs faster on its own slice, and the conflicts are left for distillation to reconcile. The third stage, multi-teacher on-policy distillation, folds all nine back into the single model you download, with each specialist teaching the student on the slice it knows best.
The reasoning-effort part is also instructive. During RL, each problem gets a token budget estimated from the cold-start model, and any trajectory that blows past a multiple of that budget has its reward overwritten with a penalty. Training walks that multiplier down in stages, from a generous max budget to tight low-effort settings, which is how you end up with a model that can be told to think hard or think cheaply and actually respects the difference. The same budgeting idea reappears on the reward side: for tasks with no automatic checker, a generative reward model scores candidates against a rubric it writes on the spot, and a candidate that runs too long automatically loses, so the model can’t win by padding its answer.
Taken one at a time, these are standard techniques. What stands out is that the shaping happens in the training loop and the reward design, not the network.
A frontier model is also a big systems project
If you turned up expecting mostly machine learning, the sheer amount of plain systems work is a jolt. Getting three awkward things to run together is most of the engineering: a linear-attention model, a 2.8-trillion-parameter sparse network and rollouts that stretch to a million tokens.
Some of it is vivid. They run a microVM sandbox on Firecracker that checkpoints in 133ms and resumes in 49ms, and they spun up 51 million sandboxes over the project. A chunk of the architecture is also shaped by how the model will be served. They train with the expert weights quantised to 4-bit in the forward pass, not just at the end, so the model learns to live at the precision it’ll be deployed at, with no train-inference mismatch to paper over later. They also train the speculative-decoding draft model from a small prediction layer built into the network, and optimise it directly for the acceptance rate that governs the speedup, instead of a proxy. Both are architecture choices driven by serving cost, not accuracy.
A few case studies make the point. K3 tuned its own attention kernels, and Moonshot says an early checkpoint was already handling most of the team’s kernel work late in the project. It wrote a Triton-like compiler end to end, and in a separate autonomous run laid out a small inference chip that closes timing at 100MHz. In one research case study it reproduced a set of astrophysics results, cross-checking more than twenty papers and thousands of lines of its own code, in a couple of hours against a human estimate of one to two weeks. Take the specific numbers with as much salt as you like. The report frames that systems work as elite human effort, and the model is already doing a fair chunk of it.
The serving lessons that transfer below frontier scale
That was building the model. The last stretch of the report is about running it, and it’s what I’d hand a working engineer first, because most of it applies well before you’re serving anything at 3 trillion parameters.
The economics set up the rest. K3’s serving cost is low by design, and most of the levers behind it are ordinary infrastructure, bent around one unusual fact about language models. Every request carries a large, reusable, expensive-to-rebuild state, and the serving design treats that state as the scarce resource.
The cheapest of those levers is prefix caching. In a long coding session, a typical request is a 400K-token prefix carrying only 4K tokens of genuinely new work. Re-run the whole prefix each step and you pay for all of it each step; cache it and a hit costs roughly a hundredth of a miss. Any agent that loops over a growing context pays this cost again and again, which makes cache-hit rate the number to design around. K3 has a wrinkle of its own here: a growing KV cache can be reused up to any point you like, but a fixed-size running state only exists where you saved a copy, so its linear-attention layers have to checkpoint their state at intervals for a later request to have anything to reuse.
Once a prefix is cached, the request is far cheaper on the machine holding that cache than on a cold one, because shipping the cache around is slow, so Moonshot pins each session to the cluster that owns its prefix and gives it a backup cluster via consistent hashing in case the primary dies.
The subtler problem is mixing request sizes. Real traffic ranges from 2K-token requests to 1M-token ones, so per-request cost spans three orders of magnitude, and a burst of the big ones will eat the box and leave short requests queuing behind them until latency collapses for everyone. Moonshot’s fix is to give each size class its own budget, so a flood of long requests can only starve itself. Two smaller ideas round it out: tiering the cache, so idle prefixes get pushed from GPU memory out to plain CPU RAM and pulled back before reuse; and setting concurrency from live signals like cache pressure and queue depth rather than a fixed guess that’s too timid early and overloaded late.
The pieces are all standard — caching, routing, admission control and backpressure, the toolkit of any busy service. The only twist is a workload where the per-request state is enormous.
What the independent numbers say
A recipe is only interesting if the result is good, and independent numbers matter more than the lab’s own. On its own benchmarks K3 trails Claude Fable 5 and GPT-5.6 Sol and beats the rest of the field, which is roughly the story the outside evaluations tell too. Artificial Analysis ranks it third among model families on its intelligence index, behind only Claude Fable 5 and GPT-5.6 Sol; it comes fourth of 580 raw entries, because two of GPT-5.6 Sol’s effort settings sit above it as separate rows. On the crowd-voted WebDev Arena it ranks first of all models, the first open model to top that board, though an arena measures which output humans prefer on one family of tasks, a narrower question than a composite index; it’s the one board where K3 leads outright. Vals AI’s industry-weighted suite has it second of thirty-nine. It’s not the best model available, but among open models it’s clearly the strongest, and the gap to the frontier is small.
Figure 4: Overall capability against peers. The Artificial Analysis Intelligence Index (v4.1) is a third-party composite where higher is better. K3 places third, behind Claude Fable 5 and GPT-5.6 Sol and ahead of the rest, and it is the strongest open-weight model in the set. The bars share a zero baseline, so their near-equal lengths reflect how close the leaders sit. 📖 Source: image by author.
The cost picture should interest anyone paying an inference bill, though both figures here are Moonshot’s own runs. On BrowseComp it posts the top score at around two dollars a task, roughly half the price of the nearest closed model at full effort and an order of magnitude under the priciest. On an internal coding benchmark its high-effort setting matches a strong closed model’s maximum-effort score at about a third of the price. These are API prices rather than compute costs, and open-weight models get hosted by competing providers bidding their margins down, so part of the gap is market structure, not engineering. Even so, it sits at or near the price-efficiency frontier.
The security results are the most specific thing in the report. On vulnerability discovery the model turned up genuine bugs in current, widely deployed software, and of the findings that went to human review, around 70% were confirmed real, including sixteen previously unknown vulnerabilities across six projects. Two were in the Linux kernel: a remotely triggerable out-of-bounds write, and a privilege-escalation bug in the RDMA subsystem where an earlier fix had dropped a permission check. On end-to-end exploit writing it solved 14 of 36 tasks against 8 for the strongest open rival, GLM, though it stalled on the hardened kernel targets. Moonshot’s own suite leaves the frontier closed models out here, because they refuse this kind of task over their public APIs. A separate assessmentby the UK AI Security Institute and the US CAISI, which can test the closed models under privileged access, landed in much the same place: K3 above the open competition, short of the frontier cyber models and unable to finish the hardest exploit chains.
Why the closed labs keep this quiet
The reason the recipe stays inside isn’t a conspiracy. The architecture is a pile of published ideas that anyone with the compute could copy. The environments, the reward machinery, the infrastructure and the data pipelines are the expensive, hard-won parts, and those are the edge that’s actually defensible. Open-sourcing the weights hands over the least protected thing a lab owns. Publishing the report, as Moonshot did, is more than most will do, and even it holds back the exact data mix and a lot of the knobs.
It also says something about where the advantage now sits. The moat used to be the model itself; now it’s the apparatus around it, the environments that can grade a task, the reward models that are hard to game, the serving stack that keeps a good model cheap. Those take years to build, and none of them leaves with the weights. An open model gets matched or distilled within months; the tooling behind it doesn’t move nearly that fast.
Open weights and offensive tooling
There’s a cost to all this openness. The same K3 that finds real vulnerabilities and writes working exploits is a free download, and once it’s on your own hardware it answers with none of the refusals or logging that a hosted API can impose. The report itself shows that split, with the frontier closed models declining cyber work at the public API while the open ones carry it out.
That “free download” deserves a caveat, though. The weights are open, but the licence is not the permissive kind. Moonshot keeps commercial conditions attached: a model-as-a-service business past roughly twenty million dollars a year has to negotiate a separate agreement, and the largest deployments have to show the Kimi K3 name in their interface. Ordinary internal use inside a company is unrestricted, so most non-tech firms can run it without a second thought. For the misuse question none of that matters, since an attacker writing malware won’t honour an attribution clause. What does raise the bar is size: at around 1.5 terabytes the weights want serious hardware, which rules out casual local use even as a determined operator, or a rented cluster, clears it without much trouble.
A recent analysis of GLM, another strong open model, sharpens the near-term picture. Its argument is that models of this class can now produce the routine building blocks of offensive tooling from fairly light prompting, the boilerplate that used to eat an operator’s afternoon. It doesn’t add capability; it removes the effort of assembling tooling operators already knew how to build. The parts that were always hard (mapping an unfamiliar network, getting past current endpoint defences, avoiding attribution) still need a skilled human. So the near-term effect is mostly volume, many more cheap, near-identical samples, which is enough on its own to blunt signature-based detection and push defenders toward behavioural methods.
Where to focus if you build with these models
If you build with these models instead of training them, the report is a fairly direct hint about where to point your attention.
The clearest signal is that evaluation and environment design are now real engineering disciplines. The single biggest slice of effort in the whole report is building tasks a machine can grade and closing the ways a model games them, which is the same skill you need to ship an agent you can trust in production. Learning to write a good verifier, and to think adversarially about how a model will cheat it, transfers directly from Moonshot’s RL loop to your own eval harness, and it’s a skill with almost no vendor lock-in.
Long-horizon agentic engineering is close behind. K3 is built for runs of hundreds to thousands of tool calls over a million tokens of context, and the work that makes those runs hold together (context management, harness design, sandbox isolation, resumable state) is its own discipline now, separate from prompting and separate from training.
Serving economics is the third, and the most immediately practical. The cost advantages in the report come from prefix caching, cache-aware routing, request-class budgeting and quantisation, and every one of those is available to a small team on a handful of machines. Making a capable model cheap to run is something the frontier labs treat as a headline result.
Systems and kernel work sits slightly apart, because it’s both clearly valuable and the thing the model is automating fastest. I’d learn enough to direct and check it rather than betting a career on hand-writing kernels the model will soon draft. And I’d spend the least effort chasing architecture novelty for its own sake: the gains there are real but incremental, they come from a team with 3-trillion-parameter ablations to run, and they’re the least reachable place for most of us to compete.
Disclaimer: The views and opinions expressed in this article are my own and do not represent those of my employer or any affiliated organizations. The content is based on personal experience and reflection, and should not be taken as professional or academic advice.
📚References
- Kimi Team, Moonshot AI. (2026). Kimi K3: Open Frontier Intelligence. The 47-page technical report this article reads throughout — an open 2.8-trillion-parameter mixture-of-experts model with 104 billion active parameters and a one-million-token context window — covering the architecture, reinforcement-learning environments, training pipeline, serving stack and evaluations discussed here.
- Kimi Team. (2025). Kimi Linear: An Expressive, Efficient Attention Architecture. Introduced Kimi Delta Attention (KDA), the fixed-size-state linear-attention mechanism together with the chunkwise kernel and context-parallel formulation, which K3 adopts as the cheap layers in its hybrid attention and which underpin the serial-recurrence-on-parallel-hardware section.
- Elango, V., et al. (2026). LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts.Introduced the latent-space routed-expert design that K3’s Stable LatentMoE builds on to scale to 896 experts with 16 active per token, decoupling model capacity from per-token compute while leaving all parameters resident in memory.
- DeepSeek-AI. (2024). DeepSeek-V3 Technical Report. Introduced the auxiliary-loss-free load-balancing scheme for large mixture-of-experts models that K3 extends with Quantile Balancing to keep nearly a thousand experts evenly used during training.
Instagram Engineering
Shopify Engineering