Workers AI runs inference for some of the best open models in the world on GPUs in Cloudflare data centers close to your users. Two of the most capable, and most demanding, are Moonshot's Kimi K-series and Z.ai's GLM. They are large, long-context, mixture-of-experts models, and they are wonderful to use. They are also very hard to serve efficiently because of memory constraints.
We've written before about how we serve large models on Workers AI and about separating the prefill and decode phases of inference to get more out of each GPU. This post looks at three techniques we layer on top of that to fit these models into memory and keep them fast: quantizing the KV cache, compressing the model weights, and, because both of those pack more requests onto shared hardware, protecting the cache those requests share. These optimizations enable us to support more customers at lower costs, with no change in model accuracy.
All our experiments and production traffic are running and benchmarked with SGLang, an open-source inference serving framework. We found that SGLang offers the best performance in the market, and we work closely with the SGLang team to upstream patches and new features to make our work available to the open-source community.
Quantizing the KV cache
As a model generates text, it stores the attention keys (K) and values (V) for every token it has already processed in a structure called the KV cache. The cache is what lets the model extend a long conversation without re-reading the entire context on every new token. For a long-context model, it grows quickly, and it is usually the KV cache, not the model's weights, that fills up GPU memory first.
By default, the cache is stored in 16-bit precision (BF16). We store it in 8-bit floating point instead (FP8, e4m3), which halves its size. On Kimi K2.6, that raises the amount of context we can hold in memory from roughly 686,000 tokens to about 1.37 million, twice as much.
It's worth being precise about where the benefit comes from, because it isn't raw speed. Quantizing the cache adds a small amount of work per token, since the FP8 attention kernel has to convert values as it reads them. What it changes is how many requests we can keep resident at once. The following measurements are for Kimi K2.6 decoding on a disaggregated H200 deployment, comparing the attention kernels directly:
|
Concurrent requests
|
BF16 KV cache (tok/s)
|
FP8 KV cache (tok/s)
|
| --- | --- | --- |
|
1
|
137
|
125
|
|
8
|
731
|
689
|
|
16
|
1,106
|
1,028
|
|
32
|
1,558
|
1,489
|
|
64
|
Out of memory
|
2,192
|
At any single concurrency level, BF16 is a few percent faster per token. But BF16 runs out of cache at 32 concurrent requests and can't admit a 33rd, while FP8 keeps going to 64 and reaches 2,192 tokens per second, about 41% higher than BF16's peak, for roughly 30% less cost per token. Because we run prefill and decode as separate pools, we can apply this where it helps most: prefill is compute-bound rather than memory-bound, so there we leave the cache in BF16 and keep its slightly higher throughput.
None of this would matter if it changed the model's answers, so we checked. Across our evaluation suite, FP8 and BF16 caches are indistinguishable:
|
Benchmark
|
BF16 KV
|
FP8 KV
|
| --- | --- | --- |
|
GSM8K
|
94.24
|
94.09
|
|
ARC-Easy
|
89.06
|
89.14
|
|
ARC-Challenge
|
66.72
|
67.49
|
|
MMLU
|
89.11
|
89.04
|
|
MMLU-Pro
|
80.29
|
79.29
|
|
mcxams (internal benchmark)
|
61 / 63
|
61 / 63
|
|
Tool-call validity
|
92.2%
|
92.6%
|
Compressing the model weights
The KV cache is one demand on GPU memory; the model's weights are the other. For GLM 5.2, we compress the weights from 8-bit floating point down to 4-bit integers (INT4) with no loss in accuracy. The checkpoint shrinks from 705 GB to 421 GB, about 40%, and per-GPU memory across an 8-way tensor-parallel deployment drops from roughly 88 GB to 52 GB, which leaves room for around 1.18 million tokens of KV cache on the same hardware.
Across our evaluation suite, INT4 and FP8 weights are indistinguishable:
|
Benchmark / Capability
|
Metric
|
FP8
|
INT4
|
| --- | --- | --- | --- |
|
GSM8K
|
Exact match
|
94.39%
|
93.56%
|
|
GSM8K
|
Flexible
|
94.24%
|
93.48%
|
|
ARC-Easy
|
Accuracy
|
86.62%
|
86.15%
|
|
ARC-Easy
|
Acc (norm)
|
84.51%
|
85.19%
|
|
ARC-Challenge
|
Accuracy
|
64.93%
|
64.85%
|
|
ARC-Challenge
|
Acc (norm)
|
67.24%
|
66.64%
|
|
MMLU
|
Average
|
86.60%
|
86.54%
|
|
MMLU-Pro
|
Exact
|
80.80%
|
80.47%
|
|
mcxams (internal benchmark)
|
Passed
|
62 / 63
|
62 / 63
|
Smaller weights make the decode phase faster, and for a clear reason: generating each token means streaming the model's weights out of GPU memory, so decode speed is limited by memory bandwidth. Move less data and every token arrives sooner. The effect is largest at low concurrency, where per-request latency matters most:
|
Concurrent requests
|
GLM FP8 (tok/s)
|
GLM INT4 (tok/s)
|
INT4 gain
|
| --- | --- | --- | --- |
|
1
|
60
|
92
|
+55%
|
|
8
|
425
|
513
|
+21%
|
|
16
|
683
|
825
|
+21%
|
|
32
|
994
|
1,267
|
+27%
|
|
64
|
1,672
|
1,933
|
+16%
|
Prefill behaves differently. It is compute-bound, and INT4 weights have to be expanded back out before the model can multiply with them, so that extra step makes prefill slower rather than faster, GLM sustains about 10,160 tokens per second of prefill in FP8 versus 8,660 in INT4. As with the KV cache, the disaggregated design turns this into a choice rather than a compromise: we run INT4 for decode, where it wins, and FP8 for prefill, where it wins. Model accuracy stays within 0.8 points of the FP8 model across every benchmark we run, making its quality indistinguishable.
Protecting a shared KV cache
Both techniques above have the same effect: they let many more requests share one GPU's memory at the same time. That efficiency is the whole point, but it also means hundreds of requests are reading and writing pages of the same physical KV cache. The mechanisms that make this fast, paged attention, continuous batching, cache reuse, all rely on getting the bookkeeping exactly right, and at our request volumes, even a one-in-a-billion mistake would show up regularly.
So we built KV cache integrity checking as a layer of defense. The idea is straightforward: every physical cache page gets a tag that changes whenever the page is reallocated, and the server records which pages and tags each request expects to use. Before supported decode operations read from the cache, those mappings are checked. If anything doesn't match, the affected request is aborted rather than allowed to return data from the wrong page.
The question that decides whether a safety check ships is what it costs. We measured it on a mid-sized production model in a two-prefill, two-decode configuration, with 8,192-token inputs and 1,000-token outputs:
|
Concurrency
|
Throughput change
|
p95 latency change
|
| --- | --- | --- |
|
1
|
−0.53%
|
+0.42%
|
|
2
|
−0.38%
|
+0.54%
|
|
4
|
−0.79%
|
+0.63%
|
|
8
|
−0.43%
|
+0.80%
|
The cost is under 1% on both throughput and tail latency, and even the upper bound of the 95% confidence interval stays near 1%. We kept it computationally cheap by running the validation as a separate batch check rather than fusing it into the attention kernel, which would have introduced a race between GPU thread groups. It's enabled per deployment, and the default path uses a no-op tracker with no measurable overhead, so deployments that don't need it pay nothing.
What's next
Serving frontier models efficiently is a moving target, and this is the ongoing work behind it. We're expanding FP8 KV caches across more of the fleet, validating NVFP4 weights on Blackwell (NVIDIA’s GPU architecture), and working toward making integrity checks something we can leave on everywhere at negligible cost. These optimizations will allow us to continue to support more customers at a lower cost and at the same accuracy.
If squeezing the best open models onto GPUs and serving them to millions of developers sounds like your kind of problem, come work with us.
Instagram Engineering
Shopify Engineering