Blog · Quantization

FP8 Cut Our TPOT by 25%. At High Concurrency, It Also Changed the Answers.

Verinfer · October 30, 2026

Update: we reran both experiments with vLLM's batch-invariant mode. It makes FP8 fully reproducible under load, at a much smaller cost than bf16 pays. See the results.

Quantization is usually sold as a free lunch: same quality, less memory, faster decoding. We wanted to check that claim before recommending FP8 anywhere, so we benchmarked it the way it actually ships — the real service, under real concurrency — instead of a single offline pass.

Setup

We served Qwen2.5-7B-Instruct on an H100 with vLLM 0.31.0, comparing bf16 against online FP8, a calibrated FP8 checkpoint, and AWQ across a concurrency sweep. Every comparison went through the same gates we use for engine upgrades: latency split into TTFT vs. TPOT, output-quality checks with significance testing, and cost converted to dollars per 1M tokens.

What we found

FP8 did what it promises on latency: TPOT dropped 23–26%.

What it also did was less comfortable. At 64 concurrent requests, self-consistency — how often the service gives the same answer to the same prompt — fell from 97.6% under bf16 to about 75% under FP8.

To find out why, we ran the same prompts at batch sizes from 1 to 128. Rerunning an input at the same batch size gave identical output on both precisions, so the engine itself is deterministic. Changing the batch size is what changed the text: FP8's output changed in 65 of 70 cases, with the first differing token at a median of 19. bf16 changed too, in 35 of 70 cases, but much later, at a median of token 77. Some batch dependence is normal; FP8 nearly doubled it and moved it much earlier in the answer.

In other words, FP8 doesn't just make decoding faster. Under load, its answers depend on how busy the server is: cheaper per token, but measurably less consistent with itself.

Why this matters

If you only watch latency, FP8 looks like an obvious win. If you watch latency, quality, and cost together, it's a tradeoff with a real price tag: lower cost per 1M tokens, paid for in output stability. That's exactly the kind of tradeoff that should be decided with data, not with a benchmark table from a launch post.

What this doesn't prove

These numbers are from one model, our prompts, and our serving setup. Quantization behavior varies by model family, and some of our capacity numbers were measured only once. Where the tokens still matched, FP8's next-token distribution barely moved, and its quality drop on our prompt set was too small to be statistically significant. We publish the methodology and the limitations alongside the results because a benchmark you can't reproduce is marketing, not evidence.

Update: batch invariance removes the drift

vLLM has a batch-invariant mode (VLLM_BATCH_INVARIANT=1) that keeps each kernel's configuration fixed regardless of batch size, and since vLLM 0.31.0 it covers the FP8 path too. We reran both experiments with it: same model, GPU and vLLM version, bf16 and FP8 each with and without the mode, prefix caching off on every server so the timings are comparable.

So FP8's instability is batch-size dependence that FP8 amplifies, not something FP8 does on its own. With the mode on, neither precision changed a single output with batch size or under load. Quality didn't move either.

The cost is where it gets interesting. bf16 gave up about half its throughput and got 2.9× slower per token; FP8 gave up about a quarter. A reproducible FP8 server served twice the tokens of a reproducible bf16 one, and came within 9% of default bf16. If you need outputs you can reproduce, FP8 is the cheaper way to get them.

One catch: the mode's outputs are fixed, but they are not the default mode's outputs. Even at batch size 1, they differed on 7–8 of our 10 prompts. Turning it on is itself a change, to gate like any engine upgrade, with the baseline recorded in the same mode. These are single runs per arm on one model, with self-consistency measured on 10 prompts, and the mode is still in beta.

What we'd do instead

Gate quantization changes the same way you'd gate an engine upgrade: run the real service against a committed baseline in CI, require statistical significance before calling anything a regression (noise gets marked inconclusive, not regressed), and check quality at the concurrency you actually serve — not at batch size 1. If you need reproducible outputs, measure batch-invariant mode as its own change: on FP8 it may cost less than you expect.