Performance CI for LLM inference.
Your test suite guards correctness. Verinfer guards latency, throughput, output quality, and cost — by benchmarking the real service you actually ship, against the baseline committed in your repo, on every change.
vs. baseline a3f91c2 committed in repo
| Metric | Baseline | Current | Δ | Verdict |
|---|---|---|---|---|
| TTFT p50 | 182 ms | 189 ms | +3.8% | Pass |
| TPOT p50 | 11.4 ms | 14.1 ms | +23.7% | Regressed |
| Throughput | 2,840 tok/s | 2,610 tok/s | −8.1% | Regressed |
| Quality score | 0.842 | 0.838 | −0.5% | Inconclusive |
| Cost / 1M tokens | $0.31 | $0.34 | +9.7% | — |
Real service. Committed baseline.
A verdict CI can act on.
Every run starts the actual inference service — not a mock, not a cached profile — and compares it to the baseline versioned alongside your code.
Spin up the real service
Launch vLLM, SGLang, or TRT-LLM exactly as deployed, with your model, flags, and quantization settings.
Run vs. baseline
Drive a reproducible workload and compare TTFT, TPOT/ITL, throughput, quality, and cost to the committed baseline.
Get a statistical verdict
Pass, regressed, or inconclusive — decided with a Mann-Whitney rank test, not a single noisy average.
Fail the build, alert Slack
Regressions beyond your threshold fail CI and notify Slack. Trends accumulate in the hosted dashboard.
Drop-in GitHub Actions
Three ready-made workflows ship with Verinfer — one per engine. Pick yours, set your thresholds and spend cap, and every pull request gets a performance verdict next to its unit tests.
- Hosted dashboard tracks every run's trend over time
- Organizations, roles, and SSO for platform teams
- --dry-run estimates GPU cost before you spend; --max-spend-usd hard-caps it
# .github/workflows/verinfer-vllm.yml
name: verinfer
on: [pull_request]
jobs:
perf:
runs-on: [self-hosted, gpu]
steps:
- uses: verinfer/verinfer-action@v1
with:
engine: vllm
baseline: ./verinfer/baseline.json
max-spend-usd: "25"
fail-on: regressed“Slower” isn’t always a regression.
Verinfer knows the difference.
A change that crosses your threshold but can’t be separated from random variation is labeled inconclusive — it won’t fail your build on noise, and a real regression won’t hide inside it either.
Delta exceeds the threshold and the Mann-Whitney test separates it from baseline noise (p < 0.01). CI fails and Slack is notified.
Three different kinds of variance — budgeted separately
A benchmark that treats all jitter the same will either cry wolf or miss real slowdowns. Verinfer’s model distinguishes where the variation comes from:
That’s why a bare threshold isn’t enough — and why Verinfer pairs every threshold with a significance test and a baseline collected under matched conditions.
Which stage got slower?
End-to-end latency hides the cause. Verinfer splits it, so you fix the right stage instead of guessing.
TTFT
Prompt processing and queueing. A TTFT regression points at prefill, scheduling, or frontend overhead — not decode.
TPOT / ITL
Per-token generation speed. A TPOT regression points at the decode path, kernels, quantization, or batching behavior.
version_bisect — find the version that introduced it
automatic binary search across engine versionsFP8 is faster. Verinfer measures
exactly what it costs you.
Compare bf16, FP8 online quantization, calibrated FP8, and AWQ side by side — token-level divergence, logprob drift, quality significance, and self-consistency, not just tokens per second.
In our case study, FP8 cut time-per-output-token substantially — and quietly changed the outputs themselves under concurrency. This is the trade-off page most benchmark tools never show you.
Self-consistency under concurrency
Same prompts, repeated runs. How often does the service give the same answer?
bf16: slower per token, markedly more stable under batching.
Case-study figures from Verinfer’s own published runs. Your numbers will differ by model and hardware — that’s the point of measuring them in CI.
How much load can you actually serve —
before TTFT and TPOT break target?
Open-loop Poisson load with rate sweeps finds your real ceiling under your SLO. The load client runs separately, multi-process — so you measure the service’s limit, not the client’s.
Case study: sustainable rate under SLO (req/s)
vLLM deployment · TTFT / TPOT targets held
Fixing the frontend took capacity from ~160 req/s to at least 400 req/s. Switching to FP8, by contrast, added only ~5% at high load. Architecture first, quantization second — Verinfer shows you which lever actually matters.
The client must not be the bottleneck
A single-process load client stops measuring accurately at ~300 req/s. Verinfer’s multi-process client stays accurate to 900 req/s with 8 processes — client and service deployed separately, so the ceiling you find is the service’s.
Quality & cost, in the same gate
Quality gates report significance, so a real quality drop is never waved through as noise. Throughput is converted to $ / 1M tokens and a monthly delta. Try the estimator:
Illustrative estimator using your inputs. In CI, Verinfer computes this from the measured run.
Every number ships with its proof.
Each conclusion maps to a reproducible, machine-readable artifact — the run config, raw samples, and statistical verdict — so a reviewer can re-derive any figure on the page.
Tests, all passing
357 in the main suite and 93 in the dashboard suite. The tool that judges your regressions is itself regression-tested.
Cost known before you run
Estimate GPU experiment cost up front and cap it with --max-spend-usd. No surprise cloud bills from a benchmark.
Machine-readable artifacts
Baselines live in your repo. Results export as structured artifacts your pipeline, dashboard, and auditors can all read.
Who it’s for
Cost-sensitive inference teams
Deciding between FP8 and bf16? Get the quantified answer: how much money you save, and exactly how much output fidelity you trade for it.
- Quantization comparisons with significance, not vibes
- $ / 1M tokens and monthly impact on every run
- Self-consistency under your real concurrency
Platform & SRE teams
Know whether an engine upgrade is safe before it ships — and when something regresses, know which stage and which version introduced it.
- Pre-merge verdicts for vLLM / SGLang / TRT-LLM upgrades
- TTFT vs. TPOT decomposition + version_bisect
- SLO capacity sweeps you can defend in review
Known limitations — stated up front
- Some capacity figures come from a single run so far and will be re-measured.
- The exact capacity ceiling with 4 frontends hasn’t been swept yet — we know it is above 400 req/s, not where it tops out.
- On Modal, server and client can only reach each other over the private network when they land on the same cloud provider; this account can’t pin the provider, so some runs wait for a suitable client to be reallocated.
Put a performance gate
on your inference stack.
Verinfer complements the engines you already run on NVIDIA GPUs — it helps teams keep vLLM, SGLang, and TensorRT-LLM deployments fast, stable, and cost-efficient across every upgrade. Tell us about your deployment and we’ll set up a baseline run.
What to include: engine + version · model · GPU · current concurrency & SLO targets.
Prefer email? Write to hello@verinfer.com
Verinfer · Austin, Texas, USA