Literature review · Pluralis · Part VI · August 2026
How speculative decoding is actually measured: the harnesses the field standardizes on, the datasets used per domain — coding, math, logic/reasoning, science, chat, summarization, translation, RAG, agentic, long-context, multilingual — the metrics that matter, and the pitfalls that make half the published numbers hard to compare. The core fact this page documents: acceptance is a property of the workload, not just the drafter — the same method can swing 2–3× across domains, and method rankings invert entirely between agentic and open-ended traffic.
Spec-Bench (arXiv 2401.07851, ACL Findings '24; repo, leaderboard) remains the default: 480 prompts — 80 each from MT-Bench (chat), WMT14 DE-EN (translation), CNN/DailyMail (summarization), Natural Questions (QA), GSM8K (math), and NQ with top-5 DPR passages (RAG) — reporting wall-clock speedup and mean accepted tokens, greedy, batch 1. Ten methods are maintained (EAGLE-1/2/3, Medusa, Hydra, classic SpS, PLD, Token Recycling, REST, Lookahead, SAM-Decoding); the leaderboard's current top is EAGLE-3 at 3.02× / τ 5.71 (Vicuna-13B, A100). There is no v2 — three newer benchmarks fill its gaps:
| Harness | Composition | What it adds / key finding |
|---|---|---|
| Spec-Bench '24 | 480 prompts, 6 tasks × 80 (MT-Bench, WMT14, CNN/DM, NQ, GSM8K, NQ+DPR) | The comparability standard; greedy, batch-1; last major update Mar '25 (EAGLE-3, SAM-Decoding) |
| SPEED-Bench '26 | 880 samples = 11 categories × 80 from 24 datasets (LiveCodeBench, HumanEval, ChatRAG-Bench, CNN/DM…), 23 languages; + 7,680-sample throughput split (ISL 1k–32k × entropy tiers, batch ≤512) | Entropy tiers (low-entropy AL 2.7 vs high 2.1); random-token synthetic workloads overestimate SD ~23%; EAGLE-3's pruned draft vocab misses ~22% of multilingual targets |
| SpecDecode-Bench '26 (site) | Production vLLM; CNN/DM, ShareGPT, InstructCoder, GSM8K, AIME 22–24, GPQA-Main; batch sweeps | Verification = 42–95% of step time; EAGLE-3 best overall; n-gram wins code-editing (AL heavy-tailed 1.1–15.0); speedups drop >40% at batch 128 |
| Test-time-scaling bench '25 | 120 prompts: AIME-24/25 (60), MATH-500 (30), GPQA (30); Best-of-N + multi-round thinking; 9 methods | SD under long reasoning traces: EAGLE-3 ≤3.04×, only −6% at T 0→0.6; SAM −22%; classic SpS hits the highest accepted length yet runs at 0.87× |
| "Lossless but Not Free" '26 | Consumer hardware (Apple silicon) anatomy across 5 model pairs | Acceptance decays 69.7%→37.8% as K 1→6; best 1.61×; 3 of 5 configs decelerate; includes a χ² losslessness test |
| Engine harnesses | vLLM blog: ShareGPT + CNN/DM at swept QPS · SGLang bench_speculative.py (sweeps steps/topk/draft-tokens/batch) · Speculators: GuideLLM on speculator_benchmarks · DeepSpec: gsm8k, math500, aime25, humaneval, mbpp, livecodebench, mt-bench, alpaca, arena-hard-v2 | What vendors actually tune on; vLLM's QPS sweep is the canonical demonstration that gains flip sign under load (n-gram: 2.8× at QPS 1 → 1.8× slowdown compute-bound) |
| InferenceX / InferenceMAX | Random-token workloads at fixed ISL/OSL (1k/1k, 1k/8k, 8k/1k); MTP on/off Pareto curves | The industry throughput-vs-interactivity reference — but per SPEED-Bench, its synthetic prompts inflate SD gains ~23%; read its curves comparatively, not absolutely |
| Domain | Datasets in actual SD use | Who uses them | Domain-specific evidence |
|---|---|---|---|
| Coding | HumanEval · MBPP · LiveCodeBench · InstructCoder · SWE-Bench · Spider/Text-to-SQL · Blazedit (90-file edit set) | EAGLE-2/3, REST, DeepSpec, SpecDecode-Bench, SuffixDecoding, Red Hat gpt-oss | Code accepts best for learned heads (EAGLE-2: 4.96× HumanEval, its top task); editing workloads are retrieval's home turf (n-gram AL heavy-tailed to 15; Blazedit ≤7.7×) |
| Math | GSM8K (ubiquitous) · MATH-500 · AIME-24/25 · MGSM (multilingual) | Spec-Bench slice, DeepSpec, TTS bench | Structured derivations accept well; gpt-oss speculator AL@k=5: math 3.29 > code 3.01 > translation 2.52 (Red Hat) |
| Logic / reasoning (TTS) | AIME-24/25 · GPQA-Diamond · LiveCodeBench · long-CoT traces | TTS bench, STAND, LongSpec | Reasoning traces are highly self-repetitive — 97% of bigrams / 80% of 4-grams recur across 16 sampled trajectories (STAND) — so retrieval/n-gram methods shine; EAGLE-3 on R1-Distill peaks on GSM8K (5.01×) but runs −34% vs its Qwen3 numbers overall |
| Science | GPQA (Diamond / Main) — effectively the only one | STAND, SpecDecode-Bench, TTS bench | Treated as a reasoning subset; no SD eval found using SciQ or domain corpora |
| Chat / instruction | MT-Bench (the default) · Alpaca · ShareGPT · UltraChat · WildChat · Arena-Hard-v2 · Dolly | Everyone; ShareGPT doubles as draft-training data; UltraChat as REST's datastore | MT-Bench numbers flatter chat-trained heads (see Pitfalls); Arena-Hard-v2 is DeepSpec's harder replacement |
| Summarization | CNN/DailyMail (standard) · MLPerf-summarization | Spec-Bench slice, vLLM blog, Red Hat | Input-grounded → prompt-lookup's best case (PLD ~2.4×); XSum/GovReport not found in SD evals |
| Translation | WMT14 DE-EN | Spec-Bench slice | Consistently the worst task for chat-trained heads — the cleanest contamination probe in the standard suite |
| QA / RAG | Natural Questions ± top-5 DPR · TriviaQA · HAGRID · ChatRAG-Bench · AntRAG (prod.) | Spec-Bench, PLD, SPEED-Bench, Ant Lookahead, LLMA | Reference overlap is the whole game: Ant Lookahead 5.36× on RAG vs 2.11× open-domain; LLMA >2× on grounded generation |
| Agentic | SWE-Bench · AgenticSQL (Snowflake) · SpiderSQL · WildChat traces · tool-use pipelines | SuffixDecoding, Red Hat gpt-oss | The ranking-inversion domain: SuffixDecoding 5.3×/τ 6.3 on AgenticSQL vs 1.7× on Spec-Bench; EAGLE-3 inverts (1.6× agentic vs 2.4× general) — see chart below |
| Long context | PG-19 · NarrativeQA · RULER · LongBench-style suites (+ AIME for long-CoT) | TriForce (2.31× @122K, α>0.9 at T=1), MagicDec (2.51× @batch 41/100K), LongSpec (≤3.26×) | The regime where speedup grows with batch (KV-bound, MagicDec) — the exception to the batch pitfall |
| Multilingual | MGSM · 11-language low-resource suites | 2510.02128, 2605.30580, SPEED-Bench (23 languages) | The cliff: 13% acceptance gap across MGSM languages (Japanese worst); low-resource languages: acceptance 0.30–0.54 → 1.02× ≈ no speedup, where n-gram beats neural drafters (1.30–1.39×); EAGLE-3's pruned vocab misses ~22% of multilingual targets |
| Metric | Definition | What it hides |
|---|---|---|
| Acceptance rate α | Per-token probability a draft token passes verification | Position-dependence — α decays sharply along the draft (69.7%→37.8% by K=6); a single α overstates deep drafts |
| Accepted length τ / MAT | Mean tokens emitted per draft–verify cycle; hardware-free | Draft cost — classic SpS posts the highest τ on the TTS bench and still runs 0.87× because its drafter is slow |
| Wall-clock speedup | Tokens/s vs autoregressive on identical hardware; the batch-1 paper standard | Batch and load — most papers report batch 1; gains shrink >40% at batch 128 and can invert under QPS load |
| TPOT / ITL / user-TPS | Serving latency per output token / per user | Cost — MTP wins user-TPS while losing tok/s/GPU at throughput-optimal points (SemiAnalysis) |
| Goodput | Load-aware expected accepted tokens per unit compute (SmartSpec); draft length chosen 0…k per step | Nothing — it is the honest serving metric; the cost is needing an online α estimator |
| Pareto curves (tok/s/GPU vs tok/s/user) | Sweep of operating points with speculation on/off (InferenceX style) | Workload realism when driven by random tokens (+23% inflation) |
Minimum honest report for a new SD setup: τ and α (with per-position decay) separately from wall-clock; wall-clock at ≥2 batch sizes and ≥2 temperatures; the workload named per number. For losslessness claims, a distributional test (the χ² procedure in "Lossless but Not Free") rather than an eyeball diff.
Vicuna-13B, greedy, batch 1 — from the EAGLE-2 paper's per-task table (arXiv 2406.16858)
SuffixDecoding paper's head-to-head (arXiv 2411.04975): same targets, two workloads
Greedy-only evaluation. The Spec-Bench leaderboard is greedy; under sampling at T=0.6, EAGLE-3 loses only ~6% but SAM-Decoding loses 22% and hybrid methods up to 42% (TTS bench). Report at least T=0 and one realistic temperature.
Batch and load reversals. vLLM's own harness shows n-gram going from 2.8× at QPS 1 to a 1.8× slowdown compute-bound; SpecDecode-Bench measures >40% speedup loss at batch 128. Long-context is the exception — MagicDec's gains grow with batch when the KV cache binds. Goodput (SmartSpec) is the metric that catches this.
Acceptance is not speed. Classic two-model SpS posts the highest accepted length on the TTS bench and still runs at 0.87× — draft cost and verification width matter as much as τ. Always pair τ with wall-clock on named hardware.
Draft-training contamination. Heads trained on ShareGPT-style chat score best on MT-Bench and worst on WMT14 — the standard suite's translation slice is the cleanest probe. Same head, same k: math 3.29 > code 3.01 > translation 2.52 (Red Hat gpt-oss speculator). Match training data to the serving distribution, and never eval only on the training domain's twin.
Synthetic-prompt inflation. Random-token workloads (InferenceMAX-style) overestimate SD throughput by ~23% (SPEED-Bench) — real text has structure that changes both acceptance and batching. Use real prompts.
The multilingual cliff. 13% acceptance spread across MGSM languages; on 11 low-resource languages neural drafters collapse to 1.02× while simple n-gram holds 1.30–1.39×; EAGLE-3's pruned draft vocabulary misses ~22% of multilingual target tokens. If traffic is multilingual, benchmark it explicitly.
For evaluating an SD setup (the WAN-PP deployment of Parts II–IV included), the composition that covers every failure mode above: Spec-Bench's 480 prompts as the comparability core; per-domain probes — HumanEval or LiveCodeBench (code), GSM8K + AIME-24 (math and long reasoning), GPQA-Diamond (science), an agentic trace suite (SWE-Bench or AgenticSQL-style pipelines — the domain where retrieval drafting inverts every ranking), PG-19 + RULER (long context, the KV-bound regime), and MGSM (the multilingual cliff); plus one real production trace (ShareGPT/WildChat-class) replayed at swept QPS. Run everything at T=0 and T≈0.6–0.8, at batch 1 and a production batch, reporting τ and per-position α separately from wall-clock, with goodput or a Pareto sweep for serving claims — and never random tokens. For a WAN deployment, add the one measurement no harness provides yet: acceptance under the deployment's own wire protocol (top-K truncation and relaxed verification change α — Part III), at real link RTTs.