Literature review · Pluralis · compiled August 2026
Speculative decoding accelerates LLM inference by guessing several future tokens cheaply, then verifying them with the target model in a single parallel forward pass. Rejection-sampling verification makes the output provably identical to standard decoding, so the speedup is (usually) free. This atlas maps ~45 published methods into five families by where the draft tokens come from, compares their claimed gains, and tracks which are open source and which ship natively in vLLM and SGLang.
Autoregressive decoding is memory-bandwidth-bound: each step moves all the weights (and KV cache) through the GPU to produce one token. But a forward pass over several positions costs nearly the same as one. Speculative decoding exploits that slack: a cheap drafter proposes K tokens, the target model scores all K+1 positions in one pass, and a rejection-sampling rule accepts the longest prefix consistent with the target's own distribution — provably preserving it exactly (Leviathan et al. 2022; Chen et al. 2023). On rejection, the target's own sample replaces the first bad token, so every cycle emits at least one token and never emits a wrong one.
Two consequences frame everything else. First, the speedup is bounded by acceptance length — how many draft tokens the target agrees with per cycle — times the cost ratio between drafter and target. Second, any drafting source works as long as verification stands: a small model, extra heads on the target, a suffix match in the prompt, or the target's own shallow layers. That choice of source is the cleanest way to categorize the field.
The original formulation (Leviathan et al. 2022 · arXiv:2211.17192; Chen et al. 2023 · arXiv:2302.01318). Successors improved the tree shape (SpecInfer, Sequoia), the draft's alignment (DistillSpec, OSD), the schedule (SpecDec++, PEARL, AMUSD, Saguaro), and the regime (TriForce and MagicDec target long context, where the KV cache — not the weights — is the bottleneck).
Medusa (arXiv:2401.10774) attached parallel FFN heads; EAGLE (1, 2, 3) made the head autoregressive at the feature level and now defines the production standard. DeepSeek-V3's MTP module showed the head can be trained during pretraining and shipped with the weights — a pattern Qwen3-Next, GLM-4-MoE, and others now follow. The 2026 frontier is DFlash (arXiv:2602.06036): a block-diffusion drafter that emits a whole draft block in one non-autoregressive pass.
When outputs overlap inputs — summarization, code editing, RAG, multi-turn chat, agentic loops — the next tokens often literally appear in the context. Prompt Lookup Decoding (repo) matches n-grams in the prompt; REST (arXiv:2311.08252) retrieves from an external datastore; SuffixDecoding (arXiv:2411.04975, NeurIPS '25 spotlight) builds suffix trees over all prior outputs and dominates agentic workloads; Token Recycling reuses the model's own discarded top-k logits, so it works even without text overlap.
Greedy decoding solves a triangular system; Jacobi iteration guesses all positions and refines them in parallel (Santilli et al., ACL '23). Vanilla Jacobi barely helps LLMs, but Lookahead Decoding (arXiv:2402.02057, LMSYS blog) harvests n-grams from the iteration trajectory, and CLLMs / Jacobi Forcing fine-tune the model so trajectories converge in a few steps.
Draft&Verify (arXiv:2309.08168) skips a searched subset of layers to draft, then verifies with the full stack — one model, one KV cache. LayerSkip (arXiv:2404.16710, Meta) bakes early-exit quality in at training time; SWIFT and CLaSp pick skip-sets on the fly with no training at all; 2026 work (SparseSpec-L) drafts with a sparsified KV cache instead of fewer layers.
assistant_early_exit.Every paper reports speedup against its own autoregressive baseline, on its own model, hardware, and workload — so the chart below is a map of claims, not a leaderboard. Bars are colored by family; the workload each number came from is shown on hover and in the master table.
Best case reported by each paper · batch-1 latency unless noted · different models/GPUs/tasks — indicative only
SGLang official benchmark · Llama-3.1-8B-Instruct · MT-Bench · 1× H100 · output tokens/s
Production engines have converged on two families: retrieval (n-gram / suffix) because it is free, and trained heads (EAGLE-3 / MTP) because they win. In vLLM everything is configured through --speculative-config '{"method": …}'; in SGLang through --speculative-algorithm. No mainstream engine ships Jacobi/Lookahead, and self-speculative methods surface only through HF Transformers.
| Approach | vLLM (V1) | SGLang | TensorRT-LLM | HF Transformers | llama.cpp |
|---|---|---|---|---|---|
| Draft model (classic SD) | ● draft_model removed in early V1, restored via PR #24322; cross-vocab greedy-only | ● STANDALONE | ● Draft-Target | ● assisted generation + UAD (cross-tokenizer) | ● |
| N-gram / prompt lookup | ● ngram · ngram_gpu | ● NGRAM (CUDA-only) | ● NGram | ● prompt_lookup_num_tokens | ● multiple ngram modes |
| EAGLE-1 / 2 | ● eagle | ● EAGLE | ● | — | — |
| EAGLE-3 | ● eagle3 | ● EAGLE3 | ● | — | ● |
| Medusa | ● medusa (de-emphasized) | — | ◐ legacy flow | — | — |
| MLP speculator (IBM) | ● mlp_speculator | — | — | — | — |
| MTP (DeepSeek, Qwen3-Next, GLM-4, …) | ● mtp / deepseek_mtp / qwen3_next_mtp … | ● MTP (via EAGLE machinery) | ● MTP (DeepSeek) | ● use_mtp | — |
| Suffix decoding | ● suffix (needs arctic-inference) | announced | ◐ Suffix Automaton | — | — |
| DFlash (block-diffusion draft) | ● dflash | ● DFLASH | ● | — | ● |
| DSpark (adaptive verification) | ● dspark | ● | — | — | ● |
| PARD (parallel draft model) | ● draft_model + parallel_drafting | — | ● | — | — |
| ReDrafter (Apple) | — | — | ◐ legacy flow | — | — |
| Lookahead (Jacobi) | — | — | ◐ legacy flow | — | — |
| LayerSkip early exit | — | — | — | ● assistant_early_exit | — |
| Custom drafter hook | ● custom_class (experimental) | — | ● UserProvided | — | — |
Every surveyed method. Lossless = output distribution provably identical to the target model. Filter by family:
| Method | Family | Year · venue | Core idea | Claimed speedup | Lossless | Training | Open source | vLLM | SGLang |
|---|---|---|---|---|---|---|---|---|---|
| Speculative Decoding (Leviathan) | draft model | '22 · ICML'23 | Small LM drafts K tokens; rejection sampling preserves target distribution | 2–3.4× (T5-XXL) | ✓ | none | 3rd-party impls | ✓ | ✓ |
| Speculative Sampling (Chen) | draft model | '23 · arXiv | Same scheme at Chinchilla-70B scale, distributed TPU setting | ~2–2.5× | ✓ | none | no official code | ✓ | ✓ |
| SpecInfer | draft model | '23 · ASPLOS'24 | Token-tree drafting + parallel tree-attention verification (first) | 1.5–2.8× | ✓ | optional boost-tune | FlexFlow | — | — |
| DistillSpec | draft model | '23 · ICLR'24 | On-policy distillation aligns draft with target, raising acceptance | +10–45% over SD | ✓ | distill draft | no | technique | technique |
| Online Spec. Decoding | draft model | '23 · ICML'24 | Continually fine-tunes draft on live traffic with spare FLOPs | 1.42–2.17× | ✓ | online | research | — | — |
| BiLD | draft model | '23 · NeurIPS'23 | Small model leads; fallback/rollback policies call the big model on demand | up to 2.12× | ≈ | none | yes (repo) | — | — |
| Cascade Spec. Drafting | draft model | '23 · arXiv | Vertical + horizontal cascades of ever-cheaper drafters | up to +81% over SD | ✓ | none | unverified | — | — |
| SpecDec++ | draft model | '24 · COLM'25 | Learned acceptance head stops drafting adaptively (MDP view) | 2.04–2.26× | ✓ | small head | research | — | — |
| Sequoia | draft model | '24 · NeurIPS'24 | DP-optimal, hardware-aware speculation tree topology | 4.04×; 9.96× offload | ✓ | none | repo | — | — |
| TriForce | draft model | '24 · COLM'24 | Hierarchical: sparse-KV self-draft + tiny drafter for 100K+ contexts | 2.31×; 7.78× offload | ✓ | none | repo | — | — |
| MagicDec | draft model | '24 · ICLR'25 | Sparse-KV drafts make SD win throughput at large batch, long context | up to 2.51× (bs 32–256) | ✓ | none | repo | — | — |
| PEARL | draft model | '24 · ICLR'25 | Pre/post-verification lets draft and target run in parallel | 4.43×; 1.5× over SD | ✓ | none | repo | P-EAGLE blog | issue open |
| AMUSD | draft model | '24 · IEEE | Draft and verify run asynchronously on separate GPUs | up to 1.96× | ✓ | none | repo | — | — |
| Saguaro (Spec.² Decoding) | draft model | '26 · ICLR'26 | Draft speculates on the verifier's outcome, so drafting never idles | up to 5×; +30% over SD | ✓ | none | unverified | — | — |
| Medusa | heads | '24 · ICML'24 | Parallel FFN heads on last hidden state + tree attention | 2.2–3.6× | ≈ (typical accept.) | heads | repo | ✓ medusa | — |
| Hydra | heads | '24 · COLM'24 | Sequentially-dependent Medusa heads | up to 2.7× | ≈ | heads | repo | — | — |
| EAGLE-1 | heads | '24 · ICML'24 | Autoregressive head at feature level, conditioned on sampled token | 2.7–3.5× | ✓ | head | repo · Apache-2.0 | ✓ eagle | ✓ EAGLE |
| EAGLE-2 | heads | '24 · EMNLP'24 | + context-aware dynamic draft tree ranked by draft confidence | 3.05–4.26× | ✓ | head | repo | ✓ | ✓ EAGLE |
| EAGLE-3 | heads | '25 · NeurIPS'25 | Multi-level feature fusion + training-time test; gains scale with data | up to ~6.5×; 2.36× measured | ✓ | head (more data) | repo | ✓ eagle3 | ✓ EAGLE3 |
| HASS | heads | '24 · ICLR'25 | Harmonized training fixes train/decode feature mismatch of EAGLE heads | 2.81–4.05× | ✓ | head | repo | — | — |
| Falcon | heads | '24 · AAAI'25 | Semi-autoregressive block drafting + glancing distillation | 2.91–3.51× | ✓ | module | unverified | — | — |
| Clover-2 | heads | '24 · arXiv (Baidu) | RNN-flavored regressive heads with attention decoder, production-oriented | +91–146% vs AR | ✓ | heads | unverified | — | — |
| GliDe + CaPE | heads | '24 · ICML'24 | Draft cross-attends the target's KV cache; confidence-expanded proposals | up to 2.61× | ✓ | draft module | promised | — | — |
| Chimera | heads | '24 · arXiv | Trigram encoder + one transformer block over target hidden states | avg 2.7× | ✓ | draft module | unverified | — | — |
| ReDrafter (Apple) | heads | '24 · arXiv | RNN draft head with beam-search drafting | ~2× | ✓ | head | yes | — | — |
| MLP Speculator (IBM) | heads | '24 · PyTorch blog | Small MLP heads conditioned on state + sampled tokens | ~2–3× | ✓ | heads | HF weights | ✓ mlp_speculator | — |
| DeepSeek MTP | heads | '24 · tech report | Multi-token module trained during pretraining = free self-draft (85–90% acceptance) | ~1.8× TPS | ✓ | at pretraining | weights incl. MTP | ✓ mtp family | ✓ MTP |
| Mixture of Attentions | heads | '24 · ICLR'25 | Draft mixes self/cross attention over target activations; on-policy training | +9.5% over EAGLE-2 | ✓ | head | unverified | — | — |
| DFlash | heads | '26 · ICML'26 | Block-diffusion drafter conditioned on target features — whole block in one pass | >6×; ≤15× Blackwell | ✓ | diffusion draft | repo | ✓ dflash | ✓ DFLASH |
| DSpark | heads | '26 · arXiv | Confidence-scheduled, variable-length verification for block drafters | on top of DFlash | ✓ | draft | via engines | ✓ dspark | ✓ |
| Prompt Lookup Decoding | retrieval | '23 · GitHub | N-gram match against the prompt; continuation becomes the draft | ~2.4× summarization | ✓ | none | repo | ✓ ngram | ✓ NGRAM |
| LLMA | retrieval | '23 · arXiv | Copies spans from reference documents (RAG, history) and verifies | >2× on RAG | ✓ | none | LMOps | — | — |
| REST | retrieval | '23 · NAACL'24 | Suffix-array datastore retrieval → Trie draft tree → tree attention | 1.62–2.36× | ✓ | none (datastore) | repo | — | — |
| Lookahead (Ant Group) | retrieval | '23 · arXiv | Trie retrieval over prompt + generation history; deployed at Alipay | ~2–5× RAG (prod.) | ✓ | none | repo | — | — |
| Ouroboros | retrieval | '24 · EMNLP'24 | Draft model + phrase pool from lookahead candidates and leftovers | up to 3.9× vs AR | ✓ | none (needs draft LM) | repo · Apache-2.0 | — | — |
| ANPD | retrieval | '24 · NAACL'24 | Adaptive multi-level n-gram stats built from live context | up to 3.67× | ✓ | none | no repo found | — | — |
| Token Recycling | retrieval | '24 · ACL'25 | Reuses discarded top-k logits as a BFS draft tree (<2 MB state) | ~2× (incl. free-form) | ✓ | none | unofficial impl | — | — |
| SuffixDecoding | retrieval | '24 · NeurIPS'25 | Suffix trees over prompts + all prior outputs; adaptive speculation length | up to 5.3× agentic | ✓ | none | ArcticInference | ✓ suffix | announced |
| SAM-Decoding | retrieval | '24 · ACL'25 | Suffix automaton gives exact longest-match retrieval in O(1)/step | +18% over retrieval SD | ✓ | none | repo | — | SA in TRT-LLM |
| PLD+ | retrieval | '24 · arXiv (IBM) | PLD + candidate ranking via attention/hidden-state signals | 3.4–5.2× input-guided | ✓ | none | unverified | — | — |
| ReSpec | retrieval | '25 · arXiv | Entropy-triggered retrieval + relaxed source-aware verification | +33% over EAGLE-2 | ≈ (relaxed mode) | none | unverified | — | — |
| Jacobi Decoding | jacobi | '23 · ACL'23 | Guess all positions, refine in parallel to the greedy fixed point | ~1× on LLMs | ✓ greedy | none | yes | — | — |
| Lookahead Decoding (LMSYS) | jacobi | '23 · ICML'24 | Harvests n-grams from Jacobi trajectories; verifies in a parallel branch | 1.5–2.3× (code best) | ✓ | none | repo | — | — |
| CLLMs | jacobi | '24 · ICML'24 | Consistency-trains the model to jump Jacobi trajectories to the fixed point | 2.4–3.4× | ≈ (fine-tuned) | full fine-tune | repo | — | — |
| Jacobi Forcing | jacobi | '25 · arXiv | Progressive distillation on the model's own parallel trajectories | 3.8–4.0× code/math | ≈ | distillation | unverified | — | — |
| Draft & Verify | self-spec | '23 · ACL'24 | Skips a searched subset of layers to draft; full stack verifies | up to 1.99× | ✓ | none (offline search) | repo | — | — |
| LayerSkip (Meta) | self-spec | '24 · ACL'24 | Layer-dropout + early-exit training; early layers draft, rest verify, shared KV | 1.8–2.16× | ✓ vs own ckpt | training recipe | repo + checkpoints | — | — |
| Kangaroo | self-spec | '24 · NeurIPS'24 | Fixed shallow subnet + 67M adapter; confidence-based early exit | up to 1.68× | ✓ | adapter | repo | — | — |
| SWIFT | self-spec | '24 · ICLR'25 | Input-adaptive layer-skip set optimized during inference itself | 1.3–1.6× | ✓ | none | repo | — | — |
| EESD | self-spec | '24 · arXiv | Early-exit head drafts; Thompson sampling sets draft length | strong on 13B/70B | ✓ | head (self-distill) | not found | — | — |
| SPACE | self-spec | '24 · arXiv | Semi-autoregressive SFT + auto-correct decoding in one pass | 2.7–4.0× code | ≈ (SFT) | full SFT | not found | — | — |
| Speculative Streaming (Apple) | self-spec | '24 · EMNLP'25 | Multi-stream attention; n-gram objective fuses drafting into the model | 1.8–3.1× | ≈ (fine-tuned) | fine-tune | unofficial | — | — |
| CLaSp | self-spec | '25 · ACL'25 | DP re-picks the layer-skip set each step from last verification's states | 1.3–1.7× | ✓ | none | unverified | — | — |
| PPSD | self-spec | '25 · arXiv | Pipelines early-exit drafting against deep-layer verification per token | 2.01–3.81× | ✓ | early-exit training | research | — | — |
| SparseSpec-L | self-spec | '26 · arXiv | Drafts with a dynamically sparsified KV cache; entropy-controlled length | long-context gains | ✓ | none | research | — | — |
✓ = provably exact · ≈ = approximate (relaxed acceptance or a modified model) · "unverified" = paper promises code but no repo was confirmed during this review. vLLM/SGLang columns show native support only; anything can run via each engine's custom-drafter hook.
The main tax on Family 2 (a head per target model) has been largely paid down by two open frameworks. SpecForge (LMSYS/SGLang, blog) trains EAGLE-3-style drafts with training-time test built in, online or offline, FSDP/TP-parallel, MoE targets included — outputs plug straight into SGLang. Speculators (Red Hat / vLLM project, docs) standardizes draft checkpoints in an HF-compatible format, converts research-repo heads, and deploys directly in vLLM; v0.5 added DFlash support and online training. vLLM also ships hidden-state extraction to generate draft-training data from a serving deployment. Arctic Inference (Snowflake) is the vLLM plugin that contributed suffix decoding upstream and combines it with trained speculators.
Pre-trained heads now cover most popular open models: the EAGLE authors' originals (yuhuili on HF — LLaMA-2/3, Vicuna, Qwen); the LMSYS SpecBundle (Llama 3.1/3.3, Llama-4 Scout & Maverick, Qwen3 dense/MoE up to Coder-480B, GPT-OSS — several contributed by Meituan, Nex, EigenAI); RedHatAI speculators in vLLM-native format (Llama, Qwen3, gpt-oss, Gemma-4); IBM's ibm-ai-platform MLP accelerators (Llama, Granite, CodeLlama); and AMD's PARD collection. Models with built-in MTP heads (DeepSeek-V3/R1, Qwen3-Next, GLM-4-MoE, Ernie, Nemotron-3) need nothing at all.
Google Research — a retrospective from the original authors · vLLM — spec decode boosts performance up to 2.8× · LMSYS — Lookahead Decoding · LMSYS — SpecForge · LMSYS — DFlash & Spec V2 · Snowflake — SuffixDecoding at production scale · PyTorch/IBM — a hitchhiker's guide to speculative decoding · Together — ATLAS, speculators that keep learning from live traffic · Together — workload-customized drafts for DeepSeek-R1 · NVIDIA — TRT-LLM spec decode up to 3.6× · NVIDIA — DFlash on Blackwell, up to 15× · Baseten — production pitfalls of draft-target SD · Red Hat — the speculators format · HF — LayerSkip in Transformers. Surveys: Xia et al. 2024, Spec. Decoding and Beyond 2025; benchmark harness: Spec-Bench; curated list: Awesome-Speculative-Decoding.
EAGLE-3 is the de-facto production baseline. Native in vLLM, SGLang, and TensorRT-LLM, with pre-trained heads for most open models and two open training frameworks (SpecForge, Speculators). Any new method should be compared against it, not against vanilla SD.
Match the family to the workload. Overlap-heavy traffic (agents, RAG, editing) → retrieval methods, free and often ahead of EAGLE there (SuffixDecoding 5.3×). Open-ended chat/reasoning → trained heads. No training budget and no overlap → n-gram still costs nothing; self-speculative gives a modest bump on any checkpoint.
Papers report best cases; engines report reality. The one same-harness comparison here (SGLang, Llama-3.1-8B) shows EAGLE-3 at 2.36× where its paper headline is ~6.5×. Batch size matters too: speculation competes with batching for the same idle FLOPs, though MagicDec shows long-context large-batch wins are real.
Losslessness is a spectrum you choose. Rejection-sampling methods are provably exact; Medusa's typical acceptance, CLLMs/Jacobi-Forcing fine-tuning, and 2025's relaxed-verification wave (ReSpec, FLy) trade exactness for speed — fine for some products, unacceptable for evals.
The frontier is structural, not incremental. 2026's biggest gains change the drafting computation itself: non-autoregressive block-diffusion drafts (DFlash, >6×), draft/verify parallelism (PEARL, Saguaro, P-EAGLE), confidence-scheduled verification (DSpark), and MTP heads shipped from pretraining — which may eventually make bolt-on speculation unnecessary for new models.