Literature review · Pluralis · Summary brief · August 2026
The capstone of a four-part review of speculative decoding for pipeline-parallel inference over WAN. One page: where the field stands, the works that matter most for this setting, and the open gaps — each gap sized and pointed at the papers it starts from. Details, evidence, and every claim's source live in Parts I–IV.
Over an N-hop WAN pipeline, decode latency is a lap of link RTTs per token, and speculative verification is the only known technique that divides the lap count — accepted length L is an RTT divisor, worth 5→25–80 tok/s on a 200 ms lap before any other optimization. The four levers are now all evidenced: amortize (trees grow acceptance ≈ log n without bound — chains cap at 1/(1−α); ~21 accepted/pass measured at 1024-node trees), shrink the wire (exact verification needs ~50 B/round, not kilobytes; 3–4-bit inter-stage activations are viable), fill bubbles (in-pipe tree schedules reach 4.5–7.8× over plain PP, with a depth wall at ~8–14 stages), and overlap (L_async = max(T_draft, T_RTT+T_verify); asynchrony beats autoregressive for any α>0). Research prototypes demonstrate every piece separately — several on real or simulated WAN links. No production engine composes speculation with layer-PP (vLLM: "not composable"; SGLang: pp_size==1 for its newest drafter); among WAN networks only BloomBee (Apr '26, a Petals descendant) ships speculation — EAGLE-2 with learned tree pruning, 1.76× over Petals at 20 Mbps — while Petals, Parallax, and exo still decode one token per lap. The field's one cautionary number: naive speculation over a real link runs at 0.44× — slower than nothing — so the wire protocol is as decisive as the drafter.
Curated from ~120 sources across Parts I–IV: the papers a WAN-PP research effort should actually read, newest first within each role.
| Work | Venue | Role in a WAN-PP design | Key number | Code |
|---|---|---|---|---|
| Intel AI-PC pipeline shards | Aug '26 · arXiv | Closest existing precedent: TCP layer shards + SD + attention-mask rollback, tested at simulated 100 ms hops | 1.79× 2-node; viable at 100 ms/hop | ✓ |
| BloomBee | Apr '26 · arXiv | First WAN substrate shipping SD: Petals descendant with DP layer placement, micro-batching, lossless 46% activation compression, EAGLE-2 + learned tree pruning (−60% bytes, 96% acceptance kept), and a break-even bandwidth formula S* | 1.76× vs Petals @ 20 Mbps; −43% latency | ✓ |
| Saguaro (SSD) | ICLR'26 | Drafting through the verify window by predicting the verification outcome; geometric fan-out cache theorem = the hedge-branch economics | ~30% over tuned SD; ≤5× vs AR | ✓ |
| SpecBranch | ICLR'26 | Confidence-scaled hedge branches at uncertain tokens — what to do with excess draft budget | ~50% fewer rollback tokens; +15% over PEARL misaligned | ✓ |
| DFlash | ICML'26 | Block-diffusion drafter: the whole draft block in one pass — the ideal per-lap payload generator (currently pp_size==1) | >6×; ≤15× on Blackwell | ✓ |
| PicoSpec | '26 · arXiv | The async latency law max(T_draft, T_RTT+T_verify) + top-K downlink; measured the 0.44× naive-SD failure | 2.9× edge↔cloud WAN | — |
| SpecEdge | NeurIPS'25 ⭐ | The only real-WAN RTT sweep (14–65 ms); depth-calibration rule verify ≈ draft + RTT | −22% worst case across sweep | ✓ |
| DSSD | ICML'25 | The exact wire protocol: ids + scalars up (<50 B), full distribution down only on rejection, residual resampled locally | 2.4× at 50 ms RTT, lossless | ✓ |
| SuffixDecoding | NeurIPS'25 ⭐ | Client-side model-free drafting — zero model access, zero extra laps; SOTA on agentic traffic | up to 5.3× agentic | ✓ |
| EAGLE-3 | NeurIPS'25 | The drafter baseline to beat; head inputs live on the last PP stage; SpecForge/Speculators industrialize training | 2.36× measured end-to-end (SGLang) | ✓ |
| SpecPipe / PipeDec | '25 · arXiv | The bubble-filling schedule: tree grows one layer per stage-hop, two-level KV — the largest reported PP gain | 4.46–7.79× vs plain PP | promised |
| PipeSpec | ACL-F'25 | The asynchrony theory: closed-form tokens/step, >AR for any α>0 — the WAN-aware version is unwritten | 2.54×; 5.8 J/token | — |
| Exponential races | '25 · arXiv | Shared-RNG coupling: token-ids-only exact speculation, ≈D_KL bits/token — the information-theoretic wire optimum, unshipped | 0 distribution bytes | — |
| DSI | ICLR'25 | Existence proof: with free comm and replicas, async speculation ≥ AR and ≥ sync SD for any drafter — the upper bound to constrain | 1.29–1.92× (sim) | ✓ |
| TAH-Quant | '25 · arXiv | 3–4-bit inter-stage activations on the PP channel — the byte floor for lever 4 (forward pass reusable at inference) | 4.3× e2e (training PP) | — |
| SpecExec | NeurIPS'24 | Mega-tree construction for expensive-traversal regimes (≈ WAN cost structure): optimal-prefix trees | ~21 accepted/pass @1024 nodes | ✓ |
| PipeInfer | SC'24 | The cancellation machinery validated on slow links: ID-only cancel, placeholder ordering, always-progress floor, multibuffered KV | 2.15× vs sync SD; best on GbE | ✓ |
⭐ = spotlight. Honorable mentions, in Parts II–IV: Sequoia (NeurIPS'24, optimal trees), FlowSpec ('25, segment pruning), AMUSD ('24, minimal async template), SwiftSpec ('25, tree re-rooting), MagicDec (ICLR'25, large-batch long-context), Batch-SD-Done-Right ('25, correctness invariants), CacheGen (SIGCOMM'24, KV migration for churn), Petals/Parallax/Helix (the substrates awaiting speculation).
Bandwidth-aware tree sizing. No published policy for draft length / tree size as a function of pipeline depth, RTT, and bandwidth — only empirical scans (SPD, SpecPipe) and single-link calibration rules (SpecEdge). BloomBee's break-even threshold S* is the first published piece; Sequoia's DP optimizer with t(n) rebuilt from RTT + n·s_h/BW terms is the full paper, and Part III's §04 model is its sketch (n* ≈ 16/64/256 at 10 Mbps/100 Mbps/1 Gbps).
WAN-latency-aware guarantees. DSI's "≥ everything" theorem assumes free communication and replicated targets; PipeSpec's closed forms assume zero link latency. Re-deriving either with an RTT term decides when asynchrony pays over WAN — nobody has.
Async exactness under stochastic sampling. Every async system evaluates greedy; seeded-RNG reproducibility across asynchronous replays and outcome-speculation under temperature is untreated. Blocks any lossless claim for a sampled WAN deployment.
Shared-RNG exponential races over a real network. Token-ids-only, exactly P-distributed, ≈D_KL bits/token — published as theory, never shipped; the open question is cross-hardware float determinism on heterogeneous volunteer nodes.
Rollback microbenchmarks. Trim vs attention-mask vs multibuffered/versioned KV — seven mechanisms published (Part IV §04), zero head-to-head latency numbers. A weekend study that every WAN protocol design needs.
Energy accounting. ~62% of drafted tokens are discarded at optimal depth; AMUSD measures energy parity, PipeSpec a 3× win — no joules-per-accepted-token comparison at matched quality exists.
A learned bottleneck for the residual stream. FrankenSplit-style variational bottleneck + entropy coding cut split-computing bitrate 60% in vision; never applied to LLM inter-stage activations. Below 8 bits/value today only fixed quantizers (TAH-Quant) exist.
Joint compression of tree batches. Sibling nodes share prefixes and correlate strongly; every system ships tree activations node-by-node. Low-rank or shared-basis coding of the batch is unexplored.
Anchor+delta activations across decode steps. CacheGen proved adjacent-token deltas have 2.4–2.9× lower variance in KV space; nobody delta-codes the per-step hidden state crossing each PP link.
The composed WAN stack. BloomBee (Apr '26) claimed first-mover — a Petals descendant shipping EAGLE-2 speculation with learned tree pruning, 1.76× over Petals at 20 Mbps — proving the niche is real. But it pulls two levers of four: no hedged async drafting through the RTT window, no exact ids-only wire protocol, no retrieval drafting, no cancellation machinery. The composed stack — those pieces plus DSSD wire rules, PipeInfer cancellation, and 4–8-bit activations on one substrate — is still unbuilt.
Engine plumbing for SD × PP. vLLM RFC #44697 (broadcast accepted-token counts) is the template; SGLang's MTP draft-cache transfer is roadmapped. Whoever lands it makes EAGLE-3/MTP-under-PP available to everyone.
Client-side retrieval drafting in a pipeline. Zero model access, zero extra laps, SOTA on agentic traffic — and no PP system has ever wired it in. Likely the highest value-to-effort ratio on this list.
Rate-match an asynchronous drafter to the lap (verify ≈ draft + RTT), drafting client-side by retrieval where traffic repeats and with an EAGLE-3/MTP head on the last stage otherwise; spend the rest of the in-flight window on 2–4 outcome-keyed hedge branches; verify link-sized trees (n* from the Part III model — roughly 16/64/256 nodes at 10/100/1000 Mbps for an 8B in int4); let idle stages carry the tree layer-by-layer, PipeDec-style, with FlowSpec index-set pruning; ship token ids and scalars, never distributions (DSSD), quantize inter-stage activations to 4–8 bits, roll back by attention mask, cancel by run-ID with an always-progress canonical run as the floor (PipeInfer); migrate KV on churn with CacheGen. Every clause is a published, mostly open-source result — the composition is the contribution. BloomBee — Apache-2.0, Petals-lineage, already carrying EAGLE-2 and a learned pruner — is the natural substrate to extend. If the effort yields one paper first, make it the bandwidth-aware tree-sizing policy: it is the smallest gap, it quantifies the whole design, and every other piece consumes its output.