Literature review · Pluralis · Part II · August 2026
Pipeline-parallel (PP) decoding has two structural taxes: (N−1)/N of stages sit idle per token (the bubble), and every token pays a full traversal of every inter-stage link — which over WAN means a full lap of internet RTTs. Speculative decoding attacks both at once: one traversal can verify K drafted tokens, and speculation gives idle stages useful work. This review maps which speculative methods are already used with PP, which are attractive but not yet used, and what a WAN-PP deployment (the Pluralis setting) should actually adopt. It builds on the companion Speculative Decoding Atlas.
In single-stream PP decoding, a token enters stage 1, flows through all N stages, and its sampled successor must return to stage 1 before anything else happens. Compute utilization is 1/N, and — decisive over WAN — latency per token ≈ Σ inter-stage RTTs + compute. With 8 volunteer stages at ~25 ms per hop plus the return link, that is ~200+ ms per token no matter how fast the GPUs are. Speculative verification changes the unit of work: a traversal carries a block of K drafted tokens and the target verifies all of them in that same lap. Accepted length L divides the number of laps — L is an RTT divisor. Extra draft tokens travel as payload bytes (cheap), not as extra laps (expensive), which is precisely the WAN trade.
Every SD × PP system in the literature pulls some combination of four levers. They compose — the strongest 2026 designs pull all four.
Two bodies of work have actually fused speculation with pipeline execution. The first splits layers across devices and speculates to fill the pipeline; the second splits draft and verify across a network link (edge drafts, remote verifier) — which is exactly a two-stage WAN pipeline, and where the wire-protocol lessons come from. Everything here exists as running code or a published measurement; none of it ships in a mainstream engine yet (§06).
| System | Year · venue | Levers | Mechanism | Reported gain (its own baseline) | Code |
|---|---|---|---|---|---|
| SpecInfer / FlexFlow Serve | '23 · ASPLOS'24 | amortize | Token-tree verification; target runs under TP+PP knobs (PP as deployment option, not co-designed) | 1.4–2.4× multi-node | FlexFlow |
| PipeInfer | '24 · SC'24 | async bubble | Continuous asynchronous speculation: a non-speculative run always in flight + speculative microbatches; early cancellation kills invalidated branches mid-pipeline | up to 2.15× vs sync SD; robust to low bandwidth & low acceptance | repo (llama.cpp+MPI) |
| AMUSD | '24 · IEEE | async | Draft and verify models run continuously on separate GPUs with rollback — a 2-stage draft/verify pipeline | 1.96× vs AR | repo |
| PipeSpec | '25 · ACL-F'25 | async | k models of increasing size as an asynchronous hierarchical pipeline (each drafts for the next), lightweight rollback breaks the draft→verify lockstep | up to 2.54× (LLaMA-2/3, multi-GPU) | repo |
| SpecPipe / PipeDec | '25 · arXiv | bubble amortize | Dynamic speculative token tree expands layer-by-layer inside the pipeline, filling every stage slot; relaxed speculative window, two-level KV | 4.46–7.79× vs plain PP decode (Llama-3.1-70B, 10 GbE, 7–21 stages) | "after review" |
| FlowSpec | '25 · arXiv | bubble | Continuous pipelined tree SD for sparse-request edge pipelines: score-based step-wise verification, pruning, dynamic expansion | 1.36–1.77× (5× Jetson) | repo |
| PPSD | '25 · arXiv | bubble async | Early-exit layers (draft) and remaining layers (verify) run as overlapping pipeline stages — verify-while-draft, per-token | 2.01–3.81× | not found |
| SPD | '26 · arXiv | bubble | Speculation module reads partially-processed features at multiple pipeline depths, timed inside the pipeline step — zero-bubble when spec layers ≤ L/n | 2.20–2.24× at 8 stages (drops to ~1.67× at 16) | claimed, URL unverified |
| prima.cpp | '25 · arXiv | amortize | Home-cluster piped-ring PP (Wi-Fi, mixed CPU/GPU) that ships with draft-model SD on top | 32B + SD = 26 tok/s; 5–17× lower TPOT vs llama.cpp/exo | repo |
| Intel AI-PC pipeline shards | '26 · arXiv | amortize wire | Per-layer OpenVINO shards over plain TCP (~16 KB/token/hop) + SD with zero-cost rollback of rejected drafts via attention mask instead of KV trim | 1.79× 2-node Llama-3.1-8B; "usable" at 100 ms/hop simulated WAN | repo |
| BloomBee | '26 · arXiv | amortize wire | Petals-descendant P2P substrate (DP layer placement, micro-batching, KV offloading) that ships EAGLE-2 speculation with a learned MLP pruner — candidates scored and dropped before transmission (tree −60%, 96% of acceptance kept) — plus a break-even bandwidth formula S* for when speculation pays at all | 1.76× vs Petals at 20 Mbps (67 vs 38 tok/s, LLaMA-30B); up to −43% latency; tested on real internet links (280–643 Mbps) | repo · Apache-2.0 |
| System | Year · venue | Levers | Mechanism | WAN-relevant numbers | Code |
|---|---|---|---|---|---|
| DSSD | '25 · ICML'25 | wire amortize | Device drafts, server verifies; resampling moved to the device so the uplink carries token ids only — distributions cross the wire only on rejection | <50 B/round uplink (vs ~61 KB); 1.5–2.4× at 0–50 ms RTT, 10–100 Mbps; naive distributed SD ≤1× | repo |
| PicoSpec | '26 · arXiv | async wire | Edge drafts block k+1 while cloud verifies block k; verification starts on partial arrival; top-K sparse distributions cut the downlink O(V)→O(K) (~1000×) | up to 2.9× (Jetson↔A100 WAN); vanilla SD on the same link = 0.44× — slower than no speculation | not seen |
| Mixture of Attentions | '25 · ICLR'25 | amortize | SD head designed for client–server split: small device model proposes, server completes with minimal calls; keeps generating on disconnect | +9.5% over EAGLE-2; client–server latencies in paper body | not seen |
| SLED | '25 · SEC'25 | amortize | Many heterogeneous edge devices draft; one shared server batch-verifies — verification-as-a-service | 2.2× throughput, 2.8× device capacity | not seen |
| CoSine | '25 · arXiv | async | Disaggregated multi-drafter cluster + verification server with confidence fusion | −27% latency, 1.31–1.62× vs SpecInfer (10 GbE) | no repo |
| DSI | '25 · ICLR'25 | async | "Speculation parallelism": replicated target/drafter instances verify overlapping lookahead segments concurrently — provably ≥ both AR and sync SD for any drafter | 1.29–1.92× over SD (sims) | repo |
| Split inference over WAN | '26 · arXiv | amortize wire | Embeddings local, middle layers remote (2-stage PP, WebSocket, 8–10 KB/token); training-free Jacobi lookahead amortizes the RTT | 8.1–10.9 tok/s at real ~80 ms RTT; acceptance 1.2–1.3 tok/step | repo |
| SpecExec | '24 · NeurIPS'24 | amortize | Deterministic mega-trees (hundreds–thousands of drafts) verified in one target pass — built for offloading, whose cost structure ≈ a WAN lap | ~20 accepted tokens per traversal; 4–6 tok/s for 50B+ on a consumer GPU | repo |
Each paper's best case against its own baseline (named per bar) — settings differ; hover for details
The strongest drafters from the Atlas have never been run inside a multi-node pipeline — mostly for engine-plumbing reasons, not fundamental ones. Ranked roughly by expected payoff in a PP/WAN setting:
Why attractive: retrieval drafters need no model access at all — no hidden states, no weights — so in a PP deployment they can run on the client, produce a draft block for free, and spend zero extra pipeline laps. On agentic/RAG traffic they out-accept EAGLE (SuffixDecoding up to 5.3×). Why unused: every published PP system drafts with a model; no one has wired an n-gram/suffix drafter into a layer-split pipeline. This is pure engineering, and the vLLM implementations are reusable. Blocker class: nobody-tried-it.
Why attractive: highest acceptance per drafted parameter (≈4–6 tokens); MTP heads ship free with DeepSeek/Qwen3-Next/GLM weights. In PP their inputs (final-layer features) live exactly where sampling already happens — the last stage — so drafting adds no extra forward lap; drafted ids ride the existing return link. Why unused: engines assume the drafter and the scheduler share one process. vLLM's fix (broadcast accepted-token counts to earlier ranks, RFC #44697) is in flight; SGLang's MTP×PP draft-cache transfer is on the roadmap. Blocker class: engine plumbing, actively being fixed.
Why attractive: SpecExec showed that when a traversal is very expensive (offloading — the same cost structure as WAN), verifying a tree of hundreds of drafts yields ~20 accepted tokens per lap. Sequoia's hardware-aware tree optimizer is exactly the right formalism — its cost model just needs RTT and bandwidth terms instead of PCIe. Why unused: both target offloading/single-node; no multi-node PP system builds trees sized to link budgets. Blocker class: cost-model transplant (research-shaped, tractable).
Why attractive: a block-diffusion drafter emits the whole K-token draft in a single forward pass, so draft latency stops competing with the pipeline clock; DSpark's confidence-scheduled variable-length verification is a natural fit for jittery WAN laps. Why unused: SGLang's DFLASH explicitly requires pp_size == 1; the draft model conditions on target context features, which in PP live on the last stage — same plumbing problem as EAGLE, one more tensor to route. Blocker class: engine constraint, declared.
Why attractive: Saguaro/PEARL-style pre/post-verification is the single-node version of what PipeInfer does across a cluster — drafting continues while verification is in flight, which over WAN means drafting through the entire RTT window for free. Adaptive-K methods should make K a function of pipeline depth and measured link jitter; today no published formula for K(depth, RTT) exists — SPD and SpecPipe only provide empirical scans. Blocker class: open research gap (and a cheap paper for whoever closes it).
Over WAN the pipeline clock is the network, not the GPU: a lap through 8 stages at ~25 ms per hop costs ~200 ms before any compute. Two published numbers frame the whole design space. Naive speculation shipped over a real link makes things worse — PicoSpec measured vanilla networked SD at 0.44× — while a WAN-aware protocol on the same link reached 2.9×. The difference is entirely in the four levers.
Illustrative model, not a measurement: effective tokens/s at a 200 ms pipeline lap (8 WAN hops), compute ≪ RTT · tok/s = L ÷ 0.2 s
Adopt speculative verification before any other optimization: it is the only technique that divides the RTT count. Size the block/tree to bandwidth, not GPU budget — a 64-node draft tree at 2 bytes/token costs nothing next to a 25 ms hop. SpecExec's mega-tree regime (~20 accepted/lap) was designed for exactly this cost structure; Sequoia's optimizer becomes network-aware by swapping PCIe terms for RTT/bandwidth in its cost model.
Verification traffic must be token ids plus occasional sparse corrections: DSSD's device-side resampling gets the uplink under 50 bytes/round (naive: ~61 KB); PicoSpec's top-K downlink is ~1000× smaller than full logits; the Intel AI-PC paper's attention-mask rollback makes rejected drafts free (no KV trimming round-trips). Between stages, quantize activations — Petals shipped dynamic blockwise 8-bit (~2×); TAH-Quant reaches 3–4 bits on the same PP channel.
While a block is in flight, the drafter should already be producing the next block against the optimistic outcome (PicoSpec's draft-ahead; PipeInfer's continuous speculation with mid-pipeline cancellation; Saguaro's verification-time drafting is the same idea one level down). Over WAN the verification window is enormous — hundreds of milliseconds of free drafting time per lap.
With a single stream, stages behind the wavefront are idle; SpecPipe/PipeDec shows the draft tree itself can expand stage-by-stage inside those slots (4.5–7.8× over plain PP). For multi-tenant swarms, FlowSpec's sparse-request variant applies.
PP's structural advantage over TP on WAN: per-stage KV caches stay resident, and only ~d_model bytes per token cross each link (~16 KB/hop for an 8B model in fp16, less quantized) versus TP's per-layer all-reduces. Placement should be network-aware (Helix's max-flow formulation; Parallax's two-phase scheduler for volunteer pools).
Petals-class volunteer substrate + client-side retrieval drafting + link-sized verification trees + DSSD/PicoSpec wire protocol (ids up, top-K down, mask rollback) + PipeInfer-style asynchronous speculation with cancellation + 4–8-bit inter-stage activations. Every ingredient is published and most are open source. The first claim on this ground is BloomBee (Apr '26, arXiv 2604.21072): a Petals descendant that ships EAGLE-2 speculation with a learned tree pruner and beats Petals 1.76× at 20 Mbps — proof the niche is real. But it pulls two levers of four: no hedged asynchronous drafting through the RTT window, no exact ids-only wire protocol, no retrieval drafting, no cancellation machinery. Petals, Parallax, and exo themselves still ship without speculation as of Aug 2026.
| Engine | SD × layer-PP status (Aug 2026) | Detail | Evidence |
|---|---|---|---|
| vLLM | not composable | Docs state SD is not composable with PP as of ≤0.15.0; drafter loads only on the last PP rank; MTP+PP "crashes or silently diverges". Active RFC proposes broadcasting accepted-token counts to earlier ranks. | docs · RFC #44697 · PR #16568 (closed) |
| SGLang | partial / in progress | DFLASH explicitly requires pp_size == 1; EAGLE/MTP carry no documented PP guarantee; PP roadmap targets prefill + PD-disaggregation, with MTP draft-cache transfer pending. Flagship DeepSeek deployment scales via EP/DP/TP, not PP. | docs · roadmap #11857 |
| TensorRT-LLM | disaggregation only | EAGLE-3 with disaggregated (prefill/decode-split) serving is supported and demonstrated via Dynamo (Llama-4 example); no explicit statement for SD with layer-PP. | docs · Dynamo example |
| llama.cpp | not integrated | Local draft-model SD works single-process; RPC distributed mode has no speculative integration — an open feature request proposes an HTTP proxy over separate draft/target servers. PipeInfer exists because upstream doesn't pipeline speculation. | docs · issue #23982 |
| FlexFlow Serve | ✓ (research) | The one system where tree speculation and TP+PP coexist behind flags — SpecInfer's home; low maintenance activity in 2025–26. | SERVE.md |
Everything surveyed, including WAN substrates without speculation and Atlas methods not yet applied to PP. Filter by primary lever:
| System / method | Lever | Year · venue | Core idea | Numbers | Uses SD | PP status | WAN fit | Open source |
|---|---|---|---|---|---|---|---|---|
| SpecInfer / FlexFlow | amortize | '23 · ASPLOS'24 | Tree verification under TP+PP deployment flags | 1.4–2.4× multi-node | ✓ | used (research) | adaptable | yes |
| PipeInfer | async | '24 · SC'24 | Continuous async speculation + mid-pipeline cancellation | 2.15× vs sync SD | ✓ | used (research) | direct | yes |
| AMUSD | async | '24 · IEEE | Async draft/verify on separate GPUs, rollback | 1.96× vs AR | ✓ | used (research) | adaptable | yes |
| PipeSpec | async | '25 · ACL-F'25 | Hierarchical async pipeline of ever-larger models | up to 2.54× | ✓ | used (research) | adaptable | yes |
| SpecPipe / PipeDec | bubble | '25 · arXiv | Draft tree expands layer-by-layer inside the pipeline | 4.46–7.79× vs plain PP | ✓ | used (research) | direct | promised |
| FlowSpec | bubble | '25 · arXiv | Pipelined tree SD for sparse-request edge pipelines | 1.36–1.77× | ✓ | used (research) | direct | yes |
| PPSD | bubble | '25 · arXiv | Early-exit draft layers ∥ verify layers as pipeline stages | 2.01–3.81× | ✓ | used (paper) | direct | not found |
| SPD | bubble | '26 · arXiv | Zero-bubble drafting from mid-pipeline features | 2.24× @ 8 stages | ✓ | used (paper) | direct | unverified |
| prima.cpp | amortize | '25 · arXiv | Home-cluster piped-ring PP shipping with draft-model SD | 32B+SD = 26 tok/s | ✓ | used (research) | LAN-direct | yes |
| Intel AI-PC shards | amortize | '26 · arXiv | TCP layer shards + SD + attention-mask rollback | 1.79× 2-node; 100 ms/hop OK | ✓ | used (research) | direct | yes |
| DSSD | wire | '25 · ICML'25 | Device resampling: token ids up, distributions only on reject | <50 B/round; 2.4× @ 50 ms | ✓ | used (2-stage) | direct | yes |
| PicoSpec | async | '26 · arXiv | Draft-ahead ∥ verification; top-K downlink; partial-arrival verify | 2.9×; vanilla netSD = 0.44× | ✓ | used (2-stage) | direct | not seen |
| Mixture of Attentions | amortize | '25 · ICLR'25 | SD head built for client–server split deployment | +9.5% over EAGLE-2 | ✓ | used (2-stage) | direct | not seen |
| SLED | amortize | '25 · SEC'25 | Many edge drafters, one shared batch-verifier | 2.2× throughput | ✓ | used (2-stage) | adaptable | not seen |
| CoSine | async | '25 · arXiv | Disaggregated multi-drafter + verifier, confidence fusion | 1.31–1.62× vs SpecInfer | ✓ | used (research) | adaptable | no repo |
| DSI | async | '25 · ICLR'25 | Speculation parallelism across replicated instances; provable | 1.29–1.92× vs SD (sim) | ✓ | used (sim) | adaptable | yes |
| Split inference over WAN | amortize | '26 · arXiv | 2-stage split + Jacobi lookahead amortizing a real 80 ms RTT | 8.1–10.9 tok/s @ 80 ms | ✓ | used (2-stage) | direct | yes |
| SpecExec | amortize | '24 · NeurIPS'24 | Mega-trees: hundreds of drafts, ~20 accepted per expensive traversal | 4–6 tok/s, 50B on consumer GPU | ✓ | offloading — not PP yet | recipe transfers | yes |
| EAGLE-3 / HASS heads (Atlas §2) | amortize | '24–25 | Best acceptance/parameter; inputs live on the last PP stage | ~4–6 accepted | ✓ | not yet — engine plumbing (vLLM RFC #44697) | direct once plumbed | yes |
| MTP heads (DeepSeek, Qwen3-Next…) | amortize | '24–26 | Draft head ships with the weights — zero training for deployers | ~1.8–2.5× single-node | ✓ | not yet — SGLang MTP×PP pending | direct once plumbed | ✓ in weights |
| SuffixDecoding / n-gram (Atlas §3) | amortize | '23–25 | Client-side drafting, no model access, zero extra laps | up to 5.3× agentic | ✓ | not yet — nobody tried in PP | ideal for WAN | yes |
| DFlash / DSpark (Atlas §2) | amortize | '26 | One drafter pass per block; adaptive verification length | >6× single-node | ✓ | blocked — SGLang pp_size==1 | attractive | yes |
| Saguaro · PEARL (Atlas §1) | async | '25–26 | Draft during verification — single-node analog of PipeInfer | 4.4–5× single-node | ✓ | not yet applied to PP | attractive | partial |
| BloomBee | amortize | '26 · arXiv | Petals-descendant WAN substrate shipping EAGLE-2 SD + learned tree pruning + lossless activation compression | 1.76× vs Petals @ 20 Mbps | ✓ | used (research) | direct | yes |
| Petals | substrate | '23 · ACL/NeurIPS | Volunteer WAN PP: routing, failover, 8-bit activations | ~2 steps/s, 70B @ ≤100 ms RTT | no SD | WAN PP substrate | direct ancestor | yes |
| Parallax (Gradient) | substrate | '25 · arXiv | Two-phase scheduler: layer allocation + per-request pipelines | — | no SD | WAN PP substrate | direct | yes |
| Helix | substrate | '25 · ASPLOS'25 | Placement/scheduling as max-flow over GPUs and network | 3.3× throughput, 24–42 nodes | no SD | placement layer | adaptable | yes |
| exo · distributed-llama · Cake | substrate | '24–26 | Home-cluster TP/PP frameworks | LAN-scale | no SD | substrates | inspiration | yes |
| TAH-Quant | wire | '25 · arXiv | Tile-wise Hadamard + entropy-adaptive 3–4-bit PP activations | 4.3× e2e (training PP; same channel) | orthogonal | channel-level | direct | unverified |
"used (research)" = running code exists that combines speculation with pipelined execution · "used (2-stage)" = draft/verify split across a network link, the minimal pipeline · "not yet" = published method with no PP deployment found during this review. WAN fit is this review's judgment, not a paper claim.
Speculative decoding matters more for WAN PP than for any other deployment. On a single GPU it buys 2×; over an 8-hop WAN lap it is the difference between 5 tok/s and 25–80 tok/s, because accepted length is the only divisor of RTT count. It should be treated as a first-class protocol feature, not an optimization.
The research exists; the engines don't have it. Ten research systems already fuse SD with pipelines (2–8× reported), but vLLM documents the combination as not composable, SGLang has it half-plumbed, and TRT-LLM only covers disaggregation. Anyone building WAN PP today must implement speculation themselves — and can, since PipeInfer, FlowSpec, DSSD, SpecExec, and the Intel shards are all open source.
The wire protocol is worth as much as the drafter. PicoSpec's 0.44× cautionary number shows naive speculation loses to no speculation over a real link; ids-up/top-K-down plus mask rollback (DSSD, PicoSpec, Intel) recovers it to 2.9×. Ship token ids, never distributions.
Best near-term stack: client retrieval drafts + last-stage head + async laps. Retrieval drafting is free and needs no model access (ideal for agentic traffic); EAGLE/MTP heads sit naturally on the last stage once accepted-token state is distributed (vLLM RFC #44697 is the template); PipeInfer-style continuous speculation hides the RTT window. The pieces compose because verification is one lap regardless of who drafted.
Two cheap papers are sitting on the table. (1) A closed-form or learned policy for draft length/tree size as a function of pipeline depth, RTT, and bandwidth — today only empirical scans exist (SPD, SpecPipe). (2) The composed WAN speculation stack: BloomBee (Apr '26) became the first WAN network to ship speculation — EAGLE-2 with learned tree pruning, 1.76× over Petals — but hedged async drafting, exact ids-only wire protocols, and retrieval drafting remain uncombined; Petals, Parallax, and exo still decode one token per lap.