Literature review · Pluralis · Part IV · August 2026
Part III sized the payload levers — how many tokens ride a lap, how few bytes a lap costs. This companion deep dive takes the two scheduling levers: fill the bubbles (idle stage-compute while the wavefront is elsewhere) and overlap async (the drafter idling while verification is in flight). They waste different resources — bubbles burn idle FLOPs, lockstep burns exposed wall-clock — and over WAN both are enormous: a measured two-hop split spends 64% of step time on the network, and an 8-hop pipeline leaves stages ~99% idle. The literature now has schedules, closed-form theory, and cancellation machinery for reclaiming both.
Lockstep speculative decoding alternates: the drafter works while the verifier (and the network) sit idle, then submits and sits idle itself for the whole round trip. In a pipeline the second waste appears too: stages behind and ahead of the wavefront hold no work. These are separate resources — bubbles are idle compute at the stages; lockstep is exposed latency at the drafter — and the cures compose: keep the drafter busy through the in-flight window (lever 3), and make the stages chew on speculative continuations while the wavefront is elsewhere (lever 2).
Other requests — the classic answer, and what engines do: PP needs ≥ N concurrent microbatches for full utilization; vLLM's V1 PP executor literally enqueues empty microbatches when traffic is thin, SGLang's chunked pipeline parallelism (Jan '26) fills prefill bubbles with 4–12K-token chunks (3.31× prefill throughput at PP4), and Sarathi's decode-maximal batches with one prefill chunk cut PP bubbles 6.29×. BloomBee ('26) shows the WAN version: micro-batching that overlaps GPU compute with CPU-mediated network transfer is worth +39.5% alone at 20 Mbps, +76.3% combined with its lossless compression. All of it is inert for a single stream with no prefill work — the volunteer-WAN regime FlowSpec calls "sparse requests." Speculation of the same stream — the subject of this section: idle stages verify speculative continuations, buying latency rather than throughput. Deferred auxiliary work — the sleeper option: EE-LLM backfills deep-layer KV inside bubbles after an early exit (≥2× with exactness caveats), and Online Speculative Decoding retrains the drafter on live traffic with spare FLOPs (acceptance +0.1→0.65, worth 1.42–2.17×) — at 99% idle, a WAN swarm gets draft-retraining nearly free.
| System | Schedule: what each stage does each tick | Flush / prune protocol | Numbers | Code |
|---|---|---|---|---|
| SpecPipe / PipeDec '25 | Dedicated draft node expands exactly one new tree layer per tick (top-k by cumulative log-prob, w=32 best) while every in-flight layer advances one stage — tree depth grows in lockstep with pipeline depth, so all stages always hold a layer | Last stage broadcasts hit_index; hit → subtree extracted by mask algebra, prediction-tree KV promoted to model KV (two-level cache); miss → full tree reset | 4.46–7.79× vs plain PP; 2.2–2.7× vs static-tree PP; 14 stages ≈ 1.64× over 7; plateaus at 21 (tree accuracy decays past depth ~20) | "after review" |
| FlowSpec '25 | EAGLE-2 tree (80 nodes, depth 6) split into ≤16-token segments sorted by cumulative confidence (parent ≥ child keeps topological validity), streamed back-to-back through the verify stages | Per-stage index sets (I_local, I_incache) prune in flight; context/score-aware expansion refills when segments run dry | 1.36–1.77× vs naive pipelined SD on 5× Jetson; beats PipeDec on Vicuna-13B (1.70× vs 1.49×) | yes |
| SPD '26 | Drafts from mid-pipeline features already computed for in-flight tokens (multi-depth anchor aggregation); per cycle one token exits and one speculated token enters stage 1 — zero-bubble when spec-module layers ≤ L/n | Per-token verification along the staircase; theoretical speedup S = α·n | n=8: 2.20–2.24× (best); n=16: 1.67–1.69× — α collapses as injected tokens see only shallow features | claimed |
| PPSD '25 | First E layers + exit head = permanent draft stage; remaining layers split into verify stages; deep stages verify token t while the exit path drafts t+1 | Per-token verification — no accumulated rollback | 1.83–3.81× (Vicuna/LLaMA-2 7B–70B, batch 1) | none |
| PipeInfer '24 | Canonical single-token run always in flight; head node launches 1–4-token speculative microbatches whenever no logits are pending — bubbles carry a rolling queue of micro-speculations | ID-only cancellation mid-pipeline; cache multibuffering, copy-on-accept | 1.5–2.15× vs sync speculative inference; largest gains on GbE, robust at 52% acceptance | yes |
| EE-LLM '24 | After an early exit, the token id loops to stage 1 immediately while the rest of the full forward continues behind it, backfilling deep-layer KV inside the bubbles | Deferred KV backfill parallel to the next token | ≥2× vs full-model inference (early-exit regime) | yes |
Bubble-filling's limit is not idle FLOPs — those are nearly infinite over WAN — but acceptance depth. PipeDec's ideal is "one token per pipeline step" (token latency collapses from a full lap to one stage-hop), which holds only while the surviving tree stays as deep as the pipe. Both measured curves hit the same wall:
SPD (arXiv 2605.30852v2), Qwen3.5-4B/9B, wall-clock vs plain PP · theoretical S = α·n until α collapses
SmartSpec/TurboSpec makes the tension explicit: speculation and batching compete for the same spare FLOPs, so its goodput controller shrinks speculation length as load rises — down to zero at saturation. The corollary for a volunteer WAN swarm is the inverse: at sparse request rates (FlowSpec's motivating regime, and Pluralis's), speculation is the only filler available, and the controller should run at its speculation-maximal end. Hybrids exist — SpecPipe-DB and DiP-SD batch multiple streams and speculate (DiP-SD: 17.9× vs AR, 1.93× vs AR with greedy batching) — the right shape once a swarm serves more than one user.
PicoSpec states the algebra: lockstep costs T_draft + T_RTT + T_verify per round; overlap costs max(T_draft, T_RTT + T_verify) — the RTT disappears entirely whenever drafting is the longer leg ("latency immunity"). SpecEdge, the one system with real WAN sweeps (14–65 ms RTT, only 22% degradation across the range), turns it into a calibration rule: pick draft depth so server-verify time ≈ edge-draft time + RTT. And PipeSpec supplies the closed form for what asynchrony buys: sync SD makes (1−α^(γ+1))/(1−α) tokens per (γ/c + 1) units of time; the async pipeline deletes the γ/c drafting-wait term outright, and produces >1 token/step — faster than autoregressive — for any α > 0, γ > 0. Measured: utilization 23.0% → 39.7%, energy 16.5 → 5.8 J/token.
Depth alone is poor spend, though: the marginal value of drafting deeper decays as α^depth (PipeSpec's exponent; NeurIPS'24 theory says batch/branch drafting provably cuts rejections most exactly when alignment is poor). So the published consensus for a 200 ms window is rate-match the depth, then hedge: SpecBranch spawns confidence-scaled parallel branches at uncertain tokens (+15% over PEARL on misaligned pairs, ~50% fewer rolled-back tokens); Saguaro speculates on the verification outcome itself, pre-computing a cache of next-drafts keyed by (accept-count, bonus token) with ~90% bonus-token prediction and a proof that the optimal cache is a geometric fan-out over outcomes. Both curves flatten at 2–4 branches.
| System | Overlap structure | Distinctive contribution | Numbers | Code |
|---|---|---|---|---|
| PEARL ICLR'25 | Pre-verify (target checks token 1 during drafting) + post-verify (keep drafting during verification) | Ablation shows post-verify is the dominant win (removing it costs 0.7–1.2× vs 0.14–0.29× for pre-verify); draft length emerges adaptively | 4.43× vs AR; 1.50× vs SD; run length between rejections 26.5–39.9 tokens | yes |
| AMUSD '24 | Fully decoupled free-running loops on two GPUs; single-writer ring buffers | Minimal template: position-based rollback with tensor-cropped KV; wins on wall-clock despite fewer accepted tokens/step (2.75 vs 5.35) | 1.96× vs AR, 1.27–1.36× vs sync SD; energy/token ≈ parity | yes |
| PipeSpec ACL-F'25 | k models in size order, every stage free-runs as producer-consumer | The closed-form theory (E[N], steady-state ρ) + rollback cascade to the last common token | 2.54× vs AR; 123.7→48.8 ms/token; 5.8 J/token | not stated |
| PipeInfer SC'24 | Canonical always-progress run + rolling micro-speculations behind it | ID-only early cancellation mid-layer; empty placeholder tensors preserve MPI ordering; confidence throttle | 1.5–2.15× vs sync SD; gains largest on slowest interconnects | yes |
| Saguaro (SSD) ICLR'26 | Drafts during verification by predicting the verification outcome | Speculation cache {outcome → ready next-draft}; geometric fan-out optimality (Thm 12); batch-adaptive fallback | ~30% mean over tuned SD, ≤2×; ≤5× vs AR; holds +20% at larger batches | yes |
| SpecBranch ICLR'26 | Parallel hedge branches at low-confidence tokens, concurrent with verification | Confidence-scaled branch count + hybrid draft-length control (H-RAD); partial reuse of rejected paths | 1.8–4.5× vs AR; +15% over PEARL misaligned; ~50% fewer rollback tokens | yes |
| SpecEdge NeurIPS'25 | Edge drafts proactively along the single best tree path while server verifies | Real-WAN RTT sweeps (15–65 ms, −22% worst case); depth calibration rule; anti-branch argument when draft compute is scarce | 1.91× cost efficiency, 2.22× server throughput, −11.2% ITL at 14 ms RTT | yes |
| SwiftSpec '25 | Draft and target disaggregated onto separate GPU groups with independent TP | Tree re-rooting: KV of the surviving branch salvaged across both models each round — rollback as cache surgery | 1.75× avg over prior SOTA systems; Llama3-70B at 348 tok/s on 8 Hoppers | not stated |
| DSI ICLR'25 | Speculation parallelism: replicas verify every draft prefix concurrently | Proof: ≥ AR and ≥ sync SD in expectation for any drafter — but the model excludes communication cost and multiplies target replicas | 1.29–1.92× vs sync SD (sims/real latencies) | yes |
| Early-exit preemption '25 | Server emits provisional early-exit tokens mid-verification; edge restarts drafting sooner | Shrinks the exposed window instead of overlapping around it; hint-not-commit hybrid unexplored | −35% latency vs cloud AR, +11% from preemption alone | not stated |
| Speculative Verification '25 | Companion model runs concurrently with verification (CUDA MPS) | Predicts acceptance from draft–companion agreement; adjusts verification length online | ≤2× over SD, 1.4× avg at batch 32–80 | not stated |
Disambiguation: ParallelSpec (arXiv 2410.05589) is not async overlap — it makes the drafter itself non-autoregressive (one pass drafts k tokens via [MASK]s). Orthogonal, and stackable with everything above. Minions (IPDPS'24) and CoSine ('25) overlap across requests (drafting cluster feeds a verification server) rather than within one stream.
Speedup over the synchronous speculative baseline each paper measured against — the isolated value of overlap
Every scheme above is output-exact in principle — draft-ahead work conditioned on an unverified prefix is discarded on rejection, so the acceptance rule never changes. In practice three things bite. Sampling: nearly all async evals run greedy; under temperature, the verifier's bonus-token RNG must be decoupled from verification timing, and no paper treats seeded-RNG reproducibility across async replays. Batching: "Batch Speculative Decoding Done Right" (arXiv 2510.22876) shows ragged acceptance desynchronizes position ids/KV mappings so badly that prior batched implementations emit gibberish (0–40% exact match vs HF); their EqSpec invariants restore 92–97% — mandatory reading for any WAN verifier batching in-flight segments. Waste: "Lossless but Not Free" measures 62% of drafted tokens discarded at the optimal depth on consumer hardware; AMUSD's energy parity vs PipeSpec's 3× energy win shows the waste verdict flips with topology — and nobody publishes joules-per-accepted-token for async vs sync at matched quality.
Asynchrony is only as cheap as its undo. Published mechanisms, cheapest first — over WAN the decisive property is that none of them requires an extra round trip:
| Mechanism | Where | How it works | Cost |
|---|---|---|---|
| Position trim / tensor crop | AMUSD; vLLM/SGLang internals | Drafter reverts to the verified position; KV cropped to the prefix | O(1) metadata |
| Attention-mask rollback | Intel AI-PC shards (Part III) | Rejected positions masked out + logical position_ids; no physical KV surgery | bit-exact; saves ~48 ms/rejection on iGPU |
| Broadcast prune by index | FlowSpec | Per-stage index sets drop tree entries + KV rows locally | one small broadcast per verify outcome |
| ID-only cancel + placeholder tensors | PipeInfer | Cancel message carrying only the run ID back-propagates; nodes abandon runs mid-layer; empty tensors keep message ordering intact | ~bytes per hop, near-zero compute residue |
| Multibuffered KV, copy-on-accept | PipeInfer | Speculative runs write to back-buffer partitions; acceptance copies to the canonical front buffer, pipelined with compute | one cache copy per accepted run |
| Tree re-rooting with KV salvage | SwiftSpec | Surviving branch's KV reorganized and reused across draft and target models each round | cache surgery, no recompute |
| Rollback cascade | PipeSpec | Rejection at stage j rolls every upstream stage back to the last common token | discards all draft-ahead above j |
Gap: no paper publishes head-to-head trim vs mask vs versioned-cache rollback latencies — a cheap microbenchmark study, and the numbers a WAN protocol design actually needs.
As of Aug 2026 no mainstream engine runs drafter and verifier concurrently on separate compute. vLLM's async scheduling became compatible with EAGLE spec decode (PR #24799, merged Nov '25) — but it hides CPU scheduling behind GPU work (+1.8–7.1% TPS), draft and verify still execute back-to-back; there is an open feature request for Saguaro-style SSD (issue #36037). SGLang's overlap scheduler gained spec-decode compatibility incrementally through Spec v2 (issues #11762, #9961), with real-world regressions still open (#12411, #13019). TensorRT-LLM enables its overlap scheduler with the chain drafter on the PyTorch backend. The true draft/verify-parallel systems — SwiftSpec, AMUSD, Saguaro, PipeInfer — all remain out-of-tree research code. For WAN PP this repeats Part II's conclusion at a second level: the scheduling layer a Pluralis-style deployment needs exists only in research repos, and the engines' "overlap" work solves a different, single-node problem.
Bubbles and lockstep waste different resources; both are ~free money over WAN. Idle stage-FLOPs (~99% in an 8-hop single-stream pipeline) should carry speculative tree layers, deferred KV backfill, and even online draft retraining; the drafter's 200 ms in-flight window should carry the next block plus hedges. The cures compose — PipeInfer and SpecPipe each implement half of the full design.
The two laws to build against: L_async = max(T_draft, T_RTT + T_verify) — rate-match drafting to the in-flight window (SpecEdge: verify ≈ draft + RTT) — and PipeSpec's theorem that asynchrony deletes the drafting-wait term entirely, beating autoregressive for any α > 0. Depth's value decays as α^depth, so spend the excess window on 2–4 outcome-keyed hedge branches (Saguaro, SpecBranch), not more depth.
In-pipe speculation hits a depth wall around 8–14 stages. SPD peaks at n=8 (2.24×) and falls at 16; PipeDec plateaus at 21 stages. Deep WAN pipelines should draft with a strong external drafter (client retrieval, EAGLE-class heads) and let stages verify, rather than drafting from mid-pipe features.
Cancellation is a solved problem — copy PipeInfer. ID-only cancel messages, empty placeholder tensors to preserve stream ordering, a canonical always-progress run as the AR-speed floor, and copy-on-accept multibuffered KV. It is the one async design validated where links are slowest, and every piece is WAN-shaped.
Three open gaps, all cheap papers: a WAN-latency-aware version of DSI/PipeSpec's guarantees (both currently assume free communication); exactness of async speculation under stochastic sampling with seeded RNG; and trim-vs-mask-vs-versioned rollback microbenchmarks. Plus the standing one: no engine ships any of this.