Literature review · Pluralis · compiled August 2026
When a model's pipeline stages live on different nodes across the internet, every decoded token pays a full lap of link RTTs — and speculative decoding is the one technique that divides the lap count: draft several future tokens cheaply, verify them all in a single parallel pass, provably losing nothing. This review maps that technique for the WAN pipeline-parallel setting: which methods exist, which survive real links, and what to build.
A summary brief plus four parts: the published method landscape, speculation meeting pipeline parallelism over WAN, and two quantitative deep dives — the payload levers that decide WAN throughput, and the scheduling levers that reclaim idle time.
Summary
The capstone, on one page: where the field stands, the seventeen works that matter most for speculative decoding in WAN pipeline-parallel inference, and twelve open gaps — theory, protocol, compression, and systems — each sized and pointed at the papers it starts from, ending with the stack to build.
Part I
Maps ~45 published methods into five families by where the draft tokens come from, compares their claimed gains, and tracks which are open source and which ship natively in vLLM and SGLang.
Part II
Pipeline-parallel decoding pays two structural taxes: the bubble, and a full traversal of every inter-stage link per token. Which speculative methods are already used with PP, which are attractive but unused, and what a WAN-PP deployment should adopt.
Part III
Deep dive into the two levers that decide WAN throughput: amortize traversals (acceptance-length theory, tree construction, verifier rules) and shrink the wire (exact ~50-byte protocols, 3–4-bit activations, byte accounting) — joined into a bandwidth-aware tree-sizing model.
Part IV
Deep dive into the two scheduling levers: fill the bubbles (in-pipe speculation schedules, the 8–14-stage depth wall, what else can ride idle stages) and overlap async (the max(·) latency law, hedge-branch economics, rollback and cancellation machinery).