Laps and Bytes

One objective, two levers

For a single decode stream on an N-stage WAN pipeline verifying a speculation tree of n nodes per lap, throughput is

tok/s  =  L(n)  ÷  T_lap(n),     T_lap(n) = Σ_i [ RTT_i + n·s_h/BW_i ] + compute(n) + T_return

where L(n) is expected accepted tokens for an n-node tree (drafter-dependent, growing ≈ logarithmically in n — Sequoia, Thm 3.6), and s_h is the per-node payload crossing each link (≈ d_model × bytes-per-value, plus protocol overhead). Lever 1 raises L(n); lever 4 shrinks s_h and T_return. Everything below is evidence about the shape of those two functions. DSD ('25) publishes the same algebra as a communication-reduction ratio — accepting k tokens per round cuts comm latency by (N−1)t₁(k−1)/k, measured ≈37% at 8 nodes — and the split-inference-over-WAN paper measured the constant term: at ~80 ms RTT, RTT is 63–64% of step time and acceptance below ≈1.15–1.20 tokens/pass does not pay at all.

Deep dive · Amortize traversals

Chains saturate: the 1/(1−α) ceiling

For a chain drafter with per-token acceptance rate α and draft length K, the classic result (Leviathan et al.) is E[L] = (1−α^K+1)/(1−α), which saturates at 1/(1−α) as K grows. An α = 0.8 drafter can never average more than 5 accepted tokens per lap no matter how long it drafts; α = 0.9 caps at 10. Over WAN — where drafting is local and nearly free while each lap costs ~200 ms — the classical cost-ratio optimum pushes K up hard, but the geometric ceiling stands:

Two quantitative warnings from the 2025 scaling-law study (arXiv 2505.07858): acceptance length grows only logarithmically in draft capacity — τ ≈ 0.74·log₁₀(draft layers) + 4.61 and τ ≈ 0.08·log₁₀(pretrain tokens) + 5.05 — so each extra accepted token costs roughly an order of magnitude more drafter; and the optimal verification width shrinks ≈ 1/√batch, so amortization budgets belong to low-batch regimes — exactly the single-stream WAN setting.

Trees keep growing: log-in-n, no ceiling

Sequoia proves that with optimal topology the expected accepted length G(n) is unbounded, growing ≈ log n — while naive k-independent-sequence trees asymptote (+33% tokens/step over them at 512 nodes). The measured frontier confirms it. EAGLE-2's dynamic tree is the accepted-per-node efficiency champion (τ ≈ 3.7–5.4 from only 60 nodes: top-10 expansion, depth 6, global rerank), tuned for GPUs where verification FLOPs bind. SpecExec is the accepted-per-lap champion, built for the offloading regime whose cost structure matches WAN — a Dijkstra-style search materializes the top-B cumulative-probability prefixes:

Free acceptance from better verification rules

A second, purely verifier-side family raises L(n) at zero draft cost — all lossless, all stackable on any tree:

At a 200 ms lap, a "+5–8%" verifier-side gain is 10–16 ms of WAN time saved per lap, from a code change that touches no model and no network.

PP-specific mechanics: depth rides the pipeline

The pipeline changes tree economics in one important way: tree depth can grow one layer per stage-hop instead of one lap per layer. SpecPipe/PipeDec's schedule generates draft layer ℓ+1 while layer ℓ is in flight, keeps a separate tree-KV per stage so each hop transmits only the newest tree layer (width × d_model, not the whole tree), and migrates accepted tokens' KV into the model cache without recompute. FlowSpec adds in-flight pruning where only index sets travel — dead branches stop consuming downstream bytes. PipeInfer sizes micro-speculations (1–4 tokens) reactively with a confidence cutoff and cancels invalidated runs mid-pipe; Speculative² Decoding pre-builds the next tree during verification so a cache hit hides all draft latency — at WAN lap times there is room to pre-speculate several plausible outcomes. DiP-SD gives the multi-stream version: jointly choose per-stream draft length and batch assignment to maximize E[accepted]/pipeline-span. One caution from SPD: with in-pipeline drafting heads, returns saturate as stage count grows because newly injected tokens carry only shallow features — deep pipelines favor a strong external drafter over in-pipe heads.

Deep dive · Shrink the wire

What exactness minimally requires

The accept test itself is nearly free: for γ drafted tokens the verifier needs token ids (~3 B each) plus the scalar draft probability q(x) of each chosen token (2 B) — ~40 bytes at γ = 8. The expensive part is rejection resampling, which needs the full target distribution p and draft distribution q co-located somewhere. Published protocols occupy exactly three exact placements — plus a fourth, information-theoretically optimal one that no system ships yet:

Activations are the real bytes: compress the residual stream

In steady-state PP decode the dominant traffic is the hidden state crossing each link — d_model values per token per hop (per tree node per hop when verifying trees). The evidence for how low it goes: Petals shipped dynamic blockwise 8-bit in production ("halves bandwidth… no noticeable effect on quality"); TAH-Quant reaches 3–4 bits on the PP channel (tile-wise Hadamard + entropy-guided bit allocation, 4.3× end-to-end — a training paper, but the forward-pass quantizer is directly reusable at inference); the TP analog (arXiv 2411.09510) lands at ≈4.25 effective bits with +1–3% perplexity, with the sensible rule "quantize only where links are slow." BloomBee ('26, arXiv 2604.21072) adds the lossless option: byte-separation coding of activations to 46% of original size (vs 69–71% for ZSTD/ZipNN), bit-exact at ~300 ms compression overhead — chosen precisely because activation outliers make quantization risky, and worth +18.4% throughput alone at 20 Mbps. The split-computing literature (FrankenSplit: learned variational bottleneck + entropy coding, −60% bitrate at no accuracy loss) has never been applied to the LLM residual stream — an open transplant. Two more unclaimed ideas: anchor+delta coding of activations across adjacent decode steps (CacheGen proved 2.4–2.9× lower variance for deltas in KV space), and joint compression of tree batches (siblings share prefixes; nobody compresses them together).

Byte accounting: when does bandwidth start to bind?

Computed for this review (framing ~20 B ignored; per token per hop = d_model × bytes/value). W* is the tree width at which per-hop serialization exceeds a 25 ms hop RTT — i.e., where bandwidth replaces latency as the binding constraint:

Readings: at ≥100 Mbps with quantized activations, latency dominates all the way to SpecExec-scale trees — width is nearly free. At 10 Mbps, int4 is what buys tree room (8B: W* 4→15); a 70B in fp16 on a 10 Mbps link is bandwidth-bound at width 2, making activation quantization mandatory before any speculation tuning. Protocol metadata (≤50 B) and top-K rejections (≤100 B) are noise everywhere. Real-WAN cross-check: the split-inference paper measured 8 KB/token (7B) and 16 KB (70B) per link, with RTT = 63–64% of step time at 80 ms.

Return path and churn

The return leg needs only accepted ids — and even those compress: range-coding tokens against the drafter's shared model costs ≈ cross-entropy bits (LLMZip: 0.71 bits/char with a 7B prior), the same ≈D_KL bound the exponential-races scheme achieves natively. Intel's shards relay the sampled token hop-by-hop instead of a 16 KB hidden state on the return path. For volunteer churn, KV migration is the one big transfer left: CacheGen's anchor+delta, layer-tiered quantization and arithmetic coding compress KV 3.7–4.3× beyond an 8-bit baseline (~15–17× vs fp16) with per-chunk bandwidth adaptation.

Joint sizing: the bandwidth-aware tree

Putting the two levers into the §01 objective yields the design chart. Take an 8-hop pipeline at 25 ms/hop (200 ms lap), an 8B model with int4 activations (2 KiB/node/hop), and a SpecExec-class drafter (L(n) ≈ 3.05·ln n − 1.3, fit to its measured curve). Sweep tree size n at three link speeds:

The same sweep explains Part II's systems: FlowSpec's segment pruning and PipeDec's one-layer-per-hop transmission both exist to keep the effective n·s_h term small while letting logical tree size grow — they move the 10 Mbps curve toward the 1 Gbps one without touching the link. BloomBee's learned pruner is the third instance — scoring candidates before transmission, it keeps 96% of acceptance while cutting tree size 60%.

Takeaways