Acceptance metrics for DeepSeek-V4-Pro with the DSpark drafter

What each number measures, how it is computed from the raw evaluation files, and what the data shows. Compiled 2 September 2026 from DeepSeek-evaluations.zip.

Setup. Target DeepSeek-V4-Pro; drafter = the shipped DSpark head (DeepSeek-V4-Pro-DSpark), vLLM, TP8, fp8 KV cache, $K = 7$ speculative tokens per step, probabilistic draft sampling, temperature 1.0, chat mode, ≤ 2,048 new tokens, seed 980406.

Run A (7_draft): one response per prompt, 3,030 responses, 435,989 verify steps; aggregate counters only. Run B (7_draft_best_of_10): ten responses per prompt, 30,296 responses, 4,344,265 verify steps, with a per-step log (spec_log.csv: request id, drafted, accepted). Run B is the primary source; run A is used as a consistency check.

Datasets. gsm8k 500, math500 500, aime25 30 (math); humaneval 164, mbpp 256, livecodebench 500 (code); mt-bench 80, alpaca 500, arena-hard-v2 500 (chat).

1. Metrics and formulas

Notation: a run consists of responses $r = 1..N$; response $r$ takes $S_r$ verify steps; step $s$ drafts $K = 7$ tokens and the target accepts the first $a_s \in \{0,\dots,K\}$ of them, then emits one bonus (resampled or next) token. Total steps $S = \sum_r S_r$.

$$\text{tokens emitted at step } s = 1 + a_s, \qquad 1 \le 1 + a_s \le K{+}1 = 8$$ spec_log.csv: one row per step, columns (request_id, 7, a_s)

Step outcome. Every step yields between 1 and 8 tokens. All other metrics are functions of the multiset $\{a_s\}$. Because DSpark is a block drafter producing all 7 tokens in one pass, the per-step draft cost is fixed and $a_s$ alone determines the step's value.

$$\tau = \frac{\sum_s (1 + a_s)}{S} = 1 + \frac{\sum_s a_s}{S}$$ run A: acceptance_length = 1 + accepted / drafts
run B: 1 + mean of column 3 of spec_log.csv

Acceptance length, step-weighted. Tokens emitted per verify step, pooled over every step in a dataset. This is the throughput-relevant quantity: wall time is $S \times t_{\text{step}}$ and tokens produced is $S\tau$, so $\tau$ is exactly tokens per unit of step time. In a WAN pipeline where one step is one lap of the pipe, $\tau$ is tokens per lap and tok/s $= \tau / T_{\text{lap}}$.

$$\tau_r = 1 + \frac{1}{S_r}\sum_{s \in r} a_s, \qquad \bar\tau_{\text{resp}} = \frac{1}{N}\sum_r \tau_r, \qquad \tau = \frac{\sum_r S_r\,\tau_r}{\sum_r S_r}$$ run B: samples.jsonl accept_len (per response); results_table.txt alen_mean

Per-response acceptance length and its mean. $\tau_r$ is the acceptance length of one response; $\bar\tau_{\text{resp}}$ weights every response equally regardless of length. The step-weighted $\tau$ is the $S_r$-weighted average of the same $\tau_r$. The two differ whenever $\tau_r$ correlates with $S_r$: if long responses draft poorly, $\bar\tau_{\text{resp}} > \tau$. Use $\bar\tau_{\text{resp}}$ only for "what does a typical prompt experience".

$$\text{verify\_rate} = \frac{\tau}{K+1}, \qquad \text{acceptance\_ratio} = \frac{\sum_s a_s}{K\,S} = \frac{\tau - 1}{K}$$ run A results_table.txt: verify_rate
run B results_table.txt: ratio_max / ratio_mean / ratio_min

Two ratios that appear in the raw tables. They are different normalisations of $\tau$ and are not comparable with each other: verify_rate is the fraction of the $K{+}1$ verified positions that produced a token; acceptance_ratio is the fraction of drafted tokens that were accepted. Both are reported here as $\tau$.

$$p_i = \Pr(\text{position } i \text{ accepted}) = \frac{\#\{s : a_s \ge i{+}1\}}{S}, \qquad i = 0..K{-}1$$ run A: accept_rates_by_position[i] = per_pos[i] / drafts

Unconditional per-position acceptance. Probability that the $i$-th drafted token (0 = first) is accepted, counting every step. Because acceptance is prefix-based ($a_s \ge i{+}1$ means positions $0..i$ were all accepted), $p_i$ is non-increasing in $i$ and $\tau = 1 + \sum_i p_i$.

$$c_i = \Pr(\text{accept } i \mid \text{accepted } 0..i{-}1) = \frac{p_i}{p_{i-1}}, \qquad p_{-1} \equiv 1$$ derived from per_pos counters / step log

Conditional per-position acceptance. The drafter's hit rate at depth $i$ given it has survived so far — the closest thing to "quality of the $i$-th draft token". If $c_i$ were a constant $\alpha$ the outcome would be geometric; the data show $c_i$ falling with $i$.

$$P(a = j) = p_{j-1} - p_j \;\;(j \lt K), \qquad P(a = K) = p_{K-1}$$ run B: histogram of column 3 of spec_log.csv
run A: differences of per_pos

Step outcome distribution. Share of steps that accepted exactly $j$ draft tokens. Recoverable from the per-position counters alone, so it is available for both runs; the two agree to within a fraction of a percent.

$$\sigma_{\text{step}} = \sqrt{\sum_{j=0}^{K} P(a=j)\,\big(j - (\tau-1)\big)^2}$$ run B: std of column 3 of spec_log.csv

Step-level standard deviation. Dispersion of tokens-per-step around $\tau$, over individual verify steps. It is the natural spread to attach to the step-weighted $\tau$ (not the per-prompt spread), and it governs latency jitter when each step has a fixed cost.

$$P_\alpha(a=j) = (1-\alpha)\alpha^{j} \;(j \lt K), \quad P_\alpha(a=K) = \alpha^K, \qquad \tau(\alpha) = \frac{1-\alpha^{K+1}}{1-\alpha}$$ α solved numerically from τ(α) = observed τ

Constant-α (geometric) reference model. The classic chain model with the same acceptance rate at every position. The "implied α" in Table 1 is the α that reproduces the observed $\tau$; comparing $P_\alpha$ and $\sigma_\alpha$ with the observed distribution shows how far the drafter is from position-independent behaviour.

$$\tau(K') = 1 + \sum_{j=0}^{K} P(a=j)\,\min(j, K'), \qquad K' \le K$$ computed from the run-B outcome distribution

Implied acceptance length at a shorter block. Truncating each observed outcome at $K'$ approximates what $\tau$ would be with fewer speculative tokens, assuming the drafter's per-position behaviour is unchanged — reasonable for a block drafter, not for an autoregressive one whose later tokens depend on earlier draft context.

$$\operatorname{Var}(\tau_r) = \underbrace{\operatorname{Var}_q\big(\bar\tau_q\big)}_{\text{between prompts}} + \underbrace{\mathbb{E}_q\big[\operatorname{Var}(\tau_r \mid q)\big]}_{\text{within prompt}}$$ run B: 10 copies per prompt q; population variances

Variance decomposition (best-of-10 run). Splits the spread of per-response $\tau_r$ into prompt difficulty (between) and sampling noise at $T = 1.0$ (within). The "mean of max / min over copies" columns are the expected best and worst of ten draws for the same prompt.

2. Acceptance length

Row markers: math code chat

dataset	family	steps (run B)	τ run A	τ run B	step std	per-response mean	p10 / p50 / p90	implied α
gsm8k	math	191,119	4.97	4.97	2.37	5.03	4.37 / 5.03 / 5.69	0.858
math500	math	499,668	5.09	5.09	2.44	5.21	4.38 / 5.25 / 5.97	0.865
aime25	math	92,076	4.41	4.45	2.43	4.57	3.68 / 4.65 / 5.38	0.822
humaneval	code	46,177	5.36	5.29	2.45	5.30	4.29 / 5.36 / 6.22	0.877
mbpp	code	398,354	4.88	4.90	2.38	4.93	4.46 / 4.93 / 5.40	0.853
livecodebench	code	494,400	4.14	4.12	2.39	5.07	3.76 / 5.09 / 6.29	0.796
mt-bench	chat	163,220	3.11	3.15	2.06	3.70	2.33 / 3.42 / 5.59	0.700
alpaca	chat	919,664	2.85	2.84	1.84	3.20	2.30 / 2.85 / 4.81	0.661
arena-hard-v2	chat	1,539,587	3.49	3.51	2.26	3.81	2.29 / 3.93 / 5.28	0.741
pooled		4,344,265	3.83	3.83	2.37	—	—	0.772

Runs A and B agree to within 0.05 on every dataset except the two smallest (aime25, humaneval), so the single-sample run was already converged at dataset level. Math and code sit at $\tau \approx 4.1$–$5.3$, chat at $2.8$–$3.5$; the pooled figure of 3.83 is chat-heavy (alpaca and arena-hard-v2 supply 57% of all steps).

Step-level standard deviation — **Figure 1.** Left: step-weighted $\tau$ with $\pm\sigma_{\text{step}}$. Right: observed $\sigma_{\text{step}}$ against the geometric model's $\sigma_\alpha$ at the same $\tau$. The observed dispersion is 0.1–0.3 tokens lower everywhere; the gap is largest on high-$\tau$ datasets where the geometric model puts 30–40% of its mass on the all-accepted outcome.

Response length vs acceptance length — **Figure 2.** Why $\bar\tau_{\text{resp}}$ and $\tau$ diverge. On livecodebench, $\tau_r$ falls steeply with response length ($r = -0.63$): long responses draft poorly and contribute most of the steps, so the per-response mean (5.07) overstates the step-weighted value (4.12) by almost a full token.

3. Acceptance probability per draft position

Unconditional $p_i$

dataset	p₀	p₁	p₂	p₃	p₄	p₅	p₆
gsm8k	0.912	0.804	0.676	0.558	0.447	0.341	0.231
math500	0.905	0.800	0.684	0.580	0.476	0.375	0.267
aime25	0.864	0.723	0.584	0.467	0.363	0.269	0.183
humaneval	0.906	0.806	0.715	0.626	0.531	0.416	0.287
mbpp	0.902	0.789	0.667	0.556	0.445	0.327	0.214
livecodebench	0.833	0.674	0.527	0.410	0.312	0.221	0.139
mt-bench	0.746	0.518	0.340	0.228	0.153	0.100	0.060
alpaca	0.718	0.469	0.285	0.172	0.103	0.060	0.033
arena-hard-v2	0.773	0.571	0.409	0.296	0.214	0.149	0.095
pooled	0.804	0.622	0.467	0.356	0.267	0.193	0.126

Conditional $c_i = p_i / p_{i-1}$

dataset	c₀	c₁	c₂	c₃	c₄	c₅	c₆	c₀ − c₆
gsm8k	0.912	0.881	0.841	0.826	0.800	0.764	0.677	0.235
math500	0.905	0.884	0.855	0.848	0.822	0.787	0.713	0.191
aime25	0.864	0.836	0.808	0.800	0.776	0.743	0.678	0.186
humaneval	0.906	0.890	0.887	0.876	0.849	0.783	0.689	0.217
mbpp	0.902	0.875	0.845	0.834	0.799	0.735	0.654	0.248
livecodebench	0.833	0.809	0.782	0.778	0.760	0.709	0.626	0.206
mt-bench	0.746	0.694	0.656	0.671	0.670	0.652	0.605	0.141
alpaca	0.718	0.653	0.607	0.605	0.598	0.578	0.552	0.166
arena-hard-v2	0.773	0.739	0.715	0.725	0.721	0.696	0.636	0.136
pooled	0.804	0.774	0.752	0.761	0.752	0.720	0.655	0.149

Per-position acceptance — **Figure 3.** Unconditional (left) and conditional (right) acceptance by draft position. The first position separates the workloads ($c_0$ 0.90–0.91 math/code vs 0.72–0.77 chat). Conditional acceptance then falls by 0.16–0.25 to position 6 on every dataset: smoothly on math/code with the steepest drop at positions 5–6, and front-loaded on chat, which drops sharply over positions 0–2 and then plateaus near 0.60–0.72. The final-position dip is partly structural — steps that end a response cannot accept late positions — so $c_6$ is a lower bound on drafter quality at depth 7.

4. Step outcome distribution

Share of verify steps (%) by number of accepted draft tokens $a_s$; the last column is the fraction of steps that hit the $K{+}1 = 8$ ceiling.

dataset	0	1	2	3	4	5	6	7
gsm8k	8.8	10.8	12.8	11.8	11.1	10.5	11.0	23.1
math500	9.5	10.5	11.6	10.4	10.3	10.1	10.8	26.7
aime25	13.6	14.1	13.9	11.7	10.5	9.3	8.7	18.3
humaneval	9.4	10.0	9.1	8.9	9.5	11.5	13.0	28.7
mbpp	9.8	11.3	12.2	11.1	11.2	11.8	11.3	21.4
livecodebench	16.7	15.9	14.7	11.7	9.8	9.1	8.3	13.9
mt-bench	25.4	22.8	17.8	11.2	7.5	5.3	3.9	6.0
alpaca	28.2	24.9	18.4	11.3	6.9	4.4	2.7	3.3
arena-hard-v2	22.7	20.1	16.3	11.2	8.3	6.5	5.4	9.5
pooled	19.6	18.2	15.4	11.2	8.8	7.5	6.6	12.6

Outcome distribution heatmap — **Figure 4.** The same table as a heatmap. Math/code distributions are roughly flat with a spike at 7; chat distributions are front-loaded with 23–28% of steps accepting nothing.

Geometric model check — **Figure 5.** Observed $P(a=j)$ against the constant-α model matched on $\tau$. The observed distribution is flatter: fewer complete rejections (pooled 19.6% vs 22.8% predicted), fewer complete acceptances (12.6% vs 16.3%), more mass in between — the signature of conditional acceptance that decays with depth. Chain-cap arithmetic $1/(1-\alpha)$ therefore overstates the return from extending $K$.

5. Implied $\tau$ at shorter blocks

dataset	K′=1	K′=2	K′=3	K′=4	K′=5	K′=6	K′=7	gain 5→7
gsm8k	1.91	2.72	3.39	3.95	4.40	4.74	4.97	+0.57
math500	1.90	2.70	3.39	3.97	4.44	4.82	5.09	+0.64
aime25	1.86	2.59	3.17	3.64	4.00	4.27	4.45	+0.45
humaneval	1.91	2.71	3.43	4.05	4.58	5.00	5.29	+0.70
mbpp	1.90	2.69	3.36	3.92	4.36	4.69	4.90	+0.54
livecodebench	1.83	2.51	3.03	3.44	3.76	3.98	4.12	+0.36
mt-bench	1.75	2.26	2.60	2.83	2.99	3.08	3.15	+0.16
alpaca	1.72	2.19	2.47	2.65	2.75	2.81	2.84	+0.09
arena-hard-v2	1.77	2.34	2.75	3.05	3.26	3.41	3.51	+0.24
pooled	1.80	2.43	2.89	3.25	3.52	3.71	3.83	+0.32

Going from $K' = 5$ to $7$ buys +0.5–0.7 tokens per step on math/code but only +0.1–0.25 on chat; the seventh position alone adds 0.23–0.29 on math/code and 0.03–0.10 on chat. In a lap-bound pipeline the whole $\tau$ is what divides the lap count, so the extra depth is worth having on math/code and nearly free of value on chat.

6. Prompt difficulty vs sampling noise

dataset	per-response std	within-prompt share	mean std over 10 copies	mean (max−min) over copies	mean of max	mean of min	corr(τ_r, length)
gsm8k	0.52	49%	0.37	1.16	5.61	4.45	-0.04
math500	0.62	25%	0.31	0.98	5.70	4.71	-0.08
aime25	0.63	22%	0.28	0.91	5.06	4.15	-0.12
humaneval	0.76	43%	0.49	1.51	6.03	4.53	+0.20
mbpp	0.36	49%	0.26	0.81	5.34	4.53	-0.02
livecodebench	0.95	26%	0.48	1.48	5.81	4.33	-0.63
mt-bench	1.27	6%	0.27	0.83	4.12	3.29	-0.21
alpaca	1.04	15%	0.32	0.99	3.72	2.73	-0.16
arena-hard-v2	1.17	5%	0.23	0.71	4.17	3.46	+0.12

Per-response distribution and variance decomposition — **Figure 6.** Left: distribution of per-response $\tau_r$ (violins) for the 10 copies × prompts of each dataset, with the step-weighted $\tau$ marked in black — the black marks are the values in Table 1. Right: share of $\operatorname{Var}(\tau_r)$ that is within-prompt, i.e. sampling noise across the 10 copies at $T = 1.0$. On math/code 22–49% of the per-response spread is sampling noise across copies of the same prompt, so prompt difficulty and noise contribute comparably; on chat only 5–15% is noise — prompt difficulty dominates (85–95%). Even so, ten resamples of one prompt move $\tau_r$ by 0.7–1.5 tokens end to end, so the "mean of max / min" columns describe a lucky or unlucky draw, not an operating point.

7. Upper bound on tokens per second in a 10-stage WAN pipeline

$$T_{\text{lap}} = N \cdot \frac{\text{RTT}}{2}, \qquad \text{tok/s} \le \frac{\tau}{T_{\text{lap}}} = \frac{2\,\tau}{N \cdot \text{RTT}}, \qquad N = 10$$ τ = step-weighted acceptance length (Table 1); RTT = per-hop round-trip latency, one-way = RTT/2

Latency-only bound. One verify step is one lap of the pipeline: the drafted block enters at stage 1, traverses all $N$ stages and the result returns to the drafter, so the lap costs $N$ one-way link traversals, each charged half the hop's RTT. Compute at every stage, drafting time, serialisation and bandwidth are all set to zero, so the numbers are ceilings: whatever a real system achieves must lie below them. The speed-up over an unspeculated pipeline (one token per lap) is exactly $\tau$ and is independent of RTT. At RTT = 25 ms the lap is 125 ms; at 100 ms it is 500 ms.

Single-stream tokens per second, $N = 10$ stages, one-way latency per hop, compute = 0:

dataset	τ	RTT 10 ms	RTT 25 ms	RTT 50 ms	RTT 100 ms	RTT 200 ms	speed-up vs no spec
gsm8k	4.97	99.4	39.8	19.9	9.9	5.0	4.97×
math500	5.09	101.7	40.7	20.3	10.2	5.1	5.09×
aime25	4.45	89.1	35.6	17.8	8.9	4.5	4.45×
humaneval	5.29	105.8	42.3	21.2	10.6	5.3	5.29×
mbpp	4.90	98.0	39.2	19.6	9.8	4.9	4.90×
livecodebench	4.12	82.3	32.9	16.5	8.2	4.1	4.12×
mt-bench	3.15	62.9	25.2	12.6	6.3	3.1	3.15×
alpaca	2.84	56.8	22.7	11.4	5.7	2.8	2.84×
arena-hard-v2	3.51	70.1	28.0	14.0	7.0	3.5	3.51×
pooled	3.83	76.7	30.7	15.3	7.7	3.8	3.83×
no speculation	1.00	20.0	8.0	4.0	2.0	1.0	1.00×

At 10 ms hops (a 50 ms lap) DSpark lifts the ceiling from 20 to 57–106 tok/s depending on workload; at 50 ms hops (a 250 ms lap) it is 11–21 tok/s; at 100 ms hops (a 500 ms lap) 5.7–10.6 tok/s, roughly reading speed on math/code and below it on chat; at 200 ms hops 2.8–5.3 tok/s. Because the bound scales as $1/\text{RTT}$, halving link latency is worth exactly as much as doubling acceptance length — and doubling $\tau$ from ≈5 to ≈10 is beyond what a $K=7$ chain can deliver (ceiling 8), which is the case for trees or wider blocks made in the laps-and-bytes note.

Tokens per second bound vs RTT — **Figure 7.** Left: the $2\tau/(10\cdot\text{RTT})$ ceiling as a function of per-hop RTT for each dataset (log–log; all lines have slope −1 and are separated only by $\tau$), with the unspeculated pipeline dashed. Right: the same ceilings at the five RTT values; dashed lines are the no-speculation ceilings at matching RTT.

8. Reading for the WAN-PP notes

Use $\tau \approx 4.9$–$5.3$ (math/code) and $2.8$–$3.5$ (chat) as the DeepSeek-V4-Pro / DSpark $K=7$ chain figures rather than a single "≈ 5". In the 10-stage one-way-latency bound of §7 that is ≈ 40 vs ≈ 25 tok/s at 25 ms hops and ≈ 20 vs ≈ 12 at 50 ms.
Rejections concentrate at positions 0–2 on chat (60–70% of all rejections), so the lever there is shallow trees or hedged branches, not deeper chains; on math/code deeper chains still pay.
Step-level dispersion is below geometric ($\sigma_{\text{step}} \approx 2.4$ at $\tau \approx 5$), which eases rate-matching an asynchronous drafter to a fixed verify RTT.

Data provenance: request ids in spec_log.csv were assigned to datasets by position (ids are contiguous per dataset in run order, 10 per prompt) and validated against samples.jsonl with a 100% match on per-request draft and accepted counts. Four alpaca requests are absent from the log. Supporting CSVs: tableA_single_sample.csv, tableB_bon10_steps.csv, tableC_bon10_responses.csv, tableD_tau_vs_K_truncation.csv, tableE_step_level_std.csv, tableF_tok_per_s_pp10.csv, bon10_per_response.csv.