1. Training facts per drafter architecture
Every competitive drafter since 2024 is trained on the same thing: prompts from a public instruction corpus with answers regenerated by the frozen target. Progress on data has been about scaling that recipe from 68K ShareGPT conversations (EAGLE-1, Medusa, HASS) to 0.5–1.4M mixed chat/code/math samples (EAGLE-3, DFlash, DSpark, SpecBundle) and, at the top end, 6B regenerated tokens (TorchSpec for Kimi K2.5) or ~96B tokens of the target's own SFT mix (Meta production EAGLE). Hardware has stayed small for the drafter and large for the target cache: a head for an 8B target trains on one 8×A100/H100/H200 node in hours to days, while target hidden-state supervision costs 100+ MB per sample, ≈38 TB for DeepSpec's Qwen3-4B cache and ≈7 GB per 128K-token Kimi sample. That I/O cost, not GPU count, is what SpecForge, Speculators, DeepSpec and TorchSpec are built around.
Sizes marked "computed" are from stated layer shapes; "not stated" means the source gives no count, so check the checkpoint's config.json.
| Drafter | Target model(s) | Drafter size | Training data | Hardware · time | Framework | Source |
|---|---|---|---|---|---|---|
| EAGLE-1 | Vicuna 7B/13B/33B, LLaMA2-Chat 7B/13B/70B | 0.24B (7B), 0.37B (13B), 0.56B (33B), 0.99B (70B): one decoder layer | ShareGPT, ≤70K dialogues (2–4B tokens), fixed text | 4× A100-40G, 1–2 days (70B); RTX 3090 node for 7–33B | EAGLE repo | arXiv 2401.15077 · repo |
| Medusa-1 | Vicuna-7B, Vicuna-13B | 5 heads × (d×d + d×V) ≈ 0.74B for 7B (computed) | ShareGPT 60K, 2 epochs | 1× A100 PCIe, 5 h (Vicuna-7B) | axolotl fork | arXiv 2401.10774 §2.2.1 |
| Medusa-2 self-distil | Vicuna-33B, Zephyr-7B | same heads + LoRA r=32 on backbone | ~100K target-generated (ShareGPT + UltraChat seeds, T=0.3) | seq 2048, batch 128 | axolotl fork | arXiv 2401.10774 §3.2 |
| HASS | LLaMA2-Chat 7B/13B, LLaMA3-Instruct 8B/70B; Qwen3-32B (reproduction) | EAGLE-sized, one layer | ShareGPT 68K, target-regenerated | 8× A100-80G, 3–4 days at 32B | HASS repo | arXiv 2408.15766 · TriSpec App. B |
| EAGLE-3 (paper) | Vicuna-13B, LLaMA-3.1-8B-Instruct, LLaMA-3.3-70B-Instruct, DeepSeek-R1-Distill-LLaMA-8B | one decoder layer + 3k→k fusion FC; size not stated | ShareGPT 68K + UltraChat-200K 464K, regenerated; + OpenThoughts-114k-math for R1-distill | not stated; AdamW lr 5e-5 | EAGLE repo | arXiv 2503.01840 §3–4 |
| EAGLE-3 (reproduction) | Qwen3-32B | 1.35B trainable | 518K samples, seq 2048, 6 epochs | 8× A100-80G, 78 h | EAGLE/HASS code | TriSpec Table 6 |
| EAGLE-3 (SpecForge recipe) | Llama/Qwen3 8–9B; Qwen3.5; gpt-oss-20B/120B | not stated | Open-PerfectBlend 1.4M regenerated, 2 ep; Nemotron 1.4M, 1 ep, 8K seq | ~48 H200-h (8–9B); 36–48 H200-h (gpt-oss) | SpecForge | arXiv 2605.09992 App. C |
| SpecForge framework | Qwen3-235B-A22B (benchmark) | not stated | ShareGPT+UltraChat 320K vs Perfect-Blend 1.4M | 8× H200; up to 9.9× faster than original code | SpecForge | arXiv 2603.18567 · repo |
| EAGLE-3 (Speculators online) | Qwen3-8B | not stated | ShareGPT / UltraChat presets, 5K-sample tutorial | 4× H100 (2 vLLM + 2 train), ~17 min | Speculators | vLLM docs |
| EAGLE-3 (TorchSpec) | Kimi K2.5, 1T MoE (hidden 7168, vocab 163,840) | not stated | 600K samples, 6B tokens, ≤128K context | 8× H200 inference + 8× H200 train, 1,500 H200-h | TorchSpec | PyTorch blog · HF model |
| EAGLE (Meta production) | Llama 3.1 8B, Llama 3.3 70B, Llama 4 Scout (109B/17B), Llama 4 Maverick (400B/17B) | 3 dense decoder layers; size not stated | Targets' own SFT set, 48K iters × 2M tokens | internal, online distillation; lr 2e-4 | Meta internal | arXiv 2508.08192 §2 |
| EAGLE-3 (NeMo recipe) | 8B-class (bring your own) | not stated | JSONL, assistant turns from target | 1K samples/10 min on 1× A100; 200K/~2 h on 8× A100 | NeMo AutoModel | NeMo docs |
| DFlash (paper) | Qwen3-4B, Qwen3-8B, Qwen3-Coder-30B-A3B, LLaMA-3.1-8B-Instruct; Qwen3.5 4B–35B, GPT-OSS 20B/120B | 5 draft layers (8 for Coder), block 16 (10 for LLaMA); total not stated | Nemotron-PT-v2 + CodeAlpaca ~800K regenerated; 6 ep, seq 3072/4096, 512 anchors | not stated; H200/B200 eval | z-lab / SpecForge | arXiv 2602.06036 §5, App. A |
| DFlash (reproduction) | Meta-Llama-3-8B-Instruct | 4 Llama-3-width layers ≈ 0.9B (computed) | ShareGPT, seq 4096, 3 epochs | 8× H100-80G, eff. batch 32 | SpecForge | arXiv 2606.11552 §5.1 |
| DFlash (NVIDIA) | MiniMax-M2.7 | not stated | Nemotron-PT-v2 prompts, 469,568 samples, <1B tokens, target-synthesised | H100 (test) | Model Optimizer | HF card |
| DFlash (Red Hat) | Mistral-Small-4-119B-2603 | 5 draft layers, sliding window 2048 | Magpie-300K + UltraChat-200k regenerated; seq 8192 | 4 GPUs vLLM + trainer | Speculators | HF card |
| DFlare | Qwen3-class | deeper than DFlash; size not stated | DFlash mix scaled 800K → 2.4M | not stated | AngelSlim | AngelSlim docs |
| DSpark (DeepSpec) | Qwen3-4B/8B/14B, Gemma-4-12B-it; production DeepSeek-V4-Flash/Pro | block 7; size not stated | Open-PerfectBlend regenerated, non-thinking | 1 node × 8 GPUs; cache ≈38 TB (Qwen3-4B) | DeepSpec | DeepSpec README · DSpark paper |
| DSpark (community) | Qwen3-4B | not stated | 1,339,649 conversations, ≤4,090 cached tokens | 38× A800, 18K steps | DeepSpec | HF card |
| DSpark (NVIDIA Nemotron-3.5) | Nemotron-3.5-Lightning-30B-A3B | not stated | Nemotron-PT-v2 + v3 prompts, 66B tokens, 2 epochs, target-synthesised | not stated | Model Optimizer | HF card |
| FastDraft | Phi-3-mini, Llama-3.1-8B-Instruct | 50M / 150M independent LM | ~10B pretrain tokens + target-synthesised alignment | 8× Gaudi 2, <24 h | Intel | arXiv 2411.11055 |
| LK losses | EAGLE-3 / Medusa / MLP heads | as base heads | Infinity-Instruct 660K, target completions | not stated | SpecForge PR #492 | arXiv 2602.23881 |
| TIDE (timing only) | per-target table in paper | — | SpecForge offline 6.16 h prefill + 9.16 h train; online 18.48 h + 9.16 h | — | SpecForge | arXiv 2602.05145 |
2. Published checkpoints: training disclosure audit
Of ~35 public drafter checkpoints audited, only four releases disclose enough to reproduce the run (data, sample count, hardware, wall-clock and recipe): DeepSeek's DeepSpec checkpoints, the community DSpark re-run, TorchSpec's Kimi-K2.5 head and the original EAGLE-1 weights. The vendor zoos (NVIDIA, Red Hat, LMSYS SpecBundle, z-lab, Nebius) name the corpus and usually the sample count but almost never the GPUs or hours; the frontier-target heads (DeepSeek-V4 DSpark, GLM-5.2/5.3 DFlash and DSpark, AQ-MedAI, incoai) state nothing about training at all, and the most-downloaded head, yuhuili's EAGLE-3 for Llama-3.1-8B, ships a 31-byte README.
Disclosure = D data corpus · N sample count · H hardware · T wall-clock · R recipe (✓ stated, ✗ absent, ~ partial, * from the paper rather than the card). "—" means the card or linked source states nothing.
| Checkpoint | Target | Drafter · size | Data · N · regenerated? | HW · time | Framework | Card eval | Disclosure | Source |
|---|---|---|---|---|---|---|---|---|
| yuhuili/EAGLE-* (Vicuna, LLaMA2-Chat, LLaMA3) | Vicuna 7/13/33B, LLaMA2-Chat 7/13/70B, LLaMA3-Instruct 8/70B | EAGLE-1 · 0.24/0.37/0.56/0.99B | ShareGPT · ≤70K · no | 4× A100-40G · 1–2 days (70B) | EAGLE repo | MT-bench 2.7–3.5× (paper) | D✓ N✓ H✓ T✓ R✓ | paper |
| yuhuili/EAGLE3-LLaMA3.1-Instruct-8B (+ Vicuna-13B, LLaMA3.3-70B, DSL-8B) | Llama-3.1-8B-Instruct | EAGLE-3 · ~0.4B (850 MB fp16) | ShareGPT + UltraChat-200K · ~532K · yes (paper; card README is 31 bytes) | — · — | EAGLE repo | τ 6.13 MT-bench, 4.40× (paper) | D✓ N✓ H✗ T✗ R✓* | paper |
| FasterDecoding Medusa heads | Vicuna-7B/13B/33B, Zephyr-7B | Medusa-1/2 · ≈0.74B (7B, computed) | ShareGPT · 60K, 2 ep · no; 33B/Zephyr ~100K · yes | 1× A100 · 5 h (Medusa-1 7B) | axolotl | 2.18–2.83× MT-bench | D✓ N✓ H✓ T✓ R✓ | paper |
| HASS heads | LLaMA2-Chat 7/13B, LLaMA3-Instruct 8/70B | HASS · EAGLE-sized | ShareGPT · 68K · yes | — · — (8× A100, 3–4 days in TriSpec re-run) | HASS repo | 2.81–4.05× | D✓ N✓ H✗ T✗ R✓ | paper |
| lmsys SpecBundle collection (Llama-3.1-8B, 3.3-70B, Llama-4 Scout/Maverick, Qwen3-30B-A3B, Qwen3-Next-80B, Qwen3-Coder-30B/480B, Kimi-K2, gpt-oss-20b/120b, Nex-N1) | as listed | EAGLE-3 · — | Open-PerfectBlend · 1.4M, 2 ep · yes (regenerated sets published; Maverick on ShareGPT+UltraChat) | — · — | SpecForge | Qwen3-235B: GSM8K 3.54, MT-Bench 3.02 | D✓ N✓ H✗ T✗ R✓ | Meituan card · list |
| lmsys/Qwen3-235B-A22B-EAGLE3 (pre-SpecBundle) | Qwen3-235B-A22B | EAGLE-3 · — | UltraChat + ShareGPT · — · — | — · — | SpecForge | GSM8K acc 0.965 (correctness only) | D✓ N✗ H✗ T✗ R✗ | card |
| nvidia/gpt-oss-120b-Eagle3 (-v2, -v3, short/long-context, throughput) | gpt-oss-120b | EAGLE-3 · — | synthesised by gpt-oss-120b; v3 = 8 Nemotron-PT-v3 subsets · — · yes | — · — (test B200) | Model Optimizer | MT-Bench AL; v3 AL@7 math 3.50, code 3.28 | D✓ N✗ H✗ T✗ R✗ | card |
| nvidia/Llama-3.3-70B-Instruct-Eagle3 | Llama-3.3-70B-Instruct | EAGLE-3 · — | Nemotron post-training mix · — · yes | — · — | Model Optimizer | MT-Bench AL | D✓ N✗ H✗ T✗ R✗ | project notes |
| nvidia/MiniMax-M2.7-DFlash | MiniMax-M2.7 | DFlash · — | Nemotron-PT-v2 prompts · 469,568 (<1B tokens) · yes | — · — (test H100) | Model Optimizer | specdec_bench, greedy, 4K gen | D✓ N✓ H✗ T✗ R✗ | card |
| nvidia/Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark | Nemotron-3.5-Lightning-30B-A3B | DSpark · — | Nemotron-PT-v2 + v3 prompts · 66B tokens, 2 ep · yes | — · — (test H100, GB200, RTX 5090, DGX Spark) | Model Optimizer | AL math 4.17, multilingual 4.55, stem 3.40 | D✓ N✓ H✗ T✗ R✗ | card |
| RedHatAI/gpt-oss-120b-speculator.eagle3 (+ gpt-oss-20b, Qwen3-30B-A3B, Qwen3-32B) | gpt-oss-120b | EAGLE-3 · — | Magpie 300K + UltraChat 200K · 500K · yes | 8× A100 or 8× H100 · — | Speculators | HumanEval 3.01, math 3.29, MT-bench 2.61 | D✓ N✓ H✓ T✗ R✓ | Red Hat study |
| RedHatAI/Mistral-Small-4-119B-2603.dflash | Mistral-Small-4-119B | DFlash · 5 layers, SW 2048 | Magpie-300K subset + UltraChat train_sft · — · yes, seq 8192 | 4 GPUs vLLM + trainer · — | Speculators (commands on card) | speedup vs Eagle3 plot, H100 | D✓ N✗ H~ T✗ R✓ | card |
| RedHatAI/GLM-5.2-speculator.dspark | GLM-5.2 (NVFP4) | DSpark · 3B | — · — · — | — · — | Speculators | AL 5.27 math / 3.78 HumanEval / 3.14 QA | D✗ N✗ H✗ T✗ R✗ | project notes |
| RedHatAI/DeepSeek-V4-Flash-speculator.dflash | DeepSeek-V4-Flash | DFlash · 2B | — · — · — | — · — | Speculators | AL ≈3.9 | D✗ N✗ H✗ T✗ R✗ | project notes |
| z-lab/Qwen3-8B-DFlash-b16 (+ Qwen3-4B, gpt-oss-20b, Qwen3.5/3.6, gemma-4-31B, Kimi-K2.5, MiniMax-M2.5, Qwen3.5-122B) | as listed | DFlash · 5 layers, block 16 · — | Nemotron-PT-v2 + CodeAlpaca · ~800K, 6 ep · yes (paper) | — · — | z-lab / SpecForge | >6× Qwen3-8B (paper) | D✓ N✓ H✗ T✗ R✓* | paper |
| z-lab/Qwen3-Coder-30B-A3B-DFlash | Qwen3-Coder-30B-A3B | DFlash · 8 layers · — | Nemotron-PT-v2 code + evol-codealpaca + Cline traces · 289K · yes | — · — | z-lab | HumanEval 3.5×, AL 8.09 | D✓ N✓ H✗ T✗ R✗ | card |
| z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat | Llama-3.1-8B-Instruct | DFlash · block 10 · — | UltraChat + ShareGPT · — · yes | — · — | z-lab | 2.2–2.8× vs EAGLE-3 1.5–2.0× (SGLang, B200) | D✓ N✗ H✗ T✗ R✗ | paper §5.5.1 |
| deepseek-ai DeepSpec checkpoints (dspark / dflash / eagle3 × Qwen3-4B, 8B, 14B, Gemma-4-12B) | as listed | DSpark b7, DFlash b7, EAGLE-3 TTT 7 · — | Open-PerfectBlend · 1.34M · yes, non-thinking | 1 node × 8 GPUs · — (cache ≈38 TB) | DeepSpec (config/) | 9 benchmarks, one harness | D✓ N✓ H✓ T✗ R✓ | README |
| alice1001/dspark_qwen3_4b_block7 | Qwen3-4B | DSpark b7 · — | Open-PerfectBlend regen · 1,339,649 · yes (dataset linked) | 38× A800 · step 18,000, loss 0.5449 | DeepSpec | accept_ratio@0 + 4-GPU eval | D✓ N✓ H✓ T~ R✓ | card |
| deepseek-ai/DeepSeek-V4-Pro-DSpark, V4-Flash-DSpark, V4-Pro-0813 | DeepSeek-V4-Pro 1.6T-A49B / V4-Flash | DSpark · ~77B sparse (community count) | — · — · — | — · — | DeepSeek internal | +57–78% per-user vs MTP-1 (paper) | D✗ N✗ H✗ T✗ R✗ | DSpark paper |
| lightseekorg/kimi-k2.5-eagle3 | Kimi K2.5 (1T MoE) | EAGLE-3 · — | kimi-mtp-dataset · 600K, 6B tokens, ≤128K ctx · yes (released) | 8× H200 infer + 8× H200 train · 1,500 H200-h | TorchSpec | +60/+30/+26% throughput at bs 1/8/16 | D✓ N✓ H✓ T✓ R✓ | PyTorch blog |
| lightseekorg/kimi-k2.6-eagle3.1-mla | Kimi K2.6 (NVFP4) | EAGLE-3.1 · — | — · — · — | — · — | TorchSpec | 2.03× per-user, GB200 TP4 | D✗ N✗ H✗ T✗ R✗ | vLLM blog |
| nebius/EAGLE3-* (Llama-3.1-8B, 3.3-70B, Qwen3-235B-2507, gpt-oss-20b/120b) + MLP, Medusa | as listed | EAGLE-3 / MLP / Medusa · — | Infinity-Instruct-0625 · 660K · yes (per-target sets released) | — · — | SpecForge PR #492 | +8–10% AL vs CE | D✓ N✓ H✗ T✗ R✓ | org · paper |
| AngelSlim EAGLE3 collection (15 heads: Qwen3 1.7B–235B, Hunyuan, Qwen3-VL, HunyuanOCR, Qwen2-Audio, CosyVoice3) | as listed | EAGLE-3 · — | — · — · — | — · — | AngelSlim | AL 1.86–2.2 at k=2 | D✗ N✗ H✗ T✗ R✗ | project notes |
| UCloud-org/GLM-5.2-FP8-DFlash | GLM-5.2 FP8 | DFlash · 3.7B | Nemotron + CodeAlpaca · — · — | — · — | SpecForge | MBPP τ 8.04 / 4.29×; MT-Bench 0.93× | D✓ N✗ H✗ T✗ R✗ | project notes |
| SubconsciousDev/glm-5.2-fp8-dflash-v1/-v2 | GLM-5.2 FP8 | DFlash · 2B | — · — · — | — · — | — | MBPP 4.9× / τ 9.03 | D✗ N✗ H✗ T✗ R✗ | project notes |
| siro1/glm-5.2-dspark-spec-v1 | GLM-5.2 | DSpark · 4B | — · — · — | — · — | Speculators fork | val τ 4.58 | D✗ N✗ H✗ T✗ R✗ | project notes |
| incoai/GLM-5.3-DFlash2, GLM-5.3-Flash-DFlash2 | GLM-5.3 / 5.3-Flash | DFlash2 · 2B / 1B | — · — · — | — · — | — (CC BY-NC-ND) | GSM8K τ 5.94 / 3.23× (4× GB300) | D✗ N✗ H✗ T✗ R✗ | project notes |
| AQ-MedAI GLM-5.1-eagle3, V4-Flash-0731-eagle3 | GLM-5.1, V4-Flash | EAGLE-3 · 0.7B | — · — · — | — · — | SpecForge | τ 2.6–3.2 | D✗ N✗ H✗ T✗ R✗ | project notes |
| thoughtworks/MiniMax-M2.5-Eagle3 | MiniMax-M2.5 | EAGLE-3 · — | — · — · — | — · — | — | matched native MTP only | D✗ N✗ H✗ T✗ R✗ | blog |
| OpenVINO FastDraft collection | Phi-3-mini, Llama-3.1-8B, Phi-4-mini | independent LM · 50M / 150M / 120M | ~10B pretrain tokens + target-synth alignment · — · yes | 8× Gaudi 2 · <24 h | Intel | code 3×, text 2× | D✓ N~ H✓ T✓ R✓ | paper |
| apple/ml-recurrent-drafter | Vicuna-class | ReDrafter · — | ShareGPT · — · — | 8× H100 · ~1.5 h | Apple repo | — | D✓ N✗ H✓ T✓ R✓ | project notes |
| facebook/layerskip-* | Llama 2/3, Code Llama | self-speculative (no drafter) | pretraining/fine-tuning recipe · — · n/a | — · — | LayerSkip | 1.8–2.06× | D~ N✗ H✗ T✗ R✓ | project notes |
| Syzygy / ManiacLabs V4-Flash EAGLE3.1 | DeepSeek-V4-Flash | EAGLE-3.1 · 0.4B | — · — · — | — · — | — | 2.63× on 4× B200 | D✗ N✗ H✗ T✗ R✗ | project notes |
Findings. Hardware and wall-clock are the missing fields (27 of 34 rows give no GPUs, 30 no time). Data is named but counts rarely are, and regeneration settings (thinking mode, temperature, target precision) are stated only by DeepSpec and Medusa even though they move acceptance by 15–25%. Every frontier-target head (DeepSeek-V4, GLM-5.x, Kimi-K2.6, MiniMax) is a black box. Evaluations are not comparable across vendors: acceptance length at k=2, 7 or 16, tokens-per-call at chain 3, and speedups on different engines and GPUs. Reproducible templates: DeepSpec, alice1001, TorchSpec, Nebius, SpecBundle. Licences gate reuse (incoai is CC BY-NC-ND; SubconsciousDev states none).
3. Datasets used for drafter training
Drafter training draws on about a dozen public instruction corpora, and in every modern recipe only the prompts are used: responses are regenerated by the target before training. Three mixes account for most released heads: ShareGPT + UltraChat (~0.5M chat), Open-PerfectBlend (1.42M, math-leaning; SpecBundle, DeepSpec) and Nemotron-Post-Training v2/v3 (~1.4M+ with code, math, STEM and chat splits; DFlash, NVIDIA, newer SpecBundle heads). Several groups publish regenerated copies for specific targets, which removes the costliest step for anyone training against the same target.
Sizes are the dataset card's counts where the card was opened; "≈" marks counts from memory or from the drafter paper.
| Dataset | Size | Categories | Used by (drafters) | Regenerated copies published | Notes |
|---|---|---|---|---|---|
| ShareGPT (Vicuna unfiltered) | ≈90K conversations; 68K after EAGLE/HASS filtering | multi-turn chat (ChatGPT logs, mixed topics) | EAGLE-1/2/3, Medusa, HASS, Apple ReDrafter, z-lab LLaMA-3.1 DFlash, SpecBundle Llama-4 Maverick, TriSpec | none official; EAGLE repo regenerates on the fly | Oldest and smallest; no code/math split; EAGLE-1 trained on the fixed text |
| UltraChat-200k | train_sft ≈208K conversations (EAGLE-3 counts 464K entries after turn splitting) | multi-turn chat: questions about the world, writing/creation, assistance on existing material | EAGLE-3, Medusa self-distil seeds, BentoML, Baseten generic recipe, Red Hat (200K slice), z-lab DFlash ablation, Snowflake MLP speculators | Red Hat regenerates per target inside Speculators | Default chat preset in Speculators and SpecForge |
| Open-PerfectBlend | 1,420,909 conversations (dedup removed 88.1K) | ultrachat_200k (chat), MetaMathQA + orca-math-200k (math), UltraInteract_sft (reasoning/code/math), ultrafeedback_binarized (chat), evol-codealpaca-v1 (code), AutoIF-instruct-61k (instruction following), lmsys-arena-55k (chat) | SpecBundle, DeepSpec (all 12 checkpoints), alice1001 DSpark, Attention-Drift EAGLE-3 runs, SpecForge paper | lmsys per-target regenerated sets; alice1001 Qwen3-4B regen | "Slight focus on math"; +23% Math500 τ over ShareGPT+UltraChat from the mix alone |
| Nemotron-Post-Training-Dataset-v2 | SFT: math 239,467 · code 175,000 · stem 355,000 · chat 627,720 (≈1.40M); plus RL data | math, code, STEM, chat; es/fr/de/it/ja; reasoning-on/off response pairs | DFlash (~800K with CodeAlpaca), z-lab Coder DFlash, NVIDIA MiniMax-M2.7 DFlash (469,568 prompts), UCloud GLM-5.2 DFlash, NeMo DFlash recipe, NVIDIA Nemotron-3.5 DSpark | NVIDIA regenerates per target, not published | CC BY 4.0; filterable by category, language, source model |
| Nemotron-Post-Training-Dataset-v3 | ≈ collection of subsets; counts on cards (not opened) | science, instruction following, code, math, SWE, multilingual, agentic/tool use, structured output | NVIDIA gpt-oss-120b-Eagle3-v3, Llama-3.3-70B Eagle3, Nemotron-3.5 DSpark, newer SpecBundle code heads | not published | Adds agentic and SWE splits that v2 lacks |
| Magpie-Llama-3.1-Pro-300K-Filtered | 300K conversations (filtered from 1M) | synthetic single-turn instructions from Llama-3.1-70B: general chat, reasoning, some code/math | Red Hat Speculators heads, RedHatAI Mistral-Small-4 DFlash, Speculators magpie preset |
Red Hat regenerates per target, not published | Responses are Llama-generated and off-policy for other targets |
| Infinity-Instruct (0625) | 660K prompts used by Nebius (full release ≈7M) | broad: chat, math, code, reasoning, multilingual | Nebius EAGLE3 / MLP / Medusa heads, LK-losses paper | nebius/*-Infinity-Instruct-0625 for Llama-3.1-8B, Llama-3.3-70B, Qwen3-235B-2507, gpt-oss-20b/120b | Largest set of ready-made regenerated corpora |
| kimi-mtp-dataset | 600K samples, 6B tokens, up to 128K context | mixed chat/code/reasoning, long context | TorchSpec Kimi-K2.5 EAGLE-3 | this is the regenerated set (Kimi K2.5) | Only public long-context drafter corpus |
| CodeAlpaca-20k / evol-codealpaca-v1 | ≈20K / ≈111K | code instructions | DFlash mix, z-lab Coder DFlash, Open-PerfectBlend | via the mixes above | Small; used to raise code share |
| OpenThoughts-114k (math) | ≈114K total; math subset used | math and reasoning with long traces | EAGLE-3 head for DeepSeek-R1-Distill-LLaMA-8B | not published | Explains why that head peaks on GSM8K |
| Cline agent traces (z-lab) | ≈2.8K | agentic coding sessions | z-lab Qwen3-Coder DFlash | not published | Only agentic data in any public drafter mix |
| LongAlign-10K | 10K; DFlash uses 1.6K | long-context QA and summarisation (8K–64K) | DFlash long-context fine-tune (Qwen3.5-27B) | not published | 3 epochs on 1.6K samples recovers acceptance at 16–32K |
| Targets' own SFT set (Meta) | ≈96B tokens over 48K steps | Llama 3/4 post-training mix | Meta production EAGLE | private | Ceiling of "match the target's data" |
| Production prompt logs (Together, Fireworks, Baseten, Nebius, Modal) | 100–300K prompts; Together 20–50M tokens | customer workload (JSON, SQL, code, résumé screening, doc extraction) | custom vendor drafters | private | 1.2–1.5× over a generic head; synthetic prompts recover 80–93% of the benefit |
| RedHatAI/speculator_benchmarks | eval only | math reasoning, coding, summarisation, RAG | Red Hat evaluation of all heads | — | Evaluation set, not training |
Coverage. Chat is over-served; math well covered; code adequate; agentic/tool-calling exists only in Nemotron-v3 and z-lab's 2.8K Cline traces; summarisation, RAG and long-context inputs have no training corpus beyond kimi-mtp-dataset and LongAlign. ShareGPT-trained heads are English-only; Nemotron-v2 adds five European/Japanese languages; no public corpus covers low-resource languages. For a new target, budget one full target pass over 1–1.4M prompts unless a regenerated copy exists (Llama-3.1-8B, Llama-3.3-70B, Llama-4 Scout, Qwen3-30B/235B/Next, Qwen3-Coder-30B, gpt-oss-20b/120b, Kimi-K2.5).
Sources
Primary pages opened for this part: EAGLE-1 (2401.15077), EAGLE-3 (2503.01840), Medusa (2401.10774), HASS (2408.15766), TriSpec (2601.23180), DFlash (2602.06036), Spec-AUF (2606.11552), SpecForge (2603.18567), Attention Drift (2605.09992), TIDE (2602.05145), Meta production EAGLE (2508.08192), FastDraft (2411.11055), LK losses (2602.23881), DSpark (2607.05147); DeepSpec README and PR #9; model cards from yuhuili, lmsys, nvidia, RedHatAI, z-lab, alice1001, lightseekorg, Nebius; dataset cards for Open-PerfectBlend, Nemotron-PT-v2, Magpie; Speculators, NeMo AutoModel and TorchSpec docs. Rows marked "project notes" come from the GLM-5.2/V4-Pro and GLM-5.3 drafter catalogs and the domain-adapted drafters survey in this collection. Companion Claude docs: Speculative drafters: training data, hardware and frameworks — a review, Published speculative-drafter checkpoints: training disclosure audit, Datasets for speculative-drafter training.