DSpark Parameter Ledger

Architecture

A semi-autoregressive draft head: hidden states from three target layers are fused once, run through three full MoE transformer blocks, and decoded by a factorized rank-512 Markov head instead of a dense lm_head. The vocabulary embedding is shared with the target model — the drafter stores no vocab tables of its own.

The plain middle block is the one that's easy to lose in scripted counts: mtp.0 is distinctive (it carries main_proj) and mtp.2 is distinctive (it carries the heads), while mtp.1 has no unique tensor patterns. Dropping it understates the drafter by exactly 25,737,249,334 parameters.

Where the parameters live

Routed experts dominate at 98.2%. The lower bar magnifies the remaining 1.80% — 1.397B parameters — so the smaller components are visible at all.

Full breakdown, both checkpoints

Weights are logical parameters and match to the element between the two repos (NVIDIA's card confirms the head was taken from DeepSeek's release unmodified, cast MXFP4 → NVFP4 bit-exact). Scale columns are the quantization metadata elements each format stores alongside them.

NV = nvidia/DeepSeek-V4-Pro-nvfp4-DSpark (drafter in the dedicated shard model-dspark-mtp.safetensors, 13,921 tensors, 41.31 GiB). DS = deepseek-ai/DeepSeek-V4-Pro-DSpark (shards 64–66, 7,009 tensors, 39.10 GiB). Expert weights are stored fp4-packed two-per-byte (U8 / I8, last dim halved) and are unpacked in the weight column.

Placement across the three blocks

The blocks are not uniform: fusion enters at mtp.0, the output heads sit on mtp.2.

Per-block MLA composition: wq_a 1536×7168 · wq_b 65536×1536 · wkv 512×7168 · wo_a 16384×4096 · wo_b 7168×16384 + q/kv norms and attention sink. Per routed expert: 3 × 3072×7168 = 66,060,288.

Tensor-level structure

Every distinct tensor pattern in the drafter, straight from the safetensors headers (block and expert indices collapsed to N). Weight tensors are identical between the two checkpoints; only the expert scale tensors differ. Note that hc_*_scale tensors are learned fp32 hyper-connection scalars — weights, not quantization scales; the scale suffix that marks quantization metadata is the .scale name component.

Expert weights are stored fp4-packed two per byte (I8 in the DeepSeek checkpoint, U8 in NVIDIA's; stored last dim is half the logical 7168 / 3072); the params column is unpacked. 1,152 tensors = 3 blocks × 384 experts.

Shared rows are the fp8 linears' 128 × 128 block scales (weight_block_size: [128, 128], ue8m0), identical in both checkpoints: 76,416 elements. Expert scales differ: DeepSeek's MXFP4 stores one power-of-two E8M0 scale per group of 32 weights; NVIDIA's NVFP4 stores one E4M3 scale per group of 16 plus per-tensor fp32 weight_scale_2 and input_scale. *Tensor counts per checkpoint: 3,528 weight tensors in each; scale tensors are DS 3,481 (of 7,009 total) and NV 10,393 (of 13,921).

Reconciling NVIDIA's “82B”

Quantization bookkeeping, side by side

Non-expert linears in both checkpoints are fp8 (E4M3) with identical 128×128 block scales; only the routed-expert format differs. NVIDIA states the experts were losslessly cast from MXFP4 to NVFP4, bit-exact.

Shard layout

How each checkpoint packages the same model, from model.safetensors.index.json. Both use one target layer per shard; they differ only in where the drafter lives.

The one-drafter-block-per-shard split in the DeepSeek repo (shards 64–66) is why a per-shard count catches a dropped mtp.1 immediately; NVIDIA's single dedicated drafter shard makes the drafter separable from the NVFP4 target without touching the other 63 files.