NVIDIA’s latest technical blog shows how to speed up mixture-of-experts (MoE) inference on one DGX Spark. The test model is Ant Group inclusionAI’s Ling-3.0-flash, a 124B-parameter MoE with 5.1B active parameters. NVIDIA stacked three optimizations: a new MoE kernel backend, an FP8 output layer and built-in speculative decoding. On all 1,319 GSM8K questions, decode throughput rose 42.5% with no statistically detectable accuracy change.
So why is it relevant for Devs? DGX Spark is a desktop system, not a data center node. NVIDIA’s results show a 124B MoE serving single-user requests faster on that desk-side box. The technical post also shows where MoE decode time goes, and how to cut it.
TL;DR
- Size: Ling-3.0-flash-fp4: 124B total, 5.1B active per token; 256K native context.
- Runs on: One NVIDIA DGX Spark (GB10, 128 GB unified memory, 273 GB/s), batch size one.
- Performance: Three stacked optimizations lift GSM8K decode throughput from 37.81 to 53.89 tokens/s.
- Best: +42.5% throughput and -24.5% mean latency, with accuracy flat (96.89% vs 96.82%).
- Lagging: Synthetic gains (+185.1%) shrink on GSM8K as MTP acceptance drops from 3.90 to 2.68.
Why is MoE decoding on DGX Spark memory-bound?
DGX Spark is NVIDIA’s desktop system built on the GB10 Grace Blackwell Superchip. It offers 128 GB of unified memory and 273 GB/s of memory bandwidth. Peak compute is up to 1 petaFLOP of sparse FP4. NVIDIA rates it for inference on models up to 200B parameters.
In the NVIDIA technical post, the NVIDIA team profiled Ling-3.0-flash-fp4 at batch size 1. Decoding was memory-bound: reading model weights drove execution time. Two operations stood out, the MoE computation and the LM head that scores the vocabulary. That points to two levers: move fewer bytes per decode step, and generate more tokens per verification round.
The baseline followed the official W4A8 recipe. It used FlashInfer MXFP4 MoE kernels, CUTLASS dense GEMM and an FP32 LM-head path. On a synthetic 32K-input, 2K-output test, it ran at 27.71 ms per token (36.09 tokens per second).
What are the 3 optimizations?
1. Humming MXFP4 MoE kernels: NVIDIA swapped FlashInfer’s MoE backend for Humming, an independent kernel library. Humming uses NVRTC to compile kernels at runtime for the actual matrix shapes, data types and GPU. That includes the narrow 768-wide expert layer. Each kernel compiles in seconds. The FlashInfer CUTLASS path compiles 97 targets with ninja, on the order of tens of minutes. Throughput rose 9.3% to 39.45 tokens per second.
2. Online FP8 LM head: The LM head is a 157,184 x 2,560 matrix stored in BF16. Every generated token reads all of it. Quantizing only this layer to FP8 at load time halves those reads. Setting SGLANG_ENABLE_FP8_LM_HEAD=1 enables it. It cannot be combined with the baseline –enable-fp32-lm-head option. Combined with Humming, throughput reached 43.94 tokens per second. That is 21.7% over baseline and 11.4% over Humming alone.
3. MTP speculative decoding with NEXTN: Ling-3.0-flash ships with a built-in NEXTN prediction layer, so no extra draft model is needed. It drafts three tokens, and the model verifies four positions at once. Verification costs about 2.26x a normal decode step. The main cost is MoE, not attention. Verifying four tokens can activate up to 32 experts instead of 8. On the synthetic test, an average of 3.90 tokens were accepted per round. Throughput jumped to 102.88 tokens per second, up 185.1%.
The three techniques do not overlap. Humming and the FP8 LM head cut the cost of each step. MTP cuts the number of steps.
The integration trap: Stacking these is not automatic. With FP8 LM head and MTP both on, the draft must share the full LM-head module. Otherwise the framework silently falls back, and the draft interprets FP8 weights as BF16. Accept length drops to 1.00, and throughput falls below the unoptimized baseline. The NVIDIA team advises checking at startup that each optimization is active. NVIDIA points to the Ling cookbook notebook for enabling both together.
What happens on a real workload?
Synthetic tests flatter speculative decoding, so the NVIDIA team also ran all 1,319 GSM8K questions. It used 5-shot prompting with chain-of-thought, temperature 0.6, top_p 0.95 and top_k 29. With all three optimizations, time per output token fell from 26.45 ms to 18.56 ms. Decode throughput rose 42.5%, from 37.81 to 53.89 tokens per second. Mean latency dropped 24.5%, from 15.21 to 11.49 seconds. Accuracy moved from 96.89% to 96.82%, a change NVIDIA calls statistically undetectable.
Accept length here was 2.68, versus 3.90 on synthetic data. The NVIDIA team gives three reasons for the gap. Synthetic outputs are more repetitive, so drafts are accepted more often. The synthetic test used greedy decoding, while GSM8K used sampling. GSM8K answers are also short, with a median near 310 tokens versus a fixed 2K.
What is Ling 3.0, and why does it suit DGX Spark?
Ling-3.0-flash belongs to Ant Group inclusionAI’s Ling 3.0 family, released under the MIT license. The open-weight lineup also includes Ling-3.0-tiny, a 7.9B MoE with 1.3B active parameters. Both come with base checkpoints and FP8, INT4 and GGUF builds; flash also has FP4. OpenRouter also lists inclusionAI’s vision-language, finance and health variants built on Ling 3.0 Flash.
Ling-3.0-flash is built to keep per-token cost low:
- Sparse MoE: Each MoE layer routes a token to 8 of 512 routed experts, along with 1 shared expert. About 4.1% of the weights are active per token.
- Hybrid linear attention: 35 Kimi Delta Attention (KDA) layers and 7 Gated MLA layers alternate 5:1. KDA, from Moonshot AI’s Kimi Linear paper, keeps a fixed-size state instead of a growing KV cache. The layout is native from the start of pretraining.
- Built-in MTP: The model was trained with multi-token prediction. That NEXTN layer is the draft model NVIDIA used.
- Agent focus: inclusionAI team reports 10,000+ interactive training environments for coding, general and deep-research agents. NVIDIA notes the native 256K context expands to 1M tokens.
