NVIDIA has released Nemotron-Labs-Sigma, a collection of 14 open checkpoints at 3B and 8B scale. Nemotron-Labs-Sigma is a continuous diffusion language model family that denoises text in a 16-D embedding space. Most diffusion LLMs, such as LLaDA and Dream, work on discrete tokens. Sigma works in a smooth latent space instead. NVIDIA team states that this makes few-step distillation and trajectory steering practical for text.
TL;DR
- Size: 3B and 8B parameters, 14 checkpoints. Context is 4,096 tokens (8,192 for Sigma-8B-instruct).
- Runs on: 1 NVIDIA GPU in bfloat16 (Ampere, Lovelace, Hopper, Blackwell). About 120 to 140 tokens/s on 1 H100. No CPU support, no quantized variants listed.
- Performance: Sigma-8B base averages 71.16 on NVIDIA’s 6-task suite, ahead of NLD-8B (70.22).
- Best: 91.96 on GSM8K (Sigma-8B-instruct, 256 DDPM steps).
- Worst: 17.00 on AIME25 (same model and sampler).
- Bottom line: Best: the 8-step distilled 8B keeps a 70.34 average. Worst: the instruct average (59.14) trails NLD-8B (59.92).
What is Nemotron-Labs-Sigma?
The research team calls Sigma the first large-scale family of continuous diffusion language models. The collection has 4 variant types at each size:
- Base: Sigma-3B and Sigma-8B.
- Instruct: Sigma-3B-instruct and Sigma-8B-instruct.
- Few-step distilled: distill-3B and distill-8B at NFE 2, 4, 8 and 16 (8 checkpoints).
- Self-speculative: distill-spec-3B and distill-spec-8B.
How does continuous diffusion for language work?
Each token maps to a point in a learned 16-D embedding space. Training adds Gaussian noise to these embeddings and teaches the model to remove it. The embedding geometry is learned jointly with the denoiser.
Generation runs blockwise. Each denoised block is committed to a KV cache before the next starts. Blocks default to 64 tokens.
The backbone is a Ministral-style decoder-only Transformer with grouped-query attention, SwiGLU and rotary embeddings. Training starts from an autoregressive warm start of about 1T tokens. About 0.3T tokens of diffusion training follow.
At inference, classifier-free guidance and score temperature matter most. The model card page recommends guidance 3 and temperature 0.3 at 32 DDIM steps.
Why does a continuous space matter?
Discrete diffusion jumps between token states, so its trajectories are hard to steer. Sigma follows smooth ODE/SDE paths instead. Per the research paper, these degrade gracefully at low step counts and distill efficiently. Embedding-space steering also tunes the quality-diversity trade-off for pass@k sampling.
How does Sigma perform on benchmarks?
On the research’s 6-task base suite, Sigma-8B with 256 DDPM steps averages 71.16. NLD-8B in diffusion mode scores 70.22, Dream-7B 58.74 and LLaDA-8B 41.78. Qwen3-8B in AR mode scores 68.91. A cheaper 32-step DDIM sampler still gives Sigma-8B a 70.18 average.
Coding is the standout. Sigma-8B scores 65.12 on HumanEval versus 62.80 for NLD-8B. MBPP is weaker, at 65.74 versus 67.00.
After SFT, Sigma-8B-instruct scores 91.96 on GSM8K, 78.40 on MATH-500 and 82.50 on HumanEval. Hard reasoning stays limited, at 23.33 on AIME24 and 17.00 on AIME25. Its 10-task average of 59.14 sits just below NLD-8B at 59.92. All scores are NVIDIA’s own evaluations.


