NVIDIA has released Nemotron-Labs-Sigma, its first large-scale continuous-diffusion language model family, in 3-billion and 8-billion parameter versions. Instead of writing text one token at a time, the models denoise whole blocks of tokens in parallel, and all 14 checkpoints are open under NVIDIA’s open model licence.
How Diffusion Writing Works
The model works in a 16-dimensional continuous token-embedding space, where the denoiser network and the token embeddings are learned together. Each block of tokens is refined over a configurable number of denoising steps, while earlier tokens are reused through a standard key-value cache.
Fewer steps trade a little quality for much larger speedups, and the smooth continuous representation suits few-step distillation and trajectory steering. NVIDIA measured roughly 120 to 140 tokens per second on a single H100 for 256 generated tokens, using 32 denoising steps for each 64-token block.
What the Release Includes
The family covers base, instruct and few-step distilled variants, plus self-speculative versions, all under the NVIDIA Nemotron Open Model License. Training used about 1.3 trillion tokens in total, and the base model is aimed at research rather than chat, with an instruct variant for that job.
The paper behind the family, ‘Large Language Continuous Diffusion Models’, is on arXiv, and the models load through standard Hugging Face Transformers with no custom kernels. A CUDA-capable NVIDIA GPU is required; CPU inference is not supported, since the sampler degrades on CPU.
Our Read
Diffusion language models are no longer a curiosity. Masked discrete diffusion models have already shown that the idea works; NVIDIA’s continuous version is the first at this scale, and its block-parallel decoding is a genuine break from token-by-token generation rather than a small tweak.
The honest limit is quality at low step counts. Below roughly 16 denoising steps the pretrained models lose quality noticeably, which is why NVIDIA ships distilled variants trained for a fixed step count. Speed claims should be read against the card’s own throughput figure, not against third-party write-ups.
Open weights are the point. NVIDIA hosts no service for these models; they run offline on the user’s own hardware, and the licence allows commercial use. That makes the family a research platform first and a product second, which is exactly how diffusion LMs will be judged.
Why It Matters
If diffusion decoding holds up at scale, it changes the economics of text generation: parallel blocks mean fewer sequential steps, and fewer steps mean faster, cheaper inference for every user. The technique also opens new controls, since trajectory steering works naturally in a continuous space.
For NVIDIA, the release is also a statement of intent. The company is steadily expanding its model footprint alongside its hardware dominance, and an open diffusion family gives researchers a genuinely new axis to compete on beyond the autoregressive mainstream that defines almost all of today’s models.
Background
NVIDIA has been building its open-model line steadily for years, and diffusion is the newest branch. Its latest earnings showed AI demand still surging, which funds exactly this kind of research bet — a different way to generate text, released openly for anyone to test.
Source: NVIDIA model card and arXiv paper


0 Comments