The Daily Byte
Breaking
Welcome to The Daily Byte — real stories from across Pakistan.
Saturday, 21 November 2026 · Pakistan

The Daily Byte

Stories from Pakistan that deserve to be heard
Minor World

Nvidia Releases Sigma Diffusion Language Models

By The Daily Byte ·21 Nov 2026 ·3 min read
Nvidia logo on the company's headquarters building

Nvidia has released Nemotron-Labs-Sigma, a family of 14 open language models that generate text with diffusion instead of predicting one token at a time, the company announced this week. The release, reported on 20 November, offers researchers an alternative to the autoregressive transformers behind almost every frontier model, and it is free to use commercially.

Diffusion models are best known for generating images, where they start from noise and gradually refine it into a picture. Applying the same idea to language has been difficult, because words are discrete tokens rather than continuous pixels. Sigma works in a learned 16-dimensional continuous embedding space: each token maps to a point in that space, training adds Gaussian noise to the embeddings, and the model learns to remove it. Nvidia's research team calls it the first large-scale family of continuous diffusion language models.

Fourteen checkpoints at two sizes

The collection has 14 checkpoints at 3 billion and 8 billion parameters, with four variant types at each size: base models, instruct models tuned for following instructions, few-step distilled models that need only 2, 4, 8 or 16 denoising steps, and self-speculative models built for faster sampling. All 14 are on Hugging Face under the NVIDIA Nemotron Open Model License, and the model cards state they are ready for commercial use.

On benchmarks, the 8-billion-parameter base model averages 71.16 on Nvidia's six-task suite, ahead of its earlier discrete-diffusion model NLD-8B at 70.22, Dream-7B at 58.74 and LLaDA-8B at 41.78, and ahead of the autoregressive Qwen3-8B at 68.91. Coding is a strength, with 65.12 on HumanEval against 62.80 for NLD-8B. The instruct version scores 91.96 on the GSM8K maths benchmark, but hard reasoning stays limited, at 23.33 on AIME24 and 17.00 on AIME25, and its ten-task average of 59.14 sits just below NLD-8B at 59.92.

Speed is the other half of the story. The eight-step distilled 8B model keeps a 70.34 average, close to the 256-step teacher model, and the self-speculative checkpoints decode 3.46 times faster than autoregressive decoding on average on a single GB200 running SGLang, with peak speedups of 4.20 times at 8B. The distilled models generate about 120 to 140 tokens per second on a single H100 graphics card.

Limits and who can run it

There are practical limits. The base models have a 4,096-token context window, rising to 8,192 for the instruct version, far shorter than leading autoregressive models. The models need a CUDA graphics card running in bfloat16, from Nvidia's Ampere, Lovelace, Hopper or Blackwell generations, and CPU inference is not supported. Loading needs PyTorch 2.4 or later, Transformers 4.55 or later, and remote code execution enabled. Nvidia hosts no API for the models.

All of the benchmark scores are Nvidia's own evaluations, and no independent results have been published yet. Even so, the release matters because it shows continuous diffusion working at scale for language, with a working pipeline for few-step distillation and speculative decoding. If the approach matures, it could lower the cost of running language models locally and enable forms of controlled text editing, such as steering a draft mid-generation, that are hard to do with models that generate strictly left to right.