NVIDIA Solves the Main Compromise of Modern LLMs

When working with any LLM, one has to choose between speed and quality. Autoregressive models like GPT provide high quality but operate slowly because they generate one token at a time.

Diffusion models work the opposite way. They generate entire blocks in parallel, making them faster, but this speed comes at the cost of quality.

This is the problem that NVIDIA researchers aimed to solve.

They took a 30B model and split it into two parts to generate tokens in parallel rather than one at a time. Thus, Nemotron-Labs-TwoTower was born — a diffusion language model from NVIDIA Research, adapted based on Nemotron-3-Nano-30B-A3B.

But before understanding how TwoTower works, it is essential to grasp why diffusion LLMs lag behind autoregressive ones.

A diffusion LLM starts with a block of masked tokens and reveals them over several passes. On each pass, it reads all the already generated text while simultaneously predicting the masked tokens in the current block. This means that the same network performs two entirely different tasks at once. It must understand the existing context while predicting tokens that do not yet exist.

One set of weights is directed at two different tasks, so the network cannot specialize fully in either.

TwoTower solves this problem by not forcing one network to perform both tasks. Instead, the pretrained autoregressive model is split into two towers.

1. Context Tower

This is the frozen part of the model that reads all the already generated text just as the original model did. It is no longer trained, so the model's intellectual capabilities are fully preserved.

2. Denoiser Tower

This is the trainable part of the model that generates each new block of tokens in parallel. To maintain context, it constantly uses cross-attention to the Context Tower.

The most interesting part is how these two towers interact. They are connected layer by layer: the fifth layer of the Denoiser Tower interacts with the fifth layer of the Context Tower, the sixth with the sixth, and so on. This allows the Denoiser to access the entire multi-level representation of the context from the base model, not just its final hidden representation.

Results:

• 2.42 times higher generation throughput
• 98.7% quality retention of the original model
• Architecture built on a hybrid 30B model Mamba–Transformer MoE
• Only about 2.1 trillion tokens were used for training — just a fraction of the 25 trillion tokens used in the pre-training of the original model

This last figure is particularly important. None of the components were trained from scratch. This is an adaptation that can be added to an already existing autoregressive model. It relatively cheaply transforms slow decoding of one token at a time into fast parallel generation of blocks.

Therefore, for any latency-sensitive systems, this is an approach that is definitely worth exploring. No longer do you have to choose between a model you trust and the speed you need.

Weights, code, and training methodology are published in open access.

Article:

https://arxiv.org/pdf/2606.26493