Yann LeCun was right: a 109M-parameter model outperformed a 72B model

Yann LeCun was right, but nobody in this industry wants to say it out loud.

A model with 109 million parameters outperformed a model with 72 billion parameters.

For years, LeCun has argued that scale alone does not force a model to understand the world. Meta bet on the opposite approach. In November, he left the company and founded his own laboratory in Paris.

Recently, he published evidence supporting his theory.

Since 2024, the industry has followed the idea that it is enough to train increasingly large models on text, images, and audio so that they eventually form a common internal representation of the world. It was believed that size solved everything. But this hypothesis was never proven.

Randall Balestriero and Yann LeCun tested it. The results showed that model size is not the decisive factor.

What matters is which data the model receives together.

A photograph and its caption are two different ways of conveying information about the same object. Models trained on such pairs reach the desired result much faster.

But two paragraphs from the same document are not two representations of the same object. Models trained on such data reach a comparable result much more slowly and require significantly more computational resources.

This second type is exactly next-token prediction. Almost all modern chatbots work on this principle.

In the experiments, the first method reached a comparable level of understanding with only 21.6 million examples. Training with next-token prediction required 900 million.

Moreover, a text model with 109 million parameters formed a representation of the world closer to that of a model trained on images than a language model with 72 billion parameters.

660 times fewer parameters.

Study