Maybe neural networks are not black boxes at all, as we tend to assume

Nick Cammarata, a prominent AI interpretability researcher, published a major essay titled Neural Networks Are Already Clean. Cammarata was among the first scientists to work on interpretability, built OpenAI's interpretability team, and authored several foundational papers on the topic. He now works at Goodfire.

He proposes the hypothesis that neural networks may actually be simpler than we assume.

Most attempts to interpret networks rely on the superposition assumption: looking at individual neurons seems almost pointless because the model supposedly piles many unrelated things into each neuron. Researchers therefore try to untangle activations with other models first — the best-known approach is SAE — and only then work with the discovered features.

Cammarata says he previously supported this approach. Now he argues that decomposition may not be needed at all, and that what neurons actually combine are not random things, but broader global concepts. They simply are not always obvious to humans.

A foundational work for interpretability is Circuits (2020). Cammarata was a coauthor, and that paper was seen as evidence for superposition in neural networks. The authors analyzed GoogLeNet for image recognition and showed that the same neuron activated both for car hoods and cat faces. This example is still considered a canonical proof that unrelated features spread almost chaotically across neurons.

In his essay, Cammarata rebuts that case. It turns out the two concepts mixed in that neuron for a reason: it was a V-shaped neuron. It activated when the model needed to distinguish elongated from round shapes, and cat faces and car hoods were just among the most common examples.

Cammarata shows that neuron readability does not even depend on model size. Smaller models simply have coarser internal features, making them harder to interpret. He calls this the fractal nature of generalization: small models give a rough explanation, while large models provide more detail for each neuron.

For language models, this also appears to work. In preliminary audits of Qwen3-32B, about 90% of neurons have one main job, around 7% have two related jobs, and only 2–4% remain unclear. Similar results are reported for models up to trillions of parameters, including MoE models.

Although all conclusions are highly tentative, Cammarata suggests a parallel with Sutton's "bitter lesson": in interpretability, the same principle that worked for deep learning may apply — ordinary neuron-level study at scale could turn out to be more effective than complex approaches.