Goodfire researcher argues neural networks are interpretable

Nick Cammarata, a researcher at the AI interpretability lab Goodfire, argues in an essay published October 15 that neural networks are much easier to read than the field has assumed. He says the field's best-known evidence for "superposition" was misread. The essay has divided researchers over whether interpretability is close to a breakthrough. Cammarata calls his claim "a working hypothesis." A technical companion paper on language-model neurons has not yet been published, according to the essay.braindetox.kr

The core claim

Under the superposition theory, a network that runs short of neurons packs unrelated concepts into the same neurons. Researchers would then need a second model, such as a sparse autoencoder, to untangle them before studying anything. Cammarata argues instead that most mixed-looking neurons come from what he calls "recursive generalization." A neuron grows offshoots related to its main job the way a branch grows sub-branches. His example is a neuron that went from detecting "iy" to "diy" to "do it yourself". "I now think I was making superposition the null hypothesis when it should have been the last hypothesis, since it's the most radical and the least testable," he writes.already-clean.goodfire.com

The essay also revisits work Cammarata did himself. He says the famous "cat and car" neuron in the InceptionV1 image model is really a detector for pointed V shapes. He also says the car-to-dog superposition circuit he identified in 2020 was an error.the-decoder.de

The numbers

Cammarata compared InceptionV1 with a ConvNeXt model of the same size trained on the same data. About 99% of the neurons in the ConvNeXt model's last layer were easily interpretable, he reports. In InceptionV1, the figure was roughly 5% with the tools of that era and about 50% with today's tools. For language models, he writes that "in preliminary audits of Qwen3-32B, we estimate that roughly 90% of neuron poles have one main job." Another 7% have two related jobs, and 2% to 4% are not yet understood.already-clean.goodfire.com

He cautions that the figures depend on where the thresholds are set. He says he is "maybe 85% sure" that nearly everything comes from ordinary generalization. The work relied on AI agents that write explanations of what neurons do and then test them, and the agents reviewed 100,000 attribution graphs, which trace how information moves through a model. By comparison, an automated OpenAI project in 2023 found only about 1.7% of neurons understandable, even by its most generous measure.the-decoder.de+1

A split reaction

Some people welcomed the essay. Latent Space's AINews newsletter ran it under the headline "Agents disprove Polysemanticity". Others were more cautious.latent.space

Neel Nanda, who leads mechanistic interpretability at Google DeepMind, called the work interesting but said it does not change the big picture. He still believes superposition is real, partly because language models store far more facts than they have neurons, and he thinks weak activations carry many of those memorized facts. He doubts that findings from small image models carry over to language models, and he argues the essay defines monosemanticity too generously. He also expects neuron-based analysis to make serious errors and miss a lot.the-decoder.de+1

The two agree on one point: models can represent more concepts than they have dimensions. Cammarata disputes only the idea that networks pack completely unrelated things together at random. Independent groups have not yet confirmed his results.the-decoder.de