How interpretability works

1 · The machine

A network is a sorting office with millions of sorters

A neural network takes something in (pixels, words) and passes it through stacked layers. Each layer holds thousands of neurons: tiny units that add up “votes” from the layer below and pass on a single number, their activation. The strength of each vote is a weight, and training is the process of tuning those weights.

Interpretability is reverse-engineering that office. Nobody designed the sorters' jobs; training invented them. Mechanistic interpretability asks two questions: what does each part respond to, and how do parts wire together into circuits that produce a behaviour.

2 · The toolkit

Seven ways to question a neuron

No single tool proves anything. Confidence comes from several tools pointing at the same answer.

• Top examplesRun millions of images or sentences through and keep the ones that make it fire hardest. The first, cheapest look.
• Feature visualisationGenerate a synthetic image that excites the neuron as much as possible (the famous psychedelic pictures). Powerful, but it can take shortcuts.
• Negative poleWhat pushes it below zero. Tells you the distinction it draws, like “pointed, not round”.
• Circuits and attributionTrace which earlier neurons feed it and which later ones read it. An attribution graph maps that flow for one input.
• AblationSwitch it off and see what the model gets wrong. Like pulling a fuse to find out what it powers.
• SteeringTurn it up artificially and read what the model starts to say. If the “Colorado” neuron is real, the model should start planning hikes.
• Training checkpointsSave the model at stages of training and watch a neuron grow. Shows whether odd extras sprouted from something nearby.

3 · Worked example

The famous “cats and cars” neuron, solved in six steps

Neuron 4e:55 in InceptionV1, an image model from 2014 that became the field's lab rat (a model organism), fired for cat faces and car fronts. For years it was the poster child for a polysemantic neuron: one that seems to have several unrelated jobs. Here is how the essay re-reads it.

Try it: feed things to a “V detector”

Tap an object. Pointed fronts push the neuron up; round things push it below zero, whatever the object is called.

4 · The big puzzle

Why do some neurons look like a jumble?

Two explanations compete, and they lead to opposite ways of doing the science.

See the difference: squeeze the number of neurons

Same 24 things in both panels; similar colours and shapes are similar ideas. Change how many neurons are available.

How a branch grows: the “iy” neuron

4b · Five suspects

“Polysemantic” often means “not explained yet”

The essay lists five reasons a neuron can look messy. Four can be tested directly. The essay's core argument is that the untestable one was treated as the default when it should be the last resort.

• Bad tools testableThe microscope is out of focus. Fix the tool and see if the neuron resolves.
• Bad explanations testableA list of nouns where one shared pattern would do.
• Bad learning testableThe model was built or trained poorly. Retrain with a better design and compare.
• Recursive generalisation testableOffshoots from a related idea. Watch it grow across training checkpoints.
• Superposition hard to testUnrelated packing. Its only symptom is “it doesn't make sense”, which every other suspect also produces.

5 · The evidence

Better tools and better designs cleaned up the “hopeless” layer

InceptionV1's final layer was the field's main exhibit for unreadable neurons. Same data, same size, different microscope or design:

Source: Cammarata (2026). Author's estimates; others put the 2014 figure near 0%.

The difference-maker was ordinary engineering meant to help deep models learn, not to make them readable. A residual connection is an express lane that lets information skip past a layer. LayerNorm keeps each layer's signal at a steady volume, like a sound engineer riding the faders.

Sometimes the mess was the microscope

• Shortcut-taking picturesFeature visualisation follows the easiest path. InceptionV1 saw so many dogs that the generator kept drifting into dogs, making car neurons look like dog neurons. The “cars hidden in dogs” circuit was really cars and dogs sharing a pose.
• Too little contextShowing only two paragraphs made every long-context neuron look random. The reason was further up the page.
• Too narrow a datasetOne “mess” neuron turned out to track tell-tale signs inside health spam pages, once the test data included enough of them.

The squeeze test

If cramming causes packing, crushing a layer should create jumbles. The essay cut a well-understood layer from 480 neurons to 10 (48× fewer) and retrained.

Source: Cammarata (2026), ConvNeXt experiments.

6 · Language models

Chatbot-style models look far tidier than earlier audits found

Not like-for-like: methods, models and definitions differ. Sources: Cammarata (2026); Bills et al. (2023).

Language neurons rarely fire on the obvious noun. Once you know the recurring patterns (motifs), many “random” neurons click into place.

Motif 1: the summary on the comma

An automated labeller called one neuron a “comma neuron”. It actually tracks Colorado. The model jots a running summary at punctuation, so the commas light up only once the text has become about Colorado. Tap run, then try steering.

Motif 2: the same idea in disguise

A neuron fired on “pr” in English, plus apparent junk: web-address code strings and Bengali text. Decode them. Tap each card.

Example strings are ours; the neuron is from the essay.

Three more patterns worth knowing

• Say, or just thinkThe final layer spends much of its effort stopping specific words from being said, so the model can hold an idea without blurting it.
• The double-negative echoIn SwiGLU layers (a common design where one signal gates another) a negative times a negative comes out positive. For about 4% of neurons this echo dominates the top examples and fools the tools.
• Specialist expertsIn mixture-of-experts models, where a router sends each word to a few of many sub-networks, the experts turned out far more specialised than reported.

How they checked the labels

For 200 random neurons, an AI agent read each label and wrote five texts that should fire it and five near-misses that shouldn't, before running them.

183 neurons with positive-side labels. Source: Cammarata (2026), via AI Wiki summary.

7 · Faint signals

Could the hidden stuff live in the quiet firing?

Most of a neuron's life is near zero. In one Qwen neuron, only about 160 of 100 million words pushed it past half strength. Sceptics argue information could hide in that quiet murmur. The essay tested it by muting it.

Image accuracy, medium ConvNeXt. Both removals take away the same total activity. Source: Cammarata (2026).

What broke when the faint part went were near-ties the model was already unsure about, like Eskimo dog versus husky. The essay itself admits there is no trusted test yet for what faint activity means in general.

8 · Why it matters

Understanding compounds, like a crossword

A single neuron in isolation is hard. Once many around it are known, each new one arrives with clues from its inputs, its readers and its neighbours. The essay argues this is why a near-empty map and a mostly-full map are the natural places to be.

• SafetySpotting deception or other risky behaviour means knowing its building blocks. Reading neurons directly makes that monitoring cheaper.
• Simpler methodIf neurons are readable, you skip training a second untangling model. The essay calls this a possible “bitter lesson” for the field: scaled-up plain investigation beats clever decompositions.
• AI doing the scienceAgents mapped neurons, tested labels and reviewed about 100,000 attribution graphs. The work scales with compute.

9 · The other side

Researchers are split, and part of the fight is definitions

Where both sides agree

Networks can represent more concepts than they have neurons, by combining neurons. The dispute is narrower: whether a single neuron packs unrelated things together. The author says he is roughly 85% sure it mostly doesn't, admits the title overstates the evidence, and leaves attention heads largely out of scope. The full language-model paper is still to come.

Glossary

Sources