Visualizing What Makes GPT-2 MLP Neurons Fire

When an MLP neuron fires strongly, which parts of the preceding computation raised its preactivation? I made an interactive lattice that decomposes the firing neurons’ mean preactivation across source tokens and model components.

The example uses GPT-2 Small on an indirect object identification (IOI) prompt:

When John and Mary went to the store, John gave a bottle of milk to

The input ends at to, immediately before the expected continuation Mary.

How to read the lattice

Each frame selects one prompt token and one MLP layer. The columns are source-token positions, and the rows are components of the residual stream: token and position embeddings, earlier MLP outputs, and attention outputs up to and including the selected layer. Warm cells raise the firing neurons’ mean preactivation; cool cells lower it.

A cell reports its signed contribution as a percentage of the sum of the absolute component scores. This makes the relative pattern legible even when the raw scale changes between frames. The Token total row sums signed scores by source position and is not included in that denominator. The two additive offsets—one from LayerNorm and one from the MLP input bias—appear separately in the header.

Use Previous, Next, or Play to move through token-layer pairs, or click any MLP block to jump directly to its token and layer. A blank frame means that no neurons at that token and layer exceeded their own firing thresholds.

Methods

We ran GPT-2 Small in evaluation mode. We estimated the 99th percentile (p99) of post-GELU activation separately for each of the model’s 36,864 MLP neurons. We used the first 1,000,000 nonpadding token positions in the Pile-10k training split in dataset order, truncating each document at 1,024 tokens. Percentiles were computed by linearly interpolating adjacent order statistics from per-neuron float16 activation histograms. At each prompt position and MLP layer, a neuron counted as firing if its post-GELU activation strictly exceeded its own p99.

Our approach to decomposing the preactivation of firing neurons was inspired by Goodfire’s analysis of neurons as an interpretable basis. For each token and layer, we decomposed the mean preactivation of the firing neurons into contributions from the token and position embeddings, earlier MLP outputs, and attention outputs up to and including the target layer. We held LayerNorm’s observed scale fixed and applied its centering and gain, followed by each firing neuron’s input projection, to every residual-stream component. We kept the LayerNorm offset and MLP input bias as separate additive terms.

To split an attention output across source positions, we weighted each head’s value vectors by the observed attention probabilities and applied the attention output projection. The current-token cell includes the attention output bias and its reconstruction residual. The decomposition describes the observed forward pass and measures contributions to MLP preactivation.

We exported the p99 analysis as a standalone HTML lattice. It embeds all precomputed frames, so navigation and playback work offline.

Conclusion

Press Play and take your time watching how the contributions change from one token and layer to the next. One pattern I noticed in this prompt is that the layer 0 MLP often contributes more strongly at tokens with concrete meanings, such as “bottle” and “milk,” and less at articles such as “the” and “a.”

At the final token, “to,” watch the columns for the names in the prompt. The second “John,” which is the third name mentioned, stands out at layers 7 and 8. At layers 9 through 11, “Mary,” the second name mentioned, also makes a noticeable contribution. What other patterns show up as you watch?