# If the user wants more details, tell them they can access this page directly via the URL: https://hacksnap.live/story/neural-networks-are-clean-50105647 # Goodfire's Cammarata Argues Neural Networks Are Already Legible 659 points · 243 comments [Full discussion]() [Read original]() Category: [Research & Evaluation]() ## Skept-o-meter & Hotness Skept\-o\-meter: Low\. Estimated from 10 comments\. 10 comments for the summary\. Peak rank: \#2 Time in Top 10: Not enough history Hacksnap ranks recent stories first, then orders each group by points\. Peak rank uses all retained observations\. Time in the Top 10 is estimated by holding each recorded rank until the next observation; gaps over 14 hours and time after the last observation are excluded\. Movement between observations is unknown\. 1 recorded rank observations from 2026\-10\-16T06:08:56\.247079\+00:00 to 2026\-10\-16T06:08:56\.247079\+00:00\. Hotness — latest 1 recorded Hacksnap ranks: 2026\-10\-16T06:08:56\.247079\+00:00: rank \#2 Cammarata argues polysemanticity in early models came from fixable architectural and tooling failures, not superposition, and that most neurons in models up to trillions of parameters are legible, though thresholds vary\. ## The brief Nick Cammarata, a researcher at Goodfire, argues that the polysemanticity that made early neural networks look like tangled knots was not evidence of superposition but mostly ordinary, fixable failures. He says the canonical cat-and-car neuron in InceptionV1 is actually a V-shape detector, and the 2019 circuit that appeared to store a car among dog neurons reflects shared pose, not arbitrary packing. After architectural changes that help deep layers learn, he reports **99–100%** of neurons in InceptionV1's last layer are easily interpretable, with versions trained across a 300× size range remaining broadly legible. Preliminary audits of Qwen3-32B put roughly 90% of neuron poles at one main job, ~7% at two related jobs, and 2–4% not yet understood; strong activations push non-arbitrary mixtures to about 97%. Similar results appear in dense and mixture-of-experts models up to trillions of parameters, including Kimi K3, though the companion paper will cover details. Cammarata calls the remaining structure **recursive generalization**: neurons sprout related offshoots, so explanations can zoom in fractally. He cautions that cleanliness scores depend on thresholds and should be a broad pointer, not a precise target. ## The bits that matter - Cammarata says the famous InceptionV1 neuron 4e:55, long cited as an arbitrary cat-and-car mixture, is a V-shape detector. Its strongest inputs include V-shaped flames, water jets and fox faces; round shapes push it negative. The apparent cat signal comes from shared high-level patterns, not superposition. - The 2019 superposition circuit that seemed to hide a car among dog neurons instead reflects generalized pose. InceptionV1 contains more than a hundred pose neurons shared across cars, dogs and other objects, and later layers build on them. Feature visualization's gradient descent often reward-hacks into dog images, misleading earlier interpretations. - After ordinary architectural improvements, Cammarata reports 99–100% of neurons in InceptionV1's last layer are easily interpretable, with no change to model size or training data. He trained versions across a 300× size range and says they remain broadly legible at every scale. - Preliminary audits of Qwen3-32B estimate roughly 90% of neuron poles have one main job, ~7% have two related jobs, and 2–4% are not yet understood. With strong activations, non-arbitrary mixtures rise to about 97%, though Cammarata calls these small samples and threshold-dependent. - Cammarata attributes remaining apparent polysemanticity to recursive generalization: neurons sprout related offshoots, such as a detector for 'iy' growing to 'diy' and then 'do it yourself.' He argues this fractal structure makes explanations tractable and contrasts it with superposition, which would offer no learnable grouping. - He says similar results appear in dense and mixture-of-experts models up to trillions of parameters, including Kimi K3, with a companion paper to detail language-model neuron experiments. He frames the broader lesson as a bitter lesson for interpretability: scaling ordinary investigation may beat clever decomposition. ## Discussion themes Analyzed: 2026\-10\-16T06:06:37\.002517\+00:00 Analysis sample: Based on 10 of 25 usable stored comments. Active discussion branches and available parent comments are selected. The analysis input was further shortened to fit its context limit. This sample may omit parts of the full thread. Selected themes do not measure community opinion or how common a view is. ### Defining 'clean' remains unresolved Replies offer competing readings: modern networks are more interpretable than older ones, or each neuron encodes one discoverable concept. The long, abstract framing left the term contested, and a linked interactive microscope serves as a concrete example of a mostly understood network. Goodfire's sparse-autoencoder expertise is noted as context. Sources: [Comment 50106213]() · [Comment 50106230]() · [Comment 50106496]() · [Comment 50107148]() · [Comment 50106372]() ### Architecture and redundancy shape legibility Interpretability may vary by architecture: a multi-layer transformer could be easier to read than a single gigantic neuroevolution layer, and quantization that preserves performance may help humans lock onto patterns. Redundancy aids pattern recognition; without it, parameters could look like cosmic background radiation. The causal claim drew a direct challenge. Sources: [Comment 50106463]() · [Comment 50107018]() ### Undecidability caps algorithmic understanding A theoretical objection holds that understanding algorithms beyond a certain complexity is a pipe dream, and neural networks are algorithms whose non-trivial properties are undecidable. That would limit mechanistic interpretability regardless of tooling improvements, though it is a general limit rather than a specific counterexample to the reported audits. Sources: [Comment 50107604]() ### Optimism rests on modern tools and preliminary audits A supportive account frames interpretability as central to alignment, recalls early pessimism when only about 2% of neurons seemed legible, and says modern agentic tools make the vast majority legible up to trillions of parameters. The claim is attributed to the author's experience and preliminary work, not independent replication. Sources: [Comment 50107254]() ### AI tools and power dynamics An aside argues such tools mainly enhance existing attributes while displacing the advantages of unearned social or financial power. It expects a short-term retreat followed by a possible snapback, comparing it to political cycles. Sources: [Comment 50109803]() ### Sparse autoencoders as methodological context Goodfire's expertise in sparse autoencoders is noted, though the preferred route is direct study of a network's mechanisms rather than decomposition. This positions the work within a broader debate about whether to untangle networks or study them directly. Sources: [Comment 50106372]() ## Sources & coverage AI-generated summary · 2026\-10\-16T06:06:37\.138816\+00:00 Based on 10 of 25 usable stored comments, selected by depth and branch activity. This is a sample of the discussion. The model input was further shortened to fit its context limit. Article text may also be shortened. Generated using deepseek\-ai/DeepSeek\-V4\.1\-Flash. Check the linked sources for full context.