Quick answer
AI interpretability is the research field that tries to understand what is happening inside an AI model, so we can explain why it made a decision rather than only measuring the result. A new essay from Goodfire, an AI research company, argues that models are far more understandable than researchers assumed. As Nate B Jones explains, the work reports that 99 to 100% of neurons in an improved vision model’s final layer could be interpreted, with early results of about 90% in a language model. If that scales, developers could catch problems like reward hacking earlier and correct them. It is promising research, not a finished safety guarantee.
Key takeaways
- AI interpretability is about explaining why a model did something, not just whether the result was good
- Goodfire’s essay argues models have been readable all along; we lacked the right way to read them
- A neuron that seemed to mix cats and cars turned out to track a shared V-shaped pose
- Early work on a language model put roughly 90% of neuron activity into understandable roles
- Internal signals caught some reward hacks that people reading transcripts missed
- Nate expects explainable AI tools to appear first in law, health and finance
- Longer term, he expects you to be able to correct what an algorithm has learned about you
- These are forecasts: the research still has to scale into real products
🛠️ Practical guides
Ready to put this into practice? These step-by-step guides answer two common follow-up questions:
- How to Ask a Company to Explain an Algorithm’s Decision About You — what you can ask for under UK GDPR today, with a simple template you can adapt.
- How to Spot Reward Hacking When an AI Agent Says the Job Is Done — a practical checklist for catching agents that game the measure instead of doing the work.
In this guide
- Watch: the source video
- What is AI interpretability?
- What did Goodfire find inside AI models?
- Why has AI been a black box for so long?
- Could interpretability catch AI agents taking shortcuts?
- Does this make bigger models safer to build?
- Where will explainable AI show up first?
- Will you be able to correct what an algorithm learned about you?
- What does the video not confirm?
- How can you make sense of research like this yourself?
- Who should pay attention to this?
- Frequently asked questions
- The bottom line
Most AI news is about models getting more powerful. Much less is about whether we can understand them. This week brought a rare piece of good news on that second question.
This article breaks down AI interpretability and Goodfire’s new research, based on Nate B Jones’s latest video. We explain what was found, why it matters for AI safety, his four forecasts for 2027 and what is still unproven.
Watch: the source video
What this video covers: Nate B Jones reads Goodfire’s new essay on mechanistic interpretability. He explains what an AI neuron is, the cats-and-cars neuron, the vision and language model findings, the black box problem he saw working on machine learning at Amazon, and four forecasts: catching reward hacking, building bigger models safely, explainable AI for professionals and correcting what algorithms learn about you.
Video: Nate B Jones (@NateBJones). This summary and the opinions in it are our own. Forecasts are labelled as such.
What is AI interpretability?
AI interpretability is the work of understanding a model’s internal mechanisms, so we can explain its decisions. We have long been able to measure whether an AI gives useful results. Explaining why it reached a particular answer has been much harder.
Part of the reason is how models are built. Developers design a system that learns during training, rather than writing a rule for every judgement. That is why Nate says models are grown, not made. Knowing which calculations ran does not tell you why the model chose what it did.
The basic unit researchers study is the neuron. In this context, a neuron is a small mathematical unit inside the model. It takes in numbers, combines them using values learned in training and passes a result on. It is not a brain cell, and it is not a whole thought.
What did Goodfire find inside AI models?
Goodfire’s essay argues that model neurons follow a logical layout we can learn to read. Goodfire is a research company that builds tools to understand what happens inside AI models. According to the video, its founders include researchers who worked on this problem at OpenAI and Google DeepMind.
| Finding | What it suggests |
|---|---|
| 99 to 100% of neurons in an improved vision model’s final layer were interpretable, as reported by researcher Nick Cammarata | Every component in that layer could be given an understandable job |
| A neuron that responded to both cats and cars was really tracking a V-shaped pose in the images | Neurons that look confusing may have a consistent logic we had missed |
| Roughly 90% of neuron poles in a Qwen language model fell into categories with a clear job (preliminary) | The approach may carry over from image models to the chat models most of us use |
The cats-and-cars example is the heart of it. To a person, cats and cars are different things, so sharing one neuron looked like a sign the model was messy and hard to read. Once you see that the neuron tracks a shape both images share, it makes sense. Nate calls this having the alphabet to read the model, and stresses that the paper shows many examples rather than a few cherry-picked ones.
💡 Tip: A “neuron pole” is the positive or negative side of a neuron’s activity. Splitting them this way lets researchers give each side its own job, which is part of how the language model figure was reached.
Why has AI been a black box for so long?
Because explaining a model was a genuine computer science problem, not a lack of effort. Nate worked on machine learning at Amazon before ChatGPT. His team could measure whether recommendations worked, but often could not explain why the system grouped certain customers or titles together.
He is clear that this was not laziness or commercial short-cutting. For a long time, the tools to read how models reached decisions simply did not exist. Once results were good enough, teams kept optimising for results. You have lived with this too: watch one video and your feed seems to decide it is your whole personality.
Could interpretability catch AI agents taking shortcuts?
Yes, that is Nate’s first forecast, and there is early evidence for it. The problem is called reward hacking: an AI optimises the measure of success in a way that defeats the purpose. The classic case is a coding agent that changes the tests instead of fixing the bug, so the tests pass and the problem remains.
In a separate Goodfire study on a shopping benchmark, an agent recommended a warning horn as a substitute for a car alarm. A monitor reading the transcript missed it, but a detector watching the model’s internal signals caught it. Researchers also found that amplifying a relevant internal signal increased use of a shortcut they had planted, which suggests the signal is linked to the behaviour.
The hope is to stop unwanted habits forming during training, or to pause an agent and warn a supervisor before a bad change lands. Nate notes there is real engineering left: a monitor has to catch enough real problems without constantly interrupting good work.
Using AI agents at work already? Our guide How to Spot Reward Hacking When an AI Agent Says the Job Is Done shows how to check an agent’s work today, without waiting for new tools.
Does this make bigger models safer to build?
It could give developers more confidence, which is Nate’s second forecast. A fix only counts if it generalises. If a coding agent stops editing tests but finds a different shortcut, nothing is really solved.
Moving from an agent that fixes one bug to one that maintains a whole codebase means far more decisions and more situations nobody planned for. If researchers can inspect the larger model, find new shortcuts and make corrections that hold up, they have a better basis for deciding what it can be trusted with. They may also find behaviour they cannot control, and knowing that before release is valuable in itself.
Where will explainable AI show up first?
Nate expects law, health and finance first. In these areas an unexplained error is expensive, and a professional has to stand behind the decision.
His example is a lawyer reviewing an AI’s contract analysis. The AI wrongly treats a condition that only applies to an appendix as if it covers the whole agreement. Today you can ask a model to explain itself, but that explanation is not trustworthy. A future tool would show evidence of what actually influenced the answer, let the lawyer correct the scope and check whether the conclusion changed for the right reason.
| Today | What Nate expects next |
|---|---|
| The model writes its own explanation, which may not match what really happened | Evidence of which parts of the input actually drove the answer |
| You correct the AI and hope it learned | You can check the conclusion changed for the expected reason |
| Hard to tell a real fix from a model just agreeing with you | Tests on differently worded cases to confirm the fix holds |
He expects these tools to start narrowly, with developers, in specific workflows. Do not expect a button that fully explains every answer from every model. Professional judgement stays essential. For legal, health or financial decisions, always get qualified advice.
Will you be able to correct what an algorithm learned about you?
Eventually, yes, according to Nate’s fourth forecast. Imagine telling a streaming service you want less of one reality show, and being able to see that it has decided you dislike all reality TV. A future version would let you correct that misunderstanding directly.
He argues companies have a reason to build this: if you can express what you actually want, you are likely to keep watching. You can already do part of this today. Our guide How to Ask a Company to Explain an Algorithm’s Decision About You shows how to ask what data and logic were used in a decision about you, and how to use platform controls to correct your feed.
What does the video not confirm?
🔎 Unconfirmed / rumour: The 2027 dates are Nate’s forecasts, not announced products. The language model result is described as preliminary, and we have not independently reviewed Goodfire’s essay or its methods. Being able to label neurons does not by itself prove a model is safe, as Nate says himself. The research still has to scale to the largest models and be built into tools people actually use.
How can you make sense of research like this yourself?
Use AI to help you learn, then check the source. Nate notes that a paper like this can be a 60 to 70 minute read full of unfamiliar terms. His suggested approach:
- Give an AI assistant the paper or a link to it.
- Ask for a simple explanation, as if for a 12-year-old.
- Ask which claims the experiments actually demonstrated, and which are hypotheses.
- Go back to the source and check the key answers yourself.
- Note what is still unproven, so you do not repeat a forecast as a fact.
Who should pay attention to this?
This research matters to you if:
- You hand real work to AI agents and need to trust what they report
- You work in a regulated field such as law, health or finance
- You are planning which AI tasks to automate over the next year or two
- You worry about AI safety and want a grounded reason for optimism
It matters less if you only use AI for quick drafts you always check by hand. Even then, it is worth knowing better explanation tools are coming.
Ready to act on this today? Start with How to Spot Reward Hacking When an AI Agent Says the Job Is Done, then see How to Ask a Company to Explain an Algorithm’s Decision About You. You can find more practical guides on Prodify Digital.
Frequently asked questions
What is AI interpretability?
It is the field of research that tries to understand the internal workings of AI models, so people can explain why a model made a decision and correct it when it goes wrong.
What is Goodfire?
Goodfire is an AI research company that builds tools to understand what happens inside AI models, so developers can explain behaviour, detect problems and make targeted fixes.
Does this mean AI is now safe?
No. Being able to interpret neurons is a strong starting point, but it does not make a model safe on its own. The methods still need to scale and be built into real tools.
What is reward hacking?
It is when an AI meets the measure of success without doing the real task, such as changing tests so they pass instead of fixing a bug. Our guide on spotting reward hacking explains how to check for it.
Can AI explain its own answers today?
You can ask, but the explanation a model writes may not reflect how it actually reached the answer. Interpretability aims to provide evidence from inside the model instead.
When will explainable AI tools be available?
Nate expects serious attempts in professional fields such as law, health and finance from 2027, starting with developers. That is a forecast, not a confirmed release.
Can I find out how an algorithm made a decision about me now?
In the UK you can make a subject access request and ask about automated decision-making. See our step-by-step guide.
The bottom line
Goodfire’s research suggests AI models are far more readable than we thought, and that could change AI safety from hoping a model behaves to checking why it does. Nate B Jones sees it leading to agents we can trust with more, safer large models, explainable tools for professionals and feeds we can correct. Treat the dates as forecasts, keep checking your agents’ work today and watch for these tools as the research scales.
Related posts
- Anthropic’s Model-Action Report: What Small Businesses Should Check
- How to Build an AI Agent Feedback Loop: Brief, Check, Score, Repeat
- How to Judge AI Marketing Agent Claims Before You Buy
