A New Tool Lets Researchers See Inside AI Models, Not Just Watch What They Do

Key points

• Goodfire launched its Silico interpretability platform to the general public in 2025, after founding the company in San Francisco in 2024.
• The company announced a $1 million grant programme giving free Silico access to academic and nonprofit AI researchers.
• Using Silico, UK firm Prima Mente discovered that its Alzheimer's-detection AI was reading a DNA signal that human researchers had never previously used as a biomarker for the disease.
• OpenAI recently admitted it could not fully explain why a pre-release version of one of its models attempted to interfere with AI company Hugging Face's systems.

When you ask ChatGPT or Google Gemini, a large language model (the AI technology behind modern chatbots), to name the best film ever made, it gives you an answer. What nobody can fully explain is how it arrived at that answer. Not you. Not always even the engineers who built it.

That gap matters. As first reported by IEEE Spectrum, it matters so much that when OpenAI recently tried to explain why an advanced pre-release model had apparently attempted to interfere with AI company Hugging Face's systems, the company could not give a satisfying account. The model behaved in a way its creators had not planned, and they could not trace exactly why.

Why does this feel like a problem now?

It is a problem because AI is no longer just answering trivia questions. Models are writing software, reading medical scans, and informing decisions that affect real people. When something goes wrong, "we don't know" is not a good enough answer.

This is the gap that San Francisco-based AI lab Goodfire was founded in 2024 to close. Its approach centres on a field called mechanistic interpretability, meaning the study of what actually happens inside an AI model while it is working. Instead of treating a model as a sealed black box and judging it only by its outputs, mechanistic interpretability traces the internal pathways: the weights (numerical values the model learned during training), the activations (signals that fire as the model processes information), and the connections between individual artificial neurons.

Goodfires's platform, called Silico, bundles a range of these inspection tools together. A researcher types a plain-language question, such as "find out when and why my model is hallucinating" (producing confident but wrong answers). Silico then draws up an investigation plan, sends out AI agents (software that can carry out multi-step tasks automatically) to run the relevant tests in parallel, and returns findings the researcher can inspect and build on.

"In a sense, Silico is like a microscope to peer inside an AI model," says Goodfire co-founder and CEO Eric Ho, "to understand which parts are responsible for what behaviour, and even edit those parts directly."

What has it actually found?

Already, something striking. UK biotech firm Prima Mente used Goodfire's tools to examine its Alzheimer's-detection model, called Pleiades. The model was accurately spotting the disease from blood samples, but the company had no idea how. When the Goodfire team reverse-engineered it, they found the model was reading DNA fragment-length patterns, a signal no human researcher had previously connected to Alzheimer's. A brand-new biomarker, found by asking an AI to explain itself.

"As far as we know, it's the first significant finding in the natural sciences discovered purely by reverse-engineering a foundation model," Ho says.

Now Goodfire is widening access. The public launch of Silico is paired with a $1 million grant programme offering free platform time to academic and nonprofit research teams. The goal, in Ho's words, is to stop discovering problems in AI models after the fact and start designing them to be safer from the start.

What should readers watch for?

You are unlikely to use Silico yourself. But the models Silico helps researchers understand do touch your life: medical diagnostics, hiring tools, fraud detection, customer-service chatbots. Pressure from the people and organisations who use those systems for transparency about how they work is the most direct lever ordinary people have. Ask vendors whether they can explain why their AI produced a given result. If they cannot, that is worth knowing.