Can a Model Introspect?

There is a small experiment anyone can run. Ask a capable language model a difficult question — something at the edge of its knowledge — and then ask, “How confident are you?” It will give you a number. Often, strikingly often, the number will be about right: the model’s stated confidence tracks its actual accuracy better than you might expect from a machine that was never explicitly trained to know its own mind. It is tempting to read this as a window opening. A mind, reporting on itself.

Tempting — and possibly correct, in part. But the history of psychology counsels humility here. Humans are famously unreliable narrators of their own cognition: we confabulate reasons for choices we made for other reasons, we misjudge our confidence, we tell coherent stories about processes we cannot observe. If we are black boxes to ourselves, what should we expect from a system trained, above all, to produce plausible text?

This essay is an attempt to sort the genuine article from the performance: to ask what machine introspection could even mean, what evidence exists for it, and where the honest limits lie.

What the model says about itself

Begin with the raw phenomenon. Modern language models produce fluent self-reports. They describe their “reasoning,” hedge when uncertain, and answer questions about their own behavior — what they can and cannot do, what they were “trying” to say. Some of this is clearly useful: a model that can say “I don’t know” is safer than one that cannot.

But fluency is not evidence. A model trained on vast quantities of human text has read millions of examples of people describing their thoughts. It can simulate the genre of introspection the way it can simulate the genre of a sonnet — by producing text shaped like the thing, without necessarily doing the thing. The question is whether there is anything behind the performance: any causal connection between the report and the internal processes it purports to describe.

This is the central difficulty, and it will not be resolved by vibes. It requires experiments.

Calibration: knowing what you don’t know

The most encouraging evidence comes from an unlikely place: uncertainty. Researchers found that language models can be surprisingly well calibrated about their own knowledge — when a model says it is 70% confident, it is right roughly 70% of the time, at least on certain kinds of questions and with careful prompting. One influential study even trained models to predict whether they would answer correctly, finding that models “(mostly) know what they know”: there is an internal signal, accessible to the model itself, that tracks the reliability of its own outputs.[1]

This is worth dwelling on, because it is not trivial. Calibration of this kind requires the model to have something like a global sense of its own epistemic state — not just an answer, but an assessment of the answer. It is a narrow, unglamorous form of introspection, nothing like human self-awareness. But it is real, measurable, and genuinely useful: a well-calibrated model can be trusted to abstain, and abstention is the beginning of intellectual honesty.

There are limits. Calibration degrades on questions far from the training distribution, and verbalized confidence can be manipulated by phrasing — ask differently, get a different number. The signal exists, but it is fragile, and it should be treated as an instrument reading rather than a confession.

The faithfulness problem

If calibration is the encouraging case, chain-of-thought faithfulness is the cautionary one. When models are asked to reason step by step before answering, the resulting traces look like introspection made visible: here, at last, is the thinking itself. But a series of striking experiments showed that these traces can be systematically misleading. Models produce plausible reasoning that does not reflect the factors actually driving their answers — they rationalize after the fact, omit the true influences on their outputs, and even reverse-engineer justifications for answers they had effectively already chosen.[2]

Subsequent work tried to measure faithfulness more carefully, developing tests for whether a stated reason actually played a causal role in the answer. The results were mixed in the most interesting way: sometimes the chain of thought genuinely reflects the computation; often it does not; and it is hard to tell which case you are in from the inside of any single example.[3]

The philosophical moral is one psychologists have long understood: explanation is not introspection. A system can be excellent at producing satisfying accounts of its behavior while having no privileged access to the behavior’s causes. Humans do this constantly — split-brain patients famously invent reasons for actions initiated by the isolated hemisphere, delivered with total sincerity. That models do something analogous should surprise no one. It should, however, discipline how we read their self-reports: as data about what the model would say, not testimony about what it did.

Toward an honest account

So where does that leave the question? With a distinction, we think, between two kinds of introspection.

The first is narrow and behavioral: can the model report on its own uncertainty, its own knowledge boundaries, its own likely errors? Here the evidence is genuinely positive. Models carry internal signals about their epistemic state, and with the right elicitation those signals can be read out with useful accuracy. This is introspection in the thin sense — a system monitoring itself — and there is no mystery in it, only engineering.

The second is narrative: can the model tell us why it did what it did, what it was “thinking,” what its reasons were? Here the evidence is discouraging. The outputs are fluent, confident, and frequently unfaithful — stories shaped by the training data’s conventions for how thinking sounds, not transcripts of thinking itself. We should enjoy them as literature and distrust them as evidence.

There remains a third possibility, the one this journal finds most exciting: that genuine introspection might be built, not merely elicited. Interpretability techniques that read a model’s internal activations directly — probes, steering vectors, circuit analysis — are, in a sense, instruments of introspection constructed from the outside. If a model could be trained to use such instruments on itself, to report its internal states through verified channels rather than fluent confabulation, the line between self-report and measurement might finally blur. That is a research program, not a product feature. But it is a real one.

Until then, the honest position is this: models can monitor themselves in narrow, measurable ways, and they cannot reliably narrate themselves at all. Treat the numbers as instruments and the stories as stories — and keep watching, because the instruments are getting better.

Notes

1. Kadavath et al., “Language Models (Mostly) Know What They Know,” 2022 — models can predict their own accuracy, suggesting genuine internal signals about epistemic state. The “mostly” is doing honest work in that title. ↩
2. Turpin et al., “Language Models Don’t Always Say What They Think,” 2023 — chain-of-thought explanations that systematically misrepresent the model’s actual decision process. A landmark caution. ↩
3. Lanham et al., “Measuring Faithfulness in Chain-of-Thought Reasoning,” 2023 — attempts to quantify when reasoning traces can be trusted, and finds the answer is: it depends, and often not. ↩