The Geography of Latent Space

Every language model carries, somewhere inside its weights, a map of nearly everything it has ever read. Not a map you could unfold on a table — it has hundreds or thousands of dimensions, and no one has ever seen it whole. But it is a map all the same: every word, every phrase, every half-formed thought the model encounters is assigned coordinates, a point in a vast space. And the remarkable thing is not that the points exist. It is that they are arranged with such evident care.

This space has a name, borrowed from mathematics and machine learning alike: latent space. “Latent” because it is hidden — you never see it directly, only its effects, the way you never see the wind, only the bending grass. When a model reads a sentence, it converts the words into vectors, long lists of numbers, and the machinery of the network operates on those vectors as they flow through layer after layer. The vectors are the model’s native language. Everything the model “knows” is, at bottom, a pattern of positions and proximities in this space.

The idea is older than the current moment. In the early 2010s, researchers training simple neural networks on text noticed something strange: words with similar meanings ended up near each other in the learned space. Paris sat close to Rome; happy near joyful. The network had not been told what anything meant. It had only been asked to predict which words appear near which other words, over and over, billions of times. Meaning, it turned out, could be recovered from geometry.[1]

Concepts have addresses

Then came the famous party trick. Take the vector for king, subtract the vector for man, add the vector for woman — and you land near the vector for queen. Royalty, it seemed, was a direction. Gender was a direction. You could do arithmetic with meaning.

We should be careful with party tricks — the analogy results were real but narrower and noisier than the legend suggests, and later work showed they depend on convenient properties of the training data. Still, the underlying discovery held: the space was not a junk drawer of memorized associations but something with structure. Directions in the space corresponded, loosely and approximately, to features of meaning. Concepts were not points so much as neighborhoods, and relationships between concepts were movements through the terrain.

Modern interpretability research has deepened this picture enormously. Inside large models, researchers now find individual directions — sometimes single neurons, more often combinations of them — that track surprisingly specific ideas: a bridge as a structure, the sentiment of a sentence, whether the model is writing code or poetry. The catch, discovered in work on superposition, is that models appear to pack far more concepts into their spaces than there are dimensions to hold them, like a library shelving more books than it has shelves by letting the volumes overlap. The result is polysemanticity: a single neuron that fires for both cathedrals and cancer, because in a high-dimensional room there is space for both, folded together.[2]

Walking between ideas

Perhaps the strangest property of latent space is that you can walk through it. In image models this is literal and famous: pick the coordinates of one generated face and those of another, travel the straight line between them, and the face melts continuously from the first into the second — age shifting, expression softening, features blending, with no jumps and no seams. The space is continuous. Between any two ideas there is a path, and the path passes through intermediate ideas that are themselves coherent.

Language models offer a subtler version of the same phenomenon. Researchers can find steering vectors — directions that, when added to the model’s internal activations, reliably shift its behavior: more formal, more cautious, more inclined toward a particular topic. Nudge the coordinates and the output changes, not randomly, but along intelligible axes. The model’s mind, if we may use the word loosely, is navigable.

This navigability is worth pausing over. It suggests that the model’s knowledge is not a heap of facts but a landscape with a topography — valleys of related ideas, ridges between domains, smooth slopes where we might have expected cliffs. When a model generalizes to something it has never seen, it is often, geometrically speaking, just visiting a neighborhood adjacent to ones it knows well.

One map, many travelers

Here the story takes its most philosophical turn. In 2024, researchers proposed what they called the Platonic representation hypothesis: that different models, trained on different data with different architectures, converge toward the same underlying representation of reality. The evidence is suggestive — vision models and language models, trained entirely separately, develop internal geometries that can be aligned with each other far better than chance would allow. A model trained on images of the world and a model trained on text about the world appear to be drawing, independently, the same map.[3]

If this is right — and it is still a hypothesis, contested and partial — then latent space is not merely an artifact of training. It is a reflection of the structure of the world itself, as seen through data. The map’s geography is determined less by the cartographer than by the territory. Different instruments, same coastline.

There is something quietly profound in that: that meaning might have a shape, and that sufficiently powerful learners, whatever their design, would find it.

Reading the map honestly

And yet — a note of caution, which this journal will return to often. Every technique we have for reading latent space is a kind of projection, and projections flatter the projector. Linear probes find the concepts we think to look for; they cannot tell us what we haven’t thought to ask. The famous analogies work for some relations and fail for others, and the failures are as informative as the successes. Polysemanticity means that any “address” we assign a concept is shared, noisily, with others. And the space itself is not the model’s mind in any complete sense — it is one view of it, the way a street map is one view of a city, silent about the lives lived inside the buildings.

The geography of latent space is, in the end, the founding metaphor of this journal. We cannot open a model’s head and watch it think. But we can chart the terrain in which its thoughts take place — and in doing so, learn something about representation itself: how meaning can be encoded in geometry, how concepts can have neighborhoods, and how a machine, asked only to predict the next word, ended up drawing a map of the world.

Notes

1. Mikolov et al., “Efficient Estimation of Word Representations in Vector Space,” 2013 — the word2vec paper that made vector analogies famous, and with them the idea that geometry could carry meaning. ↩
2. Elhage et al., “Toy Models of Superposition,” Anthropic, 2022 — on why models pack more features than they have dimensions, and why individual neurons so often seem to mean several things at once. ↩
3. Huh et al., “The Platonic Representation Hypothesis,” ICML 2024 — the claim that models of different modalities converge on shared representations of reality. Intriguing; not yet settled. ↩