In the Qwen3-1.7B language model, a single conversation is not a single point of data but a sequence of internal states where each token is represented by a vector of 2,048 numbers. By mapping thousands of these states into a three-dimensional projection, researchers discovered that English conversations occupy a remarkably flat geometric shape, with 99.3% of their variation contained within a single 3D plane.

This animation illustrates the finding that English conversation states are highly compressed into a thin geometric sheet.
The Residual Stream as a Shared Workspace
The study focuses on the 'residual stream,' which serves as the central shared workspace where a language model processes text. Each token—whether a word, part of a word, or punctuation—starts as a learned vector from an embedding table and is then refined by the model's layers based on the surrounding context.
By recording these final token states and normalizing their lengths, the researchers were able to visualize the 'paths' a conversation takes through the model's internal representations. These paths reveal how the model transforms initial embeddings into context-aware states.
“The geometry alone could not disentangle those possibilities.”
— What thousands of conversations look like inside a language model
Linguistic Geometry and Language Separation
The mapping revealed distinct spatial behaviors for different languages. While English states formed a thin sheet, Chinese states appeared in a neighboring region, visibly offset from the English cluster. This suggests that the model's internal representations carry information about both the subject matter and the specific linguistic form used to express it.
German and Croatian occupied a broad neighborhood overlapping with English, suggesting a shared geometric structure among these Latin-script languages. However, the researchers noted that the density of these regions was influenced by the data construction; for example, Croatian responses contained more tokens, resulting in a denser patch of points that does not necessarily indicate greater model capacity.

The animation demonstrates the distinct spatial behaviors and overlaps between different languages in the model's residual stream.
Code and the Vocabulary-Oriented Frame
When Python coding examples were introduced, they appeared in the same broad neighborhood as the natural language datasets. Interestingly, a simple distance comparison showed that the coding dataset was numerically closer to German and Croatian than to English, even though both code and English share similar keywords and structures.

This visual helps explain how Python coding examples reside in the same broad neighborhood as natural language datasets.
The study also demonstrated that the 'map' of these internal states is highly dependent on the perspective of the observer. Using a 'Conversation PCA'—which optimizes for the variation between conversation averages—revealed more distinguishable regions for German and Croatian than the 'Global PCA' frame, which was heavily influenced by the 151,936 vocabulary embeddings.

This visualization demonstrates how the 'map' of internal states changes when the perspective of the observer is shifted to Conversation PCA.
Chart drawn by HN Digest from figures reported by operator0x.com; values are shown as written in the source.
Limitations of 3D Projections
While the 3D visualizations provide a clear look at the model's internal geometry, they represent a significant reduction of the actual data. The researchers found that the 3D view captured only 17.6% of the total variation in the 2,048-dimensional token states, meaning over 82% of the model's internal variation remains invisible in the projection.
Furthermore, the study used a small, deliberately constructed sample of 100 topics and 90 coding examples. While this provided a controlled environment to observe linguistic geometry, it may not fully capture the vast diversity of the model's capabilities across all possible human knowledge and conversation types.