In the Qwen3-1.7B language model, a single conversation is not a single point of data but a sequence of internal states where each token is represented by a vector of 2,048 numbers. By mapping thousands of these states into a three-dimensional projection, researchers discovered that English conversations occupy a remarkably flat geometric shape, with 99.3% of their variation contained within a single 3D plane.

Animated 3D rotation of English conversation states in a Global PCA frame.
A visualization of English conversation states, showing how they occupy a remarkably flat geometric shape within the model's internal state space.operator0x.com

This animation illustrates the finding that English conversation states are highly compressed into a thin geometric sheet.

The Residual Stream as a Shared Workspace

The study focuses on the 'residual stream,' which serves as the central shared workspace where a language model processes text. Each token—whether a word, part of a word, or punctuation—starts as a learned vector from an embedding table and is then refined by the model's layers based on the surrounding context.

By recording these final token states and normalizing their lengths, the researchers were able to visualize the 'paths' a conversation takes through the model's internal representations. These paths reveal how the model transforms initial embeddings into context-aware states.

“The geometry alone could not disentangle those possibilities.”

— What thousands of conversations look like inside a language model

Linguistic Geometry and Language Separation

The mapping revealed distinct spatial behaviors for different languages. While English states formed a thin sheet, Chinese states appeared in a neighboring region, visibly offset from the English cluster. This suggests that the model's internal representations carry information about both the subject matter and the specific linguistic form used to express it.

German and Croatian occupied a broad neighborhood overlapping with English, suggesting a shared geometric structure among these Latin-script languages. However, the researchers noted that the density of these regions was influenced by the data construction; for example, Croatian responses contained more tokens, resulting in a denser patch of points that does not necessarily indicate greater model capacity.

Animated 3D rotation of English, Chinese, German, and Croatian states.
Comparative visualization of four languages, showing the spatial offset between Chinese and the overlapping Latin-script languages.operator0x.com

The animation demonstrates the distinct spatial behaviors and overlaps between different languages in the model's residual stream.

Code and the Vocabulary-Oriented Frame

When Python coding examples were introduced, they appeared in the same broad neighborhood as the natural language datasets. Interestingly, a simple distance comparison showed that the coding dataset was numerically closer to German and Croatian than to English, even though both code and English share similar keywords and structures.

Animated 3D rotation of four natural languages and Python coding examples.
A comparison of natural language datasets and Python coding examples within the Global PCA frame.operator0x.com

This visual helps explain how Python coding examples reside in the same broad neighborhood as natural language datasets.

The study also demonstrated that the 'map' of these internal states is highly dependent on the perspective of the observer. Using a 'Conversation PCA'—which optimizes for the variation between conversation averages—revealed more distinguishable regions for German and Croatian than the 'Global PCA' frame, which was heavily influenced by the 151,936 vocabulary embeddings.

Animated 3D rotation of conversation datasets under Conversation PCA.
The same datasets visualized under a Conversation PCA, which optimizes for variation between conversation averages.operator0x.com

This visualization demonstrates how the 'map' of internal states changes when the perspective of the observer is shifted to Conversation PCA.

Variation Captured by PCA Axes: Variation in 3D Global PCA 17.6%; Variation in best 2D plane 28.7%; Variation in English 3D plane 99.3%
Variation Captured by PCA Axesoperator0x.com

Chart drawn by HN Digest from figures reported by operator0x.com; values are shown as written in the source.

Limitations of 3D Projections

While the 3D visualizations provide a clear look at the model's internal geometry, they represent a significant reduction of the actual data. The researchers found that the 3D view captured only 17.6% of the total variation in the 2,048-dimensional token states, meaning over 82% of the model's internal variation remains invisible in the projection.

Furthermore, the study used a small, deliberately constructed sample of 100 topics and 90 coding examples. While this provided a controlled environment to observe linguistic geometry, it may not fully capture the vast diversity of the model's capabilities across all possible human knowledge and conversation types.