Without explicit positional information, a self-attention mechanism would treat the sentences 'The red fox jumps over the gray fox' and 'The gray fox jumps over the red fox' as identical, as the word 'fox' would receive the same embedding vector regardless of its location. To solve this, models like GPT-2 and MiniMax-M3 inject position-specific data into token vectors, allowing the transformer to understand the structural sequence of the input.

This visual explains the fundamental need for positional data by showing how identical words change meaning based on their sequence.
The Mechanism of Absolute Position Embeddings
In the GPT-2 architecture, position information is added as a distinct vector to the token embedding. The model learns a table of position vectors during training, and for each token at position m, a vector P_m is selected and added element-wise to the token's embedding vector.

This figure illustrates the specific architectural placement of positional embeddings in different model types.
This approach ensures that even if two tokens are identical (e.g., the word 'fox' appearing twice), they enter the transformer blocks with different numerical representations. However, this method is limited by the size of the pre-defined embedding table, which must be established before training begins.
“Nonetheless, GPT-2 and most newer LLMs include explicit positional information because it gives attention a direct signal about positions.”
— Understanding Positional Embeddings in Modern LLMs
Scaling Limitations and the Shift to Relative Methods
A significant trade-off of absolute position embeddings is the cost of expanding the context window. For GPT-2 Medium, the positional embedding layer accounts for only 0.30% of the weights at a 1,024-token limit; however, scaling that same context to 100,000 tokens would increase that share to over 22% of the total model weights.
Chart drawn by HN Digest from figures reported by magazine.sebastianraschka.com; values are shown as written in the source.
To circumvent this, many modern models have moved toward relative positional embeddings, such as Rotary Positional Embeddings (RoPE). Unlike the absolute method, RoPE injects positional information directly into the attention mechanism, allowing for more efficient handling of long-range dependencies without inflating the model's weight count.

This image supports the explanation of how relative methods like RoPE function within the attention mechanism.
Implicit Order vs. Explicit Signals
Causal masks in autoregressive models provide a degree of implicit order by restricting which previous tokens a current token can attend to. Theoretically, this allows a model to distinguish word order even without explicit embeddings.
Despite this, explicit positional methods remain essential because the causal mask does not provide a sense of distance. Explicit embeddings provide a direct signal that helps the model differentiate between a token's immediate neighbor and a token ten positions away, a distinction that is vital for coherent text generation.