Positional Encoding
A mechanism that injects word-order information into a Transformer, since self-attention alone is order-agnostic.
In plain language
Without positional encoding, “cat chases dog” and “dog chases cat” are the same bag of words to self-attention. Early models used sinusoidal encodings; later approaches include learned embeddings, relative position and rotary position encoding (RoPE). RoPE represents relative distance by rotating vectors and extrapolates better, making it popular in open models. Positional extrapolation is a key variable when extending context windows.
Example
Extending context from 8K to 128K requires adjusting positional encoding or continued training, or long-distance relationships degrade.
Recommended tools
Related terms
Attention Mechanism
A computation that lets a model dynamically weigh how relevant other positions are when processing one position.
Context Window
The maximum number of tokens a model can read and remember at once, including input and output.
Transformer
A neural network architecture built around self-attention and parallel training, the foundation of modern large models.
Agent Memory
A mechanism for an agent to store and retrieve user preferences, past tasks and key facts across sessions.