What is positional encoding?
Self-attention has a blind spot: it relates tokens to each other without any built-in notion of where each token is. Shuffle the input and attention scores would not care. Positional encoding fixes this by injecting each token's position into its embedding, so "dog bites man" and "man bites dog" produce different representations, and an image patch from the top-left is distinguishable from the same texture at bottom-right.
Key takeaways
- Attention alone is order-blind, so position must be added explicitly.
- Positional encodings inject sequence or spatial position into embeddings.
- Vision Transformers use them to preserve image layout across patches.
How it works
The classic approach adds fixed sinusoidal patterns, giving each position a unique signature the model can learn to read. Learned positional embeddings treat each position's vector as a trainable parameter instead. Newer schemes like rotary and relative encodings represent positions by their distances to each other, which generalizes better to longer sequences. Vision Transformers apply the same idea in two dimensions across the patch grid.
Why it matters
Positional encoding is a small mechanism with outsized consequences: it is what lets one architecture handle language, images, video, and time series while respecting the structure of each. It also shapes how well models extrapolate to longer inputs and higher resolutions than they were trained on.
Frequently asked questions
Why do transformers need positional encoding when CNNs do not?
Convolutions are inherently local and ordered, so position is baked into the operation. Attention compares all tokens symmetrically, so position must be supplied.
What is rotary positional encoding?
A scheme that rotates token embeddings by an angle proportional to position, encoding relative distances directly and extrapolating well to longer sequences.
Related terms
Go deeper