What is a transformer?
A transformer is a neural network architecture that processes its input as a set of elements, called tokens, and uses self-attention to let every token consider every other token when forming its representation. Unlike earlier sequential models, it handles all positions in parallel, which makes it efficient to train at scale and good at capturing long-range relationships.
Introduced for language, the same architecture now underpins vision, audio, and multimodal models, which is why it is the default building block of modern AI.
Key takeaways
- Transformers use self-attention to relate all input elements at once.
- They process inputs in parallel, which scales well to large datasets.
- The architecture generalizes across language, vision, and multimodal tasks.
How it works
Input is split into tokens and embedded as vectors, then passed through stacked layers that alternate self-attention with feed-forward networks. Attention assigns each token a set of weights over the others, so the model can focus on the most relevant context. Because there is no built-in notion of order, position information is added explicitly through positional encodings.
Why it matters
The transformer unlocked training on enormous unlabeled corpora, which led directly to foundation models and their transfer to countless downstream tasks. Its arrival in vision, through the Vision Transformer, reshaped visual AI by showing that attention-based models can match or beat convolutional networks given enough data.
Frequently asked questions
What makes a transformer different from a CNN?
A CNN builds features from local patches through convolutions, while a transformer relates all elements globally through attention, which captures long-range context more directly.
Are transformers only for language?
No. The same architecture powers vision, audio, and multimodal models, with the Vision Transformer being a prominent example in visual AI.
Related terms