What is a Vision Transformer?
A Vision Transformer adapts the transformer, originally built for language, to images. It divides an image into a grid of fixed-size patches, embeds each patch as a token, adds positional information, and feeds the sequence through a standard transformer. Self-attention then lets every patch relate to every other patch, giving the model a global view of the image from the first layer.
It showed that convolutions are not strictly necessary for strong visual performance, provided there is enough training data.
Key takeaways
- ViT treats image patches as tokens and applies a transformer to them.
- Self-attention gives it a global receptive field immediately.
- It matches or beats CNNs when trained on large datasets.
How it works
The image is cut into patches, each is flattened and linearly projected into an embedding, and positional encodings preserve spatial layout. The resulting token sequence passes through transformer layers, and a classification token or pooled representation produces the final output. Because ViTs lack the built-in locality of convolutions, they typically need more data or strong pretraining to reach their best performance.
Why it matters
The Vision Transformer unified vision and language under one architecture, making it far easier to build multimodal models that share components. It also serves as a powerful backbone for detection, segmentation, and retrieval, extending attention-based modeling across visual AI.
Frequently asked questions
Is a Vision Transformer better than a CNN?
It depends on data scale. ViTs excel with large datasets or heavy pretraining, while CNNs remain strong and data-efficient on smaller datasets.
Why does a ViT split images into patches?
Patches act like tokens, letting the model reuse the transformer architecture and apply self-attention across the whole image.
Related terms