What is an attention mechanism?
An attention mechanism is a way for a model to focus. For each element it is producing, it computes a set of weights over the available inputs and forms a weighted combination, so more relevant elements contribute more. This lets the model draw on distant context rather than only nearby information.
Self-attention is the special case where a sequence attends to itself, letting every element gather context from every other element, and it is the operation that defines the transformer.
Key takeaways
- Attention weights inputs by relevance, then combines them accordingly.
- Self-attention lets each element attend to all others in the same input.
- It is the core computation inside transformers.
How it works
Each element produces a query, a key, and a value. Attention compares queries to keys to get relevance scores, normalizes them into weights, and uses those weights to average the values. Doing this in parallel across all elements, often with several attention heads that capture different relationships, produces context-aware representations.
Why it matters
Attention gave models a flexible way to capture long-range dependencies without the bottlenecks of sequential processing, which is what made large-scale pretraining practical. Its generality across data types is why the same idea drives progress in language, vision, and multimodal systems.
Frequently asked questions
What is self-attention?
Self-attention is attention applied within a single input, so each element gathers context from all other elements of that same input.
What are attention heads?
Multiple attention heads run in parallel, each learning to focus on different relationships, and their outputs are combined for a richer representation.
Related terms