What is a token embedding?
Neural networks compute with numbers, not symbols, and token embeddings are the conversion step. Text is first split into tokens, and each token looks up its vector in a learned embedding table. Vision Transformers do the analogous thing with patches, projecting each into a token vector. These per-token vectors, plus positional encoding, are what actually enter the model.
Token embeddings are the model's input vocabulary in vector form, distinct from the output embeddings used for search and similarity.
Key takeaways
- Token embeddings convert discrete input units into vectors.
- They are learned jointly with the model during training.
- They are inputs to the model, unlike output embeddings used for retrieval.
How it works
For text, a tokenizer maps the input to token IDs and each ID indexes a row of a trainable embedding matrix. Training shapes these rows so tokens used similarly acquire similar vectors. For images, the patch projection plays the same role. As tokens flow through transformer layers, their representations are progressively contextualized, and pooling the final layer produces the passage- or image-level embedding.
Why it matters
Token embeddings are where representation learning begins, and their design choices, vocabulary, tokenization, and dimensionality, propagate through everything a model does. Understanding them clarifies the distinction between what enters a model and the output embeddings that power search, and why the two are not interchangeable.
Frequently asked questions
Are token embeddings the same as text embeddings?
No. Token embeddings represent individual input units inside the model, while a text embedding summarizes a whole passage for retrieval.
What is a token in a Vision Transformer?
An image patch. Each patch is projected to a vector and treated exactly like a word token in the sequence.
Related terms
Go deeper