What is a Token embedding?

What is a token embedding?

Neural networks compute with numbers, not symbols, and token embeddings are the conversion step. Text is first split into tokens, and each token looks up its vector in a learned embedding table. Vision Transformers do the analogous thing with patches, projecting each into a token vector. These per-token vectors, plus positional encoding, are what actually enter the model.

Token embeddings are the model's input vocabulary in vector form, distinct from the output embeddings used for search and similarity.

Key takeaways

Token embeddings convert discrete input units into vectors.
They are learned jointly with the model during training.
They are inputs to the model, unlike output embeddings used for retrieval.

How it works

For text, a tokenizer maps the input to token IDs and each ID indexes a row of a trainable embedding matrix. Training shapes these rows so tokens used similarly acquire similar vectors. For images, the patch projection plays the same role. As tokens flow through transformer layers, their representations are progressively contextualized, and pooling the final layer produces the passage- or image-level embedding.

Why it matters

Token embeddings are where representation learning begins, and their design choices, vocabulary, tokenization, and dimensionality, propagate through everything a model does. Understanding them clarifies the distinction between what enters a model and the output embeddings that power search, and why the two are not interchangeable.

Frequently asked questions

Are token embeddings the same as text embeddings?

No. Token embeddings represent individual input units inside the model, while a text embedding summarizes a whole passage for retrieval.

What is a token in a Vision Transformer?

An image patch. Each patch is projected to a vector and treated exactly like a word token in the sequence.

Jesse Mostipak is the SEO and Content Manager at Voxel51, where the work is helping humans find and trust what the brand knows, and teaching the Google knowledge graph and the LLMs answering on their behalf to do the same. That question, how knowledge gets built inside a system, is one Jesse has been chasing for years. Earlier versions of it ran through a New York City high school science classroom, data science and machine learning, and developer relations at Kaggle, Posit (formerly RStudio), and Baseten. The answer doesn't change much depending on whether the learner is a teenager, a software engineer, or a knowledge graph. Jesse holds a Master's in Education from CUNY Hunter College. LinkedIn

Token embedding

Building visual or physical AI? 
Let's talk.