What are Token Embeddings?

Token Embeddings are dense vector representations of individual tokens (e.g., words or subwords) in a high-dimensional space. These embeddings capture semantic relationships between tokens and are generated by models like BERT or GPT.

Token embeddings are foundational to tasks like semantic search, dense retrieval, and context-aware generation, where understanding token-level relationships is critical for performance.