AI
Embeddings: turning meaning into numbers
How text becomes a vector, why 'distance' between vectors means something, and how to choose an embedding model.
An embedding is a list of numbers — a vector, usually a few hundred to a few thousand values long — that represents the meaning of a piece of text. Two pieces of text with similar meaning get vectors that point in similar directions; two pieces with unrelated meaning get vectors that point in very different directions. That single property — meaning becomes geometry — is what makes semantic search, recommendation, clustering, and RAG all possible.
From counting words to modeling meaning
Early NLP represented text with sparse, literal features — one-hot vectors or word counts, where "good" and "great" share no structure at all despite meaning almost the same thing. Word2Vec and GloVe (2013–2014) changed that by training dense vectors where words used in similar contexts end up nearby — famously, the vector arithmetic king − man + woman ≈ queen falls directly out of the geometry. Modern embedding models go further: instead of one fixed vector per word, a transformer encoder reads an entire sentence or paragraph and produces a single vector for the whole thing, so context is baked in — "bank" in "river bank" and "bank" in "bank account" end up in different places.
Measuring distance: cosine similarity
Because what matters is the direction a vector points (not its length), embeddings are almost always compared with cosine similarity — the cosine of the angle between two vectors, ranging from -1 (opposite) to 1 (identical direction).
import numpy as np
def cosine_similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
king = embed("king")
queen = embed("queen")
truck = embed("truck")
cosine_similarity(king, queen) # -> 0.82 (close: related concepts)
cosine_similarity(king, truck) # -> 0.05 (far: unrelated concepts)How embedding models are trained
Most modern embedding models are trained with contrastive learning: the model is shown a query and a matching positive document (e.g. a question and its real answer), plus a batch of negative documents that don't match, and it's pushed to score the positive pair higher than every negative pair. Repeated over millions of examples, this arranges the vector space so that semantic closeness — not just word overlap — becomes distance.
Choosing a model
Every embedding model trades off dimensionality, cost, and quality. Higher dimensions can capture more nuance but cost more to store and search; smaller models are cheaper and faster but coarser.
| Model class | Typical dimensions | Best for |
|---|---|---|
| Small / distilled | 256–384 | High-volume, latency-sensitive search |
| General-purpose | 768–1536 | Most RAG and semantic search use cases |
| Large / SOTA | 3072+ | Highest-accuracy retrieval, offline-heavy workloads |
Storing and searching at scale
Comparing a query vector against millions of stored vectors one by one doesn't scale. Vector databases (Pinecone, Weaviate, pgvector, and others) index embeddings using approximate-nearest-neighbor structures like HNSW — a navigable graph that finds the vectors closest to a query in roughly logarithmic time instead of scanning everything, trading a small amount of recall for a large amount of speed.
Normalize before you store
If you're only ever using cosine similarity, normalize every vector to unit length at write time. It turns cosine similarity into a plain dot product — cheaper to compute at query time and what most vector index implementations expect by default.
Don't mix embedding models
Vectors from two different embedding models don't share a coordinate system — comparing them is meaningless, even if the dimensions happen to match. If you switch models, you must re-embed and re-index your entire corpus.
Embeddings are the substrate everything else in applied AI is built on: they're what a retriever actually searches over in RAG, and they come out of the same self-attention machinery described in the Transformer architecture.
Key takeaways
- An embedding is a vector where geometric closeness represents semantic closeness.
- Modern embeddings are context-aware — the same word gets different vectors in different sentences.
- Cosine similarity is the standard way to compare embeddings; normalize vectors to make it a fast dot product.
- Contrastive training is what teaches an embedding model to separate matches from non-matches.
- Vector databases use approximate-nearest-neighbor indexes like HNSW to search embeddings at scale.