01 What is a multi-vector encoder?
A regular embedding model compresses an entire text into one vector. A
multi-vector model (a.k.a. late interaction / ColBERT-style) keeps
one vector per token — so a 9-token document becomes a
9 × 128 matrix, not a 1 × 128 vector. Scoring is deferred
to query time with the MaxSim operator.
One vector each
Pools everything into a single fixed-size vector. One dot product at query time. Fast, but compression is lossy — a rare entity, an ID, or one crucial clause must all compete for room.
Full interaction
Both texts pass through the model together. Accurate but nothing can be precomputed — every document must be re-encoded for each new query.
The middle path
Documents are encoded offline and indexed, but scoring compares every query token against every document token (MaxSim). Stronger matches at retrieval time, at the cost of a bigger index.
02 MaxSim Lab
For each query token, take its highest similarity against any
document token, then sum those maxima. Because token embeddings are
L2-normalized, each cell is a cosine similarity in [-1, 1] and the total
lands in [-n_query_tokens, n_query_tokens].
Token × token similarity
Per-query-token max → sum
03 Semantic search: can MaxSim pick the right planet?
Query: — scored against 4 passages
with MaxSim (exact figures reported in the blog). Mars wins, but note
how close Saturn and Jupiter run: token-level maxima give every document a floor.
04 It isn't lexical — alignment is semantic
Encode “Where do penguins live?” against “Penguins inhabit Antarctica.” and the query token live finds its best match on the document token inhabit at 0.94 — two words that share no characters. That's what lexical retrieval (BM25) cannot do: it needs the literal term.
Late interaction keeps the other direction too: when an exact match matters (a product code, a surname, a function name), that token still sits on its own instead of being averaged into everything else. It isn't one-to-one either — several query tokens routinely align to the same document token.
05 What you gain, and what it costs
Encoding 4,874 Natural Questions passages with
lightonai/LateOn produced 608,414 token
vectors (~124.8 per passage). Compare index footprints:
| Representation | Vectors | Dim | float32 |
|---|
Raw multi-vector is ~42× MiniLM, but PLAID compresses it back near dense territory. Token pooling trims vector count; retrieve-and-rerank avoids an index entirely.
Same data, same backbone — the only difference is one vector per token vs one per document.
- Multi-requirement queries — each requirement finds its own evidence.
- One specific clause is what makes a doc relevant.
- Out-of-domain data — dense compression was tuned on other queries.
- Gap grows with document length.
- Semantic search on a few thousand docs (exhaustive MaxSim).
- Reranker stage — cheap, no index needed.
- Visual document retrieval (ColPali) — text vs page images, no OCR.
06 Retrieve & rerank — late-interaction quality, no late-interaction index
Skip the big index entirely: a fast bi-encoder narrows the corpus to a handful of candidates, then the multi-vector model rescales only those with MaxSim. Your index stays a normal dense index and the token vectors are transient.
1 · Dense retriever
Bi-encoder encodes the whole corpus once → top-50 candidates.
2 · MaxSim rerank
Multi-vector model rescales only the 50 candidates (one batched matmul).
3 · Final ranking
Reordered top results, with token-level evidence preserved.
The same role a cross-encoder plays in retrieve-and-rerank — but a multi-vector model is considerably cheaper per candidate (one matrix multiply, not one forward pass per pair).
07 Supported models
08 Code, in practice