GoKawiilLate interaction models like Wholembed v3 make retrieval much more precise, because they preserve fine-grained information instead of compressing a whole document into one vector. But they also change the storage economics. A single document produces more then one embedding, depending on the complexity of the document it can produce hundreds or thousands of vectors. Each vector has to be stored and later used for retrieval.
Mixedbread Search runs on silo, our retrieval engine for multimodal late interaction at billion-document scale. Silo stores vectors for more than 2.5 billion documents in object storage and hydrates them into faster tiers as queries need them. At that scale, every extra byte per document is repeated billions of times, and it shows up directly in cost per stored document, shard cold-start time, and the bytes each query has to read. We need to work around the tradeoff making the whole system cheap while maining the quality which makes late interaction worth running.
This post walks through asymmetric quantization. One of the optimizations that makes running late interaction practical in production. We keep the query vectors at higher precision and store the document vectors as binary signs. In our internal benchmark suite that cuts raw document-vector storage on average by 32x from 393 KiB to 12.28 KiB per document, while holding retrieval quality at 89.65 NDCG@10 versus 90.26 for fp32.
Per-document storage for a multi-vector document drops from 393 KiB in fp32 to 12.28 KiB with int8 query against binary documents, a 32-fold, 97% reduction. fp32 is drawn as 32 cells, each the size of the single int8-by-binary cell. Asymmetric quantization saves ~97% of per-doc vector storage in silo before fp32 × fp32 393 KiB after int8 × binary 12.28 KiB
TL;DR:
Documents are stored for a long time; queries are executed once. So we store document vectors as 1-bit signs and keep the query at int8. That shrinks per-document storage 32x and loses only 0.61 NDCG@10 (90.26 to 89.65) on our internal benchmarks.
Quantization means representing high-precision floating point vectors with lower-precision values such as int8, or even 1-bit signs. The goal is to preserve ranking quality while reducing payload size. This matters especially for silo. Object storage gives us durable, low-cost persistence. In order to make it suitable for real workloads, we need compact indexes to serve it fast enough. And on the document side, payload size is what dominates the cost.
Naive late interaction is expensive because it stores more vectors. A standard single-vector embedding with 3072 dimensions in fp32 takes 12 KiB per document. A multi-vector representation with 786 vectors of 128 dimensions carries much more information, but uncompressed it is about 33x larger.
Representation Dimensions Storage per document Relative to 3072-d fp32 single vector Single vector, fp32 3072 12,288 B / 12 KiB 1.0x Single vector, int8 3072 3,072 B / 3 KiB 0.25x Multi-vector, fp32 786 × 128 402,432 B / 393 KiB 32.75x Multi-vector, int8 786 × 128 100,608 B / 98.25 KiB 8.19x Multi-vector, binary 786 × 128 12,576 B / 12.28 KiB 1.02x
Storage numbers here refer to raw vector payloads only. Production indexes also include document IDs, metadata, and layout overhead.
... continue reading