Manticore Search now supports built-in chunking for embedding-based vector search, letting users add a chunk_strategy option directly to a table's vector column instead of manually splitting long documents before indexing. The database can split text into pieces, embed each chunk, and search across all of them within a single table, eliminating the need for a separate chunking pipeline or secondary table. Five strategies are offered: truncate, mean, fixed, recursive, and sentence, with the latter three requiring a float_vector_array column to store multiple vectors per document.
A technical guide argues that most retrieval-augmented generation (RAG) implementations are over-engineered, with teams reaching for embeddings, vector databases and reranking before checking if simpler tools suffice. It lays out decision factors—data freshness, corpus size and churn, query patterns, scale and team ML skill—to determine which retrieval approach fits, starting with plain full-text search like BM25, Elasticsearch or Postgres search before adding complexity.
Enterprise AI systems are largely built by connecting individual applications directly to source documents, generating separate chunks, embeddings and retrieval pipelines per use case. As companies scale up the number of AI agents and applications, teams end up duplicating work on the same documents and producing inconsistent, sometimes contradictory representations of the same business knowledge.
GoKawiil