Better Vector Search for Long Documents: Chunking Inside Manticore Search

Manticore Search now includes built-in document chunking for vector search, automatically splitting long documents into smaller chunks, embedding each chunk separately, and searching across all of them—eliminating the need for manual preprocessing or external splitting tools. The feature supports five chunking strategies (truncate, mean, fixed, recursive, and sentence) configured directly in the table definition, with testing showing significant recall improvements for content beyond a model's token window (55.1% to 83.3% recall@5). Documents remain single search results despite being chunked internally, with Manticore returning the closest matching chunk's distance while counting documents, not chunks, in result rankings.