What happens ChunkingUtils.chunkDocument splits content by collection.getChunkingMaxSize() characters (chunking method maxChars). Nothing in that path consults the maximum input length of the collection's embedding model. The internal inference server's default model, sentence-transformers/all-MiniLM-L6-v2, declares maxLength=256 word pieces in its DJL serving.properties, and the DJL HuggingFace text-embedding translator truncates beyond that silently – no warning, no error. 2000 characters of English prose is roughly 500 word pieces, and considerably more for wiki markup, code or tabular content. So with a 2000-character chunk size, very roughly the second half of every chunk never reaches the model. Why it matters This is silent, invisible data loss, and it is inconsistent between the two retrieval halves:
- the stored content field and posFirstChar / posLastChar claim the full character range
- the keyword half indexes the full text
- the vector only represents the truncated head
A chunk can therefore be found by keyword search on text that its own embedding cannot represent, and semantic search silently misses anything in the tail of a chunk. Nothing in the admin UI hints that the configured chunk size exceeds what the selected model can consume. Observed Collection configured with chunking method maxChars, chunkingMaxSize 2000, chunkingOverlapOffset 250. Retrieved chunks of 2000 / 1972 / 1966 characters; only about the first half of each is inside the model's 256 word-piece window. Suggested fix At minimum, warn when chunkingMaxSize exceeds what the selected embedding model can consume. Better: derive the character budget from the model's declared maximum input length, or add a token-based chunking method alongside maxChars. Surfacing truncation as a per-chunk warning – the errorMessage field already exists – would also make it diagnosable. |
