| Chunks are cut by character count and carry no information on where they come from. A text under "Projects > Alpha > Risks" and the same text under "Archive > 2019" get the same embedding, and a chunk often starts in the middle of a section. Add a chunking method sections that:
- keeps sections together and splits large ones at block, sentence and word boundaries;
- prefixes each chunk with a breadcrumb: the page's position in the wiki hierarchy (ancestors by page name, the page by its title) and the headings above the chunk;
- works for every syntax with a parser, as it uses the parsed document (depends on LLMAI-176).
Switching an existing collection to this method takes effect after a manual re-index (without cleaning), which embeds all chunks again. |