This issue has been created
 
 
LLM AI Integration / cid:jira-generated-image-avatar-2e1ff233-ca37-4e0c-baf2-9c39f7424b4b LLMAI-230 Open

Add a chunking method that follows the document structure and keeps the page hierarchy and headings as context

 
View issue   ยท   Add comment
 

Issue created

 
cid:jira-generated-image-avatar-db58df89-14a4-4196-803e-f515951d37f6 Gunter Ohrner created this issue on 14/Nov/26 21:46
 
Summary: Add a chunking method that follows the document structure and keeps the page hierarchy and headings as context
Issue Type: cid:jira-generated-image-avatar-2e1ff233-ca37-4e0c-baf2-9c39f7424b4b New Feature
Assignee: Unassigned
Created: 14/Nov/26 21:46
Priority: cid:jira-generated-image-static-major-4a8f42ba-8963-489c-8f53-dc1f0b1a30a0 Major
Reporter: Gunter Ohrner
Description:

Chunks are cut by character count and carry no information on where they come from. A text under "Projects > Alpha > Risks" and the same text under "Archive > 2019" get the same embedding, and a chunk often starts in the middle of a section.

Add a chunking method sections that:

  • keeps sections together and splits large ones at block, sentence and word boundaries;
  • prefixes each chunk with a breadcrumb: the page's position in the wiki hierarchy (ancestors by page name, the page by its title) and the headings above the chunk;
  • works for every syntax with a parser, as it uses the parsed document (depends on LLMAI-176).

Switching an existing collection to this method takes effect after a manual re-index (without cleaning), which embeds all chunks again.