There is 1 comment.
 
 
LLM AI Integration / cid:jira-generated-image-avatar-53bcf497-ee42-4503-9dd1-80a29af0c634 LLMAI-176 Open

Raw wiki syntax is fed to the embedder instead of rendered plain text

 
View issue   ยท   Add comment
 

1 comment

 
cid:jira-generated-image-avatar-387f2c04-7d61-43f8-a774-695cbb609b69 Gunter Ohrner on 14/Nov/26 21:52
 

Proposed approach, as part of the chunking work in LLMAI-229 (pluggable chunking methods) and LLMAI-230 (structure-aware chunking with breadcrumbs), which builds on it:

  • Index the parsed document (XDOM) instead of its source, like the platform's Solr index does, so that every syntax with a parser is supported.
  • Render it as simplified Markdown: headings, list markers, tables and code blocks are kept, links keep only their label, and emphasis, images and escaping are left out. The structure stays readable for the LLM that receives the retrieved chunks, without markup noise for the embedding model.
  • Macros are not executed during indexing, so no scripts run and no content of other pages or user-dependent output enters the index. The content of macros with wiki content (box, info, warning, ...) is prepared without execution and indexed. The content of code macros without a source parameter is kept as text. All other macros are left out.
  • XObject text areas with wiki content are handled the same way. Plain-text text areas and attachment text are indexed as text, and text areas with Velocity code are left out.
  • Existing chunks stay in the index until the collection is re-indexed. Re-indexing remains a manual decision of the administrator because of the embedding cost; the release notes will recommend re-indexing without cleaning.