What happens XWikiDocumentDocument.getContent(), which supplies the text that is chunked and embedded, uses the raw document source:
formatHeading(1, this.xWikiDocument.getRenderedTitle(this.xWikiDocument.getSyntax(), context))
+ this.xWikiDocument.getContent()
xWikiDocument.getContent() returns the unrendered source, so macro syntax, HTML, inline CSS style attributes, and image and link syntax all reach the embedder verbatim. Note that getTitle(), immediately above in the same class, does render, via Syntax.PLAIN_1_0 – so the inconsistency is within one class. Why it matters Markup is embedded as if it were prose. That dilutes the vector for the page's actual content, and it creates chunks that are almost entirely noise yet still occupy a result slot in every search. Observed A chunk retrieved and cited for an unrelated question consists almost entirely of Dark Reader style attributes that had been injected into the page source:
Other retrieved chunks carried velocity and box macro syntax and image link syntax. Suggested fix Render the document to Syntax.PLAIN_1_0 before chunking, the way getTitle() already does. Existing collections would need a re-index to benefit, which the application already supports. |
