What happens RAGChatRequestFilter.extractURLsAndformat walks the entire searchResults list and emits every unique document URL:
for (Context result : searchResults) {
String sourceURL = result.url();
if (!addedUrls.contains(sourceURL)) {
sourcesBuilder.append(String.format(SOURCE_STRING, sourceURL));
addedUrls.add(sourceURL);
}
}
chatWidget.js then splits that string on newlines and lists every entry. So "Sources" is the raw retrieval set with duplicate URLs collapsed – it has no relationship to which chunks the model actually drew on. Why it matters With the default budget the user can see up to 10 sources for a question answered entirely from one page. It reads as though the answer synthesised ten documents. That devalues the citation list – users cannot tell a precise retrieval from a noisy one, and cannot tell which link to follow to verify the answer. Observed A correct answer drawn from a single page was presented with 8 sources, 7 of them unrelated: 5 semantic + 5 keyword = 10 chunks, 1 removed as a duplicate content match, leaving 9 chunks over 8 distinct URLs. Suggested fix The prompt already numbers the chunks – DEFAULT_CHUNK_TEMPLATE emits a document element with an index attribute. So the cleanest option is to ask the model to cite those indices and render only the cited ones. Cheaper alternatives: show the similarity score next to each source, or list only sources above a relevance threshold, or visually separate "used" from "also retrieved". |
