What happens SolrConnector.keywordSearch builds its query as:
query.setQuery("%s:%s".formatted(AiLLMSolrCoreInitializer.FIELD_CONTENT_INDEX,
this.solrUtils.toCompleteFilterQueryString(textQuery)));
SolrUtils.toCompleteFilterQueryString runs ClientUtils.escapeQueryChars, which escapes whitespace. The whole question therefore arrives at the classic query parser as a single escaped term, is handed to the field analyzer, and is split into one token per word. With autoGeneratePhraseQueries unset (default false for luceneMatchVersion >= 6) and no q.op configured, those tokens are combined with the default OR operator into a BooleanQuery of SHOULD clauses ranked by BM25. Two things make this worse than it sounds:
- XWiki's text_general is created programmatically in AbstractSolrCoreInitializer.addTextGeneralFieldType() as StandardTokenizerFactory + LowerCaseFilterFactory and nothing else – there is no stopword filter. So "how", "do", "i", "and", "my", "at" are all real query terms.
- there is no mm / minimum-should-match, so a chunk matching only function words is a valid hit.
Why it matters The chat filter feeds the user's raw message into this path, and chat messages are natural-language questions. The result is that the keyword half reliably spends its whole budget on chunks that match only common words. Keyword search is meant to complement vector search on rare, exact terms; here it mostly injects noise. Observed Query "How do I install zsh and oh my zsh at rise?", keyword-only, limit 5 – 4 of the 5 returned chunks contain no occurrence of "zsh" anywhere; they match on install, at, my, rise. Scores 5.70 / 5.39 / 5.36 / 5.33 against 21.63 for the one genuinely relevant chunk. By contrast the same path with the single rare term zsh returns exactly one hit, the correct one – so the field and analyzer are fine. It is specifically multi-word input that breaks. Suggested fix Parse the query as a query rather than escaping it into one term: use edismax with qf=content_index and a sensible mm (e.g. 2<70%), optionally with pf so phrase proximity is boosted. A stopword filter on the field would help too, but mm is the part that stops function-word-only matches from consuming the result budget. |
