What happens SolrConnector.getSolrDocument writes the chunk text into two fields:
solrDocument.addField(AiLLMSolrCoreInitializer.FIELD_CONTENT, chunk.getContent());
solrDocument.addField(AiLLMSolrCoreInitializer.FIELD_CONTENT_INDEX, chunk.getContent());
Both fields are declared as text_general: content via addTextGeneralField(FIELD_CONTENT, false, false) in createSchema, and content_index via setTextGeneralField(FIELD_CONTENT_INDEX, false, false) in the migration to schema version 121000006. The migration's comment reads:
// Add another version of the text field, but indexed as regular text.
which suggests the intent was that content was not analysed as text. On a fresh install it is, so the two fields carry identical analysis of identical text. Why it matters No correctness impact – purely index size and indexing cost. The chunk text, which is the bulk of the index, is analysed and stored twice. It is also a small maintenance trap: a future reader of createSchema could reasonably conclude the two fields differ in analysis when they do not. Suggested fix Either drop content_index and search content directly, or make the two fields actually differ as the comment implies – e.g. content stored and not indexed, content_index indexed and not stored. Either way a schema migration and re-index would be needed, so this may be worth folding into the next change that already requires one. Noticed while investigating retrieval relevance in 0.9; filing separately as it is cosmetic relative to the other reports. |
