[xwiki-notifications] [xwiki-contrib/ai-llm] abb962: LLMAI-114: Duplicate chunks lead to errors and can...

Branch: refs/heads/main Home: https://github.com/xwiki-contrib/ai-llm Commit: abb9623a0c6b9eac1a83196cfdcfa6c7b24c8460 https://github.com/xwiki-contrib/ai-llm/commit/abb9623a0c6b9eac1a83196cfdcf… Author: Michael Hamann <michael.hamann(a)xwiki.com> Date: 2024-11-26 (Tue, 26 Nov 2024) Changed paths: M application-ai-llm-index/application-ai-llm-index-api/src/main/java/org/xwiki/contrib/llm/internal/DocumentIndexer.java Log Message: ----------- LLMAI-114: Duplicate chunks lead to errors and cannot be updated * Explicitly handle duplicate chunks from the index. Commit: b80090928b5097763acbdacaddb463ef9dd4ef06 https://github.com/xwiki-contrib/ai-llm/commit/b80090928b5097763acbdacaddb4… Author: Michael Hamann <michael.hamann(a)xwiki.com> Date: 2024-11-26 (Tue, 26 Nov 2024) Changed paths: M application-ai-llm-index/application-ai-llm-index-api/src/main/java/org/xwiki/contrib/llm/ChunkingUtils.java M application-ai-llm-index/application-ai-llm-index-api/src/main/java/org/xwiki/contrib/llm/SolrConnector.java Log Message: ----------- LLMAI-115: Improve the indexing speed * Don't use a regular expression for finding the last heading, just iterate over the matches of the heading pattern. * Don't commit the index after every update. Instead, use a soft commit and let Solr commit after 10 seconds. * Use a query filter instead of a query for finding existing documents. Compare: https://github.com/xwiki-contrib/ai-llm/compare/1ffb476d3a81...b80090928b50 To unsubscribe from these emails, change your notification settings at https://github.com/xwiki-contrib/ai-llm/settings/notifications