[xwiki-notifications] [xwiki-contrib/ai-llm] 0d27f8: LLMAI-79: Improve chunking by taking newlines and ...

Branch: refs/heads/main Home: https://github.com/xwiki-contrib/ai-llm Commit: 0d27f8ad69176a6b45375249e7b2b2240af10528 https://github.com/xwiki-contrib/ai-llm/commit/0d27f8ad69176a6b45375249e7b2… Author: Michael Hamann <michael.hamann(a)xwiki.com> Date: 2024-07-01 (Mon, 01 Jul 2024) Changed paths: M application-ai-llm-index/application-ai-llm-index-api/src/main/java/org/xwiki/contrib/llm/ChunkingUtils.java A application-ai-llm-index/application-ai-llm-index-api/src/test/java/org/xwiki/contrib/llm/ChunkingUtilsTest.java Log Message: ----------- LLMAI-79: Improve chunking by taking newlines and headings into account * Allow half the chunk size and half the overlap to find a better splitting point, searching for headings, empty lines, newlines, paragraphs and words until a suitable splitting point has been found. * Add a unit test for chunking. To unsubscribe from these emails, change your notification settings at https://github.com/xwiki-contrib/ai-llm/settings/notifications