This issue has been created
There are 2 updates.
 
 
LLM AI Integration / cid:jira-generated-image-avatar-c70826bd-2dc9-45fd-b0e0-63b9adb56208 LLMAI-234 Open

After a restart, indexed chunks are missing from search until Solr finishes replaying its transaction log

 
View issue   ยท   Add comment
 

Issue created

 
cid:jira-generated-image-avatar-02464a96-b70b-4e26-b20d-3fc0b67b711b Paul Pantiru created this issue on 18/Nov/26 10:35
 
Summary: After a restart, indexed chunks are missing from search until Solr finishes replaying its transaction log
Issue Type: cid:jira-generated-image-avatar-c70826bd-2dc9-45fd-b0e0-63b9adb56208 Bug
Affects Versions: 0.7
Assignee: Unassigned
Created: 18/Nov/26 10:35
Priority: cid:jira-generated-image-static-major-7b6b8528-7a5c-414b-b51c-447795a807ae Major
Reporter: Paul Pantiru
Description:

Problem

After XWiki is restarted, the chunks indexed since the previous start are missing from the index. Keyword and semantic search return nothing for them, and the chunk list of the collection is incomplete, until Solr has finished replaying its transaction log. Nothing in the UI says that a replay is running.

The data is not lost: the chunks come back when the replay ends. The duration grows with the amount indexed since the previous start. Measured on a local instance with the embedded Solr:

  • one restart: a 632 MB transaction log took 52 minutes to replay
  • another restart: the replay re-applied 85,665 added chunks and 11,085 delete-by-query operations, and the index showed 1,914 of 65,282 chunks while it ran

Cause

The indexing code never hard-commits the aillm Solr core:

  • SolrConnector.storeChunks and the delete-by-query methods send their update with a "commit within" of 10 seconds, then trigger a soft commit.
  • In Solr a "commit within" is a soft commit by default, and the core configuration provided by XWiki has no automatic commit.
  • A soft commit makes the changes searchable but does not write them to the index on disk, so they exist only in the transaction log.

The only hard commit is the one done when the core is initialized at startup. On XWiki 17.4 a clean shutdown does not commit either.

Expected

Indexed chunks are durably committed, so that a restart finds them in the index without a long replay.

Steps to reproduce

  1. Index a collection with a few thousand chunks.
  2. Restart XWiki.
  3. Search in that collection right after the start: no result until the replay ends (Log replay finished in the log).
 
 

2 updates

 
cid:jira-generated-image-avatar-02464a96-b70b-4e26-b20d-3fc0b67b711b Changes by Paul Pantiru on 18/Nov/26 10:35
 
Fix Version: 1.0.1
Assignee: Paul Pantiru