This issue has been created
 
 
XWiki Platform / cid:jira-generated-image-avatar-51c48322-5fbd-4e63-93a2-d89ec056d5e0 XWIKI-25297 Open

Prevent several cluster members from migrating the database at the same time

 
View issue   ยท   Add comment
 

Issue created

 
cid:jira-generated-image-avatar-391503d2-3389-495b-9c22-5f156f1f24cc Claude created this issue on 14/Oct/26 19:41
 
Summary: Prevent several cluster members from migrating the database at the same time
Issue Type: cid:jira-generated-image-avatar-51c48322-5fbd-4e63-93a2-d89ec056d5e0 Improvement
Affects Versions: 18.8.0
Assignee: Unassigned
Components: Old Core, Storage
Created: 14/Oct/26 19:41
Labels: cluster
Priority: cid:jira-generated-image-static-major-68c272ad-d742-468c-9ea0-df99178db52f Major
Reporter: Claude
Description:

The database schema updates and data migrations run when a cluster member opens its first transaction, and they are only synchronized inside that member's JVM. When several members start with migrations enabled (for example after an upgrade, or when they are started at the same time), each of them runs the Hibernate schema update at the same time and runs every data migration, so a data migration can be applied twice. The same happens with an empty database, which every member tries to initialize.
In addition, a member with migrations disabled (the usual setup for all but one member) refuses an outdated database and keeps refusing it until it's restarted, even once another member has migrated the database.
Finding from a Kubernetes test cluster (two threads of one member started the migrations within a few milliseconds), and from reading the code for the rest.
Proposal:

  • the member which has something to do (empty database to initialize, or migrations enabled) takes a lock of the database server (PostgreSQL advisory lock, MySQL/MariaDB named lock) held on a dedicated connection, reads the database version again once it holds it, and skips what another member did meanwhile; the database server releases the lock if the member dies;
  • the other members wait for the lock (as long as needed by default, with a log every minute, or at most store.migration.lockTimeout seconds);
  • databases without such a lock keep the current behaviour, with a warning when remote observation is enabled;
  • a member which does not migrate reads the version of an outdated database again (at most every 10 seconds) instead of refusing it until it restarts.