ActiveInstallsIT#verifyActiveInstalls is flickering

Cause of the CI failures

All CI failures of this test in the last 90 days (7 out of 1815 executions, according to Develocity) are the SocketTimeoutException on client.indices().refresh() at ActiveInstallsIT.java:70. They started on 2026-10-12, before that there were 0 failures in 1718 executions.

• They only happen on stable-16.10.x (1), stable-17.10.x (5) and stable-18.4.x (1), never on master (551 executions), stable-18.8.x or stable-18.9.x, with any browser, database and servlet container.
• These are exactly the branches that don't contain ecdd1ef0015 ("improve test so that it doesn't fail if the docker disk is almost full", 2026-06-09), which sets cluster.routing.allocation.disk.threshold_enabled=false on the Elasticsearch test container.
• By default Elasticsearch doesn't allocate shards on a node whose disk is above the high watermark (90%). The ping then creates the installs3 index, but its primary shard stays unassigned, so the index request and the test's refresh() hang until the 30s client timeout, and searches fail with 503 no_shard_available_action_exception with "node":null. The XWiki log of the failing CI run (stable-18.4.x #155) shows exactly this sequence.
• The 7 failures ran on 4 different agents (docker-a5, a6, a8, a10), so this looks like the disks of the agents filling up rather than one broken agent. This last part isn't confirmed yet, the disk usage metrics of the agents still need to be checked.

Reproduction

• Elasticsearch 8.14.3 alone, with the settings of the test and the watermark forced to 2%, replaying the requests of the ping and the test with curl: the index request and _refresh hang for 30s, searches return the same 503 as on CI, and the allocation explain API reports "the node is above the high watermark cluster setting". With threshold_enabled=false added, everything succeeds immediately.
• Full IT on stable-18.4.x at the commit of build #155, with the watermark forced to 2%: fails with the CI symptom (SocketTimeoutException at ActiveInstallsIT.java:70). With ecdd1ef0015 cherry-picked on top: passes. With only ecdd1ef0015 (no forcing): passes.

Fix

ecdd1ef0015 is now backported to stable-16.10.x (53909cda99a) and stable-17.10.x (bcd36593933). The backport to stable-18.4.x is ready and will be pushed once the 18.4.7 release is done.

Still open: the local index_not_found_exception

The index_not_found_exception on the first count() call (line 75) that you got locally hasn't happened on CI in the last 90 days, and I couldn't find a failure of this test in build #317 (the 3 configurations that ran it passed). It's a different symptom: with a full disk, refresh() would already hang on line 70. It means that the ping hadn't been indexed at all when the test started. Treating a missing index as count 0 would make the test retry, but it only helps if the ping arrives later. If the ping can't reach Elasticsearch from the servlet container at all in your setup, the test would still fail, just with "AS ping wasn't sent". The XWiki log of such a local run (the Active Installs 2 Ping Thread warnings) should tell which case it is.