Every cluster member sends the same index updates to a remote Solr server

Problem

When several XWiki instances run as a cluster (remote observation enabled) and share a remote Solr server (solr.type=remote), every cluster member updates the Solr index for every change, including the changes made on the other members. Saving one page on one member makes each of the N members index that page (the document and its translations, then each changed object, object property and attachment) and send the same update to the shared Solr cores. Deleting a page, an object or an attachment is sent N times too.

Consequences:

• the indexing work, on the Solr server and on each XWiki instance, grows with the number of cluster members, for no benefit;
• the members process their indexing queues at different times, so the same entity receives interleaved delete and add requests from several members;
• adding a language to the available locales of a wiki (wiki preferences) makes every member re-index all the documents of the parent locales.

How to observe it

1. Set up two XWiki instances with remote observation and a shared remote Solr server (solr.type=remote, solr.remote.baseURL).
2. Read the UPDATE.updateHandler.cumulativeAdds metric of the search core, for example /solr/admin/metrics?group=core&prefix=UPDATE.updateHandler.cumulativeAdds.
3. Save a page on one instance, wait for the indexing queues to be empty and read the metric again.

Expected: the metric grows by the number of Solr documents of one indexing of the page.
Actual: it grows by that number multiplied by the number of cluster members.

Cause

SolrIndexEventListener and SolrIndexAvailableLocalesListener handle every event, including the events received from the other cluster members. This is needed with the embedded Solr, where each member owns its index, but a remote Solr index is shared and the member where the change happened already updates it.

Proposed fix

When solr.type is remote, ignore the remote events in both listeners (the locales listener still refreshes its cache of the available locales) and keep the current behaviour with the embedded Solr. This is the approach used for XWIKI-15113.

An index update still waiting in the queue of the member where the change happened is lost if that member stops before processing it (the queue is in memory); today another member happens to send the same update, after the fix the next index synchronization repairs it.