Events and notifications stay disabled until restart when the remote Solr server cannot be reached during startup

Symptom

With a remote Solr server (solr.type=remote), when XWiki starts while the Solr server cannot be reached yet (Solr still starting, or on Kubernetes a Solr Service which has no ready endpoint yet), an error is logged once during the startup and then, until XWiki is restarted, on that instance:

• no event is recorded: the activity stream and the notifications miss everything done on that instance,
• notifications are not dispatched to the users,

even when the Solr server becomes available a few seconds later. Nothing else is logged afterwards.

Cause

• The event listeners are registered once, when the observation manager is used for the first time, early in the startup. A listener which cannot be looked up at that moment is skipped for the life of the instance (an error naming the listener, with the cause, is logged).
• Several listeners depend on the event store: RecordableEventListener, DocumentEventListener, UserDeletedEventListener, and UserEventDispatcherListener through the notification dispatcher. SolrEventStore gets the client of the events core in its initialize(), and RemoteSolr requests the server information in its initialize(). When Solr cannot be reached, these initializations fail, and so does the lookup of each of these listeners.
• The same happens when a core is still loading on the Solr side (XWIKI-23723).

Fix

• RemoteSolr no longer contacts the Solr server when it is initialized: the server information is requested the first time a core is requested, and requested again on the next call when it failed.
• SolrEventStore resolves the events core the first time it needs it instead of in initialize(), and again on the next task or query when it failed or when the core was not found. While Solr cannot be reached, saving an event fails (the error is reported through the returned future, as before), and the store works as soon as Solr answers.