Run the scheduler on a single cluster member and move it when that member leaves

Problem

In a cluster, every member with the scheduler enabled runs the same scheduler jobs, since all the members restore the jobs from the same job pages: each job runs once per member (for example the notification mails or the cleanup jobs). Since XWIKI-22855, the recommended workaround is to disable the scheduler (scheduler.enabled=false) on all the members but one. Then nothing runs the jobs while that member is down, and the scheduler administration is unusable on the other members.

Proposal

Use the cluster singleton node (XWIKI-25253) to run the scheduler on a single member and move it automatically:

• When remote observation is enabled and the scheduler is enabled, the scheduler is started and the jobs are restored from their pages only on the cluster singleton node, when the application is ready.
• When an instance becomes the singleton node (for example because the previous one stopped), it starts the scheduler and restores the jobs from their pages. Restoring is idempotent: the status stored in each job page decides.
• When an instance stops being the singleton node, its scheduler stops firing jobs and forgets them (running jobs are not interrupted). They are restored from the pages if the instance becomes the singleton node again.
• On a member where the scheduler is not running, the scheduler API answers from the job pages: the status is the one stored in the page and the next fire time is computed from the cron expression. Scheduling, pausing, resuming and unscheduling a job save the new status in the job page, and the singleton node applies it when it receives the document update event. This also applies the changes of status and cron expression made by editing the job page, which were ignored until the next restart.
• Triggering a job ("run now") is only possible on the member running the scheduler: nothing in the job page can ask the singleton node to execute it (see XWIKI-23249).
• scheduler.enabled keeps its meaning as a per-member switch: a member where it's false never runs the scheduler.
• When remote observation is disabled, the behaviour is unchanged: the instance starts its scheduler as soon as the plugin is initialized.

Limits

• While the singleton node changes, the previous and the new singleton node can both run the same job if their runs overlap within one cron period, so a script job can run twice. A job which fired on the previous singleton node shortly before it stopped is not run again. The notification mail job keeps its last fire time in the database, so the next run covers a run which was missed during the change.
• The previous fire time of a job is only known on the member running the scheduler.