Notification filter preference cleanup events cannot be sent to the other cluster members

Finding from the Kubernetes test cluster (several XWiki members with remote observation enabled, JGroups).

What happens

When a page is deleted and some users have a notification filter preference on that exact page (for example the automatic watch a page creator gets), the member that deleted the page logs one error per user and page, in the Notification filters clean up queue thread:

ERROR .o.i.DefaultObservationManager - Failed to send event [org.xwiki.notifications.filters.event.CleaningFilterEvent@...] to listener [org.xwiki.observation.remote.internal.LocalEventListener@...]
java.lang.IllegalArgumentException: java.io.NotSerializableException: org.xwiki.notifications.filters.internal.DefaultNotificationFilterPreference
	at org.xwiki.observation.remote.internal.jgroups.JGroupsNetworkAdapter.send(...)
	at org.xwiki.observation.remote.internal.DefaultRemoteObservationManager.notify(...)
	at org.xwiki.observation.remote.internal.LocalEventListener.onEvent(...)
	...
	at org.xwiki.notifications.filters.internal.DeletedDocumentCleanUpFilterProcessingQueue.performCleanup(...)

Deleting about 200 watched pages produced close to 400 of these errors, each with its stack trace.

Why

DeletedDocumentCleanUpFilterProcessingQueue notifies the cancelable CleaningFilterEvent (added by XWIKI-21448) with, as data, the set of filter preferences about to be deleted. The event and the set are Serializable, so the generic serializable converter of the remote observation accepts the event for the cluster, but the DefaultNotificationFilterPreference instances inside the set are not serializable and building the network message fails.

Impact

• Only instances with remote observation enabled (cluster) are affected.
• The cleanup itself still works: the error is caught for that listener only, the filter preferences are deleted, and the other members still invalidate their cached filter preferences, because the deletion itself is announced by a separate event that is serializable and sent normally.
• The logs are flooded with errors and stack traces, which hides real problems, and a serialization attempt is wasted for each user and page.
• The event never reached the other members, and it would not be useful there: it is sent before the deletion so that a listener can cancel it, and cancelling it on another member cannot prevent the deletion.

Proposed fix

Keep CleaningFilterEvent local to the member that performs the cleanup, with an event converter that stops it from being sent to the cluster (as is already done for log events). Since the event never reached other members, this keeps the current behaviour and removes the errors.