This issue has been created
There is 1 update.
 
 
XWiki Platform / cid:jira-generated-image-avatar-5bc6ff74-6254-4dce-ab1d-d9f924a1bf84 XWIKI-23419 Open

Notification pre-filtering is slow with lots of users due to lots of Solr queries
 
View issue   ยท   Add comment
 

Issue created

 
cid:jira-generated-image-avatar-0f208f93-8356-4135-aa49-afbfbcc7c15b Michael Hamann created this issue on 28/Jul/25 17:08
 
Summary: Notification pre-filtering is slow with lots of users due to lots of Solr queries
Issue Type: cid:jira-generated-image-avatar-5bc6ff74-6254-4dce-ab1d-d9f924a1bf84 Bug
Affects Versions: 16.10.0
Assignee: Unassigned
Components: Notifications
Created: 28/Jul/25 17:08
Labels: performance
Priority: cid:jira-generated-image-static-major-8ccdab08-7c67-4b23-8655-433362fbc45e Major
Reporter: Michael Hamann
Description:

Notification pre-filtering can get slow on wikis with ten thousands of users. This is noticeable when there are lots of events (not clear how many) and notifications aren't appearing anymore.

Checking thread dumps shows that the "User event dispatcher thread" frequently shows stack traces like the following in a thread dump:

"User event dispatcher thread" #291
   java.lang.Thread.State: RUNNABLE
        at java.util.TreeMap.getEntry(TreeMap.java:357)
        at java.util.TreeMap.get(TreeMap.java:279)
        at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.terms(PerFieldPostingsFormat.java:353)
        at org.apache.lucene.index.CodecReader.terms(CodecReader.java:132)
        at org.apache.lucene.index.FilterLeafReader.terms(FilterLeafReader.java:415)
        at org.apache.lucene.index.Terms.getTerms(Terms.java:41)
        at org.apache.lucene.index.TermStates.loadTermsEnum(TermStates.java:109)
        at org.apache.lucene.index.TermStates.get(TermStates.java:167)
        at org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:186)
        at org.apache.lucene.search.TermQuery$TermWeight.scorerSupplier(TermQuery.java:122)
        at org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:515)
        at org.apache.lucene.search.Weight.bulkScorer(Weight.java:165)
        at org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:368)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:759)
        at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:720)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:549)
        at org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:275)
        at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1871)
        at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1707)
        at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:710)
        at org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1696)
        at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:422)
        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:467)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:226)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2908)
        at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:225)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:234)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:926)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:939)
        at org.xwiki.eventstream.store.solr.internal.SolrEventStore.search(SolrEventStore.java:843)
        at org.xwiki.eventstream.store.solr.internal.SolrEventStore.search(SolrEventStore.java:833)
        at org.xwiki.eventstream.internal.DefaultEventStore.search(DefaultEventStore.java:356)
        at org.xwiki.notifications.notifiers.internal.UserEventDispatcher.isPrefiltered(UserEventDispatcher.java:324)
        at org.xwiki.notifications.notifiers.internal.UserEventDispatcher.isMailPrefiltered(UserEventDispatcher.java:309)
        at org.xwiki.notifications.notifiers.internal.UserEventDispatcher.dispatch(UserEventDispatcher.java:288)
        at org.xwiki.notifications.notifiers.internal.UserEventDispatcher.dispatch(UserEventDispatcher.java:338)
        at org.xwiki.notifications.notifiers.internal.UserEventDispatcher.dispatchInContext(UserEventDispatcher.java:259)
        at org.xwiki.notifications.notifiers.internal.UserEventDispatcher.dispatch(UserEventDispatcher.java:216)
        at org.xwiki.notifications.notifiers.internal.UserEventDispatcher.prefilterEvent(UserEventDispatcher.java:191)
        at org.xwiki.notifications.notifiers.internal.UserEventDispatcher.flush(UserEventDispatcher.java:175)
        at org.xwiki.notifications.notifiers.internal.UserEventDispatcherScheduler.run(UserEventDispatcherScheduler.java:124)
        at org.xwiki.notifications.notifiers.internal.UserEventDispatcherScheduler$$Lambda$2398/0x00007f9765827720.run()
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.lang.Thread.run(Thread.java:840)

An analysis of the code shows that these Solr searches are only there to avoid checking notification preferences when an event has already been pre-filtered and assigned to the user. In almost all cases, this query won't find any results and thus we'll check the user's notification preferences, anyway. It seems far more likely that most users aren't receiving a notification for the event and thus we should first check if the user is receiving a notification before checking if we have maybe already stored the user. That can only happen when XWiki was pre-filtering the event while it was restarted, so this should be super rare.
 
 

1 update

 
cid:jira-generated-image-avatar-0f208f93-8356-4135-aa49-afbfbcc7c15b Changes by Michael Hamann on 28/Jul/25 17:09
 
Assignee: Michael Hamann
 
 
This message was sent by Atlassian Jira (v9.3.0#930000-sha1:287aeb6) Atlassian logo
If image attachments aren't displayed, see this article.