This issue has been created
There is 1 update.
 
 
XWiki Platform / cid:jira-generated-image-avatar-4e9df036-117c-4ac2-b06f-a500659d0578 XWIKI-22820 Open

Solr field cache uses a lot of RAM

 
View issue   ยท   Add comment
 

Issue created

 
cid:jira-generated-image-avatar-b074b512-d03e-498f-864e-8d5708b952e8 Michael Hamann created this issue on 27/Jan/25 10:48
 
Summary: Solr field cache uses a lot of RAM
Issue Type: cid:jira-generated-image-avatar-4e9df036-117c-4ac2-b06f-a500659d0578 Bug
Affects Versions: 15.10
Assignee: Unassigned
Components: Search - Solr
Created: 27/Jan/25 10:48
Priority: cid:jira-generated-image-static-major-ebc0a61f-daa1-4775-beea-4e3f4633754d Major
Reporter: Michael Hamann
Description:

Steps to reproduce:

  1. Have a huge number of documents indexed (like a million documents) with a modest amount of RAM (like 6 GB) and internal Solr.
  2. Perform some searches with a lot of results (like *, searching for all documents)

Expected result:

The wiki is still working and most memory is still available to XWiki.

Actual result:

The wiki crashes because of an OOM error and most memory is consumed by Solr's field cache. This can be analyzed with the following Groovy snippet:

{{groovy wiki="false"}}
import org.apache.solr.common.params.ModifiableSolrParams;

def solr = services.component.getInstance(org.xwiki.search.solr.internal.api.SolrInstance)
ModifiableSolrParams params = new ModifiableSolrParams();
params.set("group", "core");
params.set("prefix", "CACHE.core.fieldCache");
params.set("qt", "/admin/metrics");

solr.query(params)
{{/groovy}}

In an instance, it reported 5.3 GiB total size. More information about the problems of this cache can be found at https://support.hcl-software.com/community?id=community_blog&sys_id=ddedbfb11be58914a2f48661cd4bcbd8

 
 

1 update

 
cid:jira-generated-image-avatar-b074b512-d03e-498f-864e-8d5708b952e8 Changes by Michael Hamann on 27/Jan/25 10:50
 
Description: *Steps to reproduce:*

# Have a huge number of documents indexed (like a million documents) with a modest amount of RAM (like 6 GB) and internal Solr.
# Perform some searches with a lot of results (like {{*}}, searching for all documents)

*Expected result:*

The wiki is still working and most memory is still available to XWiki.

*Actual result:*

The wiki crashes because of an OOM error and most memory is consumed by Solr's field cache. This can be analyzed with the following Groovy snippet:

{noformat}
{{groovy wiki="false"}}
import org.apache.solr.common.params.ModifiableSolrParams;

def solr = services.component.getInstance(org.xwiki.search.solr.internal.api.SolrInstance)
ModifiableSolrParams params = new ModifiableSolrParams();
params.set("group", "core");
params.set("prefix", "CACHE.core.fieldCache");
params.set("qt", "/admin/metrics");

solr.query(params)
{{/groovy}}
{noformat}

In an instance, it reported 5.3 GiB total size. More information about the problems of this cache can be found at https://support.hcl-software.com/community?id=community_blog&sys_id=ddedbfb11be58914a2f48661cd4bcbd8


In the concrete example, the fields used the following amounts of RAM:

{noformat}
id: 5.02 GB
space_facet: 446.97 MB
space_exact: 266.94 MB
class: 95.76 MB
attauthor: 95.76 MB
mimetype: 95.70 MB
title_sort: 81.11 MB
name_exact: 44.72 MB
creator: 27.56 MB
author: 27.56 MB
locale: 15.09 MB
doclocale: 15.01 MB
type: 8.96 MB
wiki: 5.84 MB
{noformat}