There are 10 comments.
 
 
XWiki Docker images / cid:jira-generated-image-avatar-1f8a1495-eb65-4875-8264-63f253135d53 XDOCKER-321 Closed

Container based image 16.2.x postgres-tomcat crashes regularly after upgrade due to a Solr issue

 
View issue   ยท   Add comment
 

10 comments

 
cid:jira-generated-image-avatar-ec44952c-3587-4f8c-96c2-198148a5a33d Vincent Massol on 18/Jul/26 17:18
 
The diagnosis earlier in this thread is correct, and this is indeed a Docker \ -image fix \ (the XWiki/XJetty counterpart was already done in XWIKI \ -21708, 16.4.0 \ ).

The {{No space left on device}} message is misleading: the embedded Solr suggester
\ ({{LookupFactory}} \ ) \ - and various other libraries \ - write temporary files directly to {{java.io.tmpdir}}, which is not configurable at the Solr level. In a container that default location can be small or {{tmpfs}} \ -backed, so it fills up even when the mapped {{/usr/local/xwiki/data}} volume has plenty of free space. That matches the SOLR \ -10937 pattern.

Fix made in the image:
\ * {{tomcat/setenv.sh}} now sets {{ \ -Djava.io.tmpdir=/usr/local/xwiki/data/tmp}}, i.e. on the mapped permanent volume, which always has real disk space. It stays overridable through the {{ JAVA_OPTS JAVA\_OPTS }} environment variable.
\ * Because that directory lives on the persistent volume and nothing in the container reaps temporary files \ (there is no OS tmp \ -reaper, and the volume survives restarts \ ), {{xwiki/docker \ -entrypoint.sh}} now empties it on every container start, before Tomcat launches. This keeps it from growing without bound and mirrors what XWiki's own Environment does for its {{xwiki \ -temp}} directory.

Workaround on the current image until a new one is published
\ - create the directory once and set the JVM temp dir via JAVA_OPTS JAVA\_OPTS , e.g. in docker \ -compose:
{code}
environment:
  - JAVA_OPTS=-Djava.io.tmpdir=/usr/local/xwiki/data/tmp -Xmx1024m
{code}
\ (create {{/usr/local/xwiki/data/tmp}} on the volume first, since the JVM won't create it \ ). Could you confirm this resolves the crashes on your side?
 
cid:jira-generated-image-avatar-ec44952c-3587-4f8c-96c2-198148a5a33d Vincent Massol on 18/Jul/26 17:18
 
Correction after building and running the fix on 18.5.0 \ + PostgreSQL 18:

Setting {{java.io.tmpdir}} through {{
JAVA\_OPTS JAVA_OPTS }} \ (as I described in my previous comment \ ) does NOT work. Tomcat's {{catalina.sh}} appends {{ \ -Djava.io.tmpdir=$ CATALINA\_TMPDIR CATALINA_TMPDIR }} _after_ JAVA\_OPTS JAVA_OPTS , so with the JVM's "last \ -D wins" rule the default {{/usr/local/tomcat/temp}} still won \ - I confirmed the temp files kept landing there. Please disregard the {{ JAVA\_OPTS JAVA_OPTS = \ -Djava.io.tmpdir=...}} workaround from my earlier comment.

The correct fix, now implemented and verified, is to set {{
CATALINA\_TMPDIR CATALINA_TMPDIR }} in {{tomcat/setenv.sh}} \ (catalina.sh only defaults it when empty, so it wins \ ), plus the empty \ -on \ -start cleanup in {{xwiki/docker \ -entrypoint.sh}}. Verified on the running container:
\ * JVM command line shows a single {{ \ -Djava.io.tmpdir=/usr/local/xwiki/data/tmp}} \ (confirmed in Tomcat's own VersionLoggerListener log \ ).
\ * Temporary files \ (JODConverter, etc. \ ) now land in {{/usr/local/xwiki/data/tmp}} on the mapped volume, no longer in {{/usr/local/tomcat/temp}}.
\ * The directory is emptied on every container start \ (planted a marker file, restarted, it was gone \ ).
\ * XWiki boots normally \ (HTTP 200 \ ), no {{FileSystemException}} / "No space left on device".

Corrected workaround for the current image until a new one is published
\ - set {{ CATALINA\_TMPDIR CATALINA_TMPDIR }} \ (not java.io.tmpdir via JAVA\_OPTS\ JAVA_OPTS ) and pre \ -create the directory on the volume, e.g. in docker \ -compose:
{code}
environment:
  - CATALINA_TMPDIR=/usr/local/xwiki/data/tmp
{code}
\ (create {{/usr/local/xwiki/data/tmp}} on the volume first, since the JVM won't create it \ ). Could you confirm this resolves the crashes on your side?

 
cid:jira-generated-image-avatar-ec44952c-3587-4f8c-96c2-198148a5a33d Vincent Massol on 18/Jul/26 17:18
 
The diagnosis earlier in this thread is correct, and this is indeed a Docker \ -image fix \ (the XWiki/XJetty counterpart was already done in XWIKI \ -21708, 16.4.0 \ ).

The {{No space left on device}} message is misleading: the embedded Solr suggester
\ ({{LookupFactory}} \ ) \ - and various other libraries \ - write temporary files directly to {{java.io.tmpdir}}, which is not configurable at the Solr level. In a container that default location can be small or {{tmpfs}} \ -backed, so it fills up even when the mapped {{/usr/local/xwiki/data}} volume has plenty of free space. That matches the SOLR \ -10937 pattern.

Fix made in the image:
\ * {{tomcat/setenv.sh}} now sets {{ \ -Djava.io.tmpdir=/usr/local/xwiki/data/tmp}}, i.e. on the mapped permanent volume, which always has real disk space. It stays overridable through the {{ JAVA\_OPTS JAVA_OPTS }} environment variable.
\ * Because that directory lives on the persistent volume and nothing in the container reaps temporary files \ (there is no OS tmp \ -reaper, and the volume survives restarts \ ), {{xwiki/docker \ -entrypoint.sh}} now empties it on every container start, before Tomcat launches. This keeps it from growing without bound and mirrors what XWiki's own Environment does for its {{xwiki \ -temp}} directory.

Workaround on the current image until a new one is published
\ - create the directory once and set the JVM temp dir via JAVA\_OPTS JAVA_OPTS , e.g. in docker \ -compose:
{code}
environment:
  - JAVA_OPTS=-Djava.io.tmpdir=/usr/local/xwiki/data/tmp -Xmx1024m
{code}
\ (create {{/usr/local/xwiki/data/tmp}} on the volume first, since the JVM won't create it \ ). Could you confirm this resolves the crashes on your side?
 
cid:jira-generated-image-avatar-ec44952c-3587-4f8c-96c2-198148a5a33d Vincent Massol on 18/Jul/26 17:20
 
The diagnosis earlier in this thread is correct, and this is indeed a Docker \ -image fix \ (the XWiki/XJetty counterpart was already done in XWIKI \ -21708, 16.4.0 \ ).

The {{No space left on device}} message is misleading: the embedded Solr suggester
\ ({{LookupFactory}} \ ) \ - and various other libraries \ - write temporary files directly to {{java.io.tmpdir}}, which is not configurable at the Solr level. In a container that default location can be small or {{tmpfs}} \ -backed, so it fills up even when the mapped {{/usr/local/xwiki/data}} volume has plenty of free space. That matches the SOLR \ -10937 pattern.

Fix made in the image:
\ * {{tomcat/setenv.sh}} now sets {{ \ -Djava.io.tmpdir=/usr/local/xwiki/data/tmp}}, i.e. on the mapped permanent volume, which always has real disk space. It stays overridable through the {{ JAVA_OPTS JAVA\_OPTS }} environment variable.
\ * Because that directory lives on the persistent volume and nothing in the container reaps temporary files \ (there is no OS tmp \ -reaper, and the volume survives restarts \ ), {{xwiki/docker \ -entrypoint.sh}} now empties it on every container start, before Tomcat launches. This keeps it from growing without bound and mirrors what XWiki's own Environment does for its {{xwiki \ -temp}} directory.

Workaround on the current image until a new one is published
\ - create the directory once and set the JVM temp dir via JAVA_OPTS JAVA\_OPTS , e.g. in docker \ -compose:
{code}
environment:
  - JAVA_OPTS=-Djava.io.tmpdir=/usr/local/xwiki/data/tmp -Xmx1024m
{code}
\ (create {{/usr/local/xwiki/data/tmp}} on the volume first, since the JVM won't create it \ ). Could you confirm this resolves the crashes on your side?
 
cid:jira-generated-image-avatar-ec44952c-3587-4f8c-96c2-198148a5a33d Vincent Massol on 18/Jul/26 17:20
 
The diagnosis earlier in this thread is correct, and this is indeed a Docker \ -image fix \ (the XWiki/XJetty counterpart was already done in XWIKI \ -21708, 16.4.0 \ ).

The {{No space left on device}} message is misleading: the embedded Solr suggester
\ ({{LookupFactory}} \ ) \ - and various other libraries \ - write temporary files directly to {{java.io.tmpdir}}, which is not configurable at the Solr level. In a container that default location can be small or {{tmpfs}} \ -backed, so it fills up even when the mapped {{/usr/local/xwiki/data}} volume has plenty of free space. That matches the SOLR \ -10937 pattern.

Fix made in the image:
\ * {{tomcat/setenv.sh}} now sets {{ \ -Djava.io.tmpdir=/usr/local/xwiki/data/tmp}}, i.e. on the mapped permanent volume, which always has real disk space. It stays overridable through the {{ JAVA\_OPTS JAVA_OPTS }} environment variable.
\ * Because that directory lives on the persistent volume and nothing in the container reaps temporary files \ (there is no OS tmp \ -reaper, and the volume survives restarts \ ), {{xwiki/docker \ -entrypoint.sh}} now empties it on every container start, before Tomcat launches. This keeps it from growing without bound and mirrors what XWiki's own Environment does for its {{xwiki \ -temp}} directory.

Workaround on the current image until a new one is published
\ - create the directory once and set the JVM temp dir via JAVA\_OPTS JAVA_OPTS , e.g. in docker \ -compose:
{code}
environment:
  - JAVA_OPTS=-Djava.io.tmpdir=/usr/local/xwiki/data/tmp -Xmx1024m
{code}
\ (create {{/usr/local/xwiki/data/tmp}} on the volume first, since the JVM won't create it \ ). Could you confirm this resolves the crashes on your side?
 
cid:jira-generated-image-avatar-ec44952c-3587-4f8c-96c2-198148a5a33d Vincent Massol on 18/Jul/26 17:21
 
The diagnosis earlier in this thread is correct, and this is indeed a Docker \ -image fix \ (the XWiki/XJetty counterpart was already done in XWIKI \ -21708, 16.4.0 \ ).

The {{No space left on device}} message is misleading: the embedded Solr suggester
\ ({{LookupFactory}} \ ) \ - and various other libraries \ - write temporary files directly to {{java.io.tmpdir}}, which is not configurable at the Solr level. In a container that default location can be small or {{tmpfs}} \ -backed, so it fills up even when the mapped {{/usr/local/xwiki/data}} volume has plenty of free space. That matches the SOLR \ -10937 pattern.

Fix made in the image:
\ * {{tomcat/setenv.sh}} now sets {{ \ -Djava.io.tmpdir=/usr/local/xwiki/data/tmp}}, i.e. on the mapped permanent volume, which always has real disk space. It stays overridable through the {{ JAVA_OPTS JAVA\_OPTS }} environment variable.
\ * Because that directory lives on the persistent volume and nothing in the container reaps temporary files \ (there is no OS tmp \ -reaper, and the volume survives restarts \ ), {{xwiki/docker \ -entrypoint.sh}} now empties it on every container start, before Tomcat launches. This keeps it from growing without bound and mirrors what XWiki's own Environment does for its {{xwiki \ -temp}} directory.

Workaround on the current image until a new one is published
\ - create the directory once and set the JVM temp dir via JAVA_OPTS JAVA\_OPTS , e.g. in docker \ -compose:
{code}
environment:
  - JAVA_OPTS=-Djava.io.tmpdir=/usr/local/xwiki/data/tmp -Xmx1024m
{code}
\ (create {{/usr/local/xwiki/data/tmp}} on the volume first, since the JVM won't create it \ ). Could you confirm this resolves the crashes on your side?
 
cid:jira-generated-image-avatar-ec44952c-3587-4f8c-96c2-198148a5a33d Vincent Massol on 18/Jul/26 17:21
 
The diagnosis earlier in this thread is correct, and this is indeed a Docker \ -image fix \ (the XWiki/XJetty counterpart was already done in XWIKI \ -21708, 16.4.0 \ ).

The {{No space left on device}} message is misleading: the embedded Solr suggester
\ ({{LookupFactory}} \ ) \ - and various other libraries \ - write temporary files directly to {{java.io.tmpdir}}, which is not configurable at the Solr level. In a container that default location can be small or {{tmpfs}} \ -backed, so it fills up even when the mapped {{/usr/local/xwiki/data}} volume has plenty of free space. That matches the SOLR \ -10937 pattern.

Fix made in the image:
\ * {{tomcat/setenv.sh}} now sets {{ \ -Djava.io.tmpdir=/usr/local/xwiki/data/tmp}}, i.e. on the mapped permanent volume, which always has real disk space. It stays overridable through the {{ JAVA\_OPTS JAVA_OPTS }} environment variable.
\ * Because that directory lives on the persistent volume and nothing in the container reaps temporary files \ (there is no OS tmp \ -reaper, and the volume survives restarts \ ), {{xwiki/docker \ -entrypoint.sh}} now empties it on every container start, before Tomcat launches. This keeps it from growing without bound and mirrors what XWiki's own Environment does for its {{xwiki \ -temp}} directory.

Workaround on the current image until a new one is published
\ - create the directory once and set the JVM temp dir via JAVA\_OPTS JAVA_OPTS , e.g. in docker \ -compose:
{code}
environment:
  - JAVA_OPTS=-Djava.io.tmpdir=/usr/local/xwiki/data/tmp -Xmx1024m
{code}
\ (create {{/usr/local/xwiki/data/tmp}} on the volume first, since the JVM won't create it \ ). Could you confirm this resolves the crashes on your side?
 
cid:jira-generated-image-avatar-ec44952c-3587-4f8c-96c2-198148a5a33d Vincent Massol on 18/Jul/26 17:21
 
The diagnosis earlier in this thread is correct, and this is indeed a Docker \ -image fix \ (the XWiki/XJetty counterpart was already done in XWIKI \ -21708, 16.4.0 \ ).

The {{No space left on device}} message is misleading: the embedded Solr suggester
\ ({{LookupFactory}} \ ) \ - and various other libraries \ - write temporary files directly to {{java.io.tmpdir}}, which is not configurable at the Solr level. In a container that default location can be small or {{tmpfs}} \ -backed, so it fills up even when the mapped {{/usr/local/xwiki/data}} volume has plenty of free space. That matches the SOLR \ -10937 pattern.

Fix made in the image:
\ * {{tomcat/setenv.sh}} now sets {{ \ -Djava.io.tmpdir=/usr/local/xwiki/data/tmp}}, i.e. on the mapped permanent volume, which always has real disk space. It stays overridable through the {{ JAVA\_OPTS JAVA_OPTS }} environment variable.
\ * Because that directory lives on the persistent volume and nothing in the container reaps temporary files \ (there is no OS tmp \ -reaper, and the volume survives restarts \ ), {{xwiki/docker \ -entrypoint.sh}} now empties it on every container start, before Tomcat launches. This keeps it from growing without bound and mirrors what XWiki's own Environment does for its {{xwiki \ -temp}} directory.

Workaround on the current image until a new one is published
\ - create the directory once and set the JVM temp dir via JAVA\_OPTS JAVA_OPTS , e.g. in docker \ -compose:
{code}
environment:
  - JAVA_OPTS=-Djava.io.tmpdir=/usr/local/xwiki/data/tmp -Xmx1024m
{code}
\ (create {{/usr/local/xwiki/data/tmp}} on the volume first, since the JVM won't create it \ ). Could you confirm this resolves the crashes on your side?
 
cid:jira-generated-image-avatar-ec44952c-3587-4f8c-96c2-198148a5a33d Vincent Massol on 18/Jul/26 17:21
 
The diagnosis earlier in this thread is correct, and this is indeed a Docker \ -image fix \ (the XWiki/XJetty counterpart was already done in XWIKI \ -21708, 16.4.0 \ ).

The {{No space left on device}} message is misleading: the embedded Solr suggester
\ ({{LookupFactory}} \ ) \ - and various other libraries \ - write temporary files directly to {{java.io.tmpdir}}, which is not configurable at the Solr level. In a container that default location can be small or {{tmpfs}} \ -backed, so it fills up even when the mapped {{/usr/local/xwiki/data}} volume has plenty of free space. That matches the SOLR \ -10937 pattern.

Fix made in the image:
\ * {{tomcat/setenv.sh}} now sets {{ \ -Djava.io.tmpdir=/usr/local/xwiki/data/tmp}}, i.e. on the mapped permanent volume, which always has real disk space. It stays overridable through the {{ JAVA_OPTS JAVA\_OPTS }} environment variable.
\ * Because that directory lives on the persistent volume and nothing in the container reaps temporary files \ (there is no OS tmp \ -reaper, and the volume survives restarts \ ), {{xwiki/docker \ -entrypoint.sh}} now empties it on every container start, before Tomcat launches. This keeps it from growing without bound and mirrors what XWiki's own Environment does for its {{xwiki \ -temp}} directory.

Workaround on the current image until a new one is published
\ - create the directory once and set the JVM temp dir via JAVA_OPTS JAVA\_OPTS , e.g. in docker \ -compose:
{code}
environment:
  - JAVA_OPTS=-Djava.io.tmpdir=/usr/local/xwiki/data/tmp -Xmx1024m
{code}
\ (create {{/usr/local/xwiki/data/tmp}} on the volume first, since the JVM won't create it \ ). Could you confirm this resolves the crashes on your side?
 
cid:jira-generated-image-avatar-ec44952c-3587-4f8c-96c2-198148a5a33d Vincent Massol on 18/Jul/26 17:23
 
The diagnosis earlier in this thread is correct, and this is indeed a Docker \ -image fix \ (the XWiki/XJetty counterpart was already done in XWIKI \ -21708, 16.4.0 \ ).

The {{No space left on device}} message is misleading: the embedded Solr suggester
\ ({{LookupFactory}} \ ) \ - and various other libraries \ - write temporary files directly to {{java.io.tmpdir}}, which is not configurable at the Solr level. In a container that default location can be small or {{tmpfs}} \ -backed, so it fills up even when the mapped {{/usr/local/xwiki/data}} volume has plenty of free space. That matches the SOLR \ -10937 pattern.

Fix made in the image:
\ * {{tomcat/setenv.sh}} now sets {{ \ -Djava.io.tmpdir=/usr/local/xwiki/data/tmp}}, i.e. on the mapped permanent volume, which always has real disk space. It stays overridable through the {{ JAVA\_OPTS JAVA_OPTS }} environment variable.
\ * Because that directory lives on the persistent volume and nothing in the container reaps temporary files \ (there is no OS tmp \ -reaper, and the volume survives restarts \ ), {{xwiki/docker \ -entrypoint.sh}} now empties it on every container start, before Tomcat launches. This keeps it from growing without bound and mirrors what XWiki's own Environment does for its {{xwiki \ -temp}} directory.

Workaround on the current image until a new one is published
\ - create the directory once and set the JVM temp dir via JAVA\_OPTS JAVA_OPTS , e.g. in docker \ -compose:
{code}
environment:
  - JAVA_OPTS=-Djava.io.tmpdir=/usr/local/xwiki/data/tmp -Xmx1024m
{code}
\ (create {{/usr/local/xwiki/data/tmp}} on the volume first, since the JVM won't create it \ ). Could you confirm this resolves the crashes on your side?