[xwiki-devs] [proposal] Make filesystem attachments use permanent storage directory instead of work dir.
Hi, I'd like to switch filesystem attachments to begin using the persistent storage directory now instead of the work directory. This means there's a new way of calculating where the attachments will be stored so it might fail on upgrade. I would like to not do any migration and just add to the release notes because: #1, it doesn't cause any permanent harm so long as nobody adds attachments while it's in what is an obviously broken state. #2 administrators who have FS attachments enabled are probably going to know what's going on. #3 migration code is scary, it requires lots of work and lots of review and even if it works, people might feel violated having files shuffled around on their system without their permission. WDYT? Caleb
On 02/16/2012 05:07 AM, Caleb James DeLisle wrote:
Hi,
I'd like to switch filesystem attachments to begin using the persistent storage directory now instead of the work directory. This means there's a new way of calculating where the attachments will be stored so it might fail on upgrade. I would like to not do any migration and just add to the release notes because: #1, it doesn't cause any permanent harm so long as nobody adds attachments while it's in what is an obviously broken state. #2 administrators who have FS attachments enabled are probably going to know what's going on. #3 migration code is scary, it requires lots of work and lots of review and even if it works, people might feel violated having files shuffled around on their system without their permission.
WDYT?
+1 for changing the location. I'm slightly in favor of writing a migrator. -- Sergiu Dumitriu http://purl.org/net/sergiu/
On Thu, Feb 16, 2012 at 11:07, Caleb James DeLisle <[email protected]
wrote:
Hi,
I'd like to switch filesystem attachments to begin using the persistent storage directory now instead of the work directory. This means there's a new way of calculating where the attachments will be stored so it might fail on upgrade. I would like to not do any migration and just add to the release notes because: #1, it doesn't cause any permanent harm so long as nobody adds attachments while it's in what is an obviously broken state. #2 administrators who have FS attachments enabled are probably going to know what's going on. #3 migration code is scary, it requires lots of work and lots of review and even if it works, people might feel violated having files shuffled around on their system without their permission.
WDYT?
+1 for this in 4.0M1 Putting this late in the 3.5 cycle risk to cause less attention to the release notes, and more mistake. What is really bad, is that the permanent directory is not defaulted to the work directory. So, without attention, it will surely get broken in most cases. On the other hand, I agree that moving file on the filesystem is the job of the admin, and having this done almost silently is not good. If attachements are large and the permanent directory is not setup correctly, this could cause disk write or disk full issue and could be worse than the above issues. -- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO
On Thu, Feb 16, 2012 at 11:07 AM, Caleb James DeLisle <[email protected]> wrote:
Hi,
I'd like to switch filesystem attachments to begin using the persistent storage directory now instead of the work directory. This means there's a new way of calculating where the attachments will be stored so it might fail on upgrade. I would like to not do any migration and just add to the release notes because: #1, it doesn't cause any permanent harm so long as nobody adds attachments while it's in what is an obviously broken state. #2 administrators who have FS attachments enabled are probably going to know what's going on. #3 migration code is scary, it requires lots of work and lots of review and even if it works, people might feel violated having files shuffled around on their system without their permission.
WDYT?
+1 and +1 for no migration by default too. If admin did his job properly, work dir and permanent dir are supposed to be the same anyway and if the permanent dir is not set then admin just need to set it and it OK.
Caleb
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne
On Thu, Feb 16, 2012 at 11:40, Thomas Mortagne <[email protected]>wrote:
On Thu, Feb 16, 2012 at 11:07 AM, Caleb James DeLisle <[email protected]> wrote:
Hi,
I'd like to switch filesystem attachments to begin using the persistent storage directory now instead of the work directory. This means there's a new way of calculating where the attachments will be stored so it might fail on upgrade. I would like to not do any migration and just add to the release notes because: #1, it doesn't cause any permanent harm so long as nobody adds attachments while it's in what is an obviously broken state. #2 administrators who have FS attachments enabled are probably going to know what's going on. #3 migration code is scary, it requires lots of work and lots of review and even if it works, people might feel violated having files shuffled around on their system without their permission.
WDYT?
+1 and +1 for no migration by default too. If admin did his job properly, work dir and permanent dir are supposed to be the same
I do not fully agree. My work dir has always been been stored in a place like /var/cache, where the data is not backup and has not real importance (I was not using filesystem attachement obviously). This was the default on a debian install. On the other hand, permanentDirectory should be in /var/lib, and be backed up. This is my setup right now with the patch of Caleb.
anyway and if the permanent dir is not set then admin just need to set it and it OK.
Caleb
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO
On Thu, Feb 16, 2012 at 2:44 PM, Denis Gervalle <[email protected]> wrote:
On Thu, Feb 16, 2012 at 11:40, Thomas Mortagne <[email protected]>wrote:
On Thu, Feb 16, 2012 at 11:07 AM, Caleb James DeLisle <[email protected]> wrote:
Hi,
I'd like to switch filesystem attachments to begin using the persistent storage directory now instead of the work directory. This means there's a new way of calculating where the attachments will be stored so it might fail on upgrade. I would like to not do any migration and just add to the release notes because: #1, it doesn't cause any permanent harm so long as nobody adds attachments while it's in what is an obviously broken state. #2 administrators who have FS attachments enabled are probably going to know what's going on. #3 migration code is scary, it requires lots of work and lots of review and even if it works, people might feel violated having files shuffled around on their system without their permission.
WDYT?
+1 and +1 for no migration by default too. If admin did his job properly, work dir and permanent dir are supposed to be the same
I do not fully agree. My work dir has always been been stored in a place like /var/cache, where the data is not backup and has not real importance (I was not using filesystem attachement obviously). This was the default on a debian install. On the other hand, permanentDirectory should be in /var/lib, and be backed up. This is my setup right now with the patch of Caleb.
xwiki.work.dir was already supposed to be used for persisted datas and there was xwiki.temp.dir for temporary stuff. What's in xwiki.properties is just a new component oriented access to the same thing and nothing new. Anyway you would have use it properly if you were using filesystem attachment so it does not really change the need for a migration.
anyway and if the permanent dir is not set then admin just need to set it and it OK.
Caleb
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne
On Thu, Feb 16, 2012 at 15:07, Thomas Mortagne <[email protected]>wrote:
On Thu, Feb 16, 2012 at 2:44 PM, Denis Gervalle <[email protected]> wrote:
On Thu, Feb 16, 2012 at 11:40, Thomas Mortagne < [email protected]>wrote:
On Thu, Feb 16, 2012 at 11:07 AM, Caleb James DeLisle <[email protected]> wrote:
Hi,
I'd like to switch filesystem attachments to begin using the persistent storage directory now instead of the work directory. This means there's a new way of calculating where the attachments will be stored so it might fail on upgrade. I would like to not do any migration and just add to the release notes because: #1, it doesn't cause any permanent harm so long as nobody adds attachments while it's in what is an obviously broken state. #2 administrators who have FS attachments enabled are probably going to know what's going on. #3 migration code is scary, it requires lots of work and lots of review and even if it works, people might feel violated having files shuffled around on their system without their permission.
WDYT?
+1 and +1 for no migration by default too. If admin did his job properly, work dir and permanent dir are supposed to be the same
I do not fully agree. My work dir has always been been stored in a place like /var/cache, where the data is not backup and has not real importance (I was not using filesystem attachement obviously). This was the default on a debian install. On the other hand, permanentDirectory should be in /var/lib, and be backed up. This is my setup right now with the patch of Caleb.
xwiki.work.dir was already supposed to be used for persisted datas and there was xwiki.temp.dir for temporary stuff.
The default of the first, is the second however ! And, AFAIK, apart from the attachment storage, what was stored there where mainly data that could be freely deleted.
What's in xwiki.properties is just a new component oriented access to the same thing and nothing new.
Anyway you would have use it properly if you were using filesystem attachment so it does not really change the need for a migration.
Sure.
anyway and if the permanent dir is not set then admin just need to set it and it OK.
Caleb
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO
On Thu, Feb 16, 2012 at 4:41 PM, Denis Gervalle <[email protected]> wrote:
On Thu, Feb 16, 2012 at 15:07, Thomas Mortagne <[email protected]>wrote:
On Thu, Feb 16, 2012 at 2:44 PM, Denis Gervalle <[email protected]> wrote:
On Thu, Feb 16, 2012 at 11:40, Thomas Mortagne < [email protected]>wrote:
On Thu, Feb 16, 2012 at 11:07 AM, Caleb James DeLisle <[email protected]> wrote:
Hi,
I'd like to switch filesystem attachments to begin using the persistent storage directory now instead of the work directory. This means there's a new way of calculating where the attachments will be stored so it might fail on upgrade. I would like to not do any migration and just add to the release notes because: #1, it doesn't cause any permanent harm so long as nobody adds attachments while it's in what is an obviously broken state. #2 administrators who have FS attachments enabled are probably going to know what's going on. #3 migration code is scary, it requires lots of work and lots of review and even if it works, people might feel violated having files shuffled around on their system without their permission.
WDYT?
+1 and +1 for no migration by default too. If admin did his job properly, work dir and permanent dir are supposed to be the same
I do not fully agree. My work dir has always been been stored in a place like /var/cache, where the data is not backup and has not real importance (I was not using filesystem attachement obviously). This was the default on a debian install. On the other hand, permanentDirectory should be in /var/lib, and be backed up. This is my setup right now with the patch of Caleb.
xwiki.work.dir was already supposed to be used for persisted datas and there was xwiki.temp.dir for temporary stuff.
The default of the first, is the second however !
It's just a fallback to always provide something and permanentDirectory has the exact same behavior...
And, AFAIK, apart from the attachment storage, what was stored there where mainly data that could be freely deleted.
No, xwiki.work.dir always been used for datas you wanted to keep after a restard, the fact that it was not too critical to loose Lucene index does not make it ok to rebuild the whole index all the time especially for a big farm.
What's in xwiki.properties is just a new component oriented access to the same thing and nothing new.
Anyway you would have use it properly if you were using filesystem attachment so it does not really change the need for a migration.
Sure.
anyway and if the permanent dir is not set then admin just need to set it and it OK.
Caleb
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne
On Thu, Feb 16, 2012 at 16:57, Thomas Mortagne <[email protected]>wrote:
On Thu, Feb 16, 2012 at 4:41 PM, Denis Gervalle <[email protected]> wrote:
On Thu, Feb 16, 2012 at 15:07, Thomas Mortagne < [email protected]>wrote:
On Thu, Feb 16, 2012 at 2:44 PM, Denis Gervalle <[email protected]> wrote:
On Thu, Feb 16, 2012 at 11:40, Thomas Mortagne < [email protected]>wrote:
On Thu, Feb 16, 2012 at 11:07 AM, Caleb James DeLisle <[email protected]> wrote:
Hi,
I'd like to switch filesystem attachments to begin using the persistent storage directory now instead of the work directory. This means there's a new way of calculating where the attachments will be stored so it might fail on upgrade. I would like to not do any migration and just add to the release notes because: #1, it doesn't cause any permanent harm so long as nobody adds attachments while it's in what is an obviously broken state. #2 administrators who have FS attachments enabled are probably going to know what's going on. #3 migration code is scary, it requires lots of work and lots of review and even if it works, people might feel violated having files shuffled around on their system without their permission.
WDYT?
+1 and +1 for no migration by default too. If admin did his job properly, work dir and permanent dir are supposed to be the same
I do not fully agree. My work dir has always been been stored in a place like /var/cache, where the data is not backup and has not real importance (I was not using filesystem attachement obviously). This was the default on a debian install. On the other hand, permanentDirectory should be in /var/lib, and be backed up. This is my setup right now with the patch of Caleb.
xwiki.work.dir was already supposed to be used for persisted datas and there was xwiki.temp.dir for temporary stuff.
The default of the first, is the second however !
It's just a fallback to always provide something and permanentDirectory has the exact same behavior...
And, AFAIK, apart from the attachment storage, what was stored there where mainly data that could be freely deleted.
No, xwiki.work.dir always been used for datas you wanted to keep after a restard, the fact that it was not too critical to loose Lucene index does not make it ok to rebuild the whole index all the time especially for a big farm.
From FHS standard:
"/var/cache is intended for cached data from applications. Such data is locally generated as a result of time-consuming I/O or calculation. The application must be able to regenerate or restore the data." This exactly what I am talking about, and by default on debian, work dir goes there. But this one does not fit attachment, which should go to: "/var/lib : Variable state information. State information is generally used to preserve the condition of an application (or a group of inter-related applications) between invocations and between different instances of the same application." So even if the permanent directory was not intended that way, it fill the gap allow this configuration.
What's in xwiki.properties is just a new component oriented access to the same thing and nothing new.
Anyway you would have use it properly if you were using filesystem attachment so it does not really change the need for a migration.
Sure.
anyway and if the permanent dir is not set then admin just need to
set
it and it OK.
Caleb
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO
participants (4)
-
Caleb James DeLisle -
Denis Gervalle -
Sergiu Dumitriu -
Thomas Mortagne