[xwiki-devs] [Brainstorming] Use cases for an alternative filesystem format (alternative to XAR)
27 Aug
2016
27 Aug
'16
1:01 p.m.
Hi, This is a follow-up on the threads: * "Designing the perfect FS representation of a wiki”: http://markmail.org/message/3yghqwetmdt5woez * "XAR source projects should allow source files”: http://markmail.org/message/432o36r4klh7yv24 It’s also a continuation of the work done here: http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiAppl... Once we get convergence on those thread (or even if we don’t), I’ll update design.xwiki.org with the results. The goal is to define the use case for an alternate filesystem to XAR. Issues with the XAR format ====================== * XML is not an easy to edit format and doesn’t allow use a specific editor to edit content * XML also requires content to be XML-encoded and thus is really not easy to make modification (there’s a risk of breaking the XML easily) Can you see more issues? Use cases for an alternative filesystem format =================================== (some UC taken from http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiAppl...) * UC1: the structure should be (as) easy (as possible) to navigate in an IDE style view * UC2: it should be easy to add content (a new script or attachment on an existing structure). It should allow using specific editors for different content types, e.g. if a page content is in markdown, it should be editable with a MD editor, js and css should be editable with web editors, etc. * UC3: It should be possible to build a packaged version of the sources with Maven * UC4: It should be possible to import the packaged version into a running XWiki instance * UC5: It should be possible to export a portion of a running XWiki instance in this format * UC6: This format should be able to fully replace the XAR format . The new format should support at least all features supported by the XAR format (versioned, etc). Note: XE will need to be refactor a bit so that the XAR format can be swapped out by introducing extension points/APIs. The idea would be to deprecate the XAR format and introduce this new format instead, and the 2 formats should be avle to cohabit next to each other in XWiki. * UC7: When importing in a wiki and exporting again (without making any change in the wiki), it should generate an identical structure and content, with no difference. Any more? Thanks -Vincent
27 Aug
27 Aug
1:44 p.m.
New subject: [xwiki-devs] [Brainstorming] Use cases for an alternative filesystem format (alternative to XAR)
Thanks for having this extra thread. I think this thread is much more important than starting to design something already.
Issues with the XAR format ======================
* XML is not an easy to edit format and doesn’t allow use a specific editor to edit content * XML also requires content to be XML-encoded and thus is really not easy to make modification (there’s a risk of breaking the XML easily) I completely disagree with these two statements. XML is easy to edit and is supported by very very many editors and IDEs. It can also be validated. XML can be written in a very elegant and readable fashion if you care for it. Generally however, XML is produced form other structured information in a way that does not help readability. Can you see more issues? The problem is how we put *everything* into XML. (you get the same horror if you use Caleb's tools xardump and do not tune anything: the resulting javascript is horrible.) Use cases for an alternative filesystem format ===================================
(some UC taken from http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiAppl...)
* UC1: the structure should be (as) easy (as possible) to navigate in an IDE style view * UC2: it should be easy to add content (a new script or attachment on an existing structure). It should allow using specific editors for different content types, e.g. if a page content is in markdown, it should be editable with a MD editor, js and css should be editable with web editors, etc. UC2.1: Attachments should also be present as standalone files. * UC3: It should be possible to build a packaged version of the sources with Maven * UC4: It should be possible to import the packaged version into a running XWiki instance * UC5: It should be possible to export a portion of a running XWiki instance in this format * UC6: This format should be able to fully replace the XAR format . The new format should support at least all features supported by the XAR format (versioned, etc). Note: XE will need to be refactor a bit so that the XAR format can be swapped out by introducing extension points/APIs. The idea would be to deprecate the XAR format and introduce this new format instead, and the 2 formats should be avle to cohabit next to each other in XWiki. * UC7: When importing in a wiki and exporting again (without making any change in the wiki), it should generate an identical structure and content, with no difference. I do believe that UC7 is not doable in full generality.
Any more? UC8: the core representation should be using a syntax that is widely spread and completely specified (i.e. we should not invent another syntax for this) UC9: the system should make an archival process a widespread practice, in the form of zipped files probably. UC10: developers should have the discretion to decide when a component needs to be in a separate file or not. That is, small text fragments and even small attachments should be includable within bigger files UC11: there should be the possibility to share content of a file between several files or components (e.g. creator elements) UC12: (transformation) simple search and replace operation should be supported by the build mechanism, especially when dependencies are followed. UC13: it would be good that the format can be specified by a grammer against which one can validate (e.g. RelaxNG) Paul (FYI UC10, UC11, and UC12 follow the architecture recommendation to be composable vs contextual so as to give us greater flexibility)
2:46 p.m.
On 27 Aug 2016, at 15:44, Paul Libbrecht <[email protected]> wrote:
Thanks for having this extra thread. I think this thread is much more important than starting to design something already.
Issues with the XAR format ======================
* XML is not an easy to edit format and doesn’t allow use a specific editor to edit content * XML also requires content to be XML-encoded and thus is really not easy to make modification (there’s a risk of breaking the XML easily) I completely disagree with these two statements.
I probably didn’t use the right words because that’s the reason why I think that's you started the xinclude proposal :) AFAIK you started the xinclude proposal because you wanted to be able to edit content with specific editor (js, css, etc)… which is exactly what the second part of the first point is about. Let me rephrase the first sentence: * XML is not an easy to edit format for human beings (it’s very verbose and easy to make mistakes: missing encoding, etc). It’s also very hard to edit with a plain text editor. As for the 2nd sentence, I don’t see how you can disagree since it’s a fact...
XML is easy to edit and is supported by very very many editors and IDEs. It can also be validated.
What you’re saying is very theoretical. The practice (that we’ve had for 10 years of using the XAR format) is that our XML format that is hard to edit and can break easily (as was proven numerous times by our committers and contributors). It’s actually so dangerous that we’ve had to develop a process which goes like this: * Never edit the xml directly * Always import it into a running XWiki instance, make the modifications there and export from the wiki into XAR * Then unpack the XAR into the source directory and run mvn xar:format to go back to the original format. Nobody is using a pure XML editor with validation. We are all using Java IDEA (IntelliJ IDEA, Eclipse, etc) and they all allow you to edit XML as plain text and that’s what we’re doing. No developer I know of is using an XML editor in a structured way (just too painful and complex to navigate the structure). (see below)
XML can be written in a very elegant and readable fashion if you care for it. Generally however, XML is produced form other structured information in a way that does not help readability.
Can you see more issues? The problem is how we put *everything* into XML. (you get the same horror if you use Caleb's tools xardump and do not tune anything: the resulting javascript is horrible.) Use cases for an alternative filesystem format ===================================
(some UC taken from http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiAppl...)
* UC1: the structure should be (as) easy (as possible) to navigate in an IDE style view * UC2: it should be easy to add content (a new script or attachment on an existing structure). It should allow using specific editors for different content types, e.g. if a page content is in markdown, it should be editable with a MD editor, js and css should be editable with web editors, etc. UC2.1: Attachments should also be present as standalone files. * UC3: It should be possible to build a packaged version of the sources with Maven * UC4: It should be possible to import the packaged version into a running XWiki instance * UC5: It should be possible to export a portion of a running XWiki instance in this format * UC6: This format should be able to fully replace the XAR format . The new format should support at least all features supported by the XAR format (versioned, etc). Note: XE will need to be refactor a bit so that the XAR format can be swapped out by introducing extension points/APIs. The idea would be to deprecate the XAR format and introduce this new format instead, and the 2 formats should be avle to cohabit next to each other in XWiki. * UC7: When importing in a wiki and exporting again (without making any change in the wiki), it should generate an identical structure and content, with no difference. I do believe that UC7 is not doable in full generality.
Why not, this is what we have with the XAR format actually and UC7 is actually already contained in UC6 (but it’s better to be explicit)?
Any more?
UC8: the core representation should be using a syntax that is widely spread and completely specified (i.e. we should not invent another syntax for this)
This is not a requirement for me. The syntax should be easy to write into, especially using a plain text editor. YAML for example is a perfectly valid syntax for me.
UC9: the system should make an archival process a widespread practice, in the form of zipped files probably.
That’s UC3 for me. I hesitated to put ZIP in the requirement for UC3 since I didn’t want to limit us to that but it’s probably going to be zip anyway.
UC10: developers should have the discretion to decide when a component needs to be in a separate file or not. That is, small text fragments and even small attachments should be includable within bigger files
I don’t agree with this one. I’d like convention over flexibility, i.e. a fixed format on which tools can easily build upon. This is similar to Maven vs Ant. This is why in my proposal for a format I’ve proposed fixed file names. Allowing discretion means everyone will do it differently and we’ll need to define best practices and best practices are hard to enforce and always cause problems.
UC11: there should be the possibility to share content of a file between several files or components (e.g. creator elements)
I wouldn’t phrase it this way. I’d prefer to say that it should be possible to apply default values to missing elements in metadata. The way I see this for example for the format I’ve proposed in the other thread, is by having default properties that can be put on the filesystem, for example in default.properties file) so that when an element is missing the default would be used. Now I’m not sure we want this requirement and for me it’s an optional requirement and not a mandatory one. It makes it much harder to develop an exporter.
UC12: (transformation) simple search and replace operation should be supported by the build mechanism, especially when dependencies are followed.
Could you explain more, I don’t understand?
UC13: it would be good that the format can be specified by a grammer against which one can validate (e.g. RelaxNG)
I don’t agree in the way it’s phrased since it’s too limiting. I’d change it to: it should be possible for the packager tool to validate the content (what xar:verify does right now but that could be extended to verify that the required metadata are defined). I don’t think we need a formal grammar. The important part is that we have validation. Thanks -Vincent
Paul
(FYI UC10, UC11, and UC12 follow the architecture recommendation to be composable vs contextual so as to give us greater flexibility)
3:50 p.m.
On 27 Aug 2016, at 16:46, Vincent Massol <[email protected]> wrote:
On 27 Aug 2016, at 15:44, Paul Libbrecht <[email protected]> wrote:
Thanks for having this extra thread. I think this thread is much more important than starting to design something already.
Issues with the XAR format ======================
* XML is not an easy to edit format and doesn’t allow use a specific editor to edit content * XML also requires content to be XML-encoded and thus is really not easy to make modification (there’s a risk of breaking the XML easily) I completely disagree with these two statements.
I probably didn’t use the right words because that’s the reason why I think that's you started the xinclude proposal :)
AFAIK you started the xinclude proposal because you wanted to be able to edit content with specific editor (js, css, etc)… which is exactly what the second part of the first point is about.
Let me rephrase the first sentence:
* XML is not an easy to edit format for human beings (it’s very verbose and easy to make mistakes: missing encoding, etc). It’s also very hard to edit with a plain text editor.
As for the 2nd sentence, I don’t see how you can disagree since it’s a fact...
XML is easy to edit and is supported by very very many editors and IDEs. It can also be validated.
What you’re saying is very theoretical. The practice (that we’ve had for 10 years of using the XAR format) is that our XML format that is hard to edit and can break easily (as was proven numerous times by our committers and contributors). It’s actually so dangerous that we’ve had to develop a process which goes like this: * Never edit the xml directly * Always import it into a running XWiki instance, make the modifications there and export from the wiki into XAR * Then unpack the XAR into the source directory and run mvn xar:format to go back to the original format.
Nobody is using a pure XML editor with validation. We are all using Java IDEA (IntelliJ IDEA, Eclipse, etc) and they all allow you to edit XML as plain text and that’s what we’re doing. No developer I know of is using an XML editor in a structured way (just too painful and complex to navigate the structure).
To be more specific, the main issue we’ve had is contributors/committers who committed unencoded content, such as && instead of && or > instead of > Now, to be accurate, if you remove the problem with the encoding (which can be removed IMO with CDATA) then we never touch much of the rest of the metadata. In practice it would be nice if most of it could be generated by the maven plugin. In practice we don’t need much specific data (for a pure doc, it’s a bit more for xclass/xobjects): title, reference, syntax, parent, hidden, and language/translation. Syntax and reference could be computed from the directory structure. Parent could too. And hidden could default to visible by default. That said it doesn’t matter that much since the process is to export from a running xwiki instance (we need that to validate that it works at execution time, or we’d need good unit/integration tests for pages). So once we take the content out, the format of the metadata doesn’t matter that much probably since we’re not going to author it from scratch anyway (it’ll come from exported wiki pages). So I guess XML, even though very verbose, could still be ok. But XML for doc content or xproperty textareas, or attachments is a sure no go. Thanks -Vincent
(see below)
XML can be written in a very elegant and readable fashion if you care for it. Generally however, XML is produced form other structured information in a way that does not help readability.
Can you see more issues? The problem is how we put *everything* into XML. (you get the same horror if you use Caleb's tools xardump and do not tune anything: the resulting javascript is horrible.) Use cases for an alternative filesystem format ===================================
(some UC taken from http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiAppl...)
* UC1: the structure should be (as) easy (as possible) to navigate in an IDE style view * UC2: it should be easy to add content (a new script or attachment on an existing structure). It should allow using specific editors for different content types, e.g. if a page content is in markdown, it should be editable with a MD editor, js and css should be editable with web editors, etc. UC2.1: Attachments should also be present as standalone files. * UC3: It should be possible to build a packaged version of the sources with Maven * UC4: It should be possible to import the packaged version into a running XWiki instance * UC5: It should be possible to export a portion of a running XWiki instance in this format * UC6: This format should be able to fully replace the XAR format . The new format should support at least all features supported by the XAR format (versioned, etc). Note: XE will need to be refactor a bit so that the XAR format can be swapped out by introducing extension points/APIs. The idea would be to deprecate the XAR format and introduce this new format instead, and the 2 formats should be avle to cohabit next to each other in XWiki. * UC7: When importing in a wiki and exporting again (without making any change in the wiki), it should generate an identical structure and content, with no difference. I do believe that UC7 is not doable in full generality.
Why not, this is what we have with the XAR format actually and UC7 is actually already contained in UC6 (but it’s better to be explicit)?
Any more?
UC8: the core representation should be using a syntax that is widely spread and completely specified (i.e. we should not invent another syntax for this)
This is not a requirement for me. The syntax should be easy to write into, especially using a plain text editor. YAML for example is a perfectly valid syntax for me.
UC9: the system should make an archival process a widespread practice, in the form of zipped files probably.
That’s UC3 for me. I hesitated to put ZIP in the requirement for UC3 since I didn’t want to limit us to that but it’s probably going to be zip anyway.
UC10: developers should have the discretion to decide when a component needs to be in a separate file or not. That is, small text fragments and even small attachments should be includable within bigger files
I don’t agree with this one. I’d like convention over flexibility, i.e. a fixed format on which tools can easily build upon. This is similar to Maven vs Ant.
This is why in my proposal for a format I’ve proposed fixed file names.
Allowing discretion means everyone will do it differently and we’ll need to define best practices and best practices are hard to enforce and always cause problems.
UC11: there should be the possibility to share content of a file between several files or components (e.g. creator elements)
I wouldn’t phrase it this way. I’d prefer to say that it should be possible to apply default values to missing elements in metadata.
The way I see this for example for the format I’ve proposed in the other thread, is by having default properties that can be put on the filesystem, for example in default.properties file) so that when an element is missing the default would be used.
Now I’m not sure we want this requirement and for me it’s an optional requirement and not a mandatory one. It makes it much harder to develop an exporter.
UC12: (transformation) simple search and replace operation should be supported by the build mechanism, especially when dependencies are followed.
Could you explain more, I don’t understand?
UC13: it would be good that the format can be specified by a grammer against which one can validate (e.g. RelaxNG)
I don’t agree in the way it’s phrased since it’s too limiting. I’d change it to: it should be possible for the packager tool to validate the content (what xar:verify does right now but that could be extended to verify that the required metadata are defined). I don’t think we need a formal grammar. The important part is that we have validation.
Thanks -Vincent
Paul
(FYI UC10, UC11, and UC12 follow the architecture recommendation to be composable vs contextual so as to give us greater flexibility)
4:37 p.m.
New subject: [xwiki-devs] [Brainstorming] Use cases for an alternative filesystem format (alternative to XAR)
Vincent, I think we disagree on two major aspects which are kind of related: - writing document structure and metadata should be doable by hand editing as opposed to writing into the wiki alone: - I believe this is a fundamental aspect of source editing and should be supported (hence the wish to a validatable widespread syntax) - you seem to indicate that the current process you've agreed upon has discouraged it (but, at the same time, you indicate you wish to do it for YAML) - writing XML can be done by hand and is easy: - I am a strong defender of this: all editors I've worked with (IntelliJ IDEA, jEdit, BBEdit, XCode, maybe not vim) support XML very well so that it's just careless to commit a && (not to mention that mvn would not validate it, and would not build it if with my patch, simply because it parses). Clearly, editing in a textbox can get you there, same with MS Word! - you seem to say that bad things happen with such commits as not even valid XML documents and seem to motivate that to move to another format (I feel YAML a zillion less predictable than XML because of whitespace handling btw). Paul PS: when I say editing XML is for sane people, I don't mean everything should be XML, which is why I proposed the XAR extension to include XInclude, textinclude and binaryInclude. PPS: another, maybe deeper, disagreement might be the need to round-trip (server, source) vs the need to source-edit: hand-tuning the format of a file (e.g. in XML or in JSON) is easily in contradiction with a server generation from a structured source. Maybe this is the hard bone. > Vincent Massol <mailto:[email protected]> > 27 August 2016 at 17:50 >> On 27 Aug 2016, at 16:46, Vincent Massol <[email protected]> wrote: >> >>> On 27 Aug 2016, at 15:44, Paul Libbrecht <[email protected]> wrote: >>> >>> Thanks for having this extra thread. >>> I think this thread is much more important than starting to design >>> something already. >>>> Issues with the XAR format >>>> ====================== >>>> >>>> * XML is not an easy to edit format and doesn’t allow use a specific >>>> editor to edit content >>>> * XML also requires content to be XML-encoded and thus is really not >>>> easy to make modification (there’s a risk of breaking the XML easily) >>> I completely disagree with these two statements. >> I probably didn’t use the right words because that’s the reason why I think that's you started the xinclude proposal :) >> >> AFAIK you started the xinclude proposal because you wanted to be able to edit content with specific editor (js, css, etc)… which is exactly what the second part of the first point is about. >> >> Let me rephrase the first sentence: >> >> * XML is not an easy to edit format for human beings (it’s very verbose and easy to make mistakes: missing encoding, etc). It’s also very hard to edit with a plain text editor. >> >> As for the 2nd sentence, I don’t see how you can disagree since it’s a fact... >> >>> XML is easy to edit and is supported by very very many editors and IDEs. >>> It can also be validated. >> What you’re saying is very theoretical. The practice (that we’ve had for 10 years of using the XAR format) is that our XML format that is hard to edit and can break easily (as was proven numerous times by our committers and contributors). It’s actually so dangerous that we’ve had to develop a process which goes like this: >> * Never edit the xml directly >> * Always import it into a running XWiki instance, make the modifications there and export from the wiki into XAR >> * Then unpack the XAR into the source directory and run mvn xar:format to go back to the original format. >> >> Nobody is using a pure XML editor with validation. We are all using Java IDEA (IntelliJ IDEA, Eclipse, etc) and they all allow you to edit XML as plain text and that’s what we’re doing. No developer I know of is using an XML editor in a structured way (just too painful and complex to navigate the structure). > > To be more specific, the main issue we’ve had is contributors/committers who committed unencoded content, such as && instead of && or > instead of > > > Now, to be accurate, if you remove the problem with the encoding (which can be removed IMO with CDATA) then we never touch much of the rest of the metadata. > > In practice it would be nice if most of it could be generated by the maven plugin. In practice we don’t need much specific data (for a pure doc, it’s a bit more for xclass/xobjects): title, reference, syntax, parent, hidden, and language/translation. Syntax and reference could be computed from the directory structure. Parent could too. And hidden could default to visible by default. That said it doesn’t matter that much since the process is to export from a running xwiki instance (we need that to validate that it works at execution time, or we’d need good unit/integration tests for pages). > > So once we take the content out, the format of the metadata doesn’t matter that much probably since we’re not going to author it from scratch anyway (it’ll come from exported wiki pages). > > So I guess XML, even though very verbose, could still be ok. But XML for doc content or xproperty textareas, or attachments is a sure no go. > > Thanks > -Vincent > >> (see below) >> >>> XML can be written in a very elegant and readable fashion if you care >>> for it. >>> Generally however, XML is produced form other structured information in >>> a way that does not help readability. >>>> Can you see more issues? >>> The problem is how we put *everything* into XML. >>> (you get the same horror if you use Caleb's tools xardump and do not >>> tune anything: the resulting javascript is horrible.) >>>> Use cases for an alternative filesystem format >>>> =================================== >>>> >>>> (some UC taken from >>>> http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications) >>>> >>>> * UC1: the structure should be (as) easy (as possible) to navigate in >>>> an IDE style view >>>> * UC2: it should be easy to add content (a new script or attachment on >>>> an existing structure). It should allow using specific editors for >>>> different content types, e.g. if a page content is in markdown, it >>>> should be editable with a MD editor, js and css should be editable >>>> with web editors, etc. >>> UC2.1: Attachments should also be present as standalone files. >>>> * UC3: It should be possible to build a packaged version of the >>>> sources with Maven >>>> * UC4: It should be possible to import the packaged version into a >>>> running XWiki instance >>>> * UC5: It should be possible to export a portion of a running XWiki >>>> instance in this format >>>> * UC6: This format should be able to fully replace the XAR format . >>>> The new format should support at least all features supported by the >>>> XAR format (versioned, etc). Note: XE will need to be refactor a bit >>>> so that the XAR format can be swapped out by introducing extension >>>> points/APIs. The idea would be to deprecate the XAR format and >>>> introduce this new format instead, and the 2 formats should be avle to >>>> cohabit next to each other in XWiki. >>>> * UC7: When importing in a wiki and exporting again (without making >>>> any change in the wiki), it should generate an identical structure and >>>> content, with no difference. >>> I do believe that UC7 is not doable in full generality. >> Why not, this is what we have with the XAR format actually and UC7 is actually already contained in UC6 (but it’s better to be explicit)? >> >>> Any more? >>> >>> UC8: the core representation should be using a syntax that is widely >>> spread and completely specified (i.e. we should not invent another >>> syntax for this) >> This is not a requirement for me. The syntax should be easy to write into, especially using a plain text editor. YAML for example is a perfectly valid syntax for me. >> >>> UC9: the system should make an archival process a widespread practice, >>> in the form of zipped files probably. >> That’s UC3 for me. I hesitated to put ZIP in the requirement for UC3 since I didn’t want to limit us to that but it’s probably going to be zip anyway. >> >>> UC10: developers should have the discretion to decide when a component >>> needs to be in a separate file or not. That is, small text fragments and >>> even small attachments should be includable within bigger files >> I don’t agree with this one. I’d like convention over flexibility, i.e. a fixed format on which tools can easily build upon. This is similar to Maven vs Ant. >> >> This is why in my proposal for a format I’ve proposed fixed file names. >> >> Allowing discretion means everyone will do it differently and we’ll need to define best practices and best practices are hard to enforce and always cause problems. >> >>> UC11: there should be the possibility to share content of a file between >>> several files or components (e.g. creator elements) >> I wouldn’t phrase it this way. I’d prefer to say that it should be possible to apply default values to missing elements in metadata. >> >> The way I see this for example for the format I’ve proposed in the other thread, is by having default properties that can be put on the filesystem, for example in default.properties file) so that when an element is missing the default would be used. >> >> Now I’m not sure we want this requirement and for me it’s an optional requirement and not a mandatory one. It makes it much harder to develop an exporter. >> >>> UC12: (transformation) simple search and replace operation should be >>> supported by the build mechanism, especially when dependencies are followed. >> Could you explain more, I don’t understand? >> >>> UC13: it would be good that the format can be specified by a grammer >>> against which one can validate (e.g. RelaxNG) >> I don’t agree in the way it’s phrased since it’s too limiting. I’d change it to: it should be possible for the packager tool to validate the content (what xar:verify does right now but that could be extended to verify that the required metadata are defined). I don’t think we need a formal grammar. The important part is that we have validation. >> >> Thanks >> -Vincent >> >>> Paul >>> >>> (FYI UC10, UC11, and UC12 follow the architecture recommendation to be >>> composable vs contextual so as to give us greater flexibility) > > _______________________________________________ > devs mailing list > [email protected] > http://lists.xwiki.org/mailman/listinfo/devs > Vincent Massol <mailto:[email protected]> > 27 August 2016 at 16:46 >> On 27 Aug 2016, at 15:44, Paul Libbrecht <[email protected]> wrote: >> >> Thanks for having this extra thread. >> I think this thread is much more important than starting to design >> something already. >>> Issues with the XAR format >>> ====================== >>> >>> * XML is not an easy to edit format and doesn’t allow use a specific >>> editor to edit content >>> * XML also requires content to be XML-encoded and thus is really not >>> easy to make modification (there’s a risk of breaking the XML easily) >> I completely disagree with these two statements. > > I probably didn’t use the right words because that’s the reason why I think that's you started the xinclude proposal :) > > AFAIK you started the xinclude proposal because you wanted to be able to edit content with specific editor (js, css, etc)… which is exactly what the second part of the first point is about. > > Let me rephrase the first sentence: > > * XML is not an easy to edit format for human beings (it’s very verbose and easy to make mistakes: missing encoding, etc). It’s also very hard to edit with a plain text editor. > > As for the 2nd sentence, I don’t see how you can disagree since it’s a fact... > >> XML is easy to edit and is supported by very very many editors and IDEs. >> It can also be validated. > > What you’re saying is very theoretical. The practice (that we’ve had for 10 years of using the XAR format) is that our XML format that is hard to edit and can break easily (as was proven numerous times by our committers and contributors). It’s actually so dangerous that we’ve had to develop a process which goes like this: > * Never edit the xml directly > * Always import it into a running XWiki instance, make the modifications there and export from the wiki into XAR > * Then unpack the XAR into the source directory and run mvn xar:format to go back to the original format. > > Nobody is using a pure XML editor with validation. We are all using Java IDEA (IntelliJ IDEA, Eclipse, etc) and they all allow you to edit XML as plain text and that’s what we’re doing. No developer I know of is using an XML editor in a structured way (just too painful and complex to navigate the structure). > > (see below) > >> XML can be written in a very elegant and readable fashion if you care >> for it. >> Generally however, XML is produced form other structured information in >> a way that does not help readability. >>> Can you see more issues? >> The problem is how we put *everything* into XML. >> (you get the same horror if you use Caleb's tools xardump and do not >> tune anything: the resulting javascript is horrible.) >>> Use cases for an alternative filesystem format >>> =================================== >>> >>> (some UC taken from >>> http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications) >>> >>> * UC1: the structure should be (as) easy (as possible) to navigate in >>> an IDE style view >>> * UC2: it should be easy to add content (a new script or attachment on >>> an existing structure). It should allow using specific editors for >>> different content types, e.g. if a page content is in markdown, it >>> should be editable with a MD editor, js and css should be editable >>> with web editors, etc. >> UC2.1: Attachments should also be present as standalone files. >>> * UC3: It should be possible to build a packaged version of the >>> sources with Maven >>> * UC4: It should be possible to import the packaged version into a >>> running XWiki instance >>> * UC5: It should be possible to export a portion of a running XWiki >>> instance in this format >>> * UC6: This format should be able to fully replace the XAR format . >>> The new format should support at least all features supported by the >>> XAR format (versioned, etc). Note: XE will need to be refactor a bit >>> so that the XAR format can be swapped out by introducing extension >>> points/APIs. The idea would be to deprecate the XAR format and >>> introduce this new format instead, and the 2 formats should be avle to >>> cohabit next to each other in XWiki. >>> * UC7: When importing in a wiki and exporting again (without making >>> any change in the wiki), it should generate an identical structure and >>> content, with no difference. >> I do believe that UC7 is not doable in full generality. > > Why not, this is what we have with the XAR format actually and UC7 is actually already contained in UC6 (but it’s better to be explicit)? > >> Any more? >> >> UC8: the core representation should be using a syntax that is widely >> spread and completely specified (i.e. we should not invent another >> syntax for this) > > This is not a requirement for me. The syntax should be easy to write into, especially using a plain text editor. YAML for example is a perfectly valid syntax for me. > >> UC9: the system should make an archival process a widespread practice, >> in the form of zipped files probably. > > That’s UC3 for me. I hesitated to put ZIP in the requirement for UC3 since I didn’t want to limit us to that but it’s probably going to be zip anyway. > >> UC10: developers should have the discretion to decide when a component >> needs to be in a separate file or not. That is, small text fragments and >> even small attachments should be includable within bigger files > > I don’t agree with this one. I’d like convention over flexibility, i.e. a fixed format on which tools can easily build upon. This is similar to Maven vs Ant. > > This is why in my proposal for a format I’ve proposed fixed file names. > > Allowing discretion means everyone will do it differently and we’ll need to define best practices and best practices are hard to enforce and always cause problems. > >> UC11: there should be the possibility to share content of a file between >> several files or components (e.g. creator elements) > > I wouldn’t phrase it this way. I’d prefer to say that it should be possible to apply default values to missing elements in metadata. > > The way I see this for example for the format I’ve proposed in the other thread, is by having default properties that can be put on the filesystem, for example in default.properties file) so that when an element is missing the default would be used. > > Now I’m not sure we want this requirement and for me it’s an optional requirement and not a mandatory one. It makes it much harder to develop an exporter. > >> UC12: (transformation) simple search and replace operation should be >> supported by the build mechanism, especially when dependencies are followed. > > Could you explain more, I don’t understand? > >> UC13: it would be good that the format can be specified by a grammer >> against which one can validate (e.g. RelaxNG) > > I don’t agree in the way it’s phrased since it’s too limiting. I’d change it to: it should be possible for the packager tool to validate the content (what xar:verify does right now but that could be extended to verify that the required metadata are defined). I don’t think we need a formal grammar. The important part is that we have validation. > > Thanks > -Vincent > >> Paul >> >> (FYI UC10, UC11, and UC12 follow the architecture recommendation to be >> composable vs contextual so as to give us greater flexibility) > > _______________________________________________ > devs mailing list > [email protected] > http://lists.xwiki.org/mailman/listinfo/devs > Paul Libbrecht <mailto:[email protected]> > 27 August 2016 at 15:44 > Thanks for having this extra thread. > I think this thread is much more important than starting to design > something already. >> Issues with the XAR format >> ====================== >> >> * XML is not an easy to edit format and doesn’t allow use a specific >> editor to edit content >> * XML also requires content to be XML-encoded and thus is really not >> easy to make modification (there’s a risk of breaking the XML easily) > I completely disagree with these two statements. > XML is easy to edit and is supported by very very many editors and IDEs. > It can also be validated. > XML can be written in a very elegant and readable fashion if you care > for it. > Generally however, XML is produced form other structured information in > a way that does not help readability. >> Can you see more issues? > The problem is how we put *everything* into XML. > (you get the same horror if you use Caleb's tools xardump and do not > tune anything: the resulting javascript is horrible.) >> Use cases for an alternative filesystem format >> =================================== >> >> (some UC taken from >> http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications) >> >> * UC1: the structure should be (as) easy (as possible) to navigate in >> an IDE style view >> * UC2: it should be easy to add content (a new script or attachment on >> an existing structure). It should allow using specific editors for >> different content types, e.g. if a page content is in markdown, it >> should be editable with a MD editor, js and css should be editable >> with web editors, etc. > UC2.1: Attachments should also be present as standalone files. >> * UC3: It should be possible to build a packaged version of the >> sources with Maven >> * UC4: It should be possible to import the packaged version into a >> running XWiki instance >> * UC5: It should be possible to export a portion of a running XWiki >> instance in this format >> * UC6: This format should be able to fully replace the XAR format . >> The new format should support at least all features supported by the >> XAR format (versioned, etc). Note: XE will need to be refactor a bit >> so that the XAR format can be swapped out by introducing extension >> points/APIs. The idea would be to deprecate the XAR format and >> introduce this new format instead, and the 2 formats should be avle to >> cohabit next to each other in XWiki. >> * UC7: When importing in a wiki and exporting again (without making >> any change in the wiki), it should generate an identical structure and >> content, with no difference. > I do believe that UC7 is not doable in full generality. > > Any more? > > UC8: the core representation should be using a syntax that is widely > spread and completely specified (i.e. we should not invent another > syntax for this) > > UC9: the system should make an archival process a widespread practice, > in the form of zipped files probably. > > UC10: developers should have the discretion to decide when a component > needs to be in a separate file or not. That is, small text fragments and > even small attachments should be includable within bigger files > > UC11: there should be the possibility to share content of a file between > several files or components (e.g. creator elements) > > UC12: (transformation) simple search and replace operation should be > supported by the build mechanism, especially when dependencies are followed. > > UC13: it would be good that the format can be specified by a grammer > against which one can validate (e.g. RelaxNG) > > Paul > > (FYI UC10, UC11, and UC12 follow the architecture recommendation to be > composable vs contextual so as to give us greater flexibility) > _______________________________________________ > devs mailing list > [email protected] > http://lists.xwiki.org/mailman/listinfo/devs > Vincent Massol <mailto:[email protected]> > 27 August 2016 at 15:01 > Hi, > > This is a follow-up on the threads: > * "Designing the perfect FS representation of a wiki”: > http://markmail.org/message/3yghqwetmdt5woez > * "XAR source projects should allow source files”: > http://markmail.org/message/432o36r4klh7yv24 > > It’s also a continuation of the work done here: > http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications > > Once we get convergence on those thread (or even if we don’t), I’ll > update design.xwiki.org with the results. > > The goal is to define the use case for an alternate filesystem to XAR. > > Issues with the XAR format > ====================== > > * XML is not an easy to edit format and doesn’t allow use a specific > editor to edit content > * XML also requires content to be XML-encoded and thus is really not > easy to make modification (there’s a risk of breaking the XML easily) > > Can you see more issues? > > Use cases for an alternative filesystem format > =================================== > > (some UC taken from > http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications) > > * UC1: the structure should be (as) easy (as possible) to navigate in > an IDE style view > * UC2: it should be easy to add content (a new script or attachment on > an existing structure). It should allow using specific editors for > different content types, e.g. if a page content is in markdown, it > should be editable with a MD editor, js and css should be editable > with web editors, etc. > * UC3: It should be possible to build a packaged version of the > sources with Maven > * UC4: It should be possible to import the packaged version into a > running XWiki instance > * UC5: It should be possible to export a portion of a running XWiki > instance in this format > * UC6: This format should be able to fully replace the XAR format . > The new format should support at least all features supported by the > XAR format (versioned, etc). Note: XE will need to be refactor a bit > so that the XAR format can be swapped out by introducing extension > points/APIs. The idea would be to deprecate the XAR format and > introduce this new format instead, and the 2 formats should be avle to > cohabit next to each other in XWiki. > * UC7: When importing in a wiki and exporting again (without making > any change in the wiki), it should generate an identical structure and > content, with no difference. > > Any more? > > Thanks > -Vincent > > > > > > > > _______________________________________________ > devs mailing list > [email protected] > http://lists.xwiki.org/mailman/listinfo/devs
29 Aug
29 Aug
7:40 a.m.
> On 27 Aug 2016, at 18:37, Paul Libbrecht <[email protected]> wrote: > > Vincent, > > I think we disagree on two major aspects which are kind of related: > > - writing document structure and metadata should be doable by hand > editing as opposed to writing into the wiki alone: I don’t disagree on this. I just gave some arguments against doing it to avoid problems. Can you tell me how you validate that the changes work? For me it means deploying it in a wiki and if you deploy you’ll need to make changes and export the changes back (that’s the safest). And if you deploy you might as well start in the wiki to save you time. Side note: this is why it’s important that we have good editing tools inside the wiki (hence XWebIDE for example). We need devs to be able to write code in the wiki. BTW that was one reason I was suggesting YAML and not XML since XML is not a human-writable syntax (it’s not made for humans, it’s made for machines). > - I believe this is a fundamental aspect of source editing and should > be supported (hence the wish to a validatable widespread syntax) > - you seem to indicate that the current process you've agreed upon has > discouraged it (but, at the same time, you indicate you wish to do it > for YAML) It’s just very hard to write a full wiki page from scratch in sources and brings little value since you need to validate it anyway. > - writing XML can be done by hand and is easy: > - I am a strong defender of this: all editors I've worked with > (IntelliJ IDEA, jEdit, BBEdit, XCode, maybe not vim) support XML very > well so that it's just careless to commit a && (not to mention that mvn > would not validate it, and would not build it if with my patch, simply > because it parses). This is very nice (and something we should have implemented a long time ago) if your patch makes validation on the XML. > Clearly, editing in a textbox can get you there, > same with MS Word! > - you seem to say that bad things happen with such commits as not even > valid XML documents and seem to motivate that to move to another format > (I feel YAML a zillion less predictable than XML because of whitespace > handling btw). AFAIK whitespaces are not significant in YAML and you can align the elements which makes it even more readable. > Paul > > PS: when I say editing XML is for sane people, I don't mean everything > should be XML, which is why I proposed the XAR extension to include > XInclude, textinclude and binaryInclude. > > PPS: another, maybe deeper, disagreement might be the need to round-trip > (server, source) vs the need to source-edit: hand-tuning the format of a > file (e.g. in XML or in JSON) is easily in contradiction with a server > generation from a structured source. Maybe this is the hard bone. IMO you’ll be able to get away from roundtrip only when we’ll have an extra easy way of unit testing pages without having to deploy it in a full-fledged running XWiki. But even then, you won’t be able to see the impact on other stuff so functional tests is more what we’ll need and this requires deployment. Now, as I mentioned in the other thread, having a mvn xar:deploy and mvn xar:fetch mojos would help a lot for writing changes directly in the source tree. My only point is that we need to support both use cases: - round trip strategy (i.e. write in wiki, test, export, save) - source tree changes strategy (i.e. write in source, push, test, fetch, save) Thanks -Vincent > Vincent Massol <mailto:[email protected]> >> 27 August 2016 at 17:50 >>> On 27 Aug 2016, at 16:46, Vincent Massol <[email protected]> wrote: >>> >>>> On 27 Aug 2016, at 15:44, Paul Libbrecht <[email protected]> wrote: >>>> >>>> Thanks for having this extra thread. >>>> I think this thread is much more important than starting to design >>>> something already. >>>>> Issues with the XAR format >>>>> ====================== >>>>> >>>>> * XML is not an easy to edit format and doesn’t allow use a specific >>>>> editor to edit content >>>>> * XML also requires content to be XML-encoded and thus is really not >>>>> easy to make modification (there’s a risk of breaking the XML easily) >>>> I completely disagree with these two statements. >>> I probably didn’t use the right words because that’s the reason why I think that's you started the xinclude proposal :) >>> >>> AFAIK you started the xinclude proposal because you wanted to be able to edit content with specific editor (js, css, etc)… which is exactly what the second part of the first point is about. >>> >>> Let me rephrase the first sentence: >>> >>> * XML is not an easy to edit format for human beings (it’s very verbose and easy to make mistakes: missing encoding, etc). It’s also very hard to edit with a plain text editor. >>> >>> As for the 2nd sentence, I don’t see how you can disagree since it’s a fact... >>> >>>> XML is easy to edit and is supported by very very many editors and IDEs. >>>> It can also be validated. >>> What you’re saying is very theoretical. The practice (that we’ve had for 10 years of using the XAR format) is that our XML format that is hard to edit and can break easily (as was proven numerous times by our committers and contributors). It’s actually so dangerous that we’ve had to develop a process which goes like this: >>> * Never edit the xml directly >>> * Always import it into a running XWiki instance, make the modifications there and export from the wiki into XAR >>> * Then unpack the XAR into the source directory and run mvn xar:format to go back to the original format. >>> >>> Nobody is using a pure XML editor with validation. We are all using Java IDEA (IntelliJ IDEA, Eclipse, etc) and they all allow you to edit XML as plain text and that’s what we’re doing. No developer I know of is using an XML editor in a structured way (just too painful and complex to navigate the structure). >> >> To be more specific, the main issue we’ve had is contributors/committers who committed unencoded content, such as && instead of && or > instead of > >> >> Now, to be accurate, if you remove the problem with the encoding (which can be removed IMO with CDATA) then we never touch much of the rest of the metadata. >> >> In practice it would be nice if most of it could be generated by the maven plugin. In practice we don’t need much specific data (for a pure doc, it’s a bit more for xclass/xobjects): title, reference, syntax, parent, hidden, and language/translation. Syntax and reference could be computed from the directory structure. Parent could too. And hidden could default to visible by default. That said it doesn’t matter that much since the process is to export from a running xwiki instance (we need that to validate that it works at execution time, or we’d need good unit/integration tests for pages). >> >> So once we take the content out, the format of the metadata doesn’t matter that much probably since we’re not going to author it from scratch anyway (it’ll come from exported wiki pages). >> >> So I guess XML, even though very verbose, could still be ok. But XML for doc content or xproperty textareas, or attachments is a sure no go. >> >> Thanks >> -Vincent >> >>> (see below) >>> >>>> XML can be written in a very elegant and readable fashion if you care >>>> for it. >>>> Generally however, XML is produced form other structured information in >>>> a way that does not help readability. >>>>> Can you see more issues? >>>> The problem is how we put *everything* into XML. >>>> (you get the same horror if you use Caleb's tools xardump and do not >>>> tune anything: the resulting javascript is horrible.) >>>>> Use cases for an alternative filesystem format >>>>> =================================== >>>>> >>>>> (some UC taken from >>>>> http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications) >>>>> >>>>> * UC1: the structure should be (as) easy (as possible) to navigate in >>>>> an IDE style view >>>>> * UC2: it should be easy to add content (a new script or attachment on >>>>> an existing structure). It should allow using specific editors for >>>>> different content types, e.g. if a page content is in markdown, it >>>>> should be editable with a MD editor, js and css should be editable >>>>> with web editors, etc. >>>> UC2.1: Attachments should also be present as standalone files. >>>>> * UC3: It should be possible to build a packaged version of the >>>>> sources with Maven >>>>> * UC4: It should be possible to import the packaged version into a >>>>> running XWiki instance >>>>> * UC5: It should be possible to export a portion of a running XWiki >>>>> instance in this format >>>>> * UC6: This format should be able to fully replace the XAR format . >>>>> The new format should support at least all features supported by the >>>>> XAR format (versioned, etc). Note: XE will need to be refactor a bit >>>>> so that the XAR format can be swapped out by introducing extension >>>>> points/APIs. The idea would be to deprecate the XAR format and >>>>> introduce this new format instead, and the 2 formats should be avle to >>>>> cohabit next to each other in XWiki. >>>>> * UC7: When importing in a wiki and exporting again (without making >>>>> any change in the wiki), it should generate an identical structure and >>>>> content, with no difference. >>>> I do believe that UC7 is not doable in full generality. >>> Why not, this is what we have with the XAR format actually and UC7 is actually already contained in UC6 (but it’s better to be explicit)? >>> >>>> Any more? >>>> >>>> UC8: the core representation should be using a syntax that is widely >>>> spread and completely specified (i.e. we should not invent another >>>> syntax for this) >>> This is not a requirement for me. The syntax should be easy to write into, especially using a plain text editor. YAML for example is a perfectly valid syntax for me. >>> >>>> UC9: the system should make an archival process a widespread practice, >>>> in the form of zipped files probably. >>> That’s UC3 for me. I hesitated to put ZIP in the requirement for UC3 since I didn’t want to limit us to that but it’s probably going to be zip anyway. >>> >>>> UC10: developers should have the discretion to decide when a component >>>> needs to be in a separate file or not. That is, small text fragments and >>>> even small attachments should be includable within bigger files >>> I don’t agree with this one. I’d like convention over flexibility, i.e. a fixed format on which tools can easily build upon. This is similar to Maven vs Ant. >>> >>> This is why in my proposal for a format I’ve proposed fixed file names. >>> >>> Allowing discretion means everyone will do it differently and we’ll need to define best practices and best practices are hard to enforce and always cause problems. >>> >>>> UC11: there should be the possibility to share content of a file between >>>> several files or components (e.g. creator elements) >>> I wouldn’t phrase it this way. I’d prefer to say that it should be possible to apply default values to missing elements in metadata. >>> >>> The way I see this for example for the format I’ve proposed in the other thread, is by having default properties that can be put on the filesystem, for example in default.properties file) so that when an element is missing the default would be used. >>> >>> Now I’m not sure we want this requirement and for me it’s an optional requirement and not a mandatory one. It makes it much harder to develop an exporter. >>> >>>> UC12: (transformation) simple search and replace operation should be >>>> supported by the build mechanism, especially when dependencies are followed. >>> Could you explain more, I don’t understand? >>> >>>> UC13: it would be good that the format can be specified by a grammer >>>> against which one can validate (e.g. RelaxNG) >>> I don’t agree in the way it’s phrased since it’s too limiting. I’d change it to: it should be possible for the packager tool to validate the content (what xar:verify does right now but that could be extended to verify that the required metadata are defined). I don’t think we need a formal grammar. The important part is that we have validation. >>> >>> Thanks >>> -Vincent >>> >>>> Paul >>>> >>>> (FYI UC10, UC11, and UC12 follow the architecture recommendation to be >>>> composable vs contextual so as to give us greater flexibility) >> >> _______________________________________________ >> devs mailing list >> [email protected] >> http://lists.xwiki.org/mailman/listinfo/devs >> Vincent Massol <mailto:[email protected]> >> 27 August 2016 at 16:46 >>> On 27 Aug 2016, at 15:44, Paul Libbrecht <[email protected]> wrote: >>> >>> Thanks for having this extra thread. >>> I think this thread is much more important than starting to design >>> something already. >>>> Issues with the XAR format >>>> ====================== >>>> >>>> * XML is not an easy to edit format and doesn’t allow use a specific >>>> editor to edit content >>>> * XML also requires content to be XML-encoded and thus is really not >>>> easy to make modification (there’s a risk of breaking the XML easily) >>> I completely disagree with these two statements. >> >> I probably didn’t use the right words because that’s the reason why I think that's you started the xinclude proposal :) >> >> AFAIK you started the xinclude proposal because you wanted to be able to edit content with specific editor (js, css, etc)… which is exactly what the second part of the first point is about. >> >> Let me rephrase the first sentence: >> >> * XML is not an easy to edit format for human beings (it’s very verbose and easy to make mistakes: missing encoding, etc). It’s also very hard to edit with a plain text editor. >> >> As for the 2nd sentence, I don’t see how you can disagree since it’s a fact... >> >>> XML is easy to edit and is supported by very very many editors and IDEs. >>> It can also be validated. >> >> What you’re saying is very theoretical. The practice (that we’ve had for 10 years of using the XAR format) is that our XML format that is hard to edit and can break easily (as was proven numerous times by our committers and contributors). It’s actually so dangerous that we’ve had to develop a process which goes like this: >> * Never edit the xml directly >> * Always import it into a running XWiki instance, make the modifications there and export from the wiki into XAR >> * Then unpack the XAR into the source directory and run mvn xar:format to go back to the original format. >> >> Nobody is using a pure XML editor with validation. We are all using Java IDEA (IntelliJ IDEA, Eclipse, etc) and they all allow you to edit XML as plain text and that’s what we’re doing. No developer I know of is using an XML editor in a structured way (just too painful and complex to navigate the structure). >> >> (see below) >> >>> XML can be written in a very elegant and readable fashion if you care >>> for it. >>> Generally however, XML is produced form other structured information in >>> a way that does not help readability. >>>> Can you see more issues? >>> The problem is how we put *everything* into XML. >>> (you get the same horror if you use Caleb's tools xardump and do not >>> tune anything: the resulting javascript is horrible.) >>>> Use cases for an alternative filesystem format >>>> =================================== >>>> >>>> (some UC taken from >>>> http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications) >>>> >>>> * UC1: the structure should be (as) easy (as possible) to navigate in >>>> an IDE style view >>>> * UC2: it should be easy to add content (a new script or attachment on >>>> an existing structure). It should allow using specific editors for >>>> different content types, e.g. if a page content is in markdown, it >>>> should be editable with a MD editor, js and css should be editable >>>> with web editors, etc. >>> UC2.1: Attachments should also be present as standalone files. >>>> * UC3: It should be possible to build a packaged version of the >>>> sources with Maven >>>> * UC4: It should be possible to import the packaged version into a >>>> running XWiki instance >>>> * UC5: It should be possible to export a portion of a running XWiki >>>> instance in this format >>>> * UC6: This format should be able to fully replace the XAR format . >>>> The new format should support at least all features supported by the >>>> XAR format (versioned, etc). Note: XE will need to be refactor a bit >>>> so that the XAR format can be swapped out by introducing extension >>>> points/APIs. The idea would be to deprecate the XAR format and >>>> introduce this new format instead, and the 2 formats should be avle to >>>> cohabit next to each other in XWiki. >>>> * UC7: When importing in a wiki and exporting again (without making >>>> any change in the wiki), it should generate an identical structure and >>>> content, with no difference. >>> I do believe that UC7 is not doable in full generality. >> >> Why not, this is what we have with the XAR format actually and UC7 is actually already contained in UC6 (but it’s better to be explicit)? >> >>> Any more? >>> >>> UC8: the core representation should be using a syntax that is widely >>> spread and completely specified (i.e. we should not invent another >>> syntax for this) >> >> This is not a requirement for me. The syntax should be easy to write into, especially using a plain text editor. YAML for example is a perfectly valid syntax for me. >> >>> UC9: the system should make an archival process a widespread practice, >>> in the form of zipped files probably. >> >> That’s UC3 for me. I hesitated to put ZIP in the requirement for UC3 since I didn’t want to limit us to that but it’s probably going to be zip anyway. >> >>> UC10: developers should have the discretion to decide when a component >>> needs to be in a separate file or not. That is, small text fragments and >>> even small attachments should be includable within bigger files >> >> I don’t agree with this one. I’d like convention over flexibility, i.e. a fixed format on which tools can easily build upon. This is similar to Maven vs Ant. >> >> This is why in my proposal for a format I’ve proposed fixed file names. >> >> Allowing discretion means everyone will do it differently and we’ll need to define best practices and best practices are hard to enforce and always cause problems. >> >>> UC11: there should be the possibility to share content of a file between >>> several files or components (e.g. creator elements) >> >> I wouldn’t phrase it this way. I’d prefer to say that it should be possible to apply default values to missing elements in metadata. >> >> The way I see this for example for the format I’ve proposed in the other thread, is by having default properties that can be put on the filesystem, for example in default.properties file) so that when an element is missing the default would be used. >> >> Now I’m not sure we want this requirement and for me it’s an optional requirement and not a mandatory one. It makes it much harder to develop an exporter. >> >>> UC12: (transformation) simple search and replace operation should be >>> supported by the build mechanism, especially when dependencies are followed. >> >> Could you explain more, I don’t understand? >> >>> UC13: it would be good that the format can be specified by a grammer >>> against which one can validate (e.g. RelaxNG) >> >> I don’t agree in the way it’s phrased since it’s too limiting. I’d change it to: it should be possible for the packager tool to validate the content (what xar:verify does right now but that could be extended to verify that the required metadata are defined). I don’t think we need a formal grammar. The important part is that we have validation. >> >> Thanks >> -Vincent >> >>> Paul >>> >>> (FYI UC10, UC11, and UC12 follow the architecture recommendation to be >>> composable vs contextual so as to give us greater flexibility) >> >> _______________________________________________ >> devs mailing list >> [email protected] >> http://lists.xwiki.org/mailman/listinfo/devs >> Paul Libbrecht <mailto:[email protected]> >> 27 August 2016 at 15:44 >> Thanks for having this extra thread. >> I think this thread is much more important than starting to design >> something already. >>> Issues with the XAR format >>> ====================== >>> >>> * XML is not an easy to edit format and doesn’t allow use a specific >>> editor to edit content >>> * XML also requires content to be XML-encoded and thus is really not >>> easy to make modification (there’s a risk of breaking the XML easily) >> I completely disagree with these two statements. >> XML is easy to edit and is supported by very very many editors and IDEs. >> It can also be validated. >> XML can be written in a very elegant and readable fashion if you care >> for it. >> Generally however, XML is produced form other structured information in >> a way that does not help readability. >>> Can you see more issues? >> The problem is how we put *everything* into XML. >> (you get the same horror if you use Caleb's tools xardump and do not >> tune anything: the resulting javascript is horrible.) >>> Use cases for an alternative filesystem format >>> =================================== >>> >>> (some UC taken from >>> http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications) >>> >>> * UC1: the structure should be (as) easy (as possible) to navigate in >>> an IDE style view >>> * UC2: it should be easy to add content (a new script or attachment on >>> an existing structure). It should allow using specific editors for >>> different content types, e.g. if a page content is in markdown, it >>> should be editable with a MD editor, js and css should be editable >>> with web editors, etc. >> UC2.1: Attachments should also be present as standalone files. >>> * UC3: It should be possible to build a packaged version of the >>> sources with Maven >>> * UC4: It should be possible to import the packaged version into a >>> running XWiki instance >>> * UC5: It should be possible to export a portion of a running XWiki >>> instance in this format >>> * UC6: This format should be able to fully replace the XAR format . >>> The new format should support at least all features supported by the >>> XAR format (versioned, etc). Note: XE will need to be refactor a bit >>> so that the XAR format can be swapped out by introducing extension >>> points/APIs. The idea would be to deprecate the XAR format and >>> introduce this new format instead, and the 2 formats should be avle to >>> cohabit next to each other in XWiki. >>> * UC7: When importing in a wiki and exporting again (without making >>> any change in the wiki), it should generate an identical structure and >>> content, with no difference. >> I do believe that UC7 is not doable in full generality. >> >> Any more? >> >> UC8: the core representation should be using a syntax that is widely >> spread and completely specified (i.e. we should not invent another >> syntax for this) >> >> UC9: the system should make an archival process a widespread practice, >> in the form of zipped files probably. >> >> UC10: developers should have the discretion to decide when a component >> needs to be in a separate file or not. That is, small text fragments and >> even small attachments should be includable within bigger files >> >> UC11: there should be the possibility to share content of a file between >> several files or components (e.g. creator elements) >> >> UC12: (transformation) simple search and replace operation should be >> supported by the build mechanism, especially when dependencies are followed. >> >> UC13: it would be good that the format can be specified by a grammer >> against which one can validate (e.g. RelaxNG) >> >> Paul >> >> (FYI UC10, UC11, and UC12 follow the architecture recommendation to be >> composable vs contextual so as to give us greater flexibility) >> _______________________________________________ >> devs mailing list >> [email protected] >> http://lists.xwiki.org/mailman/listinfo/devs >> Vincent Massol <mailto:[email protected]> >> 27 August 2016 at 15:01 >> Hi, >> >> This is a follow-up on the threads: >> * "Designing the perfect FS representation of a wiki”: >> http://markmail.org/message/3yghqwetmdt5woez >> * "XAR source projects should allow source files”: >> http://markmail.org/message/432o36r4klh7yv24 >> >> It’s also a continuation of the work done here: >> http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications >> >> Once we get convergence on those thread (or even if we don’t), I’ll >> update design.xwiki.org with the results. >> >> The goal is to define the use case for an alternate filesystem to XAR. >> >> Issues with the XAR format >> ====================== >> >> * XML is not an easy to edit format and doesn’t allow use a specific >> editor to edit content >> * XML also requires content to be XML-encoded and thus is really not >> easy to make modification (there’s a risk of breaking the XML easily) >> >> Can you see more issues? >> >> Use cases for an alternative filesystem format >> =================================== >> >> (some UC taken from >> http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications) >> >> * UC1: the structure should be (as) easy (as possible) to navigate in >> an IDE style view >> * UC2: it should be easy to add content (a new script or attachment on >> an existing structure). It should allow using specific editors for >> different content types, e.g. if a page content is in markdown, it >> should be editable with a MD editor, js and css should be editable >> with web editors, etc. >> * UC3: It should be possible to build a packaged version of the >> sources with Maven >> * UC4: It should be possible to import the packaged version into a >> running XWiki instance >> * UC5: It should be possible to export a portion of a running XWiki >> instance in this format >> * UC6: This format should be able to fully replace the XAR format . >> The new format should support at least all features supported by the >> XAR format (versioned, etc). Note: XE will need to be refactor a bit >> so that the XAR format can be swapped out by introducing extension >> points/APIs. The idea would be to deprecate the XAR format and >> introduce this new format instead, and the 2 formats should be avle to >> cohabit next to each other in XWiki. >> * UC7: When importing in a wiki and exporting again (without making >> any change in the wiki), it should generate an identical structure and >> content, with no difference. >> >> Any more? >> >> Thanks >> -Vincent >>
7:58 a.m.
> On 29 Aug 2016, at 09:40, Vincent Massol <[email protected]> wrote: > >> >> On 27 Aug 2016, at 18:37, Paul Libbrecht <[email protected]> wrote: >> >> Vincent, >> >> I think we disagree on two major aspects which are kind of related: >> >> - writing document structure and metadata should be doable by hand >> editing as opposed to writing into the wiki alone: > > I don’t disagree on this. I just gave some arguments against doing it to avoid problems. Can you tell me how you validate that the changes work? For me it means deploying it in a wiki and if you deploy you’ll need to make changes and export the changes back (that’s the safest). And if you deploy you might as well start in the wiki to save you time. Side note: this is why it’s important that we have good editing tools inside the wiki (hence XWebIDE for example). We need devs to be able to write code in the wiki. > > BTW that was one reason I was suggesting YAML and not XML since XML is not a human-writable syntax (it’s not made for humans, it’s made for machines). > >> - I believe this is a fundamental aspect of source editing and should >> be supported (hence the wish to a validatable widespread syntax) >> - you seem to indicate that the current process you've agreed upon has >> discouraged it (but, at the same time, you indicate you wish to do it >> for YAML) > > It’s just very hard to write a full wiki page from scratch in sources and brings little value since you need to validate it anyway. > >> - writing XML can be done by hand and is easy: >> - I am a strong defender of this: all editors I've worked with >> (IntelliJ IDEA, jEdit, BBEdit, XCode, maybe not vim) support XML very >> well so that it's just careless to commit a && (not to mention that mvn >> would not validate it, and would not build it if with my patch, simply >> because it parses). > > This is very nice (and something we should have implemented a long time ago) if your patch makes validation on the XML. > >> Clearly, editing in a textbox can get you there, >> same with MS Word! >> - you seem to say that bad things happen with such commits as not even >> valid XML documents and seem to motivate that to move to another format >> (I feel YAML a zillion less predictable than XML because of whitespace >> handling btw). > > AFAIK whitespaces are not significant in YAML and you can align the elements which makes it even more readable. > >> Paul >> >> PS: when I say editing XML is for sane people, I don't mean everything >> should be XML, which is why I proposed the XAR extension to include >> XInclude, textinclude and binaryInclude. >> >> PPS: another, maybe deeper, disagreement might be the need to round-trip >> (server, source) vs the need to source-edit: hand-tuning the format of a >> file (e.g. in XML or in JSON) is easily in contradiction with a server >> generation from a structured source. Maybe this is the hard bone. > > IMO you’ll be able to get away from roundtrip only when we’ll have an extra easy way of unit testing pages without having to deploy it in a full-fledged running XWiki. But even then, you won’t be able to see the impact on other stuff so functional tests is more what we’ll need and this requires deployment. > > Now, as I mentioned in the other thread, having a mvn xar:deploy and mvn xar:fetch mojos would help a lot for writing changes directly in the source tree. > > My only point is that we need to support both use cases: > - round trip strategy (i.e. write in wiki, test, export, save) > - source tree changes strategy (i.e. write in source, push, test, fetch, save) To be more precise: - round trip strategy (i.e. write in wiki, [test, make changes]*, export, save) - source tree changes strategy (i.e. write in sources, [push, test in wiki, make changes locally]*, save) Thanks -Vincent > > Thanks > -Vincent > >> Vincent Massol <mailto:[email protected]> >>> 27 August 2016 at 17:50 >>>> On 27 Aug 2016, at 16:46, Vincent Massol <[email protected]> wrote: >>>> >>>>> On 27 Aug 2016, at 15:44, Paul Libbrecht <[email protected]> wrote: >>>>> >>>>> Thanks for having this extra thread. >>>>> I think this thread is much more important than starting to design >>>>> something already. >>>>>> Issues with the XAR format >>>>>> ====================== >>>>>> >>>>>> * XML is not an easy to edit format and doesn’t allow use a specific >>>>>> editor to edit content >>>>>> * XML also requires content to be XML-encoded and thus is really not >>>>>> easy to make modification (there’s a risk of breaking the XML easily) >>>>> I completely disagree with these two statements. >>>> I probably didn’t use the right words because that’s the reason why I think that's you started the xinclude proposal :) >>>> >>>> AFAIK you started the xinclude proposal because you wanted to be able to edit content with specific editor (js, css, etc)… which is exactly what the second part of the first point is about. >>>> >>>> Let me rephrase the first sentence: >>>> >>>> * XML is not an easy to edit format for human beings (it’s very verbose and easy to make mistakes: missing encoding, etc). It’s also very hard to edit with a plain text editor. >>>> >>>> As for the 2nd sentence, I don’t see how you can disagree since it’s a fact... >>>> >>>>> XML is easy to edit and is supported by very very many editors and IDEs. >>>>> It can also be validated. >>>> What you’re saying is very theoretical. The practice (that we’ve had for 10 years of using the XAR format) is that our XML format that is hard to edit and can break easily (as was proven numerous times by our committers and contributors). It’s actually so dangerous that we’ve had to develop a process which goes like this: >>>> * Never edit the xml directly >>>> * Always import it into a running XWiki instance, make the modifications there and export from the wiki into XAR >>>> * Then unpack the XAR into the source directory and run mvn xar:format to go back to the original format. >>>> >>>> Nobody is using a pure XML editor with validation. We are all using Java IDEA (IntelliJ IDEA, Eclipse, etc) and they all allow you to edit XML as plain text and that’s what we’re doing. No developer I know of is using an XML editor in a structured way (just too painful and complex to navigate the structure). >>> >>> To be more specific, the main issue we’ve had is contributors/committers who committed unencoded content, such as && instead of && or > instead of > >>> >>> Now, to be accurate, if you remove the problem with the encoding (which can be removed IMO with CDATA) then we never touch much of the rest of the metadata. >>> >>> In practice it would be nice if most of it could be generated by the maven plugin. In practice we don’t need much specific data (for a pure doc, it’s a bit more for xclass/xobjects): title, reference, syntax, parent, hidden, and language/translation. Syntax and reference could be computed from the directory structure. Parent could too. And hidden could default to visible by default. That said it doesn’t matter that much since the process is to export from a running xwiki instance (we need that to validate that it works at execution time, or we’d need good unit/integration tests for pages). >>> >>> So once we take the content out, the format of the metadata doesn’t matter that much probably since we’re not going to author it from scratch anyway (it’ll come from exported wiki pages). >>> >>> So I guess XML, even though very verbose, could still be ok. But XML for doc content or xproperty textareas, or attachments is a sure no go. >>> >>> Thanks >>> -Vincent >>> >>>> (see below) >>>> >>>>> XML can be written in a very elegant and readable fashion if you care >>>>> for it. >>>>> Generally however, XML is produced form other structured information in >>>>> a way that does not help readability. >>>>>> Can you see more issues? >>>>> The problem is how we put *everything* into XML. >>>>> (you get the same horror if you use Caleb's tools xardump and do not >>>>> tune anything: the resulting javascript is horrible.) >>>>>> Use cases for an alternative filesystem format >>>>>> =================================== >>>>>> >>>>>> (some UC taken from >>>>>> http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications) >>>>>> >>>>>> * UC1: the structure should be (as) easy (as possible) to navigate in >>>>>> an IDE style view >>>>>> * UC2: it should be easy to add content (a new script or attachment on >>>>>> an existing structure). It should allow using specific editors for >>>>>> different content types, e.g. if a page content is in markdown, it >>>>>> should be editable with a MD editor, js and css should be editable >>>>>> with web editors, etc. >>>>> UC2.1: Attachments should also be present as standalone files. >>>>>> * UC3: It should be possible to build a packaged version of the >>>>>> sources with Maven >>>>>> * UC4: It should be possible to import the packaged version into a >>>>>> running XWiki instance >>>>>> * UC5: It should be possible to export a portion of a running XWiki >>>>>> instance in this format >>>>>> * UC6: This format should be able to fully replace the XAR format . >>>>>> The new format should support at least all features supported by the >>>>>> XAR format (versioned, etc). Note: XE will need to be refactor a bit >>>>>> so that the XAR format can be swapped out by introducing extension >>>>>> points/APIs. The idea would be to deprecate the XAR format and >>>>>> introduce this new format instead, and the 2 formats should be avle to >>>>>> cohabit next to each other in XWiki. >>>>>> * UC7: When importing in a wiki and exporting again (without making >>>>>> any change in the wiki), it should generate an identical structure and >>>>>> content, with no difference. >>>>> I do believe that UC7 is not doable in full generality. >>>> Why not, this is what we have with the XAR format actually and UC7 is actually already contained in UC6 (but it’s better to be explicit)? >>>> >>>>> Any more? >>>>> >>>>> UC8: the core representation should be using a syntax that is widely >>>>> spread and completely specified (i.e. we should not invent another >>>>> syntax for this) >>>> This is not a requirement for me. The syntax should be easy to write into, especially using a plain text editor. YAML for example is a perfectly valid syntax for me. >>>> >>>>> UC9: the system should make an archival process a widespread practice, >>>>> in the form of zipped files probably. >>>> That’s UC3 for me. I hesitated to put ZIP in the requirement for UC3 since I didn’t want to limit us to that but it’s probably going to be zip anyway. >>>> >>>>> UC10: developers should have the discretion to decide when a component >>>>> needs to be in a separate file or not. That is, small text fragments and >>>>> even small attachments should be includable within bigger files >>>> I don’t agree with this one. I’d like convention over flexibility, i.e. a fixed format on which tools can easily build upon. This is similar to Maven vs Ant. >>>> >>>> This is why in my proposal for a format I’ve proposed fixed file names. >>>> >>>> Allowing discretion means everyone will do it differently and we’ll need to define best practices and best practices are hard to enforce and always cause problems. >>>> >>>>> UC11: there should be the possibility to share content of a file between >>>>> several files or components (e.g. creator elements) >>>> I wouldn’t phrase it this way. I’d prefer to say that it should be possible to apply default values to missing elements in metadata. >>>> >>>> The way I see this for example for the format I’ve proposed in the other thread, is by having default properties that can be put on the filesystem, for example in default.properties file) so that when an element is missing the default would be used. >>>> >>>> Now I’m not sure we want this requirement and for me it’s an optional requirement and not a mandatory one. It makes it much harder to develop an exporter. >>>> >>>>> UC12: (transformation) simple search and replace operation should be >>>>> supported by the build mechanism, especially when dependencies are followed. >>>> Could you explain more, I don’t understand? >>>> >>>>> UC13: it would be good that the format can be specified by a grammer >>>>> against which one can validate (e.g. RelaxNG) >>>> I don’t agree in the way it’s phrased since it’s too limiting. I’d change it to: it should be possible for the packager tool to validate the content (what xar:verify does right now but that could be extended to verify that the required metadata are defined). I don’t think we need a formal grammar. The important part is that we have validation. >>>> >>>> Thanks >>>> -Vincent >>>> >>>>> Paul >>>>> >>>>> (FYI UC10, UC11, and UC12 follow the architecture recommendation to be >>>>> composable vs contextual so as to give us greater flexibility) >>> >>> _______________________________________________ >>> devs mailing list >>> [email protected] >>> http://lists.xwiki.org/mailman/listinfo/devs >>> Vincent Massol <mailto:[email protected]> >>> 27 August 2016 at 16:46 >>>> On 27 Aug 2016, at 15:44, Paul Libbrecht <[email protected]> wrote: >>>> >>>> Thanks for having this extra thread. >>>> I think this thread is much more important than starting to design >>>> something already. >>>>> Issues with the XAR format >>>>> ====================== >>>>> >>>>> * XML is not an easy to edit format and doesn’t allow use a specific >>>>> editor to edit content >>>>> * XML also requires content to be XML-encoded and thus is really not >>>>> easy to make modification (there’s a risk of breaking the XML easily) >>>> I completely disagree with these two statements. >>> >>> I probably didn’t use the right words because that’s the reason why I think that's you started the xinclude proposal :) >>> >>> AFAIK you started the xinclude proposal because you wanted to be able to edit content with specific editor (js, css, etc)… which is exactly what the second part of the first point is about. >>> >>> Let me rephrase the first sentence: >>> >>> * XML is not an easy to edit format for human beings (it’s very verbose and easy to make mistakes: missing encoding, etc). It’s also very hard to edit with a plain text editor. >>> >>> As for the 2nd sentence, I don’t see how you can disagree since it’s a fact... >>> >>>> XML is easy to edit and is supported by very very many editors and IDEs. >>>> It can also be validated. >>> >>> What you’re saying is very theoretical. The practice (that we’ve had for 10 years of using the XAR format) is that our XML format that is hard to edit and can break easily (as was proven numerous times by our committers and contributors). It’s actually so dangerous that we’ve had to develop a process which goes like this: >>> * Never edit the xml directly >>> * Always import it into a running XWiki instance, make the modifications there and export from the wiki into XAR >>> * Then unpack the XAR into the source directory and run mvn xar:format to go back to the original format. >>> >>> Nobody is using a pure XML editor with validation. We are all using Java IDEA (IntelliJ IDEA, Eclipse, etc) and they all allow you to edit XML as plain text and that’s what we’re doing. No developer I know of is using an XML editor in a structured way (just too painful and complex to navigate the structure). >>> >>> (see below) >>> >>>> XML can be written in a very elegant and readable fashion if you care >>>> for it. >>>> Generally however, XML is produced form other structured information in >>>> a way that does not help readability. >>>>> Can you see more issues? >>>> The problem is how we put *everything* into XML. >>>> (you get the same horror if you use Caleb's tools xardump and do not >>>> tune anything: the resulting javascript is horrible.) >>>>> Use cases for an alternative filesystem format >>>>> =================================== >>>>> >>>>> (some UC taken from >>>>> http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications) >>>>> >>>>> * UC1: the structure should be (as) easy (as possible) to navigate in >>>>> an IDE style view >>>>> * UC2: it should be easy to add content (a new script or attachment on >>>>> an existing structure). It should allow using specific editors for >>>>> different content types, e.g. if a page content is in markdown, it >>>>> should be editable with a MD editor, js and css should be editable >>>>> with web editors, etc. >>>> UC2.1: Attachments should also be present as standalone files. >>>>> * UC3: It should be possible to build a packaged version of the >>>>> sources with Maven >>>>> * UC4: It should be possible to import the packaged version into a >>>>> running XWiki instance >>>>> * UC5: It should be possible to export a portion of a running XWiki >>>>> instance in this format >>>>> * UC6: This format should be able to fully replace the XAR format . >>>>> The new format should support at least all features supported by the >>>>> XAR format (versioned, etc). Note: XE will need to be refactor a bit >>>>> so that the XAR format can be swapped out by introducing extension >>>>> points/APIs. The idea would be to deprecate the XAR format and >>>>> introduce this new format instead, and the 2 formats should be avle to >>>>> cohabit next to each other in XWiki. >>>>> * UC7: When importing in a wiki and exporting again (without making >>>>> any change in the wiki), it should generate an identical structure and >>>>> content, with no difference. >>>> I do believe that UC7 is not doable in full generality. >>> >>> Why not, this is what we have with the XAR format actually and UC7 is actually already contained in UC6 (but it’s better to be explicit)? >>> >>>> Any more? >>>> >>>> UC8: the core representation should be using a syntax that is widely >>>> spread and completely specified (i.e. we should not invent another >>>> syntax for this) >>> >>> This is not a requirement for me. The syntax should be easy to write into, especially using a plain text editor. YAML for example is a perfectly valid syntax for me. >>> >>>> UC9: the system should make an archival process a widespread practice, >>>> in the form of zipped files probably. >>> >>> That’s UC3 for me. I hesitated to put ZIP in the requirement for UC3 since I didn’t want to limit us to that but it’s probably going to be zip anyway. >>> >>>> UC10: developers should have the discretion to decide when a component >>>> needs to be in a separate file or not. That is, small text fragments and >>>> even small attachments should be includable within bigger files >>> >>> I don’t agree with this one. I’d like convention over flexibility, i.e. a fixed format on which tools can easily build upon. This is similar to Maven vs Ant. >>> >>> This is why in my proposal for a format I’ve proposed fixed file names. >>> >>> Allowing discretion means everyone will do it differently and we’ll need to define best practices and best practices are hard to enforce and always cause problems. >>> >>>> UC11: there should be the possibility to share content of a file between >>>> several files or components (e.g. creator elements) >>> >>> I wouldn’t phrase it this way. I’d prefer to say that it should be possible to apply default values to missing elements in metadata. >>> >>> The way I see this for example for the format I’ve proposed in the other thread, is by having default properties that can be put on the filesystem, for example in default.properties file) so that when an element is missing the default would be used. >>> >>> Now I’m not sure we want this requirement and for me it’s an optional requirement and not a mandatory one. It makes it much harder to develop an exporter. >>> >>>> UC12: (transformation) simple search and replace operation should be >>>> supported by the build mechanism, especially when dependencies are followed. >>> >>> Could you explain more, I don’t understand? >>> >>>> UC13: it would be good that the format can be specified by a grammer >>>> against which one can validate (e.g. RelaxNG) >>> >>> I don’t agree in the way it’s phrased since it’s too limiting. I’d change it to: it should be possible for the packager tool to validate the content (what xar:verify does right now but that could be extended to verify that the required metadata are defined). I don’t think we need a formal grammar. The important part is that we have validation. >>> >>> Thanks >>> -Vincent >>> >>>> Paul >>>> >>>> (FYI UC10, UC11, and UC12 follow the architecture recommendation to be >>>> composable vs contextual so as to give us greater flexibility) >>> >>> _______________________________________________ >>> devs mailing list >>> [email protected] >>> http://lists.xwiki.org/mailman/listinfo/devs >>> Paul Libbrecht <mailto:[email protected]> >>> 27 August 2016 at 15:44 >>> Thanks for having this extra thread. >>> I think this thread is much more important than starting to design >>> something already. >>>> Issues with the XAR format >>>> ====================== >>>> >>>> * XML is not an easy to edit format and doesn’t allow use a specific >>>> editor to edit content >>>> * XML also requires content to be XML-encoded and thus is really not >>>> easy to make modification (there’s a risk of breaking the XML easily) >>> I completely disagree with these two statements. >>> XML is easy to edit and is supported by very very many editors and IDEs. >>> It can also be validated. >>> XML can be written in a very elegant and readable fashion if you care >>> for it. >>> Generally however, XML is produced form other structured information in >>> a way that does not help readability. >>>> Can you see more issues? >>> The problem is how we put *everything* into XML. >>> (you get the same horror if you use Caleb's tools xardump and do not >>> tune anything: the resulting javascript is horrible.) >>>> Use cases for an alternative filesystem format >>>> =================================== >>>> >>>> (some UC taken from >>>> http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications) >>>> >>>> * UC1: the structure should be (as) easy (as possible) to navigate in >>>> an IDE style view >>>> * UC2: it should be easy to add content (a new script or attachment on >>>> an existing structure). It should allow using specific editors for >>>> different content types, e.g. if a page content is in markdown, it >>>> should be editable with a MD editor, js and css should be editable >>>> with web editors, etc. >>> UC2.1: Attachments should also be present as standalone files. >>>> * UC3: It should be possible to build a packaged version of the >>>> sources with Maven >>>> * UC4: It should be possible to import the packaged version into a >>>> running XWiki instance >>>> * UC5: It should be possible to export a portion of a running XWiki >>>> instance in this format >>>> * UC6: This format should be able to fully replace the XAR format . >>>> The new format should support at least all features supported by the >>>> XAR format (versioned, etc). Note: XE will need to be refactor a bit >>>> so that the XAR format can be swapped out by introducing extension >>>> points/APIs. The idea would be to deprecate the XAR format and >>>> introduce this new format instead, and the 2 formats should be avle to >>>> cohabit next to each other in XWiki. >>>> * UC7: When importing in a wiki and exporting again (without making >>>> any change in the wiki), it should generate an identical structure and >>>> content, with no difference. >>> I do believe that UC7 is not doable in full generality. >>> >>> Any more? >>> >>> UC8: the core representation should be using a syntax that is widely >>> spread and completely specified (i.e. we should not invent another >>> syntax for this) >>> >>> UC9: the system should make an archival process a widespread practice, >>> in the form of zipped files probably. >>> >>> UC10: developers should have the discretion to decide when a component >>> needs to be in a separate file or not. That is, small text fragments and >>> even small attachments should be includable within bigger files >>> >>> UC11: there should be the possibility to share content of a file between >>> several files or components (e.g. creator elements) >>> >>> UC12: (transformation) simple search and replace operation should be >>> supported by the build mechanism, especially when dependencies are followed. >>> >>> UC13: it would be good that the format can be specified by a grammer >>> against which one can validate (e.g. RelaxNG) >>> >>> Paul >>> >>> (FYI UC10, UC11, and UC12 follow the architecture recommendation to be >>> composable vs contextual so as to give us greater flexibility) >>> _______________________________________________ >>> devs mailing list >>> [email protected] >>> http://lists.xwiki.org/mailman/listinfo/devs >>> Vincent Massol <mailto:[email protected]> >>> 27 August 2016 at 15:01 >>> Hi, >>> >>> This is a follow-up on the threads: >>> * "Designing the perfect FS representation of a wiki”: >>> http://markmail.org/message/3yghqwetmdt5woez >>> * "XAR source projects should allow source files”: >>> http://markmail.org/message/432o36r4klh7yv24 >>> >>> It’s also a continuation of the work done here: >>> http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications >>> >>> Once we get convergence on those thread (or even if we don’t), I’ll >>> update design.xwiki.org with the results. >>> >>> The goal is to define the use case for an alternate filesystem to XAR. >>> >>> Issues with the XAR format >>> ====================== >>> >>> * XML is not an easy to edit format and doesn’t allow use a specific >>> editor to edit content >>> * XML also requires content to be XML-encoded and thus is really not >>> easy to make modification (there’s a risk of breaking the XML easily) >>> >>> Can you see more issues? >>> >>> Use cases for an alternative filesystem format >>> =================================== >>> >>> (some UC taken from >>> http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApplications) >>> >>> * UC1: the structure should be (as) easy (as possible) to navigate in >>> an IDE style view >>> * UC2: it should be easy to add content (a new script or attachment on >>> an existing structure). It should allow using specific editors for >>> different content types, e.g. if a page content is in markdown, it >>> should be editable with a MD editor, js and css should be editable >>> with web editors, etc. >>> * UC3: It should be possible to build a packaged version of the >>> sources with Maven >>> * UC4: It should be possible to import the packaged version into a >>> running XWiki instance >>> * UC5: It should be possible to export a portion of a running XWiki >>> instance in this format >>> * UC6: This format should be able to fully replace the XAR format . >>> The new format should support at least all features supported by the >>> XAR format (versioned, etc). Note: XE will need to be refactor a bit >>> so that the XAR format can be swapped out by introducing extension >>> points/APIs. The idea would be to deprecate the XAR format and >>> introduce this new format instead, and the 2 formats should be avle to >>> cohabit next to each other in XWiki. >>> * UC7: When importing in a wiki and exporting again (without making >>> any change in the wiki), it should generate an identical structure and >>> content, with no difference. >>> >>> Any more? >>> >>> Thanks >>> -Vincent >>> > > _______________________________________________ > devs mailing list > [email protected] > http://lists.xwiki.org/mailman/listinfo/devs
1:36 p.m.
New subject: [xwiki-devs] [Brainstorming] Use cases for an alternative filesystem format (alternative to XAR)
Vincent Massol wrote:
My only point is that we need to support both use cases:
- round trip strategy (i.e. write in wiki, test, export, save) - source tree changes strategy (i.e. write in source, push, test, fetch, save)
To be more precise: - round trip strategy (i.e. write in wiki, [test, make changes]*, export, save) - source tree changes strategy (i.e. write in sources, [push, test in wiki, make changes locally]*, save) I agree we need to support both. The more fundamental question is: do we need to support both with the same sources?
My claim is that this is close to impossible, especially if one wants to leave a lot of freedom to the developers to layout the source the way they want. If the sources do not need to be equal (i.e. a source project does not need round-tripping equality) then I think we're safe. Paul
4 Sep
4 Sep
2:02 p.m.
New subject: [xwiki-devs] [Brainstorming] Use cases for an alternative filesystem format (alternative to XAR)
Hi All, Can admin please unsubscribe from this group. I have made this number of times in past but somehow it went unheard. Request admin to do the needful. Thanks and Regards Mohit On Mon, Aug 29, 2016 at 1:28 PM, Vincent Massol <[email protected]> wrote: > > > On 29 Aug 2016, at 09:40, Vincent Massol <[email protected]> wrote: > > > >> > >> On 27 Aug 2016, at 18:37, Paul Libbrecht <[email protected]> wrote: > >> > >> Vincent, > >> > >> I think we disagree on two major aspects which are kind of related: > >> > >> - writing document structure and metadata should be doable by hand > >> editing as opposed to writing into the wiki alone: > > > > I don’t disagree on this. I just gave some arguments against doing it to > avoid problems. Can you tell me how you validate that the changes work? For > me it means deploying it in a wiki and if you deploy you’ll need to make > changes and export the changes back (that’s the safest). And if you deploy > you might as well start in the wiki to save you time. Side note: this is > why it’s important that we have good editing tools inside the wiki (hence > XWebIDE for example). We need devs to be able to write code in the wiki. > > > > BTW that was one reason I was suggesting YAML and not XML since XML is > not a human-writable syntax (it’s not made for humans, it’s made for > machines). > > > >> - I believe this is a fundamental aspect of source editing and should > >> be supported (hence the wish to a validatable widespread syntax) > >> - you seem to indicate that the current process you've agreed upon has > >> discouraged it (but, at the same time, you indicate you wish to do it > >> for YAML) > > > > It’s just very hard to write a full wiki page from scratch in sources > and brings little value since you need to validate it anyway. > > > >> - writing XML can be done by hand and is easy: > >> - I am a strong defender of this: all editors I've worked with > >> (IntelliJ IDEA, jEdit, BBEdit, XCode, maybe not vim) support XML very > >> well so that it's just careless to commit a && (not to mention that mvn > >> would not validate it, and would not build it if with my patch, simply > >> because it parses). > > > > This is very nice (and something we should have implemented a long time > ago) if your patch makes validation on the XML. > > > >> Clearly, editing in a textbox can get you there, > >> same with MS Word! > >> - you seem to say that bad things happen with such commits as not even > >> valid XML documents and seem to motivate that to move to another format > >> (I feel YAML a zillion less predictable than XML because of whitespace > >> handling btw). > > > > AFAIK whitespaces are not significant in YAML and you can align the > elements which makes it even more readable. > > > >> Paul > >> > >> PS: when I say editing XML is for sane people, I don't mean everything > >> should be XML, which is why I proposed the XAR extension to include > >> XInclude, textinclude and binaryInclude. > >> > >> PPS: another, maybe deeper, disagreement might be the need to round-trip > >> (server, source) vs the need to source-edit: hand-tuning the format of a > >> file (e.g. in XML or in JSON) is easily in contradiction with a server > >> generation from a structured source. Maybe this is the hard bone. > > > > IMO you’ll be able to get away from roundtrip only when we’ll have an > extra easy way of unit testing pages without having to deploy it in a > full-fledged running XWiki. But even then, you won’t be able to see the > impact on other stuff so functional tests is more what we’ll need and this > requires deployment. > > > > Now, as I mentioned in the other thread, having a mvn xar:deploy and mvn > xar:fetch mojos would help a lot for writing changes directly in the source > tree. > > > > My only point is that we need to support both use cases: > > - round trip strategy (i.e. write in wiki, test, export, save) > > - source tree changes strategy (i.e. write in source, push, test, fetch, > save) > > To be more precise: > - round trip strategy (i.e. write in wiki, [test, make changes]*, export, > save) > - source tree changes strategy (i.e. write in sources, [push, test in > wiki, make changes locally]*, save) > > Thanks > -Vincent > > > > > Thanks > > -Vincent > > > >> Vincent Massol <mailto:[email protected]> > >>> 27 August 2016 at 17:50 > >>>> On 27 Aug 2016, at 16:46, Vincent Massol <[email protected]> wrote: > >>>> > >>>>> On 27 Aug 2016, at 15:44, Paul Libbrecht <[email protected]> wrote: > >>>>> > >>>>> Thanks for having this extra thread. > >>>>> I think this thread is much more important than starting to design > >>>>> something already. > >>>>>> Issues with the XAR format > >>>>>> ====================== > >>>>>> > >>>>>> * XML is not an easy to edit format and doesn’t allow use a specific > >>>>>> editor to edit content > >>>>>> * XML also requires content to be XML-encoded and thus is really not > >>>>>> easy to make modification (there’s a risk of breaking the XML > easily) > >>>>> I completely disagree with these two statements. > >>>> I probably didn’t use the right words because that’s the reason why I > think that's you started the xinclude proposal :) > >>>> > >>>> AFAIK you started the xinclude proposal because you wanted to be able > to edit content with specific editor (js, css, etc)… which is exactly what > the second part of the first point is about. > >>>> > >>>> Let me rephrase the first sentence: > >>>> > >>>> * XML is not an easy to edit format for human beings (it’s very > verbose and easy to make mistakes: missing encoding, etc). It’s also very > hard to edit with a plain text editor. > >>>> > >>>> As for the 2nd sentence, I don’t see how you can disagree since it’s > a fact... > >>>> > >>>>> XML is easy to edit and is supported by very very many editors and > IDEs. > >>>>> It can also be validated. > >>>> What you’re saying is very theoretical. The practice (that we’ve had > for 10 years of using the XAR format) is that our XML format that is hard > to edit and can break easily (as was proven numerous times by our > committers and contributors). It’s actually so dangerous that we’ve had to > develop a process which goes like this: > >>>> * Never edit the xml directly > >>>> * Always import it into a running XWiki instance, make the > modifications there and export from the wiki into XAR > >>>> * Then unpack the XAR into the source directory and run mvn > xar:format to go back to the original format. > >>>> > >>>> Nobody is using a pure XML editor with validation. We are all using > Java IDEA (IntelliJ IDEA, Eclipse, etc) and they all allow you to edit XML > as plain text and that’s what we’re doing. No developer I know of is using > an XML editor in a structured way (just too painful and complex to navigate > the structure). > >>> > >>> To be more specific, the main issue we’ve had is > contributors/committers who committed unencoded content, such as && instead > of && or > instead of > > >>> > >>> Now, to be accurate, if you remove the problem with the encoding > (which can be removed IMO with CDATA) then we never touch much of the rest > of the metadata. > >>> > >>> In practice it would be nice if most of it could be generated by the > maven plugin. In practice we don’t need much specific data (for a pure doc, > it’s a bit more for xclass/xobjects): title, reference, syntax, parent, > hidden, and language/translation. Syntax and reference could be computed > from the directory structure. Parent could too. And hidden could default to > visible by default. That said it doesn’t matter that much since the process > is to export from a running xwiki instance (we need that to validate that > it works at execution time, or we’d need good unit/integration tests for > pages). > >>> > >>> So once we take the content out, the format of the metadata doesn’t > matter that much probably since we’re not going to author it from scratch > anyway (it’ll come from exported wiki pages). > >>> > >>> So I guess XML, even though very verbose, could still be ok. But XML > for doc content or xproperty textareas, or attachments is a sure no go. > >>> > >>> Thanks > >>> -Vincent > >>> > >>>> (see below) > >>>> > >>>>> XML can be written in a very elegant and readable fashion if you care > >>>>> for it. > >>>>> Generally however, XML is produced form other structured information > in > >>>>> a way that does not help readability. > >>>>>> Can you see more issues? > >>>>> The problem is how we put *everything* into XML. > >>>>> (you get the same horror if you use Caleb's tools xardump and do not > >>>>> tune anything: the resulting javascript is horrible.) > >>>>>> Use cases for an alternative filesystem format > >>>>>> =================================== > >>>>>> > >>>>>> (some UC taken from > >>>>>> http://design.xwiki.org/xwiki/bin/view/Design/ > DirectoryStructureforXWikiApplications) > >>>>>> > >>>>>> * UC1: the structure should be (as) easy (as possible) to navigate > in > >>>>>> an IDE style view > >>>>>> * UC2: it should be easy to add content (a new script or attachment > on > >>>>>> an existing structure). It should allow using specific editors for > >>>>>> different content types, e.g. if a page content is in markdown, it > >>>>>> should be editable with a MD editor, js and css should be editable > >>>>>> with web editors, etc. > >>>>> UC2.1: Attachments should also be present as standalone files. > >>>>>> * UC3: It should be possible to build a packaged version of the > >>>>>> sources with Maven > >>>>>> * UC4: It should be possible to import the packaged version into a > >>>>>> running XWiki instance > >>>>>> * UC5: It should be possible to export a portion of a running XWiki > >>>>>> instance in this format > >>>>>> * UC6: This format should be able to fully replace the XAR format . > >>>>>> The new format should support at least all features supported by the > >>>>>> XAR format (versioned, etc). Note: XE will need to be refactor a bit > >>>>>> so that the XAR format can be swapped out by introducing extension > >>>>>> points/APIs. The idea would be to deprecate the XAR format and > >>>>>> introduce this new format instead, and the 2 formats should be avle > to > >>>>>> cohabit next to each other in XWiki. > >>>>>> * UC7: When importing in a wiki and exporting again (without making > >>>>>> any change in the wiki), it should generate an identical structure > and > >>>>>> content, with no difference. > >>>>> I do believe that UC7 is not doable in full generality. > >>>> Why not, this is what we have with the XAR format actually and UC7 is > actually already contained in UC6 (but it’s better to be explicit)? > >>>> > >>>>> Any more? > >>>>> > >>>>> UC8: the core representation should be using a syntax that is widely > >>>>> spread and completely specified (i.e. we should not invent another > >>>>> syntax for this) > >>>> This is not a requirement for me. The syntax should be easy to write > into, especially using a plain text editor. YAML for example is a perfectly > valid syntax for me. > >>>> > >>>>> UC9: the system should make an archival process a widespread > practice, > >>>>> in the form of zipped files probably. > >>>> That’s UC3 for me. I hesitated to put ZIP in the requirement for UC3 > since I didn’t want to limit us to that but it’s probably going to be zip > anyway. > >>>> > >>>>> UC10: developers should have the discretion to decide when a > component > >>>>> needs to be in a separate file or not. That is, small text fragments > and > >>>>> even small attachments should be includable within bigger files > >>>> I don’t agree with this one. I’d like convention over flexibility, > i.e. a fixed format on which tools can easily build upon. This is similar > to Maven vs Ant. > >>>> > >>>> This is why in my proposal for a format I’ve proposed fixed file > names. > >>>> > >>>> Allowing discretion means everyone will do it differently and we’ll > need to define best practices and best practices are hard to enforce and > always cause problems. > >>>> > >>>>> UC11: there should be the possibility to share content of a file > between > >>>>> several files or components (e.g. creator elements) > >>>> I wouldn’t phrase it this way. I’d prefer to say that it should be > possible to apply default values to missing elements in metadata. > >>>> > >>>> The way I see this for example for the format I’ve proposed in the > other thread, is by having default properties that can be put on the > filesystem, for example in default.properties file) so that when an element > is missing the default would be used. > >>>> > >>>> Now I’m not sure we want this requirement and for me it’s an optional > requirement and not a mandatory one. It makes it much harder to develop an > exporter. > >>>> > >>>>> UC12: (transformation) simple search and replace operation should be > >>>>> supported by the build mechanism, especially when dependencies are > followed. > >>>> Could you explain more, I don’t understand? > >>>> > >>>>> UC13: it would be good that the format can be specified by a grammer > >>>>> against which one can validate (e.g. RelaxNG) > >>>> I don’t agree in the way it’s phrased since it’s too limiting. I’d > change it to: it should be possible for the packager tool to validate the > content (what xar:verify does right now but that could be extended to > verify that the required metadata are defined). I don’t think we need a > formal grammar. The important part is that we have validation. > >>>> > >>>> Thanks > >>>> -Vincent > >>>> > >>>>> Paul > >>>>> > >>>>> (FYI UC10, UC11, and UC12 follow the architecture recommendation to > be > >>>>> composable vs contextual so as to give us greater flexibility) > >>> > >>> _______________________________________________ > >>> devs mailing list > >>> [email protected] > >>> http://lists.xwiki.org/mailman/listinfo/devs > >>> Vincent Massol <mailto:[email protected]> > >>> 27 August 2016 at 16:46 > >>>> On 27 Aug 2016, at 15:44, Paul Libbrecht <[email protected]> wrote: > >>>> > >>>> Thanks for having this extra thread. > >>>> I think this thread is much more important than starting to design > >>>> something already. > >>>>> Issues with the XAR format > >>>>> ====================== > >>>>> > >>>>> * XML is not an easy to edit format and doesn’t allow use a specific > >>>>> editor to edit content > >>>>> * XML also requires content to be XML-encoded and thus is really not > >>>>> easy to make modification (there’s a risk of breaking the XML easily) > >>>> I completely disagree with these two statements. > >>> > >>> I probably didn’t use the right words because that’s the reason why I > think that's you started the xinclude proposal :) > >>> > >>> AFAIK you started the xinclude proposal because you wanted to be able > to edit content with specific editor (js, css, etc)… which is exactly what > the second part of the first point is about. > >>> > >>> Let me rephrase the first sentence: > >>> > >>> * XML is not an easy to edit format for human beings (it’s very > verbose and easy to make mistakes: missing encoding, etc). It’s also very > hard to edit with a plain text editor. > >>> > >>> As for the 2nd sentence, I don’t see how you can disagree since it’s a > fact... > >>> > >>>> XML is easy to edit and is supported by very very many editors and > IDEs. > >>>> It can also be validated. > >>> > >>> What you’re saying is very theoretical. The practice (that we’ve had > for 10 years of using the XAR format) is that our XML format that is hard > to edit and can break easily (as was proven numerous times by our > committers and contributors). It’s actually so dangerous that we’ve had to > develop a process which goes like this: > >>> * Never edit the xml directly > >>> * Always import it into a running XWiki instance, make the > modifications there and export from the wiki into XAR > >>> * Then unpack the XAR into the source directory and run mvn xar:format > to go back to the original format. > >>> > >>> Nobody is using a pure XML editor with validation. We are all using > Java IDEA (IntelliJ IDEA, Eclipse, etc) and they all allow you to edit XML > as plain text and that’s what we’re doing. No developer I know of is using > an XML editor in a structured way (just too painful and complex to navigate > the structure). > >>> > >>> (see below) > >>> > >>>> XML can be written in a very elegant and readable fashion if you care > >>>> for it. > >>>> Generally however, XML is produced form other structured information > in > >>>> a way that does not help readability. > >>>>> Can you see more issues? > >>>> The problem is how we put *everything* into XML. > >>>> (you get the same horror if you use Caleb's tools xardump and do not > >>>> tune anything: the resulting javascript is horrible.) > >>>>> Use cases for an alternative filesystem format > >>>>> =================================== > >>>>> > >>>>> (some UC taken from > >>>>> http://design.xwiki.org/xwiki/bin/view/Design/ > DirectoryStructureforXWikiApplications) > >>>>> > >>>>> * UC1: the structure should be (as) easy (as possible) to navigate in > >>>>> an IDE style view > >>>>> * UC2: it should be easy to add content (a new script or attachment > on > >>>>> an existing structure). It should allow using specific editors for > >>>>> different content types, e.g. if a page content is in markdown, it > >>>>> should be editable with a MD editor, js and css should be editable > >>>>> with web editors, etc. > >>>> UC2.1: Attachments should also be present as standalone files. > >>>>> * UC3: It should be possible to build a packaged version of the > >>>>> sources with Maven > >>>>> * UC4: It should be possible to import the packaged version into a > >>>>> running XWiki instance > >>>>> * UC5: It should be possible to export a portion of a running XWiki > >>>>> instance in this format > >>>>> * UC6: This format should be able to fully replace the XAR format . > >>>>> The new format should support at least all features supported by the > >>>>> XAR format (versioned, etc). Note: XE will need to be refactor a bit > >>>>> so that the XAR format can be swapped out by introducing extension > >>>>> points/APIs. The idea would be to deprecate the XAR format and > >>>>> introduce this new format instead, and the 2 formats should be avle > to > >>>>> cohabit next to each other in XWiki. > >>>>> * UC7: When importing in a wiki and exporting again (without making > >>>>> any change in the wiki), it should generate an identical structure > and > >>>>> content, with no difference. > >>>> I do believe that UC7 is not doable in full generality. > >>> > >>> Why not, this is what we have with the XAR format actually and UC7 is > actually already contained in UC6 (but it’s better to be explicit)? > >>> > >>>> Any more? > >>>> > >>>> UC8: the core representation should be using a syntax that is widely > >>>> spread and completely specified (i.e. we should not invent another > >>>> syntax for this) > >>> > >>> This is not a requirement for me. The syntax should be easy to write > into, especially using a plain text editor. YAML for example is a perfectly > valid syntax for me. > >>> > >>>> UC9: the system should make an archival process a widespread practice, > >>>> in the form of zipped files probably. > >>> > >>> That’s UC3 for me. I hesitated to put ZIP in the requirement for UC3 > since I didn’t want to limit us to that but it’s probably going to be zip > anyway. > >>> > >>>> UC10: developers should have the discretion to decide when a component > >>>> needs to be in a separate file or not. That is, small text fragments > and > >>>> even small attachments should be includable within bigger files > >>> > >>> I don’t agree with this one. I’d like convention over flexibility, > i.e. a fixed format on which tools can easily build upon. This is similar > to Maven vs Ant. > >>> > >>> This is why in my proposal for a format I’ve proposed fixed file names. > >>> > >>> Allowing discretion means everyone will do it differently and we’ll > need to define best practices and best practices are hard to enforce and > always cause problems. > >>> > >>>> UC11: there should be the possibility to share content of a file > between > >>>> several files or components (e.g. creator elements) > >>> > >>> I wouldn’t phrase it this way. I’d prefer to say that it should be > possible to apply default values to missing elements in metadata. > >>> > >>> The way I see this for example for the format I’ve proposed in the > other thread, is by having default properties that can be put on the > filesystem, for example in default.properties file) so that when an element > is missing the default would be used. > >>> > >>> Now I’m not sure we want this requirement and for me it’s an optional > requirement and not a mandatory one. It makes it much harder to develop an > exporter. > >>> > >>>> UC12: (transformation) simple search and replace operation should be > >>>> supported by the build mechanism, especially when dependencies are > followed. > >>> > >>> Could you explain more, I don’t understand? > >>> > >>>> UC13: it would be good that the format can be specified by a grammer > >>>> against which one can validate (e.g. RelaxNG) > >>> > >>> I don’t agree in the way it’s phrased since it’s too limiting. I’d > change it to: it should be possible for the packager tool to validate the > content (what xar:verify does right now but that could be extended to > verify that the required metadata are defined). I don’t think we need a > formal grammar. The important part is that we have validation. > >>> > >>> Thanks > >>> -Vincent > >>> > >>>> Paul > >>>> > >>>> (FYI UC10, UC11, and UC12 follow the architecture recommendation to be > >>>> composable vs contextual so as to give us greater flexibility) > >>> > >>> _______________________________________________ > >>> devs mailing list > >>> [email protected] > >>> http://lists.xwiki.org/mailman/listinfo/devs > >>> Paul Libbrecht <mailto:[email protected]> > >>> 27 August 2016 at 15:44 > >>> Thanks for having this extra thread. > >>> I think this thread is much more important than starting to design > >>> something already. > >>>> Issues with the XAR format > >>>> ====================== > >>>> > >>>> * XML is not an easy to edit format and doesn’t allow use a specific > >>>> editor to edit content > >>>> * XML also requires content to be XML-encoded and thus is really not > >>>> easy to make modification (there’s a risk of breaking the XML easily) > >>> I completely disagree with these two statements. > >>> XML is easy to edit and is supported by very very many editors and > IDEs. > >>> It can also be validated. > >>> XML can be written in a very elegant and readable fashion if you care > >>> for it. > >>> Generally however, XML is produced form other structured information in > >>> a way that does not help readability. > >>>> Can you see more issues? > >>> The problem is how we put *everything* into XML. > >>> (you get the same horror if you use Caleb's tools xardump and do not > >>> tune anything: the resulting javascript is horrible.) > >>>> Use cases for an alternative filesystem format > >>>> =================================== > >>>> > >>>> (some UC taken from > >>>> http://design.xwiki.org/xwiki/bin/view/Design/ > DirectoryStructureforXWikiApplications) > >>>> > >>>> * UC1: the structure should be (as) easy (as possible) to navigate in > >>>> an IDE style view > >>>> * UC2: it should be easy to add content (a new script or attachment on > >>>> an existing structure). It should allow using specific editors for > >>>> different content types, e.g. if a page content is in markdown, it > >>>> should be editable with a MD editor, js and css should be editable > >>>> with web editors, etc. > >>> UC2.1: Attachments should also be present as standalone files. > >>>> * UC3: It should be possible to build a packaged version of the > >>>> sources with Maven > >>>> * UC4: It should be possible to import the packaged version into a > >>>> running XWiki instance > >>>> * UC5: It should be possible to export a portion of a running XWiki > >>>> instance in this format > >>>> * UC6: This format should be able to fully replace the XAR format . > >>>> The new format should support at least all features supported by the > >>>> XAR format (versioned, etc). Note: XE will need to be refactor a bit > >>>> so that the XAR format can be swapped out by introducing extension > >>>> points/APIs. The idea would be to deprecate the XAR format and > >>>> introduce this new format instead, and the 2 formats should be avle to > >>>> cohabit next to each other in XWiki. > >>>> * UC7: When importing in a wiki and exporting again (without making > >>>> any change in the wiki), it should generate an identical structure and > >>>> content, with no difference. > >>> I do believe that UC7 is not doable in full generality. > >>> > >>> Any more? > >>> > >>> UC8: the core representation should be using a syntax that is widely > >>> spread and completely specified (i.e. we should not invent another > >>> syntax for this) > >>> > >>> UC9: the system should make an archival process a widespread practice, > >>> in the form of zipped files probably. > >>> > >>> UC10: developers should have the discretion to decide when a component > >>> needs to be in a separate file or not. That is, small text fragments > and > >>> even small attachments should be includable within bigger files > >>> > >>> UC11: there should be the possibility to share content of a file > between > >>> several files or components (e.g. creator elements) > >>> > >>> UC12: (transformation) simple search and replace operation should be > >>> supported by the build mechanism, especially when dependencies are > followed. > >>> > >>> UC13: it would be good that the format can be specified by a grammer > >>> against which one can validate (e.g. RelaxNG) > >>> > >>> Paul > >>> > >>> (FYI UC10, UC11, and UC12 follow the architecture recommendation to be > >>> composable vs contextual so as to give us greater flexibility) > >>> _______________________________________________ > >>> devs mailing list > >>> [email protected] > >>> http://lists.xwiki.org/mailman/listinfo/devs > >>> Vincent Massol <mailto:[email protected]> > >>> 27 August 2016 at 15:01 > >>> Hi, > >>> > >>> This is a follow-up on the threads: > >>> * "Designing the perfect FS representation of a wiki”: > >>> http://markmail.org/message/3yghqwetmdt5woez > >>> * "XAR source projects should allow source files”: > >>> http://markmail.org/message/432o36r4klh7yv24 > >>> > >>> It’s also a continuation of the work done here: > >>> http://design.xwiki.org/xwiki/bin/view/Design/ > DirectoryStructureforXWikiApplications > >>> > >>> Once we get convergence on those thread (or even if we don’t), I’ll > >>> update design.xwiki.org with the results. > >>> > >>> The goal is to define the use case for an alternate filesystem to XAR. > >>> > >>> Issues with the XAR format > >>> ====================== > >>> > >>> * XML is not an easy to edit format and doesn’t allow use a specific > >>> editor to edit content > >>> * XML also requires content to be XML-encoded and thus is really not > >>> easy to make modification (there’s a risk of breaking the XML easily) > >>> > >>> Can you see more issues? > >>> > >>> Use cases for an alternative filesystem format > >>> =================================== > >>> > >>> (some UC taken from > >>> http://design.xwiki.org/xwiki/bin/view/Design/ > DirectoryStructureforXWikiApplications) > >>> > >>> * UC1: the structure should be (as) easy (as possible) to navigate in > >>> an IDE style view > >>> * UC2: it should be easy to add content (a new script or attachment on > >>> an existing structure). It should allow using specific editors for > >>> different content types, e.g. if a page content is in markdown, it > >>> should be editable with a MD editor, js and css should be editable > >>> with web editors, etc. > >>> * UC3: It should be possible to build a packaged version of the > >>> sources with Maven > >>> * UC4: It should be possible to import the packaged version into a > >>> running XWiki instance > >>> * UC5: It should be possible to export a portion of a running XWiki > >>> instance in this format > >>> * UC6: This format should be able to fully replace the XAR format . > >>> The new format should support at least all features supported by the > >>> XAR format (versioned, etc). Note: XE will need to be refactor a bit > >>> so that the XAR format can be swapped out by introducing extension > >>> points/APIs. The idea would be to deprecate the XAR format and > >>> introduce this new format instead, and the 2 formats should be avle to > >>> cohabit next to each other in XWiki. > >>> * UC7: When importing in a wiki and exporting again (without making > >>> any change in the wiki), it should generate an identical structure and > >>> content, with no difference. > >>> > >>> Any more? > >>> > >>> Thanks > >>> -Vincent > >>> > > > > _______________________________________________ > > devs mailing list > > [email protected] > > http://lists.xwiki.org/mailman/listinfo/devs > > _______________________________________________ > devs mailing list > [email protected] > http://lists.xwiki.org/mailman/listinfo/devs >
4:29 p.m.
New subject: [xwiki-devs] Unsubscribing (was Re: [Brainstorming] Use cases for an alternative filesystem format (alternative to XAR))
Hi Mohit,
On 04 Sep 2016, at 16:02, Mohit Gupta <[email protected]> wrote:
Hi All,
Can admin please unsubscribe from this group. I have made this number of times in past but somehow it went unheard. Request admin to do the needful.
Next time, would be nice if you could start a new thread (and not hijack an existing mail thread) to post on a completely different topic. Now on the unsubscription part, this is not something that anyone handles here. You subscribed by yourself to the list and it’s the same for unsubscribing, you should do it yourself. As mentioned on http://dev.xwiki.org/xwiki/bin/view/Community/MailingLists (second paragraph) you can go to http://lists.xwiki.org/mailman/listinfo/devs to unsubscribe. Thanks -Vincent [snip]
3644
Age (days ago)
3652
Last active (days ago)
9 comments
3 participants
participants (3)
-
Mohit Gupta -
Paul Libbrecht -
Vincent Massol