Note that an alternative is to add a mojo in the XAR plugin (or write a new Maven plugin) based on Filter module to produce a XAR package from http://extensions.xwiki.org/xwiki/bin/view/Extension/XFF+filter which already been designed for what you want: provide a format that can be edited from filesystem easily. In theory all it takes is to put XFF filter in input and XAR filter in output and run.

On 23 Aug 2016, at 11:00, Thomas Mortagne <thomas.mortagne(a)xwiki.com> wrote: I can't give you much details on the XFF format, I just know the general goal. I only helped Jean on the "how to use Filter API" side. On Tue, Aug 23, 2016 at 10:57 AM, Paul Libbrecht <paul(a)hoplahup.net> wrote: > Thomas, > > I had seen this approach yesterday only. > I think my attempt is a tick more conservative. > But indeed, it seems to be the objective of the XFF filter. > > Also, what I could not tell fully from the XFF attempt is whether the > optionality was there: are there property values or even complete > objects that I can leave gouped in some page XML? It seems to me that > itreally does not make sense to include them all in a file each. (e.g. > objects of the XWiki.TranslationDocumentClass). > > paul > > > > > > Thomas Mortagne wrote: >> Note that an alternative is to add a mojo in the XAR plugin (or write a >> new Maven plugin) based on Filter module to produce a XAR package from >> http://extensions.xwiki.org/xwiki/bin/view/Extension/XFF+filter which >> already been designed for what you want: provide a format that can be >> edited from filesystem easily. In theory all it takes is to put XFF >> filter in input and XAR filter in output and run. > > __________

On 23 Aug 2016, at 11:13, Paul Libbrecht <paul(a)hoplahup.net> wrote: Let's ask things differently: what's missing XFF for others to use?

paul > Vincent Massol <mailto:vincent@massol.net> > 23 August 2016 at 11:01 > The format seems to be defined here: > http://extensions.xwiki.org/xwiki/bin/view/Extension/XFF+filter#HFormat

Vincent Massol <mailto:vincent@massol.net> 23 August 2016 at 11:35 Actually just noticed that Jean has started one: https://github.com/xwiki-contrib/api-xff/tree/master/xff-maven I don’t know its status though. So what needs to be done is try the full chain and see what works/what doesn’t. Unless jean could tell us what’s the status? (I’ll ping him on skype). Thanks -Vincent _______________________________________________ devs mailing list devs(a)xwiki.org http://lists.xwiki.org/mailman/listinfo/devs Vincent Massol <mailto:vincent@massol.net> 23 August 2016 at 11:15 I think it’s just missing a maven plugin to use it. All the other parts are ready. Thanks -Vincent _______________________________________________ devs mailing list devs(a)xwiki.org http://lists.xwiki.org/mailman/listinfo/devs Paul Libbrecht <mailto:paul@hoplahup.net> 23 August 2016 at 11:13 Let's ask things differently: what's missing XFF for others to use? paul _______________________________________________ devs mailing list devs(a)xwiki.org http://lists.xwiki.org/mailman/listinfo/devs Vincent Massol <mailto:vincent@massol.net> 23 August 2016 at 11:01 The format seems to be defined here: http://extensions.xwiki.org/xwiki/bin/view/Extension/XFF+filter#HFormat Thanks -Vincent _______________________________________________ devs mailing list devs(a)xwiki.org http://lists.xwiki.org/mailman/listinfo/devs Thomas Mortagne <mailto:thomas.mortagne@xwiki.com> 23 August 2016 at 11:00 I can't give you much details on the XFF format, I just know the general goal. I only helped Jean on the "how to use Filter API" side.

Not trough XAR tools no, it's completely different format.

On Tue, Aug 23, 2016 at 12:17 PM, Paul Libbrecht &lt;paul(a)hoplahup.net&gt; wrote: Which is what I indicated: you just use XFF filter (which already exist) in input and XAR filter (which also already exist) as output. > We'll see what Jean says. > > > > In the meantime, do I understand at least that an enrichment to the > xar-mojo is a more conservative approach that might have a better chance > of uptake?

> What does it take for me to get the patch accepted?

-- Thomas Mortagne

Thomas Mortagne wrote: both are claimed. There are unit tests. The only danger of breaking anything is if dom4j does not faithfully restore the XML source. I believe this can be ignored just as it has been ignored by the implementation of transformations. No, it is enabled by default. I could replace the dom4j output by a file copy if no relevant elements are found. Note however, that it operates on elements that were never allowed before. I believe this is best practice to insure that the wiki does not insert you surprise elements such as the author name or the edit comment.

Is this practice not corresponding at all what others are doing? Using an XML-diff might be answering Guillaume's point... That's not completely trivial. Paul _______________________________________________ devs mailing list devs(a)xwiki.org http://lists.xwiki.org/mailman/listinfo/devs

Guillaume Delhumeau <mailto:guillaume.delhumeau@xwiki.com> 23 August 2016 at 13:19 Of course, we could still change our workflow and copy/paste source contents manually. Not sure it's gonna be more effective. So I prefer the XFF approach (except that I would love having a Nested Document terminology instead of the current Space/Document one). 2016-08-23 13:11 GMT+02:00 Guillaume Delhumeau < Guillaume Delhumeau <mailto:guillaume.delhumeau@xwiki.com> 23 August 2016 at 13:11 2016-08-23 12:36 GMT+02:00 Paul Libbrecht &lt;paul(a)hoplahup.net&gt;et>: In practice, this is the only manual step. I do a diff of the modifications I have made and I revert all changes that are meaningless. But it's quick & easy using my IDE (IntelliJ). I know some developers here do the same. The only issue is when the XAR format have changed: some fields are not ordered in the same way, and it makes harder to compute the real diff. Paul Libbrecht <mailto:paul@hoplahup.net> 23 August 2016 at 12:36 Thomas Mortagne wrote: both are claimed. There are unit tests. The only danger of breaking anything is if dom4j does not faithfully restore the XML source. I believe this can be ignored just as it has been ignored by the implementation of transformations. No, it is enabled by default. I could replace the dom4j output by a file copy if no relevant elements are found. Note however, that it operates on elements that were never allowed before. I believe this is best practice to insure that the wiki does not insert you surprise elements such as the author name or the edit comment. Is this practice not corresponding at all what others are doing? Using an XML-diff might be answering Guillaume's point... That's not completely trivial. Paul _______________________________________________ devs mailing list devs(a)xwiki.org http://lists.xwiki.org/mailman/listinfo/devs Thomas Mortagne <mailto:thomas.mortagne@xwiki.com> 23 August 2016 at 12:28 On Tue, Aug 23, 2016 at 12:26 PM, Thomas Mortagne &lt;thomas.mortagne(a)xwiki.com&gt; wrote: It takes the following IMO: * it works * it does not break any retro compatibility (probably not enabled by default for example) Now Guillaume has a point. The issue you will quickly have as soon as you start using this is that you HAVE to modify your XAR trough filesystem because you can't edit it in XWiki and export it anymore. That is if you don't provide any tool to export this kind of XAR extension. Thomas Mortagne <mailto:thomas.mortagne@xwiki.com> 23 August 2016 at 12:26 On Tue, Aug 23, 2016 at 12:17 PM, Paul Libbrecht &lt;paul(a)hoplahup.net&gt; wrote: Which is what I indicated: you just use XFF filter (which already exist) in input and XAR filter (which also already exist) as output.

I would be very concerned if we have wiki pages with dead code in some repository. I am very skeptical, because if we cannot rely on the standard exporter to generate the source correctly, who will use these source files? When you develop an application, do you write it in the XML page in your repository or directly in your wiki? I am a bit conservative here and I don't want to block you, but I really wonder if the advantages you mention worth the pain of being sure we don't have dead code, and manually cutting/pasting some bit of the code from a file to an other everytime you commit a page (and I do it often).

On 26 Aug 2016, at 18:51, Vincent Massol &lt;vincent(a)massol.net&gt; wrote: Useful but not enough IMO. This looks like just a workaround and it still means some manual action to do after you’ve export your XAR into the source tree. You still need to manually convert the document content which is XML-encoded as a non-xml encoded content in the separate file. BTW on this topic, our XAR export should really use CDATA construct and stop encoding the content. This is just a major PITA. Another idea is to offer 2 new Mojos in the XAR plugin: * mvn xar:deploy. This would push the sources into a running XWiki. The URL to be used would be defined as maven properties (and so would be the import options: overwrite, backup pack, etc). * mvn xar:fetch. This would do the opposite and it would accept 2 modes. Mode 1: it would fetch new versions of existing files in the source directory structure by fetching them from a running XWiki. Mode 2: there could be some maven property to define the name of a space it should fetch from and it would fetch all content inside that space. Now let’s come back to Paul’s proposal. The fetch operation could check the current source file in the filesystem prior to the fetching an find all xinclude and when it fetches a new version of that file (i.e. of the page), it would replace the content with the xinclude it had found and copy the content(s) in their separate file(s). This would allow not loosing the xincludes. And it would automate the whole process even more. WDYT? FTR I find Paul’s idea quite clever with XInclude. I don’t think it’s our final solution but it’s a clever one that addresses one need: that of being able to use desktop tools to edit various file types (css, js, plain text). So to conclude and while waiting for more elaborate and final solutions, I’d agree with Paul’s idea but I think we need: * Modify XWiki’s export to use CDATA instead of xml-encoding the contents. This allows to more easily and manually copy content into separate files. * Implement the 2 mojos I mentioned above which should automate nicely our process + avoid loosing xincludes. * Paul’s idea of generating a warning if there’s a file in the tree that doesn’t have a corresponding xinclude. WDYT?

Thanks -Vincent > Offering a diff is tempting but it's a lot more complex I feel (e.g. > it's better if it's three way which developers rarely all have on a > single disk, one or two of them being on a versioning server). > > Paul > > > Guillaume Delhumeau wrote: >> I would be very concerned if we have wiki pages with dead code in some >> repository. >> >> I am very skeptical, because if we cannot rely on the standard exporter to >> generate the source correctly, who will use these source files? When you >> develop an application, do you write it in the XML page in your repository >> or directly in your wiki? >> >> I am a bit conservative here and I don't want to block you, but I really >> wonder if the advantages you mention worth the pain of being sure we don't >> have dead code, and manually cutting/pasting some bit of the code from a >> file to an other everytime you commit a page (and I do it often).

BTW on a related note, I was wondering if XInclude allows including non-XML encoded content. I’ve found https://msdn.microsoft.com/en-us/library/aa302291.aspx#xinc_topic4 which seems to indicate it’s supported (with parse=text and by using the encoding attribute too).

Guillaume Delhumeau <mailto:guillaume.delhumeau@xwiki.com> 23 August 2016 at 10:51 To be extra-clear, let me explain my workflow when I design an application in XWiki. - First, I write my code directly in an instance of XWiki. - When it's ready, I export all the pages related to my application in a XAR. - Then, I uncompress my XAR inside my repository. - I clean the XML files using mvn xar:format. - I commit. And when I fix something, I do it directly in the wiki, then export the page in a XAR, uncompress it in my repository, clean the xml files manually (reverting the changed dates, etc...). What I don't want is to be forced to manually remove the source code from the XML to put it in the other file every time. Thanks,

On 26 Aug 2016, at 14:03, Paul Libbrecht &lt;paul(a)hoplahup.net&gt; wrote: Hello VIncent, Well. That brings the question back: how many are using it? Obviously, I see almost no-one.

As long as: - it is not complete - it is not used by developers big time ... it will not be considered useful. So... requesting to "go away" is very close to what you're asking, I feel.

In the contrary, I believe that changing the XAR plugin in a fully conservative way is the way to go if one wants incremental progress.

Can it be that you did not notice, it is fully backwards compatible?

But obviously the time has not been invested about it. All issues are closed for XFF.

That's a moot criterion. My code is really not a big change and also not very high risk.

Paul

On 26 Aug 2016, at 16:04, Paul Libbrecht &lt;paul(a)hoplahup.net&gt; wrote: Vincent Massol wrote: http://extensions.xwiki.org/xwiki/bin/view/Extension/DownloadAttachments/ is an example source with it. This can also be browsed at: http://direct.hoplahup.net/tmp/DownloadAttachments/src/main/resources/Macro… where DownloadAttachments.xml includes DownloadAttachments_content.groovy.

I guess that the way to test it is to build and install the project I have (which would be 8.3-SNAPHOT): http://direct.hoplahup.net/tmp/xwiki-commons-tool-xar-plugin-8.3-SNAPSHOT-p… (which you'd obtain by applying the patch I've put in the jira). I'd like to avoid forking. This creates niches. My laptop is full of such attempts. Creating a github fork, a maven polxar or something like this is fairly equivalent.

Paul

Is the warning proposal made to Guillaume sufficient?

On 26 Aug 2016, at 13:50, Vincent Massol &lt;vincent(a)massol.net&gt; wrote: Hi Paul, I’ve finally taken the time to read this discussion (a bit quickly for the last mail though, I admit ;)). Let me give my POV: * What I understand is that you’re not proposing to change the XAR format but to introduce some tooling to generate a XAR from a set of source files, at build time. Correct? * IMO we should not touch the existing XAR plugin. This is current the standard way to generate XAR files and we need to keep, if only for backward compatibility with existing sources. * IMO you should develop your tool as a new maven plugin in xwiki-contrib and use it on some contrib projects to validate and show to others how it works. Of course you can depend on the existing XAR plugin if you need tor reuse some code (it’s even possible to introduce a new api module if there’s an important need to share common code). * FTR this is exactly what Jean Simard as done with the XFF tools (https://github.com/xwiki-contrib/api-xff). He’s done is separately to validate it and show it. We need to know where we stand on XFF since it’s currently the most advanced POC that we have on introducing a new format for storing wiki pages in the SCM. It also supports both directions: from source to XAR and from wiki pages to source. I’m planning to check what’s missing in that progress to be able to use it (not much IMO).

* If you were to develop your maven plugin, you would just compete with the XFF code. This is fine, although I’d have preferred that we spend our time finishing the XFF work. It’s your time so your choice though. In the future, we’ll be able to choose which solution we want to use between the 3 that will exist (current XAR plugin, XFF solution and your solution). * FTR Fabio had also worked on this topic in the past, see http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApp…. Caleb has also worked on something related: https://github.com/xwiki-contrib/xwiki-tools-node So for me the important part is for you to develop your solution in a separate module in xwiki-contrib as a demonstator and possibly start using it on some contrib project to validate it. The alternative is to work on XFF and finish it. Personally I’m not a big fan at all of modifying the existing XAR plugin for various reasons: * We don’t know yet what solution we’ll choose in the future. At this point in time I even have a preference for XFF. So it wouldn’t be fair and nice to modify the existing XAR plugin with your solution and then have to remove the code if we don’ want it in the future. * It will be much simpler for you to develop your own project in xwiki-contrib since you’ll have commit access (which you won’t have in xwiki-commons). WDYT? Thanks -Vincent > On 22 Aug 2016, at 15:59, Paul Libbrecht &lt;paul(a)hoplahup.net&gt; wrote: > > > Hello fellow developers, > > many of you are developing XAR-packaged resources. The exchangeability > of these archives is one of the precious aspects of XWiki, I believe. > > However, if one see all the many GitHub-contributed xar packages, the > source is not there: it's in a Wiki with which one would "open it". A > pity since much of the content is code. > > I'd like to propose to allow XARs to be built form source code which is > true source that one edits. > Code files with their proper syntax coloring, editors, auto-completion, > search. Media-files that are edited with our tool of choice by a simple > double-click and not a complex pull, edit, and replace. > > Thus, my proposal is to allow inclusions within the XML files. > Inclusions should be of type: > - text to include page content or object properties (e.g. css of cssx > objects) > - attachments to include attached objects > > My current setup is documented here:http://jira.xwiki.org/browse/XWIKI-13643 > It does binaryInclude to generate base64 and textInclude to generate > XML-text. > Thus, I can advertise a github project which is a macro and whose > complex function is in a velocity file that is in a .vm file. However, I > do not need to explode everything, e.g. the text-content describing the > macro or (in this case too small) translation set. > > What do you think? > Should I rather generate complete attachment elements (name, length, > modif date, ...) instead of doing a binaryInclude that just generates > base64 ? > Is fetching files for inclusion beyond the boundaries of the project of > any use? (it is currently prohibited for safety) > > thanks in advance. > > Paul

On 26 Aug 2016, at 18:33, Vincent Massol &lt;vincent(a)massol.net&gt; wrote: More notes: - Caleb’s solution is working and is being used for some projects. For example: https://github.com/xwiki-contrib/realtime-wysiwyg/tree/master/src/xwiki Cons of Caleb solutions are listed at http://design.xwiki.org/xwiki/bin/view/Design/DirectoryStructureforXWikiApp…. Main issue points IMO: * There is no automated way to create this structure from code in the Wiki (TODO) • Still lacking a Maven job for building extensions (TODO). Requires nodejs to build. - There’s also work done by Fabio here that I had missed: https://github.com/fmancinelli/xwikifs. Fabio has chosen YAML, see https://github.com/fmancinelli/xwikifs/tree/master/xwikifs-maven-plugin/src… an example. However I don’t know what is its status. There seems to be a Maven XAR mojo written to generate a XAR file.

So we have plenty of stuff that have been tried but none has been finished and none are addressing the full needs. To summarize, so far I’ve found: * Caleb’s nodejs tool: https://github.com/xwiki-contrib/xwiki-tools-node * Fabio’s xwikifs: https://github.com/fmancinelli/xwikifs * Jean’s XFF: https://github.com/xwiki-contrib/api-xff * Paul’s XInclude extension to the XAR plugin: http://jira.xwiki.org/browse/XWIKI-13643 They all seem to be missing the ability to export wiki pages from a running wiki into the source filesystem. Thanks -Vincent > So for me the important part is for you to develop your solution in a separate module in xwiki-contrib as a demonstator and possibly start using it on some contrib project to validate it. The alternative is to work on XFF and finish it. > > Personally I’m not a big fan at all of modifying the existing XAR plugin for various reasons: > * We don’t know yet what solution we’ll choose in the future. At this point in time I even have a preference for XFF. So it wouldn’t be fair and nice to modify the existing XAR plugin with your solution and then have to remove the code if we don’ want it in the future. > * It will be much simpler for you to develop your own project in xwiki-contrib since you’ll have commit access (which you won’t have in xwiki-commons). > > WDYT? > > Thanks > -Vincent > > >> On 22 Aug 2016, at 15:59, Paul Libbrecht &lt;paul(a)hoplahup.net&gt; wrote: >> >> >> Hello fellow developers, >> >> many of you are developing XAR-packaged resources. The exchangeability >> of these archives is one of the precious aspects of XWiki, I believe. >> >> However, if one see all the many GitHub-contributed xar packages, the >> source is not there: it's in a Wiki with which one would "open it". A >> pity since much of the content is code. >> >> I'd like to propose to allow XARs to be built form source code which is >> true source that one edits. >> Code files with their proper syntax coloring, editors, auto-completion, >> search. Media-files that are edited with our tool of choice by a simple >> double-click and not a complex pull, edit, and replace. >> >> Thus, my proposal is to allow inclusions within the XML files. >> Inclusions should be of type: >> - text to include page content or object properties (e.g. css of cssx >> objects) >> - attachments to include attached objects >> >> My current setup is documented here:http://jira.xwiki.org/browse/XWIKI-13643 >> It does binaryInclude to generate base64 and textInclude to generate >> XML-text. >> Thus, I can advertise a github project which is a macro and whose >> complex function is in a velocity file that is in a .vm file. However, I >> do not need to explode everything, e.g. the text-content describing the >> macro or (in this case too small) translation set. >> >> What do you think? >> Should I rather generate complete attachment elements (name, length, >> modif date, ...) instead of doing a binaryInclude that just generates >> base64 ? >> Is fetching files for inclusion beyond the boundaries of the project of >> any use? (it is currently prohibited for safety) >> >> thanks in advance. >> >> Paul