Re: [xwiki-devs] [SUMMARY] Re: [Discussion] Designing the new Rendering/Parsing component/API

I've discussed this email below with Mikhail. We discussed 2 points: * How to support the following: The solution we've come to is the following for rendering the above: a) When parsed with wikimodel, the velocity macro is called b) the velocity macro uses an XML parser to parse its content (by adding a top level xml wrapping element for example) c) for each text content found during the XML parsing, call wikimodel on it so that the wiki syntax can be interpreted Same for the {html} macro. We could also use parameters to decide if XML parsing should be done, if wiki syntax should be interpreted, etc. For example: {html xml=true|false wikisyntax=true|false ...}, {velocity xml=true|false wikisyntax=true|false ...}. * How to speed up document rendering? We propose several caches: - level 1: Parse documents using a wikisyntax DOM parser which produces a DOM tree which is cached. This tree is composed of nodes representing macros (unparsed or XML-parsed in a XML DOM tree which is cached). - level 2: Cache the non macro blocks after they are rendered (XHTML) since they are static. - level 3: Caching at the level of the rendered XHTML. This would be a timed-cache (the cache is refreshed every N minutes). This is absolutely required for heavy sites (Imagine Apache.org using XWiki). I think we would need to cache pages for users not logged in at least. Not sure how to cache pages when users are logged in. Note1: level 2 and level 3 caches do not require a DOM tree. Note2: we need the DOM tree to speed up all macros that act on content, like the TOC macro as otherwise it means the content will need to be parsed (which is hard and slow where traversing the DOM tree is easy and fast). Since the DOM tree is cached the rendering for these macros will be fast. Note3: The TOC macro reminds me that we need somehow to support macros that should be rendered last since they operate on the full DOM tree. This is easy to do with a DOM tree but would quite harder if we were only using a stream. Any comment? Do you agree? Any other idea? Thanks -Vincent On Oct 1, 2007, at 12:51 PM, Mikhail Kotelnikov wrote: > Hi! Excuse me please for the late response... > > On 9/26/07, Vincent Massol &lt;vincent(a)massol.net&gt; wrote: > Hi xwiki devs, > > This is a summary of the decisions so far and the remaining > questions. This is also the outcome of my discussion of today with > Mikhail on skype. > > 1) We'll be able to import all syntaxes. > > Yes > > > 2) An XWiki instance will use a single syntax at a time. Once the > database is created using that syntax it won't be possible to > change it (except by doing an export and reimport). > > We also need to decide what syntax we use by default OOB. I propose > we use the current xwiki syntax for some time and then switch to > the Wikimodel one (Common Syntax) later on. > > I have no choice - I have to vote for the "CommonSyntax" :-) > > > 3) All pages will be able to be exported to any syntax. Some > elements have no equivalent in other syntaxes and when this happens > a warning will be displayed and the elements in question ignored. > > Yes. You can loose data only when you *export* your data from the > CommonSyntax, not when you *import* it. The WikiModel/CommonSyntax > support a super-set of elements defined in other wikis. > > BTW: I think that it would be useful to involve Max Völkel (http:// > xam.de/) in the discussion about wiki imports/exports. Max is one > of authors of the Semantic Media WIki. He proposed a Wiki > Interchange Format (WIF) <http://eyaloren.org/pubs/semwiki2006- > wif.pdf> which covers the topic. At least it would be useful to > know his opinion. > > > 4) Mikhail has agreed to modify wikiModel to add macro block > recognition.The syntax isn't fully defined yet but it'll be > something like: > > {xxx param1=value1 param2=value2} > ... > {/xxx} > > (param1='value value' and param1="value value" will also be supported) > > This means we'll be able to have a common syntax for xwiki's macros > and also for groovy code: > > {groovy} > ... > {/groovy} > > And also for HTML blocks: > > {html} > ... > {/html} > > It is already done and committed. You can use "embedded" macro > blocks like > {xxx param1=value1 param2="long value 2" param3='this is a > parameter 3'} > ... {yyy} ... {/yyy} ... > {/xxx} > > Even embedded elements with the same names are possible: > {xxx} > ... > {xxx} > ... {yyy} ... {/yyy} ... > {/xxx} > ... > {/xxx} > > > 5) We need to decide if we want: > > A) No velocity block but document properties/metadata to tell xwiki > to render the page using velocity. A user putting velocity code in > a page will have to check a box somewhere to say that this is a > velocity page. > > Pros: > * Slightly easier to enter velocity code > > Cons: > * Exception case compared to groovy, macros, etc > * User must not forget to check the checkbox saying the page > contains velocity code > > OR > > B) Velocity blocks same as what exists for groovy/macros/html, namely: > > {velocity} > ... velocity code with wiki syntax allowed > {/velocity} > > Note1: For B) we would allow putting wiki syntax inside the > velocity block. Technically we'll apply the velocity rendering and > then re-apply wikimodel on the result. > Note2: For backward compatibility we can have a config flag > ( xwiki.compatibility = 1) that automatically adds the {velocity}{/ > velocity} marker around the whole page. The only downside is that > it'll be as slow as it currently is (actually it'll be faster since > wikimodel is going to be faster than radeox) > > Pros: > * Speed. Since we know the blocks that use velocity we can cache > all the wiki syntax not inside the velocity/macros/groovy blocks > which will speed up considerably the rendering of pages > * Consistency with macros and groovy blocks. > > My preferences goes to B) and I'm proposing to use that. > > My +1 for B) > > > 6) Mikhail is going to add support for recognizing XML tags in the > wikimodel parser so that onOpenXmlTag()/onCloseXmlTag() events are > called in listeners). This is needed for point 7 below. > > Hmm... Yes, I said that I'll do it. Technically it is simple > (simpler than to add "macro" blocks). But conceptually it breaks > everything. > Explanations: Imagine that the listener already has the onOpenXmlTag > (String name, WikiParameters params)/onCloseXmlTag(String name) > methods. > Then the text: > ---------------------- > <table>... > > This is a content of the table > > ...</table> > ---------------------- > > will be reported as following: > ---------------------- > - onBeginParagraph > - onOpenXmlTag: => "table" > - onEndParagraph > > - onBeginParagraph > - onWord/onSpace => "This is a content of the table" > - onEndParagraph > > - onBeginParagraph > - onCloseXmlTag: => "table" > - onEndParagraph > ---------------------- > > It means that the calls onOpenXmlTag/onCloseXmlTag will cross the > borders of multiple wiki paragraphs. And this is a BIG problem. > It means that we have to chose one of the following: > (A) Report HTML tags "as is" inside of wiki structural elements; In > the example above opening and closing "table" tags are reported in > two separate wiki paragraphs. > pro: It is the simplest solution. > con: if somebody want to treat these elements and create a well > formed document then it is up to the him/her to do it by hands and > to ignore non-appropriate structural wiki elements; > (B) Ignore wiki formatting inside of XML tags. In the example above > all wiki paragraphs will be skipped and only HTML "table" tags will > be taken into account; in this case WikiModel can not guaranties > that each opening element was really closed. > pro: It is doable. > con: a) It breaks completely the idea of the WikiModel - to give > access to a well-formed structure of wiki documents; b) the grammar > will be bigger; c) It is not so simple to implement > (C) Add some HTML tags as markup elements for the CommonSyntax. In > this case each '<table>...</table>' tag pair will be interpreted in > the same way as normal wiki tables. The same for "ol/ul/li/dd/dl/dt/ > p/span/div/..." elements. > pro: you can mix your wiki and HTML markup with the same meaning > and all of them will be reported in the same way to the listeners. > con: a) the grammar will be bigger; b) I have to do much more > additional validations of documents by hand to guarantee that the > document is well-formed; c) the parsing will be much slower (as the > consequence of a and b); d) it is much more difficult to > implement; e) the number of possible HTML elements have to be fixed > in advance > > From my point of view neither option is good. > > One another approach to resolve this problem - see below, in the > response to the next question. > > 7) We need to allow intermixing velocity/HTML and wiki syntax > easily. For this our listener (the code that listens to {velocity} > events) will evaluate the content using velocity and will call wiki > model again on the resulting code. Since wikimodel requires HTML to > be in a block ({html} for us) we'll use a different wikimodel > listener that intercepts the onOpenXmlTag/onCloseXmlTag so that > it'll output XML tags with no modifications (the standard HTML > listener generates &lt; and &gt; for < and >). This will allow > writing: > > > I propose to interpret the content of such blocks in the following > manner: > - The given text is interpreted as a "relaxed-XML" where all > opening XML tags have to be closed (or a tag has to be empty) > - If an XML tag has a text content (not only spaces) then this > content is interpreted as a wiki syntax. In this case the wiki > content can be handled by a normal wiki parser. > > In this case the scenario of usage can be following: > 1. You parse your initial wiki document and create a backbone > structure of the document containing macro blocks > 2. All macro blocks corresponding to template blocks are handled by > the corresponding engine (ie velocity) > 3. The output from template engines is used as an input for such a > "relaxed-XML" parser > 4. A not-empty text content of tags is interpreted as wiki content > using separate wiki parser instances; the content formed by these > wiki blocks should be inserted in the document from the step 3. > > pro: > * It seems that it is the simplest solution and it resolves > problems with inter-mixing of the wiki syntax/XML/HTML from the > previous point (if such inter-mixing is available only in macro > blocks). > * For such a "relaxed-XML" content existing XML parsers can be > used. It is possible just to add the "<xml>...</xml>" pair around > the content and use directly a normal XML parser. But there is a > risk that this document is not a well-formed XML. Or an HTML > cleaner (JTidy/NekoHTML/...) can be used to get a well-formed XHTML > before the XML parsing. > * All steps gives well-formed structures > con: More parsers in the chain of the page treatment => the > treatment is slower (this can be partially compensated by > additional caches in the future) > > Personally I definitely prefer this approach. And from my point of > view it resolves the problems with onOpenXmlTag/onCloseXmlTag > mentioned above. > > 8) Documents are stored in textual format in the DB (i.e. as the > user sees them). Portions of them will be cached after they're > rendered for the first time (see option B above for the best > caching option). > > Are you ok on these points and especially about using the 5B solution? > > Anything else I've forgotten? > > If we agree, then my next steps are: > * Understand the wikimodel API in more details > * Understand the doxia API too (it's a "competitor" to wikimodel). > The reason is that I'd like to see two implementations to ensure > that the XWiki Interfaces can be implemented using different > implementations so that XWiki is independent of the underlying > rendering/parsing framework used. Jason Van Zyl is also interested > in implementing the doxia part for XWiki in the future. > > I think that in any case it is a good idea to have a specialized > interface (API) for each type of functionalities to isolate the > core of the system from external libraries/frameworks. > > BTW: Thank you for the pointer on Doxia! I will see it more in > details. After a brief look I saw that WikiModel contains very > similar modules with the "Sink" object of Doxia. I think that it is > very simple to create a WikiModel/Doxia bridge. And it seems > possible to add the APT ( http://maven.apache.org/doxia/references/ > apt-format.html) format support directly to the WikiModel. > > Best regards > Mikhail > > * Propose a XWiki API > * Propose an integration path > * Implement it using WikiModel > > Thanks > -Vincent