[xwiki-devs] Parser Issuer

[malaka ekanayake]

Hi Fabio,/Vincent/Venkatesh

On Wed, Jul 2, 2008 at 12:30 AM, Fabio Mancinelli <
fabio.mancinelli at xwiki.com> wrote:

> On 1 juil. 08, at 11:12, Vincent Massol wrote:
>
>  Just thought about a potential use case for storing the offset: error
>> reporting.
>>
>> We'll also need it if we want to do auto completion in the wiki editor
>> which I think would be nice to have.
>>
>> Ok I'll try to add offset/length to the rendering module. It's going
>> to cost us some performance actually since when there are block
>> transformations you need to recompute the blocks offsets... I still
>> need to know how it's best to store them for your needs Malaka.
>> Storing them in the Blocks themselves might cost too much to retrieve
>> them. I could also store them in some other structure with references
>> to the blocks. Let me know what you need.
>>
>>  To speed up things I've done a bit of investigations (Malaka, please
> integrate with your own findings).
>
> I've looked at how these things are done in other complex projects, and
> maybe I think it will not be possible to use the actual rendering engine's
> parser as the basis for the Eclipse infrastructure.
>
> The Eclipse JDT uses a FastJavaPartitionScanner (
> http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.jdt.ui/ui/org/eclipse/jdt/internal/ui/text/FastJavaPartitionScanner.java?revision=1.13)
> that basically implements an ad-hoc scanner for isolating blocks (there is
> plenty of while(true), etc., i.e. clearly it isn't a standard Java parser)
>
> The Groovy editor uses the standard plain RuleBasedPartitionScanner (
> http://svn.codehaus.org/groovy/trunk/groovy/ide/groovy-eclipse/org.codehaus.groovy.eclipse.ui/src/org/codehaus/groovy/eclipse/editor/GroovyPartitionScanner.java)
> suitably customized for its own needs, with a lot of ad-hoc rule-based
> scanners (e.g.,
> http://svn.codehaus.org/groovy/trunk/groovy/ide/groovy-eclipse/org.codehaus.groovy.eclipse.ui/src/org/codehaus/groovy/eclipse/editor/GroovyTagScanner.java
> ).
>
> Apparently the parsing of the file with the real parser is done only in the
> generation of the content outline (and in the error reporting). This is
> noticeable in the Java IDE: the content outline is updated only when the
> file is saved (i.e., compiled and so parsed with the actual Java parser).
> The same is also true in the Groovy code and in the XMLEditor example where
> the SAXParser is called in the OutlineContentProvider when the document is
> saved.


Thank you very much for the  finding.I think what you are saying is
correct.I wrote a wrapper class of FastPartitioner and see what are the
methods which are called when typeing.It shows that for each keyperss the
following methodes woll be called to repartition the document

getContentType
documentAboutToBeChanged
getPartition
documentChanged
getPartition
computePartitioning

  If I'm going to use the rendering passer then I need to recompute the XDOM
for each keypress and it would be too costly to do.

>
>
> Why this architecture is necessary, imho, is because the editor's parser
> should be as loose as possible and resilient to (transient) errors. In fact,
> when the user is typing the document is almost always in an inconsistent
> state (which makes the real parser fail). Examining the document while
> editing with the actual parser also means to parse the document at every
> keystroke: that's what happens in practice: the partitioner is asked to
> return next tokens starting from arbitrary positions of the text: a request
> that is not very well fulfilled by the real parser.


Now I think I needed to write a some kind of xwiki parser which suit the
requirments.I dont think available RuleBasedPartitionScanner alone cannot be
use to define the partitions because of the recursive nature of the xwiki
syntax.

>
>
> From these considerations, I think that probably using the rendering
> engine's parser is not feasible, and we will need to backtrack and build a
> "parallel" infrastructure for parsing XWiki documents. Infrastructure that
> has already been put in place by Venkatesh and that Malaka, at this point,
> will finish and improve for supporting all the necessary features and
> variants.


I will first look at the  sample parseres you have given above and  try to
write a xwiki parser that suite the job.Is that ok.Then for the syntax
coloring,we can use the rule base partitioning which is already available in
jface because it is done in per partition basis.

>
>
> Malaka could you comment on this?
> And in general, WDYT?


>
> -Fabio

WDYT ? .

--Malaka Ekanayake
CSE UOM