(resending since I got a failure from xwiki's mail content filter ) ---------- Forwarded message ---------- From: Vincent Massol <[email protected]> Date: Sun, Jul 26, 2009 at 10:21 AM Subject: Re: [xwiki-devs] adding a new jsr-223 scripting language to xwiki (clojure) To: XWiki Developers <[email protected]> Hi Niels, On Jul 26, 2009, at 10:13 AM, Niels Mayer wrote:

In http://code.xwiki.org/xwiki/bin/view/Snippets/DisplayFormatedCodeAttachments...

...

VincentMassol | 2009/05/05 20:03

...

Niels, no there's no plan but we can now support any language using macros

so feel free to provide a clojure macro if you're interested. Shouldn't be too hard to do, especially if there's a JSR-223 implementation (in which case simply dropping the jar in WEB-INF/lib should be enough - you'd then use it using the script macro).

Clojure <http://groups.google.com/group/clojure> has aJSR-223 implementation according to: http://github.com/pmf/clojure-jsr223/tree/master http://groovy.codehaus.org/JSR-223+access+to+other+JVM+languages http://sayspy.blogspot.com/2009/03/interacting-between-jvm-lang-here-and.htm...

Are there any examples, documentation, or suggestions of writing a "script macro" to call a new jar in WEB-INF/lib ?

Well all you need is drop the JSR223 jar in WEB-INF/lib and then you can use the new language using the script macro: http://code.xwiki.org/xwiki/bin/view/Macros/ScriptMacro Then if you want you can write a wrapping macro such as we've done for groovy and jruby so that it can be called with {{clojure}} instead of {{script language="clojure"}}.

Any other special magic I should know? Does anything from the above suggest that Clojure couldn't be used as an Xwiki scripting language, replacing cases where Groovy scripting might normally be employed?

Niels http://nielsmayer.com

PS: Here's examples of easy "scripted java" programming you can do in Clojure (note the helpful parallelism constructs): http://travis-whitton.blogspot.com/2009/07/network-sweeping-with-clojure.htm... http://travis-whitton.blogspot.com/2009/06/clojures-agents-scientists-monkey...

It could be very useful to employ massive parallelism via such Clojure scripts, which could achieve a xwiki-based web portal performance akin to Yahoo's, Google's, etc. For example, the following describes how Yahoo works -- and would be quite easy to implement this kind of processing "for free" in Clojure with very little code:

http://research.yahoo.com/files/pnuts.pdf

I'm definitely +1 to make it part of the XWiki platform (same as for jruby) if you develop it. Similar to jruby I don't think we should package it in the default WAR though but as an optional download to install (will be even easier with the app manager). Thanks -Vincent

The component responsible for multi-record requests is called the

...

scatter-gather engine, and is a component of the router. The scatter-gather engine receives a multi-record request, splits it into multiple individual requests for single records or single tablet scans, and initiates those requests in parallel. As the requests return success or failure, the scatter-gather engine assembles the results and then passes them to the client. In our implementation, the engine can begin streaming some results back to the client as soon as they appear. We chose a server-side approach instead of having the client initiate multiple parallel requests for several reasons. First, at the TCP/IP layer, it is preferable to have one connection per client to the PNUTS service; since there are many clients (and many concurrent processes per client machine) opening one connection to PNUTS for each record being requested in parallel overloads the network stack. Second, placing this functionality on the server side allows us to optimize, for example by grouping multiple requests to the same storage server in the same web service call.

Range queries and table scans are also handled by the scatter gather engine. Typically there is only a single client process retrieving the results for a query. The scatter gather engine will scan only one tablet at a time and return results to the client; this is about as fast as a typical client can process results. In the case of a range scan, this mecha- nism simplifies the process of returning the top-K results (a frequently requested feature), since we only need to scan enough tablets to provide K results. After returning the first set of results, the scatter-gather engine constructs and returns a continuation object, which allows the client to re- trieve the next set of results. The continuation object con- tains a modified range query, which, when executed, restarts the range scan at the point the previous results left off. Con- tinuation objects allow us to have cursor state on the client side rather than the server. In a shared service such as PNUTS, it is essential to minimize the amount of server- side state we have to manage on behalf of clients.