July 2018 - xwiki-devs@xwiki.org

[xwiki-devs] [ANN] XWiki 10.6RC1 released
by Marius Dumitru Florea 16 Jul '18

16 Jul '18

The XWiki development team is proud to announce the availability of XWiki 10.6RC1. This release adds a RSS view to the Notifications macro and improves the user picker with compact display and in-line selection. The developers should check the new Page API and the improvements to the existing Icon API. In addition there have been 23 bugs fixed and a couple of small improvements done. You can download it here: http://www.xwiki.org/xwiki/bin/view/Main/Download Make sure to review the release notes: http://www.xwiki.org/xwiki/bin/view/ReleaseNotes/Data/XWiki/10.6RC1/ Thanks for your support -The XWiki dev team

1 0

[xwiki-devs] [Contrib] New repo and JIRA for an OpenStreetMap macro
by Stéphane Laurière 12 Jul '18

12 Jul '18

Hi devs, We have a MapMacro already [1], for displaying Google Maps. I'd like to create a distinct one for creating maps based on OpenStreetMap. Another option would be to enhance the existing MapMacro for supporting the display of OSM maps in addition, but this would make the code more complex, and in general an administrator willing to use maps in his wiki is likely to opt for one map type, probably not both, what do you think? If you agree, the proposed repo name and JIRA would be: - Repo: macro-openstreetmap - JIRA project: OSM [1] https://extensions.xwiki.org/xwiki/bin/view/Extension/Map%20Macro Thank you, Stéphane -- Stéphane Laurière XWiki www.xwiki.com @slauriere

6 15

[xwiki-devs] First batch of the new experimental Page oriented APIs
by Thomas Mortagne 09 Jul '18

09 Jul '18

Hi xwikiers, Some of us discussed that for a while and created/improved http://design.xwiki.org/xwiki/bin/view/Proposal/DeprecatingSpaceAndSpaceRef… and I finally started working on it. So here is what I pushed on master already: = PAGE EntityType and Page*Reference classes First thing first we now have a few new type in EntityType and the corresponding typed Page*Reference helpers. One for each of the element you can have in a page: * PAGE * PAGE_ATTACHMENT * PAGE_OBJECT * PAGE_OBJECT_PROPERTY * PAGE_CLASS_PROPERTY = Corresponding resolvers/serializer/providers Each of the already existing DOCUMENT oriented resolvers and serializers have their PAGE oriented implementation. = Conversion from DOCUMENT to PAGE world The entity resolvers automatically convert from/to DOCUMENT from/to PAGE world references. Do yes using a resolver is the official way to do that conversion. = Model script service The $services.model API also got his own new helpers to manipulate pages = New syntax Last but not least: pages reference have a very different syntax. To summarize it's a filesystem path syntax with support for parameters. Here is are a few examples: wiki:page1/page2;param1=value1;param2=value2;en_US wiki:page1/page2/attachment ../siblingpage * separator: "/" between all elements except WIKI which is still ":" * escaping: still "\" * current, parent support: like in a filesystem you can refer to the current page or parent page/wiki using "." and ".." * parameters: we now have syntax to express the parameters of an EntityReference. Each parameter is separated by a ";" at the end of the entity name * default parameter: the syntax have the concept of default parameter which mean a parameter for which you don't need to indicate the name. For example PAGE reference default parameter is "locale" = TODO The next elements on the short term TODO list are: * support for page references in XWiki#getDocument (which essentially means add a fallback) * support for page references in various macros (generally means adding a "page" parameter as alternative to the existing "reference" or "document" parameter) Thanks, -- Thomas Mortagne

4 9

[xwiki-devs] [GSoC] Reminder: 2nd evaluation deadline on July 13, 16:00 UTC (less than 1 week)
by Eduard Moraru 07 Jul '18

07 Jul '18

Hello, GSoC students and mentors. Quick reminder about the 2nd evaluation that is quickly coming up [1]: July 9 16:00 UTC Mentors and students can begin submitting Phase 2 evaluations July 13 16:00 UTC Phase 2 Evaluation deadline Same as last time, make sure you are clear on the expectations and *do not forget to fill in your evaluations* when asked by Google. You should be quite far now into your projects and things should have already begun taking shape. Don't be shy in sharing your progress ;) As always, have fun and build cool stuff! Thanks, Eduard ---------- [1] https://developers.google.com/open-source/gsoc/timeline

1 0

[xwiki-devs] Request to move Blockly fork to Contrb
by Vivek Iyer 07 Jul '18

07 Jul '18

Hi everyone! I would like to request my Blockly fork (https://github.com/Remorax/blockly) to be moved to xwiki-contrib. Please do the needful ASAP :) Thanks and regards, Vivek Iyer

2 1

[xwiki-devs] [VOTE] Add new check to measure quality of tests
by Vincent Massol 06 Jul '18

06 Jul '18

Hi devs, As part of the STAMP research project, we’ve developed a new tool (Descartes, based on Pitest) to measure the quality of tests. It generates a mutation score for your tests, defining how good the tests are. Technical Descartes performs some extreme mutations on the code under test (e.g. remove content of void methods, return true for methods returning a boolean, etc - See https://github.com/STAMP-project/pitest-descartes) If the test continues to pass then it means it’s not killing the mutant and thus its mutation score decreases. So in short: * Jacoco/Clover: measure how much of the code is tested * Pitest/Descartes: measure how good the tests are Both provide a percentage value. I’m proposing to compute the current mutation scores for xwiki-commons and xwiki-rendering and fail the build when new code is added that reduce the mutation score threshold (exactly the same as our jacoco threshold and strategy). I consider this is an experiment to push the limit of software engineering a bit further. I don’t know how well it’ll work or not. I propose to do the work and test this for over 2-3 months and see how well it works or not. At that time we can then decide whether it works or not (i.e whether the gains it brings are more important than the problems it causes). Here’s my +1 to try this out. Some links: * pitest: http://pitest.org/ * descartes: https://github.com/STAMP-project/pitest-descartes * http://massol.myxwiki.org/xwiki/bin/view/Blog/ControllingTestQuality * http://massol.myxwiki.org/xwiki/bin/view/Blog/MutationTestingDescartes If you’re curious, you can see a screenshot of a mutation score report at http://massol.myxwiki.org/xwiki/bin/download/Blog/MutationTestingDescartes/… Please cast your votes. Thanks -Vincent

6 12

[xwiki-devs] [Brainstorming] Preparation for XWiki 10.7 aka the "Quality Release"
by Vincent Massol 05 Jul '18

05 Jul '18

Hi devs, We’ll soon start XWiki 10.7 (see https://markmail.org/message/qjemnip7hjva2rjd) Goals ===== There are 2 goals for this release: 1) Close as many bugs as possible (note that I didn’t say “fix” ;), the goal is really to reduce the number of open bugs and thus to close won’t fix, duplicates, etc and also to fix low hanging fruits, i.e. easy bugs). The goal is quantity. To try to reduce our "bug lag" Our current status today: * -47 bugs over 120 days (4 months), i.e. we need to close 47 bugs to have created bugs # = closed bugs # * -95 bugs over 365 days (1 year) * -160 bugs over 500 days (between 1 and 2 years) * -331 bugs over 1600 days (4.3 years) A good result would be to close 47 bugs during 10.7 and an excellent goal would be to close 95 bugs during 10.7 (i.e. have as many bugs closed than opened for the past year). 2) Improve our tests and make sure that our global TPC is increasing again and not going down. See * http://markmail.org/message/up2gc2zzbbe4uqgn * http://markmail.org/message/grphwta63pp5p4l7 * http://markmail.org/message/hqumkdiz7jm76ya6 I think the following activities would be good one for 10.7: * Increase coverage especially for modules that have lost coverage. See all the lines in red on https://up1.xwikisas.com/#-GNXv9QYlBWPXTHNnvQD2g which should be high priority modules. * Add tests for modules that don’t have tests yet (for example I added some functional tests last week to the xwiki-platform-menu module which didn’t have any test at all) * Once coverage has been increased, up the jacoco threshold wherever possible ;) * IMPORTANT: Fix known flickering tests * (easy, to relax ;)) Convert JUnit3 and JUnit4 tests to JUnit5 Repartition of Work =============== Ideally we should spend 1/2 on BFD and 1/2 on Tests. We could say that the 1st 15 days are on Tests and the last 15 days on BFD (or the opposite), or just let everyone handle his own time table and just make sure we do roughly half of both activities. I don’t think it would be good to have devs focus only on tests and others only on BFD. I’d really prefer that each dev does 1/2 of both. My preference goes to leave each dev choose when they work on BFD and on tests with an agreement that we will try to do half of each. WDYT? Thanks -Vincent

3 4

[xwiki-devs] Fwd: [Brainstorm] Notifications filters capabilities and performances.
by Guillaume Delhumeau 05 Jul '18

05 Jul '18

I resend this message that was accidentally sent to Sergiu only. ---------- Forwarded message ---------- From: Guillaume Delhumeau <guillaume.delhumeau(a)xwiki.com> Date: 2018-07-02 16:06 GMT+02:00 Subject: Re: [xwiki-devs] [Brainstorm] Notifications filters capabilities and performances. To: Sergiu Dumitriu <sergiu(a)xwiki.org> Hi and thanks you for your answers. 2018-06-29 5:41 GMT+02:00 Sergiu Dumitriu <sergiu(a)xwiki.org>: > The two main problems that I see are that you're mixing two separate > things: > > 1. Tags which are stored in one place, and events which are stored in a > different place > 2. Again tags, and document locations. They may seem similar, but they > are completely different as they are implemented. That's why saying "not > in this wiki, except for this tag" cannot ever be implemented in a sane > way since the exception is on a different plane. > > The logical approach would be to also store the tags in the events, but > this is not a good thing to do. Also, a tag could be added to a page AFTER the event has been triggered. In this case, if you still want to fetch these pages, it won't work. > Where do we stop? For future-proofing > filters, we would have to store the entire document in the event, in > case someone wants to add a filter on "documents with a specific type of > object attached", or "documents with a specific value for a specific > property of a specific object", or "documents with attachments ending in > .extension". > > On 06/28/2018 04:44 PM, Guillaume Delhumeau wrote: > > For the tags filter, I can also: > > > > * perform a query to fetch all documents that correspond to the given > tags > > (it doesn't need to be permanent, but it would be better to cache it) > > * add a HQL filter on these pages (OR document IN :list_of_pages). > > > > It's a little variation of solution A. It's ugly but it could do the job. > > It's the sanest proposal so far, but still impossible. > Apparently it's the way it's done on Activity Stream: https://github.com/xwiki/xwiki-platform/blob/553afe7287dcc4ce8b588276e639bf 283352e805/xwiki-platform-core/xwiki-platform-activitystream/xwiki-platform- activitystream-ui/src/main/resources/Main/Activity.xml#L1008 > > Since "not in LOCATION except WITH TAG" doesn't work because it mixes > two types of information in the same query fragment, how about making > this a one-dimensional query as "only WITH TAG". And since this > information is in a different place, pre-querying it is needed. However, > if there are thousands or tens/hundreds of thousands of documents with a > tag, won't the query crash anyway? > Sure, it won't scale. > > > 2018-06-27 12:58 GMT+02:00 Guillaume Delhumeau < > > guillaume.delhumeau(a)xwiki.com>: > > > >> Hi developers. > >> > >> I am trying to add a new filter to the notifications to be able to > follow > >> pages > >> that are marked with a given tag. And it leads me to some questions > about > >> the > >> technical implementation of the notifications. > >> > >> To remind the context: notifications are computed on top of the events > >> recorded > >> by the event stream (a.k.a activity stream). We take events from the > event > >> stream SQL table, we apply some transformations on them, and we display > >> them to > >> the user. > >> > >> Then we have implemented the ability to filter on these events: for > >> example > >> "don't show events concerning the document A nor the wiki B". Filters > are > >> implemented with 2 distinct ways: > >> > >> 1/ SQL injections: each filter can add SQL elements in the query we > make > >> to > >> fetch the events from the event stream table. We made this > mechanism > >> so we > >> can let the database do a lot of the filtering process. After all, > >> it's its > >> job and it's supposed to perform well. To be precise, Clement has > >> even > >> created an Abstract Syntax Tree (AST) so it's easier to inject some > >> content > >> in the query and it creates an abstraction over the SQL language so > >> we can > >> even consider to change the storage of the event stream someday. > >> > >> The bad thing is that some complex filtering are difficult to write > >> with > >> the SQL language (event with the AST) or even impossible. > >> > >> 2/ Post-filtering: after the events have been fetched from the > database, > >> each > >> filter can still decide to keep or filter them. This is useful for > >> complex filtering that cannot be expressed with the SQL language. > It > >> is > >> also needed by the real-time notification email sender, because it > >> takes > >> the events immediately when they occurs without fetching them in > the > >> database (so SQL filters are bypassed). > >> > >> The bad thing is that some events are loaded in the memory to > finally > >> be > >> rejected, and these filters can perform costly operations such as > >> loading > >> documents. > >> > >> Until now, this double mechanism was working quite well, with each > >> mechanism > >> filling the lacks of the other. > >> > >> However, we still have technical limitations in our design: > >> > >> 1/ Users who have a lot of filter preferences can end up with a giant > >> SQL > >> query that is almost impossible to perform by the database. > Actually > >> we had > >> a user complaining about an OutOfMemory problem in the HQL to SQL > >> translator ! > >> > >> 2/ I cannot implement the tag filter ! > >> > >> The tag filter is supposed to show events that concern pages that hold a > >> given > >> tag, EVEN IF THE PAGE WAS EXCLUDED BY THE USER. Example of use-case: "I > >> don't > >> want to receive notifications about wiki A except for pages marked with > >> the tag > >> T". > >> > >> And it is not working. First because it is difficult to write a SQL > query > >> for > >> that. It requires to make a join clause with the document and the object > >> tables, > >> which our SQL injection mechanism does not support. Even if it were > >> possible, > >> creating a SQL join with the document table will de-facto filter events > >> that do > >> not concern any pages or pages that do not have any objects: so many > other > >> filters will be broken. I also don't consider creating a SQL subquery, I > >> think > >> the whole query would became too big. So I decided to just not inject > any > >> SQL > >> code for this filter and only implement the post-filtering mechanism. > >> > >> But the other filter "EXCLUDE WIKI A" generates a SQL injection such as > >> "WIKI <> 'WIKI A'" so the events concerning the wiki A are not fetched > >> from the > >> database. Consequence: the tag filter never see the events that it is > >> supposed > >> to keep. It would be actually possible to by-pass the first SQL > injections > >> by > >> injecting something like "OR 1=1". But doing something like this is like > >> dropping the all SQL injections mechanism. > >> > >> I see some solutions for this problem: > >> > >> A/ For each tag, create a permanent list of pages that hold it. So I > can > >> inject "OR document IN (that_list)". I think this is heavy. > > Definitely not. You'll have to maintain this list in parallel to the > canonical storage for tags, the xwikidoc table itself, with all the > headaches that data duplication brings. > I agree. > > >> B/ Drop the SQL injection mechanism and only rely on the > post-filtering > >> mechanism. It would require to load from the database A LOT of > events, > >> but maybe we could cache this. > > Nope, filtering millions of events by inspecting them one by one is a > performance nightmare. > > >> C/ Don't drop the SQL injection mechanism completely but use it as > >> little as > >> possible (for example, do not use it for LOCATION filtering). Seems > >> hard to > >> determine when a filter should use this feature or not. > > I think location filtering should still happen in the query itself, > since it is a lot faster. > > >> D/ Don't implement the "tags" filter, since it is the root of the > issue. > >> But > >> it is like sweeping dirt under the carpet! > > Well, not implementing something is always the fastest, least intrusive > solution to any problem... > > Humor aside, I think that in this case it is the best approach. > Thanks for saying it :) > > >> Since we have the OutOfMemory problem with the SQL injections becoming > too > >> huge, > >> I am more in favor of solution B or C. But I'm not sure for now, since I > >> do not > >> know how much it would impact the performances and the scalability of > the > >> whole > >> notifications feature. > >> > >> This is a complex topic, but I hope this message will inspire you some > >> suggestions or things I have not seen with my own eyes. > >> > >> Thanks for your help, > >> > >> Guillaume > >> > > > > > > > Other options: > > E/ Drop SQL altogether, move the event stream into a nosql database, and > do stream filtering; kind of like the B proposal > That's unfortunately out of the scope right now. > > F/ Drop SQL and querying altogether, switch to a pub-sub mechanism where > a subscriber is created for every user and his filters, and matching > events are placed in a queue until they are all sent (consumed) in a > notification. Obviously, this only works for email notifications, not > for browsing past events in a webpage. > Indeed it cannot replace Activity Stream. However, it was my first design attempt: http://design.xwiki.org/xwiki/bin/view/Proposal/ NotificationCenterforAppsImplementation#HIteration1 But then we decided to rely on what we had (the event stream) and to kill 2 birds with the same stones (having notifications + replacing activity stream). Thanks, Guillaume > > -- > Sergiu Dumitriu > http://purl.org/net/sergiu/ > -- Guillaume Delhumeau (guillaume.delhumeau(a)xwiki.com) Research & Development Engineer at XWiki SAS Committer on the XWiki.org project -- Guillaume Delhumeau (guillaume.delhumeau(a)xwiki.com) Research & Development Engineer at XWiki SAS Committer on the XWiki.org project

1 0

[xwiki-devs] [XWiki Day] BFD#183
by Alex Cotiugă 05 Jul '18

05 Jul '18

Hello devs, This Thursday is BFD#183: http://dev.xwiki.org/xwiki/bin/view/Community/XWikiDays#HBugfixingday <http://dev.xwiki.org/xwiki/bin/view/Community/XWikiDays#HBugfixingdays> Our current status is: * -47 bugs over 120 days (4 months), i.e. we need to close 47 bugs to have created bugs # = closed bugs # * -95 bugs over 365 days (1 year) * -160 bugs over 500 days (between 1 and 2 years) * -331 bugs over 1600 days (4.3 years) See https://jira.xwiki.org/secure/Dashboard.jspa?selectPageId=10352 Here's the BFD#183 dashboard to follow the progress during the day: https://jira.xwiki.org/secure/Dashboard.jspa?selectPageId=14308 Happy Bug Fixing Day, Alex

1 0

[xwiki-devs] [Brainstorm] Notifications filters capabilities and performances.
by Guillaume Delhumeau 02 Jul '18

02 Jul '18

Hi developers. I am trying to add a new filter to the notifications to be able to follow pages that are marked with a given tag. And it leads me to some questions about the technical implementation of the notifications. To remind the context: notifications are computed on top of the events recorded by the event stream (a.k.a activity stream). We take events from the event stream SQL table, we apply some transformations on them, and we display them to the user. Then we have implemented the ability to filter on these events: for example "don't show events concerning the document A nor the wiki B". Filters are implemented with 2 distinct ways: 1/ SQL injections: each filter can add SQL elements in the query we make to fetch the events from the event stream table. We made this mechanism so we can let the database do a lot of the filtering process. After all, it's its job and it's supposed to perform well. To be precise, Clement has even created an Abstract Syntax Tree (AST) so it's easier to inject some content in the query and it creates an abstraction over the SQL language so we can even consider to change the storage of the event stream someday. The bad thing is that some complex filtering are difficult to write with the SQL language (event with the AST) or even impossible. 2/ Post-filtering: after the events have been fetched from the database, each filter can still decide to keep or filter them. This is useful for complex filtering that cannot be expressed with the SQL language. It is also needed by the real-time notification email sender, because it takes the events immediately when they occurs without fetching them in the database (so SQL filters are bypassed). The bad thing is that some events are loaded in the memory to finally be rejected, and these filters can perform costly operations such as loading documents. Until now, this double mechanism was working quite well, with each mechanism filling the lacks of the other. However, we still have technical limitations in our design: 1/ Users who have a lot of filter preferences can end up with a giant SQL query that is almost impossible to perform by the database. Actually we had a user complaining about an OutOfMemory problem in the HQL to SQL translator ! 2/ I cannot implement the tag filter ! The tag filter is supposed to show events that concern pages that hold a given tag, EVEN IF THE PAGE WAS EXCLUDED BY THE USER. Example of use-case: "I don't want to receive notifications about wiki A except for pages marked with the tag T". And it is not working. First because it is difficult to write a SQL query for that. It requires to make a join clause with the document and the object tables, which our SQL injection mechanism does not support. Even if it were possible, creating a SQL join with the document table will de-facto filter events that do not concern any pages or pages that do not have any objects: so many other filters will be broken. I also don't consider creating a SQL subquery, I think the whole query would became too big. So I decided to just not inject any SQL code for this filter and only implement the post-filtering mechanism. But the other filter "EXCLUDE WIKI A" generates a SQL injection such as "WIKI <> 'WIKI A'" so the events concerning the wiki A are not fetched from the database. Consequence: the tag filter never see the events that it is supposed to keep. It would be actually possible to by-pass the first SQL injections by injecting something like "OR 1=1". But doing something like this is like dropping the all SQL injections mechanism. I see some solutions for this problem: A/ For each tag, create a permanent list of pages that hold it. So I can inject "OR document IN (that_list)". I think this is heavy. B/ Drop the SQL injection mechanism and only rely on the post-filtering mechanism. It would require to load from the database A LOT of events, but maybe we could cache this. C/ Don't drop the SQL injection mechanism completely but use it as little as possible (for example, do not use it for LOCATION filtering). Seems hard to determine when a filter should use this feature or not. D/ Don't implement the "tags" filter, since it is the root of the issue. But it is like sweeping dirt under the carpet! Since we have the OutOfMemory problem with the SQL injections becoming too huge, I am more in favor of solution B or C. But I'm not sure for now, since I do not know how much it would impact the performances and the scalability of the whole notifications feature. This is a complex topic, but I hope this message will inspire you some suggestions or things I have not seen with my own eyes. Thanks for your help, Guillaume

3 4