August 2018 - xwiki-devs@xwiki.org

[xwiki-devs] [VOTE] Use discourse for the "dev" mailing list
by Guillaume Delhumeau 18 Sep '19

18 Sep '19

Hi devs, I think we are now satisfied with our experimentation with the "users" mailing list on discourse. I propose you to move the "dev" ML too. My main point is that it allows advanced formatting (which is good when you have complex proposals), meanwhile the formatting is lost with our archiving tool like markmail (it's plain text). Here is my +1. Thanks, -- Guillaume Delhumeau (guillaume.delhumeau(a)xwiki.com) Research & Development Engineer at XWiki SAS Committer on the XWiki.org project

3 5

[xwiki-devs] [proposal] Process to handle Pull Requests
by Ludovic Dubost 10 May '19

10 May '19

Hi devs, I don't think there is currently a process that is in place to handle pull requests and I have the feeling that the way there are handled today is a bit random. There are usually comments sent out on each pull request but sometimes it seems that some pull requests are going in sleep mode and it's not clear who is in charge. I would like to suggest that a process is put in place where it's clear who is responsible for a pull request and a status is given to the contributors that propose that pull request. Something like: Assigned developer: XXXX Status: New -> new pull request, not yet assigned Assigned -> assigned to a developer, he is in charge of reviewing the pull request and ask for modifications or accept it. The developer can auto assign it to himself. If nobody does, we need to decide how they will be taken into account. ModificationsRequired -> for now rejected with comments. Contributor needs to apply comments and then change back to Assigned for further evaluation VoteRequired -> there are no more comments, but a vote is required as the changes to XWiki core are important WaitingFinalAuthorization -> optional step for complex patches where a additional authorization would be required (need to define who would be the persons that give the authorization) WaitingApplication -> there are no more comments and no changes or vote required. The pull request can be applied and is waiting for a developer to apply it Abandoned -> contributors is abandoning the pull request (cannot do the changes, no more time, etc..) Rejected -> pull request is rejected (quality not enough, etc..) Applied -> pull request is applied What do you think ? Ludovic -- Ludovic Dubost Founder and CEO Blog: http://blog.ludovic.org/ XWiki: http://www.xwiki.com Skype: ldubost GTalk: ldubost

7 12

[xwiki-devs] [contrib] Page Preview Popover extension request
by Stéphane Laurière 19 Feb '19

19 Feb '19

Hi all, I would like to contribute an extension that will display page preview popovers when hovering wiki links, similarly to what MediaWiki offers: https://www.mediawiki.org/wiki/Page_Previews https://blog.wikimedia.org/2018/05/09/page-previews-documentation/ Its name could be 'application-page-preview-popover' - what do you think? As discussed with Caty yesterday, the extension will use the Bootstrap popovers. Should you have any need or suggestion, please let me know. If the name is ok, can I ask you for the creation of a repository and JIRA project? Stéphane -- Stéphane Laurière XWiki www.xwiki.com @slauriere

6 10

[xwiki-devs] [VOTE] Drop support for IE11
by Ecaterina Moraru (Valica) 24 Jan '19

24 Jan '19

IE is the last browser we are supporting at a particular version, see http://dev.xwiki.org/xwiki/bin/view/Community/BrowserSupportStrategy . For all the other browsers, we are supporting just the latest release. We are supporting the latest version of Microsoft Edge since XWiki 9.9, starting from 24/Oct/17. Microsoft Edge replaced Internet Explorer as the default web browser since the release of Windows 10 (launched Jul 2015). Internet Explorer 11 is maintained on Windows 10 for compatibility purposes, but is deprecated in favor of Edge and is no longer actively developed. We discussed dropping IE11 in the past Oct 6, 2017 (see https://xwiki.markmail.org/thread/qsrkka6zfrmyxv3d) but the vote failed. Some updated statistics: -------------------------------------------------- XWiki.org's 1 year traffic (between Sep 30, 2016 - Sep 30, 2017) compared to Oct 1, 2017 - May 31, 2018: - Internet Explorer dropped from (5.84%) to (4.65%) -- from this percentage, IE 11 usage has increased from (88.43%) to (88.76%) - Edge increased from (1.56%) to (1.84%). In terms of Worldwide market share for Desktops, for the past 12 months (according to http://gs.statcounter.com/browser-market-share/desktop/ ), we have Sep, 2016 - Sep, 2017 compared to May 2017 - May 2018: - Internet Explorer dropped from 8.21% to 6.97% - Edge dropped from 4.3% to 4.15% In May 2018, Global Market Share Worldwide for All Platforms contains: - Chrome 58.09% - Safari 13.78% - UC Browser 8.16% - Firefox 5.27% - Opera 3.68% - Internet Explorer combined is at 3.08% (with IE 11 having 2.65%. IE11 in August 2017 had 3.04%) - Edge combined having 1.91% (Edge combined in August 2017 had 1.8%) - etc. -------------------------------------------------- In terms of statistics, IE11 is more used than Microsoft Edge. The only argument of dropping IE11 is that it's a browser that is deprecated for about 3 years, and it's limiting the usage of newer specifications (like CSS Flex, Grid, etc.) I'm curios about how this period of supporting both IE11 and Edge went, and if you have other examples of things you wanted to use and are not supported by IE11. Thanks, Caty

3 4

[xwiki-devs] [Proposal] Get rid of sonar.xwiki.org?
by Vincent Massol 07 Jan '19

07 Jan '19

Hi devs, Context: * I lost the config on ci.xwiki.org to push maven results to sonar.xwiki.org. I can find them back for sure but will take me a bit of time. * We don’t use sonar.xwiki.org that much * We now have https://sonarcloud.io/organizations/xwiki/projects to which we can push results from our builds * https://sonarcloud.io/organizations/xwiki/projects is always up to date in term of sonar version and we don’t need to maintain it So my proposal is to decommission sonar.xwiki.org and to configure https://sonarcloud.io/organizations/xwiki/projects the way we want (I already did that a bit but we can tune it more) and modify our maven build to push to it. WDYT? Thanks -Vincent

6 9

[xwiki-devs] [Discussion] Coding practice for location of internal package name
by Vincent Massol 31 Dec '18

31 Dec '18

Hi devs, On Matrix/IRC, I’ve posted the following: " * Guillaume Delhumeau: BTW your naming is strange for the internal package * for ex: package org.xwiki.notifications.preferences.internal.email; * normally we put internal just after the main package part * ie. * org.xwiki.notifications.internal.* * and org.xwiki.notifications.* for public classes * see http://dev.xwiki.org/xwiki/bin/view/Community/CodeStyle/JavaCodeStyle/#HPac… * General rule is org.xwiki.(module name).internal. * I see thomas has done the same “error" for org.xwiki.job.handler.internal and org.xwiki.job.handler.internal.question . So maybe there's something to discuss/change * I guess we have a mix of both now so we should discuss it and adjust our rules if need be “ So I think we don’t have all the same rules/understanding of the definition at http://dev.xwiki.org/xwiki/bin/view/Community/CodeStyle/JavaCodeStyle/#HPac… I’d like to discuss with you to see what you prefer and adjust our rules so that it matches what we do in practice. Any take in this? Thanks -Vincent

7 8

[xwiki-devs] [Brainstorming] Finish refactoring for xwiki configuration directories
by Vincent Massol 25 Dec '18

25 Dec '18

Hi devs, Context ====== Clemens fixed a bug at https://jira.xwiki.org/browse/XWIKI-15163 and while doing so inroduced a new system property to override the location of the xwiki.properties file. I commented at https://jira.xwiki.org/browse/XWIKI-15163?focusedCommentId=98075&page=com.a… Even though it’s not necessary to introduce a new system property just for the need of this test (it’s easy to refactor the code to not need this IMO), it raises the question of what we want to do to make the configuration simpler in XWiki (simpler config, simpler upgrades, etc). History ====== We discussed this a few times in the past: * March 2010: http://markmail.org/message/6cvm5hocvtbqtgp6 * June 2012: https://markmail.org/message/3aq2bjrb6a2ip2ri Note that the June 2012 proposal was agreed. Globally this is what we implemented since the June 2012 proposal: * XCOMMONS-187: The Permanent Data directory resolver should support System Property "xwiki.data.dir". More specifically the code is here: https://github.com/xwiki/xwiki-commons/blob/55569d3466dc0ea36f6964474973f7a… * XWIKI-13867: Search xwiki.cfg in /etc/xwiki/ first. Code is here: https://github.com/xwiki/xwiki-platform/blob/973d4e9c6ad02dbb31d94fe96df9c1… * XWIKI-13868: Search xwiki.properties in /etc/xwiki/ first. Code is here: https://github.com/xwiki/xwiki-platform/blob/93f02215783ac0f4030fe3062cac4d… I’d like to note that I don’t remember discussions/proposals for XWIKI-13867/XWIKI-13868 and I commented on http://jira.xwiki.org/browse/XWIKI-13867 and didn’t get any response from my various comments. Current behavior ============= So right now the behavior is the following on config files (I’m excluding the recent change of Clements, see below in actions): * If xwiki.cfg exists under the. "java:comp/env” JNDI key, then it’s used * If not found, then search for it in /etc/xwiki/xwiki.cfg * If not found, then default to WEB-INF/xwiki.cfg * If /etc/xwiki.xwiki.properties exist then it’s used * if not found, search in WEB-INF/xwiki.properties (as a ServletContext resource) * If not found, then default to an empty configuration Discussion/Proposal ================ * I think we should ask Clemens to rollback the introduction of the xwiki.properties.default.dir system property and to just make the test work without introducing any system property. I can help Clemens do that. * I propose that instead we continue implementing the June 2012 proposal defined at https://markmail.org/message/3aq2bjrb6a2ip2ri and introduce the xwiki.config.dir system property. * Right now I don’t like the solution introduced by /etc/xwiki/* because they don’t allow supporting several instances of XWiki on the same machine. However, the introduction of xwiki.config.dir system property would fix it. * We could also introduce the user home dir location as a location where xwiki config files would be looked for. * We also need to generalize the config files to hibernate.cfg.xml and clustering config files (jgroup files), and logback. See https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Backup#HConfi… WDYT? Thanks -Vincent PS: Sorry for the long mail, I had to do a lot of archeology to research this… Took me a while ;)

2 2

[xwiki-devs] [Need proposal] How to show "conflicting" macro parameters
by Thomas Mortagne 16 Nov '18

16 Nov '18

Hi xwikiers, In the contact of bringing new Page concept (OK 7.4 is starting to get old) to the API and macros too we decided (1) to introduce a "page" shortcut property (even if we keep the reference/type for other types). While it's nicer for wiki syntax, one issue is that on WYSIWYG macros UI side, which display all properties, it means ending up with conflicting parameters that needs to be displayed as such. I don't really have much clue on how best to display this so I'm searching for ideas :) Then I will add in the macro descriptor what's required for whatever UI we want to build (group and sub groups of properties, etc.). 1: http://design.xwiki.org/xwiki/bin/view/Proposal/DeprecatingSpaceAndSpaceRef… Thanks, -- Thomas Mortagnes

4 37

[xwiki-devs] [STAMP/Test] Metrics we need to improve + strategy
by Vincent Massol 17 Oct '18

17 Oct '18

Hi devs (and anyone else interested to improve the tests of XWiki), History ====== It all started when I analyzed our global TPC and found that it was going down globally even though we have the fail-build-on-jacoco-threshold strategy. I sent several email threads: - Loss of TPC: http://markmail.org/message/hqumkdiz7jm76ya6 - TPC evolution: http://markmail.org/message/up2gc2zzbbe4uqgn - Improve our TPC strategy: http://markmail.org/message/grphwta63pp5p4l7 Note: As a consequence of this last thread, I implemented a Jenkins Pipeline to send us a mail when the global TPC of an XWiki module goes down so that we fix it ASAP. This is still a development in progress. A first version is done and running at https://ci.xwiki.org/view/Tools/job/Clover/ but I need to debug it and fix it (it’s not working ATM). As a result of the global TPC going down/stagnating, I have proposed to have 10.7 focused on Tests + BFD. - Initially I proposed to focus on increasing the global TPC by looking at the reports from 1) above (http://markmail.org/message/qjemnip7hjva2rjd) See the last report at https://up1.xwikisas.com/#mJ0loeB6nBrAgYeKA7MGGw (we need to fix the red parts). - Then with the STAMP mid-term review, a bigger urgency surfaced and I asked if we could instead focus on fixing tests as reported by Descartes to increase both coverage and mutation score (ie test quality), since those are 2 metrics/KPIs measured by STAMP and since XWiki participates to STAMP we need to work on them and increase them substantially. See http://markmail.org/message/ejmdkf3hx7drkj52 The results of XWiki 10.7 has been quite poor on test improvements (more focus on BFD than tests, lots of devs on holidays, etc). This forces us to have a different strategy. Full Strategy proposal ================= 1) As many XWiki SAS devs as possible (and anyone else from the community who’s interested ofc! :)) should spend 1 day per week working on improving STAMP metrics * Currently the agreement is that Thomas and myself will do this for the foreseeable future till we get some good-enough metric progress * Some other devs from XWiki SAS will help out for XWiki 10.8 only FTM (Marius, Adel if he can, Simon in the future). The idea is to see where that could get us by using substantial manpower. 2) All committers: More generally the global TPC failure is also already active and dev need to modify modules that see their global TPC go down. 3) All committers: Of course, the jacoco strategy is also active at each module level. STAMP tools ========== There are 4 tools developed by STAMP: * Descartes: Improves quality of tests by increasing their mutation scores. See http://markmail.org/message/bonb5f7f37omnnog and also https://massol.myxwiki.org/xwiki/bin/view/Blog/MutationTestingDescartes * DSpot: Automatically generate new tests, based on existing tests. See https://massol.myxwiki.org/xwiki/bin/view/Blog/TestGenerationDspot * CAMP: Takes a Dockerfile and generates mutations of it, then deploys and execute tests on the software to see if the mutation works or not. Note this is currently not fitting the need of XWiki and thus I’ve been developing another tool as an experiment (which may go back in CAMP one day), based on TestContainers, see https://massol.myxwiki.org/xwiki/bin/view/Blog/EnvironmentTestingExperiment… * EvoCrash: Takes a stack trace from production logs and generates a test that, when executed, reproduces the crash. See https://markmail.org/message/v74g3tsmflquqwra. See also https://github.com/SERG-Delft/EvoCrash Since XWiki is part of the STAMP research project, we need to use those 4 tools to increase the KPIs associated with the tools. See below. Objectives/KPIs/Metrics for STAMP =========================== The STAMP project has defined 9 KPIs that all partners (and thus XWiki) need to work on: 1) K01: Increase test coverage * Global increase by reducing by 40% the non-covered code. For XWiki since we’re at about 70%, this means reaching about 80% before the end of STAMP (ie. before end of 2019) * Increase the coverage contributions of each tool developed by STAMP. Strategy: * Primary goal: ** Increase coverage by executing Descartes and improving our tests. This is http://markmail.org/message/ejmdkf3hx7drkj52 ** Don’t do anything with DSpot. I’ll do that part. Note that the goal is to write a Jenkins pipeline to automatically execute DSpot from time to time and commit the generated tests in a separate test source and have our build execute both src/test/java and this new test source. ** Don’t do anything with TestContainers FTM since I need to finish a first working version. I may need help in the future to implement docker images for more configurations (on Oracle, in a cluster, with LibreOffice, with an external SOLR server, etc). ** For EvoCrash: We’ll count contributions of EvoCrash to coverage in K08. * Secondary goal: ** Increase our global TPC as mentioned above by fixing the modules in red. 2) K02: Reduce flaky tests. * Objective: reduce the number of flaky tests by 20% Strategy: * Record flaky tests in jira * Fix the max number of them 3) K03: Better test quality * Objective: increase mutation score by 20% Strategy: * Same strategy as K01. 4) K04: More configuration-related paths tested * Objective: increase the code coverage of configuration-related paths in our code by 20% (e.g. DB schema creation, cluster)related code, SOLR-related code, LibreOffice-related code, etc). Strategy: * Leave it to FTM. The idea is to measure Clover TPC with the base configuration, then execute all other configurations (with TestContainers) and regenerate the Clover report to see how much the TPC has increased. 5) K05: Reduce system-specific bugs * Objective: 30% improvement Strategy: * Run TestContainers, execute existing tests and find new bugs related to configurations. Record them 6) K06: More configurations/Faster tests * Objective: increase the number of automatically tested configurations by 50% Strategy: * Increase the # of configurations we test with TestContainers. I’ll do that part initially. * Reduce time it takes to deploy the software under a given configuration vs time it used to take when done manually before STAMP. I’ll do this one. I’ve already worked on it in the past year with the dockerization of XWiki. 7) K07: Pending, nothing to do FTM 8) K08: More crash replicating test cases * Objective: increase the number of crash replicating test cases by at least 70% Strategy: * For all issues that are still open and that have stack traces and for all issues closed but without tests, run EvoCrash on them to try to generate a test. * Record and count the number of successful EvoCrash-generated test cases. * Derive a regression test (which can be very different from the negative of the test generated by evocrash!). * Measure the new coverage increase * Note that I haven’t experimented much with this yet myself. 9) K09: Pending, nothing to do FTM. Conclusion ========= Right now, I need your help for the following KPIs: K01, K02, K03, K08. Since there’s a lot to understand in this email, I’m open to: * Organizing a meeting on youtube live to discuss all this * Answering any questions on this thread ofc * Also feel free to ask on IRC/Matrix. Here’s an extract from STAMP which has more details about the KPIs/metrics: https://up1.xwikisas.com/#QJyxqspKXSzuWNOHUuAaEA Thanks -Vincent

4 15

[xwiki-devs] State of Velocity 2.0 upgrade
by Thomas Mortagne 15 Oct '18

15 Oct '18

Hi devs, Some of you may know that I started working on upgrading our scripting engine to Velocity 2.0. The corresponding Jira issue is https://jira.xwiki.org/browse/XCOMMONS-1296. Those of you who want to take a look at breakages with standard Velocity 2.0 upgrade (if you directly use Velocity API) can take a look at http://velocity.apache.org/engine/2.0/upgrading.html. The following details are about XWiki Velocity "engine": = The code You can see the current state of the upgrade on the following branches: * https://github.com/xwiki/xwiki-commons/tree/feature-velocity2 * https://github.com/xwiki/xwiki-rendering/tree/feature-velocity2 * https://github.com/xwiki/xwiki-platform/tree/feature-velocity2 = Bad news == Definitive (probably) breakages * it was not easy in Velocity 1.7 but it's now impossible in Velocity 2.0 to use the hack we currently use to make macro "return" something. I mitigated this a bit by working on a #setVariable directive (the name of the helper we currently have in macros.vm) but it's not yet working in all situations and any code that was not going trough #setVariable will be broken anyway * the hypen ( - ) cannot be used in variable names anymore Those changes make impossible to upgrade to Velocity 2.0 in 10.x branch IMO, to big to be done in the middle of a cycle. == Temporary (hopefully) breakages * impossible to have $ or # at the end of a " based string literal (we have that a lot in various regex for example). This looks like an unplanned regression (since it's not documented) so I'm waiting for some kind reaction in the issues I reported (https://issues.apache.org/jira/browse/VELOCITY-897 and https://issues.apache.org/jira/browse/VELOCITY-896) == Not too impacting (hopefully) breakages The Velocity API changed quite a lot, fortunately not in classes we expose publicly (basically all we expose is VelocityContext). Also reminder: directly manipulating even XWiki Velocity API is considered legacy things (better use templates or wiki content and ScriptContext). = Good news == Macros handling The way to manage namespaces and macros has been completely changed (it does mean that lots of API has been broken in the process but we don't expose this API) and the good news is the new design will allow us to (finally) do proper caching of velocity scripts. I also removed a few hacks we had to do to not end up with endless memory leaks and milti-thread issues caused by the old namespace handling system. == Less stuff on our side I was able to move the following to legacy: * an equivalent of our chainable uberspector system has been implemented in Velocity standard * a DeprecatedCheckUberspector has been implemented in Velocity standard == Retro compatibility I was able to add * XWiki will keep supporting $velocityCount and $velocityHasNext (with a warning). It imply using XWikiVelocityContext instead of VelocityContext if you create it directly (which is a very bad idea) but you don't need to worry about that if you get your context from VelocityManager * a new #setVariable directive help supporting a few use case of macro "return" but unfortunately not all yet (Velocity AST is not super obvious to navigate...) == Performances Other than the caching on our side I already mentioned (huge boost planned when implemented) there is some memory and performance improvements claims in Velocity 2.0 release note (but I did not had time to do any testing myself yet). = TODO a) waiting for answer about the regressions I reported b) try find a way to make #setVariable directive works in a macro which is itself called in another macro (but it may not be possible) c) send a mail to discuss when to merge the Velocity 2 branch when at least a) is resolved Thanks, -- Thomas Mortagne

1 1