Question: Do we need the captcha module to function without the DB as well? This would make it more difficult to create the configuration UI (in Administration) and would also mean that some captchas implementations might not respect this constraint (i.e. choosing to implement their UI in a wiki page instead). IMO, it's not really worth the trouble and we should proceed to extending the current captcha module into 2 parts: -api and -ui (the current module would be converted from "jar" to "pom" and the current code will be deprecated and moved to the new -api module).

Thanks, Eduard On Tue, May 22, 2018 at 8:35 PM, Eduard Moraru <enygma2002(a)gmail.com> wrote: > Hi, devs, > > XWiki's current CAPTCHA module [1] is very old and outdated for a while > now and this is not news for anyone. > > I see 2 major problems: > 1) The obvious one is that we just need a technologically better CAPTCHA > implementation that the current JCaptcha-based one and JCaptcha is > discontinued. > 2) The current architecture is outdated as well (i.e. wrapped around > Struts actions and initializing the VelocityContext with a custom binding) > and hard to use (i.e. you have to write your CAPTCHA UI every time you use > it, there is no rendering helper). > > For 1), as both determined a few years ago [2] and also observed from > practice, the industry standard is now Google's reCaptcha service (either > its v2 or invisible version, or both), so we definitely need an > implementation that makes it easy to use in XWiki [3]. > > For 2), we need: > * to allow CAPTCHA implementations to provide their own rendering > ** Can be done using a template .vm file (rendered with the > TemplateManager), a wiki page or directly from Java code. > ** The implementation-specific parameters could be passed to > control/customize a particular rendering. > ** The syntax ID may not be needed, since the only relevant output would > be html. > * to move to a ScriptService-based system and to leave the resource > (image/audio) accessing concern to the individual implementation (that may > choose to continue with Struts actions, or may choose to use something > lighter like a REST resource or even or even something more inventive, like > temporary attachments on some fixed page). > * an administration UI that configures the default CAPTCHA service and its > configuration > * to allow callers to use a different CAPTCHA than the default configured > one, as long as it is available (i.e. installed) > > Examples: > > = Display > $services.captcha.display() -- default implementation, current > configuration > $services.captcha.recaptcha.display() -- explicitly requested > implementation, current/default configuration > $services.captcha.recaptcha.display({'theme' : 'dark', 'size' : > 'compact', 'sitekey' : 'xyz'}) -- explicitly requested implementation and > custom configuration overwrites > $services.captcha.jcaptcha.display() -- other implementation, > current/default configuration > > = Verification > $services.captcha.isValid() -- default implementation, current > configuration > $services.captcha.recaptcha.isValid() -- explicitly requested > implementation, current/default configuration > $services.captcha.recaptcha.isValid({'secret' : 'xyz'}) -- explicitly > requested implementation and custom configuration overwrites > $services.captcha.jcaptcha.isValid() -- explicitly requested > implementation and custom configuration overwrites > > Basically, the ScriptService API would be highly simplified to just > displaying and validating, while the components would consist of 2 parts: > * a generic Manager for the various implementations (listing, > getting/setting the default) > * various implementations, each responsible with rendering a form-friendly > CAPTCHA input and using the current request for extracting the information > they needs to read the user's answer and validate it. > > WDYT?

> Thanks, > Eduard > > ---------- > [1] http://extensions.xwiki.org/xwiki/bin/view/Extension/Captcha%20Module > [2] http://design.xwiki.org/xwiki/bin/view/Proposal/CAPTCHAinvestigation70 > [3] https://jira.xwiki.org/browse/XWIKI-12964 >

On 25 May 2018, at 08:51, Thomas Mortagne <thomas.mortagne(a)xwiki.com> wrote: I agree that the default captcha we provide should not depend on an external service but it should just be a default. IMO the most important is to make sure it's easy to choose a different captcha implementation and we should most probably provide the Google one as an option (through a contrib extension to better follow Google captcha API evolution). On Fri, May 25, 2018 at 12:37 AM, Clément Aubin <aubincleme(a)gmail.com> wrote: -- Thomas Mortagne

On 25 May 2018, at 14:02, Eduard Moraru <enygma2002(a)gmail.com> wrote: Hi, On Fri, May 25, 2018 at 10:32 AM, Vincent Massol <vincent(a)massol.net> wrote: I wouldn't directly say that I disagree with this statement, however I am going to reiterate the conclusions we (at least I) had back when I did the initial XWiki 7.0 CAPTCHA investigation regarding self hosted vs service solution: The conclusion was that the need for a CAPTCHA system arises in a public environment/website. There is no plausible scenario where a company/organization would want to setup a CAPTCHA on a private intranet or internal system because it would be poitless to try to protect your system from, e.g., bots registering fake accounts from inside your company's VPN (or something like that). So, if the usecase is public endpoints/systems that need to be protected by CAPTCHAs, then using a public service (such as Google's reCAPTCHA) is perfectly plausible and not creating any technical problems, since the client has access to download the necessary javascript and the server (most likely) has external access to validate the response.

The above conclusion is also most likely the reason why self-hosted CAPTCHA solutions have pretty much died out along the years, mostly due to Google's reCAPTCHA that is enjoying a huge success, both in adoption and in efficiency.

I even found an asnwer from actual self-hosted CAPTCHA authors recommending to use Google's service solution instead, even mentioning the fact that it was helping (at some point) improve OCR on digializing text books (now Google seems to have moved more to image recognition instead of OCR, but still...). https://stackoverflow.com/questions/810493/recommendations-for-java-captcha… (check https://stackoverflow.com/a/4528230/5198689 for a reply from SimpleCaptcha's author) SimpleCaptcha does seem to pop up in many discussions as the best Java alternative so far. We might want to try and offer a contrib extension with it as well. Clement's suggestion of custom CAPTCHA solutions is not bad either. They should be much easier to implement and switch to, with the new system. Maybe it would be more appropriate at a per-depoyment level, instead of at a product level, though. If you ask me, it's Google's reCAPTCHA that we should use as a default (since it's now the industry standard), while still bundling (at least for a while) the old JCaptcha implementation (for backwards compatibility, until users switch to the new system) and the new JCaptcha implementation (with the new system). IMO, it would be nicer to offer the best solution as a default and have the few cases that need the self-hosted solution switch to JCaptcha or install some contrib extension based on SimpleCaptcha, or whatever else is available. Now, as Thomas mentioned, it would make more sense to implement it as a Contrib extension (so that, in the near future, changes to reCAPTCHA could be still used by older vesions; older as in stil using the new system, i.e. 10.5+, but not the latest XWiki version), so the only solution to have it as default as well would mean for the Standard Flavor to bundle the reCAPTCHA implementaion, as a Contrib extension. (Vincent, insert "-1" here :) )

--- Side Note: Taking an approach, in terms of Admin UI, that is similar to what we did for Search (transition and choice from Solr to Lucene), with configuration options specific to each implementation.

--- Now, CAPTCHA is not enabled by default in XWiki, so admins still need to do this. Once they enable it, if reCAPTCHA is the default, it won't work until it's configured (with a Google API key, even for localhost). An interesting thing we could do to avoid this problem and have the CAPTCHA not fail completely, is to use Google's test keys (originally intended to not break automated testing) by default which will make the CAPTCHA always validate, while still being explicit that the configuration needs to be done. This would make sure there is no friction in the process and we'll probably do this by default for this implementation, regardless if we choose to use it as XWiki's default or if it needs to be manually installed (from contrib). https://developers.google.com/recaptcha/docs/faq#id-like-to-run-automated-t… Thanks, Eduard > > Thanks > -Vincent > >> On 25 May 2018, at 08:51, Thomas Mortagne <thomas.mortagne(a)xwiki.com> > wrote: >> >> I agree that the default captcha we provide should not depend on an >> external service but it should just be a default. IMO the most >> important is to make sure it's easy to choose a different captcha >> implementation and we should most probably provide the Google one as >> an option (through a contrib extension to better follow Google captcha >> API evolution). >> >> On Fri, May 25, 2018 at 12:37 AM, Clément Aubin <aubincleme(a)gmail.com> > wrote: >>> Hi, >>> >>> On 05/24/2018 05:15 PM, Eduard Moraru wrote: >>>> Question: >>>> e >>>> Do we need the captcha module to function without the DB as well? >>>> >>>> This would make it more difficult to create the configuration UI (in >>>> Administration) and would also mean that some captchas implementations >>>> might not respect this constraint (i.e. choosing to implement their UI > in a >>>> wiki page instead). >>>> >>>> IMO, it's not really worth the trouble and we should proceed to > extending >>>> the current captcha module into 2 parts: -api and -ui (the current > module >>>> would be converted from "jar" to "pom" and the current code will be >>>> deprecated and moved to the new -api module). >>> >>> The only reason we could not to depend on the DB would be to transfer >>> parts of the captcha module to xwiki-commons IMO. For now, I don't see >>> any particular use case that could support this migration. >>> >>>> Thanks, >>>> Eduard >>>> >>>> On Tue, May 22, 2018 at 8:35 PM, Eduard Moraru <enygma2002(a)gmail.com> > wrote: >>>> >>>>> Hi, devs, >>>>> >>>>> XWiki's current CAPTCHA module [1] is very old and outdated for a > while >>>>> now and this is not news for anyone. >>>>> >>>>> I see 2 major problems: >>>>> 1) The obvious one is that we just need a technologically better > CAPTCHA >>>>> implementation that the current JCaptcha-based one and JCaptcha is >>>>> discontinued. >>>>> 2) The current architecture is outdated as well (i.e. wrapped around >>>>> Struts actions and initializing the VelocityContext with a custom > binding) >>>>> and hard to use (i.e. you have to write your CAPTCHA UI every time > you use >>>>> it, there is no rendering helper). >>>>> >>>>> For 1), as both determined a few years ago [2] and also observed from >>>>> practice, the industry standard is now Google's reCaptcha service > (either >>>>> its v2 or invisible version, or both), so we definitely need an >>>>> implementation that makes it easy to use in XWiki [3]. >>>>> >>>>> For 2), we need: >>>>> * to allow CAPTCHA implementations to provide their own rendering >>>>> ** Can be done using a template .vm file (rendered with the >>>>> TemplateManager), a wiki page or directly from Java code. >>>>> ** The implementation-specific parameters could be passed to >>>>> control/customize a particular rendering. >>>>> ** The syntax ID may not be needed, since the only relevant output > would >>>>> be html. >>>>> * to move to a ScriptService-based system and to leave the resource >>>>> (image/audio) accessing concern to the individual implementation > (that may >>>>> choose to continue with Struts actions, or may choose to use something >>>>> lighter like a REST resource or even or even something more > inventive, like >>>>> temporary attachments on some fixed page). >>>>> * an administration UI that configures the default CAPTCHA service > and its >>>>> configuration >>>>> * to allow callers to use a different CAPTCHA than the default > configured >>>>> one, as long as it is available (i.e. installed) >>>>> >>>>> Examples: >>>>> >>>>> = Display >>>>> $services.captcha.display() -- default implementation, current >>>>> configuration >>>>> $services.captcha.recaptcha.display() -- explicitly requested >>>>> implementation, current/default configuration >>>>> $services.captcha.recaptcha.display({'theme' : 'dark', 'size' : >>>>> 'compact', 'sitekey' : 'xyz'}) -- explicitly requested implementation > and >>>>> custom configuration overwrites >>>>> $services.captcha.jcaptcha.display() -- other implementation, >>>>> current/default configuration >>>>> >>>>> = Verification >>>>> $services.captcha.isValid() -- default implementation, current >>>>> configuration >>>>> $services.captcha.recaptcha.isValid() -- explicitly requested >>>>> implementation, current/default configuration >>>>> $services.captcha.recaptcha.isValid({'secret' : 'xyz'}) -- explicitly >>>>> requested implementation and custom configuration overwrites >>>>> $services.captcha.jcaptcha.isValid() -- explicitly requested >>>>> implementation and custom configuration overwrites >>>>> >>>>> Basically, the ScriptService API would be highly simplified to just >>>>> displaying and validating, while the components would consist of 2 > parts: >>>>> * a generic Manager for the various implementations (listing, >>>>> getting/setting the default) >>>>> * various implementations, each responsible with rendering a > form-friendly >>>>> CAPTCHA input and using the current request for extracting the > information >>>>> they needs to read the user's answer and validate it. >>>>> >>>>> WDYT? >>> >>> I do agree that JCaptcha is getting quite old now, however, we might >>> have a hard time finding new FOSS and self-hosted captcha solutions (at >>> least, there's not a lot of interesting artifacts on maven central [1]). >>> >>> IMO, we shouldn't rely on a captcha using an external verification >>> service (such as Google's captchas) as we might end up blocking users >>> that access their wiki with a filtered / controlled internet access. One >>> option could be to provide a challenge which asks the user to perform a >>> simple math operation, or respond to a control question instead, but I'm >>> not convinced of the efficiency of those solutions against real >>> captchas. As a captcha is usually quite simple to implement (apply >>> operations on a text written in an image in order to make it >>> non-readable by OCR software), we could also support our own version, >>> maybe based on the generation mechanism given by JCaptcha. >>> >>> Thanks, >>> Clément >>> >>> [1] https://search.maven.org/#search%7Cga%7C1%7Ccaptcha >>> >>>>> Thanks, >>>>> Eduard >>>>> >>>>> ---------- >>>>> [1] http://extensions.xwiki.org/xwiki/bin/view/Extension/ > Captcha%20Module >>>>> [2] http://design.xwiki.org/xwiki/bin/view/Proposal/ > CAPTCHAinvestigation70 >>>>> [3] https://jira.xwiki.org/browse/XWIKI-12964 >>>>> >> >> >> >> -- >> Thomas Mortagne > >

On 25 May 2018, at 18:24, Eduard Moraru <enygma2002(a)gmail.com> wrote:

Remember that Registration and Comments both allow enabling Captchas individually, so it's a feature that XS still needs, at least at the API/configuration level.

Sure, however the problem with the current Captcha Module ( http://extensions.xwiki.org/xwiki/bin/view/Extension/Captcha%20Module#HGett… ) is that it is implemented with a Struts Action (bound in struts-config.xml for "/imagecaptcha/“)

that anyone asking for a CAPTCHA should use to generate the CAPTCHA image and use it in an <img src=".../imagecaptcha/"/> tag. That makes it impossible to extract it as an extension.

So we have a backwards compatibility problem in the fact that we need to keep supporting: * Users of the "/imagecaptcha/" action * Users of custom CAPTCHA systems, build on top of the old CaptchaVerifier system ( http://extensions.xwiki.org/xwiki/bin/view/Extension/Captcha%20Module#HHowa… ) ...and give them time to transition. Because of this, I plan to move everything found in the current xwiki-platform-captcha module to xwiki-platform-captcha-jcaptcha-api, together with the integration with the new system, and continue bundling it in XS. This way, both implementations should still work. So we end up with the following layout: xwiki-platform-captcha + xwiki-platform-captcha-api + xwiki-platform-captcha-ui + xwiki-platform-captcha-jcaptcha ++ xwiki-platform-captcha-jcaptcha-api ++ xwiki-platform-captcha-jcaptcha-ui captcha-recaptcha (contrib, depends on xwiki-platform-captcha) + captcha-recaptcha-api + captcha-recaptcha-ui

Thanks, Eduard > > The other option is to introduce UIXP instead and fully move the code to > the extension(s). > > Thanks > -Vincent > >> --- >> >> Now, CAPTCHA is not enabled by default in XWiki, so admins still need to > do >> this. Once they enable it, if reCAPTCHA is the default, it won't work > until >> it's configured (with a Google API key, even for localhost). An > interesting >> thing we could do to avoid this problem and have the CAPTCHA not fail >> completely, is to use Google's test keys (originally intended to not > break >> automated testing) by default which will make the CAPTCHA always > validate, >> while still being explicit that the configuration needs to be done. This >> would make sure there is no friction in the process and we'll probably do >> this by default for this implementation, regardless if we choose to use > it >> as XWiki's default or if it needs to be manually installed (from > contrib). >> >> https://developers.google.com/recaptcha/docs/faq#id-like-to- > run-automated-tests-with-recaptcha-v2-what-should-i-do >> >> Thanks, >> Eduard >> >> >>> >>> Thanks >>> -Vincent >>> >>>> On 25 May 2018, at 08:51, Thomas Mortagne &lt;thomas.mortagne(a)xwiki.com&gt; >>> wrote: >>>> >>>> I agree that the default captcha we provide should not depend on an >>>> external service but it should just be a default. IMO the most >>>> important is to make sure it's easy to choose a different captcha >>>> implementation and we should most probably provide the Google one as >>>> an option (through a contrib extension to better follow Google captcha >>>> API evolution). >>>> >>>> On Fri, May 25, 2018 at 12:37 AM, Clément Aubin &lt;aubincleme(a)gmail.com&gt; >>> wrote: >>>>> Hi, >>>>> >>>>> On 05/24/2018 05:15 PM, Eduard Moraru wrote: >>>>>> Question: >>>>>> e >>>>>> Do we need the captcha module to function without the DB as well? >>>>>> >>>>>> This would make it more difficult to create the configuration UI (in >>>>>> Administration) and would also mean that some captchas > implementations >>>>>> might not respect this constraint (i.e. choosing to implement their > UI >>> in a >>>>>> wiki page instead). >>>>>> >>>>>> IMO, it's not really worth the trouble and we should proceed to >>> extending >>>>>> the current captcha module into 2 parts: -api and -ui (the current >>> module >>>>>> would be converted from "jar" to "pom" and the current code will be >>>>>> deprecated and moved to the new -api module). >>>>> >>>>> The only reason we could not to depend on the DB would be to transfer >>>>> parts of the captcha module to xwiki-commons IMO. For now, I don't see >>>>> any particular use case that could support this migration. >>>>> >>>>>> Thanks, >>>>>> Eduard >>>>>> >>>>>> On Tue, May 22, 2018 at 8:35 PM, Eduard Moraru &lt;enygma2002(a)gmail.com >> >>> wrote: >>>>>> >>>>>>> Hi, devs, >>>>>>> >>>>>>> XWiki's current CAPTCHA module [1] is very old and outdated for a >>> while >>>>>>> now and this is not news for anyone. >>>>>>> >>>>>>> I see 2 major problems: >>>>>>> 1) The obvious one is that we just need a technologically better >>> CAPTCHA >>>>>>> implementation that the current JCaptcha-based one and JCaptcha is >>>>>>> discontinued. >>>>>>> 2) The current architecture is outdated as well (i.e. wrapped around >>>>>>> Struts actions and initializing the VelocityContext with a custom >>> binding) >>>>>>> and hard to use (i.e. you have to write your CAPTCHA UI every time >>> you use >>>>>>> it, there is no rendering helper). >>>>>>> >>>>>>> For 1), as both determined a few years ago [2] and also observed > from >>>>>>> practice, the industry standard is now Google's reCaptcha service >>> (either >>>>>>> its v2 or invisible version, or both), so we definitely need an >>>>>>> implementation that makes it easy to use in XWiki [3]. >>>>>>> >>>>>>> For 2), we need: >>>>>>> * to allow CAPTCHA implementations to provide their own rendering >>>>>>> ** Can be done using a template .vm file (rendered with the >>>>>>> TemplateManager), a wiki page or directly from Java code. >>>>>>> ** The implementation-specific parameters could be passed to >>>>>>> control/customize a particular rendering. >>>>>>> ** The syntax ID may not be needed, since the only relevant output >>> would >>>>>>> be html. >>>>>>> * to move to a ScriptService-based system and to leave the resource >>>>>>> (image/audio) accessing concern to the individual implementation >>> (that may >>>>>>> choose to continue with Struts actions, or may choose to use > something >>>>>>> lighter like a REST resource or even or even something more >>> inventive, like >>>>>>> temporary attachments on some fixed page). >>>>>>> * an administration UI that configures the default CAPTCHA service >>> and its >>>>>>> configuration >>>>>>> * to allow callers to use a different CAPTCHA than the default >>> configured >>>>>>> one, as long as it is available (i.e. installed) >>>>>>> >>>>>>> Examples: >>>>>>> >>>>>>> = Display >>>>>>> $services.captcha.display() -- default implementation, current >>>>>>> configuration >>>>>>> $services.captcha.recaptcha.display() -- explicitly requested >>>>>>> implementation, current/default configuration >>>>>>> $services.captcha.recaptcha.display({'theme' : 'dark', 'size' : >>>>>>> 'compact', 'sitekey' : 'xyz'}) -- explicitly requested > implementation >>> and >>>>>>> custom configuration overwrites >>>>>>> $services.captcha.jcaptcha.display() -- other implementation, >>>>>>> current/default configuration >>>>>>> >>>>>>> = Verification >>>>>>> $services.captcha.isValid() -- default implementation, current >>>>>>> configuration >>>>>>> $services.captcha.recaptcha.isValid() -- explicitly requested >>>>>>> implementation, current/default configuration >>>>>>> $services.captcha.recaptcha.isValid({'secret' : 'xyz'}) -- > explicitly >>>>>>> requested implementation and custom configuration overwrites >>>>>>> $services.captcha.jcaptcha.isValid() -- explicitly requested >>>>>>> implementation and custom configuration overwrites >>>>>>> >>>>>>> Basically, the ScriptService API would be highly simplified to just >>>>>>> displaying and validating, while the components would consist of 2 >>> parts: >>>>>>> * a generic Manager for the various implementations (listing, >>>>>>> getting/setting the default) >>>>>>> * various implementations, each responsible with rendering a >>> form-friendly >>>>>>> CAPTCHA input and using the current request for extracting the >>> information >>>>>>> they needs to read the user's answer and validate it. >>>>>>> >>>>>>> WDYT? >>>>> >>>>> I do agree that JCaptcha is getting quite old now, however, we might >>>>> have a hard time finding new FOSS and self-hosted captcha solutions > (at >>>>> least, there's not a lot of interesting artifacts on maven central > [1]). >>>>> >>>>> IMO, we shouldn't rely on a captcha using an external verification >>>>> service (such as Google's captchas) as we might end up blocking users >>>>> that access their wiki with a filtered / controlled internet access. > One >>>>> option could be to provide a challenge which asks the user to perform > a >>>>> simple math operation, or respond to a control question instead, but > I'm >>>>> not convinced of the efficiency of those solutions against real >>>>> captchas. As a captcha is usually quite simple to implement (apply >>>>> operations on a text written in an image in order to make it >>>>> non-readable by OCR software), we could also support our own version, >>>>> maybe based on the generation mechanism given by JCaptcha. >>>>> >>>>> Thanks, >>>>> Clément >>>>> >>>>> [1] https://search.maven.org/#search%7Cga%7C1%7Ccaptcha >>>>> >>>>>>> Thanks, >>>>>>> Eduard >>>>>>> >>>>>>> ---------- >>>>>>> [1] http://extensions.xwiki.org/xwiki/bin/view/Extension/ >>> Captcha%20Module >>>>>>> [2] http://design.xwiki.org/xwiki/bin/view/Proposal/ >>> CAPTCHAinvestigation70 >>>>>>> [3] https://jira.xwiki.org/browse/XWIKI-12964 >>>>>>> >>>> >>>> >>>> >>>> -- >>>> Thomas Mortagne