This issue has been created
 
 
XWiki Platform / cid:jira-generated-image-avatar-b6335996-b0e4-4e85-9b38-5152d7dcf958 XWIKI-24846 Open

Make the code macro's syntax highlighter pluggable and use a client-side one by default

 
View issue   ·   Add comment
 

Issue created

 
cid:jira-generated-image-avatar-99419cc3-2462-419b-a4a3-d133ddcef5bb Charpentier Lucas created this issue on 09/Sep/26 13:51
 
Summary: Make the code macro's syntax highlighter pluggable and use a client-side one by default
Issue Type: cid:jira-generated-image-avatar-b6335996-b0e4-4e85-9b38-5152d7dcf958 New Feature
Affects Versions: 18.7.0
Assignee: Unassigned
Components: Rendering - Code Macro
Created: 09/Sep/26 13:51
Labels: usability
Priority: cid:jira-generated-image-static-major-006efc00-e12b-427b-baf0-34f5add9cfb0 Major
Reporter: Charpentier Lucas
Description:

Context

This implements the proposal accepted on
Change the code macro implementation
(+1 from mleduc, tmortagne, mflorea and Wardenburg), whose conclusion was:

1) We agreed to move to a javascript-based syntax highlighting in XS
2) We agreed to keep the code macro and introduce an Admin UI to configure which implementation
to use and by default to use the javascript-based one. And once done and working well, to move the
pygments-based one to contrib.

The reasons recorded there:

  • Performance: the current highlighter is written in Python and executed through Jython. It is very
    slow on first use, while Jython loads, and still slow afterwards, especially on large content.
  • Missing features: line numbering, a button to copy the content (XWIKI-21364), and others, which
    are far easier to provide client side.
  • The historical reason for keeping the highlighting server side – the PDF exporter not supporting
    JavaScript – no longer applies, since the current PDF exporter does support it.

A client-side implementation already exists in contrib and works:
Prism Macro, along with
org.xwiki.contrib.prismjs:prismjs-code which substitutes it for the code macro
(https://github.com/xwiki-contrib/prismjs/tree/master/prismjs-code). What is left is doing it
properly in XWiki Standard.

Current state in the code

Most of the pluggability is already there:

  • CodeMacro resolves a HighlightParser component by language hint, and otherwise falls back
    to componentManager.getInstance(HighlightParser.class, "default") – the fallback hint is
    hardcoded.
  • PygmentsParser is declared as {{@Component(roles = {HighlightParser.class}

    )}} with no
    @Named, so it is that default hint.

  • HighlightParser#highlight(String syntaxId, Reader source) receives only the language and the
    content: not the macro parameters, and not whether the call is inline.
  • xwiki-platform-rendering-macro-code declares jython-slim and pygments as direct
    dependencies, so the generic macro and the Pygments implementation ship in the same module.
  • The source parameter needs no work: the generic part of the macro already resolves it and hands
    the resulting content to the HighlightParser.

Proposal

  1. Replace the hardcoded "default" fallback hint by a configured one, and add the Admin UI to
    choose among the HighlightParser implementations present in the instance. Per-language
    HighlightParser components keep taking precedence, as today.
  2. Extract the Pygments implementation into its own module, under the hint pygments, so that
    xwiki-platform-rendering-macro-code stops depending on Jython and Pygments.
  3. Add a client-side HighlightParser implementation to XS, based on Prism.js, and make it the
    configured default.
  4. Once that is stable, move the Pygments module to contrib.

Also to be decided while doing 1: whether HighlightParser#highlight needs the macro parameters
and an inline flag, which a client-side implementation may need and the current signature cannot
carry.

Backward compatibility

The risk identified on the thread is language coverage, not architecture:

  • Pygments supports far more languages than the JavaScript frameworks. The two that matter to XWiki
    are Velocity – supported by Pygments only because the project asked for it – and Groovy. Prism.js
    supports both, which is a concrete argument for Prism over highlight.js, which does not.
  • Language identifiers do not map one to one. Pygments treats bash, sh, ksh, zsh and
    shell as synonyms; Prism does not, and zsh would need mapping onto shell. The agreed
    approach is to handle such cases individually inside the Prism implementation rather than build a
    generic mapping layer.

Existing content using the code macro must keep being highlighted after the switch.

Consequences for other issues

  • XWIKI-21364 (copy button) and XWIKI-22832 (line numbers) become much cheaper client side.
  • XWIKI-8456 (cache the highlighting result for performance) largely dissolves.
  • XWIKI-17107 (upgrade Pygments) and XWIKI-18369 (Pygments lexer patch) become contrib concerns once
    the Pygments implementation moves out.