The code macro syntax highlighting is slow and scales badly when several pages are rendered in parallel

Pages with many code macros are slow to render, and they get much slower when many of them are rendered at the same time (for example while crawlers visit a wiki): each code macro is highlighted on the server by Pygments running in Jython, and the highlighting has costs that don't depend on the code being highlighted and that limit how well it runs on several threads.

Causes

• PygmentsParser passes the Python script that drives Pygments as a string to the script engine for each highlight, so Jython parses and compiles it again every time and loads a new Java class for each highlight.
• Pygments lexers keep their token rules in Python lists shared by all threads, and Jython synchronizes each step of the iteration over a list. The lexer iterates over these rules at each position of the code, so the threads highlighting code at the same time block each other.
• The text of each token is parsed with the plain text parser, which allocates a 16 KB read buffer for each call. This one is fixed in the plain text parser, see the linked XRENDERING issue.

Measurements

Highlighting a 25-line Java snippet on an 8-core laptop, with prototypes of the changes:

• Currently: 240 highlights per second on 1 thread, 630 per second on 8 threads.
• Compiling the script once: 450 highlights per second on 1 thread, 960 per second on 8 threads.
• Also avoiding the shared lists and the plain text parser buffer: 470 highlights per second on 1 thread, 1570 per second on 8 threads.