[xwiki-users] which HTML parsing libs are already using/shiipped with XWiki ? - xwiki-users@xwiki.org - lists.xwiki.org

List overview All Threads
Download

[xwiki-users] which HTML parsing libs are already using/shiipped with XWiki ?

[xwiki-users] OpenOffice auto-start

[xwiki-users] [Video] XWiki-G3...

Arioch

4 Jul 2012 4 Jul '12

12:41 p.m.

which HTML parsers do u have onboard ? Jericho, JTidy, whatever ? -- View this message in context: http://xwiki.475771.n2.nabble.com/which-HTML-parsing-libs-are-already-using… Sent from the XWiki- Users mailing list archive at Nabble.com.

Reply

Show replies by date

Vincent Massol

4 Jul 4 Jul

12:45 p.m.

New subject: [xwiki-users] which HTML parsing libs are already using/shiipped with XWiki ?

Hi, On Jul 4, 2012, at 12:41 PM, Arioch wrote:

which HTML parsers do u have onboard ? Jericho, JTidy, whatever ?

Please don't cross post. And don't use JIRA as a place to get answers! It's there to report issues… We use our own parse based on SAX in the Rendering module. We also use htmlcleaner to clean up HTML. Thanks -Vincent

Reply

Arioch

1:13 p.m.

New subject: [xwiki-users] which HTML parsing libs are already using/shiipped with XWiki ?

okay,. maybe you'd better devise the code ? i can only copy-paste from googled sources without real Java knowledge and real ability to test. So even if i do something - it still would have to be reviewed and maybe even would not compile. http://www.benmccann.com/dev-blog/java-html-parsing-library-comparison/ Here i can see how to create DOM, yet it would be overkill, SAX is proper better approach here. But can SAX be run over HTML not XML ? java-sources.net suggest to use hotsax.sf.net, but it probably lacks auto-detection. another HTML SAX is JTagSoup, it also lacks auto-detection yet suggests looking at jchardet.sourceforge.net For what i can see, OpenOffice does not offer UTF-16 or such exports, so we have to choose between UTF-8, UTF-7 and single-byte encodings... That should replace hardcoded " htmlReader = new InputStreamReader(htmlStream, "UTF-8");" at https://github.com/xwiki/xwiki-platform/blob/master/xwiki-platform-core/xwi… We maybe can assume any charset initially, for we need only Latin1 tags and values. Yet... Some tag parameters values might be non-Latin and if tags order would be different, they might come up before the encoding tag... Like in <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <HTML> <HEAD> <META HTTP-EQUIV="CONTENT-TYPE" CONTENT="text/html; charset=utf-8"> <TITLE></TITLE> <META NAME="GENERATOR" CONTENT="OpenOffice.org 3.4 (Win32)"> <META NAME="AUTHOR" CONTENT="РўРµСЃС‚РѕРІС‹Р№ РјРµРЅРµРґР¶РµСЂ"> <META NAME="CREATED" CONTENT="20120525;11540000"> <META NAME="CHANGEDBY" CONTENT="РўРµСЃС‚РѕРІС‹Р№ РјРµРЅРµРґР¶РµСЂ"> Here u can see that charset is specified above all the rest. If we can assume that as a traditional behaviour, then we can even just offset few bytes from beginning and get directly to '=utf-8"' part :-) -- View this message in context: http://xwiki.475771.n2.nabble.com/which-HTML-parsing-libs-are-already-using… Sent from the XWiki- Users mailing list archive at Nabble.com.

Reply

Arioch

1:31 p.m.

New subject: [xwiki-users] which HTML parsing libs are already using/shiipped with XWiki ?

HTML cleaner tells to have auto-detection in one of its methods http://htmlcleaner.sourceforge.net/doc/org/htmlcleaner/HtmlCleaner.html#cle… .... Okay, this probably might be copy-pasted almost non-modified (if HtmlCleaner's 3-clause BSD license allows it, http://htmlcleaner.sourceforge.net/license.php) Potential extension might be the loop, if there are multiple charset declarations (i saw such malformed HTMLs in the wild, though i have doubts OpenOffice would ever do such a thing, but who knows what HTML importer might get reused for later?), breaking out on 1st `supported` charset. Or just copy-paste like that, to return 1st match and no more guessing... .... org/htmlcleaner/Utils.java public static String getCharsetFromContent(URL url) throws IOException { InputStream stream = url.openStream(); byte chunk[] = new byte[2048]; int bytesRead = stream.read(chunk); if (bytesRead > 0) { String startContent = new String(chunk); String pattern = "\\<meta\\s*http-equiv=[\\\"\\']content-type[\\\"\\']\\s*content\\s*=\\s*[\"']text/html\\s*;\\s*charset=([a-z\\d\\-]*)[\\\"\\'\\>]"; Matcher matcher = Pattern.compile(pattern, Pattern.CASE_INSENSITIVE).matcher(startContent); if (matcher.find()) { String charset = matcher.group(1); if (Charset.isSupported(charset)) { return charset; } } } return null; } ----------------- Another approach might be to use HTML parser. http://htmlparser.sourceforge.net/faq.html#encodingchangeexception This sounds like the target, the parser able to made some assumptions of charset and re-scan if proven wrong. -- View this message in context: http://xwiki.475771.n2.nabble.com/which-HTML-parsing-libs-are-already-using… Sent from the XWiki- Users mailing list archive at Nabble.com.

Reply

4510

days inactive

4510

days old

xwiki-users@xwiki.org

Manage subscription

3 comments

2 participants

tags (0)

participants (2)

Arioch
Vincent Massol