4 ms·
There are lots of attempts to write new Wikipedia parsers that just do "the useful stuff", like getting the text. They all fail, for the simple reason that som
by mkl 14y ago
There are lots of attempts to write new Wikipedia parsers that just do "the useful stuff", like getting the text. They all fail, for the simple reason that some of the text comes from MediaWiki templates.
E.g.
about {{convert|55|km|0|abbr=on}} east of
will turn into
about 55 km (34 mi) east of
and
{{As of|2010|7|5}}
will turn into
As of 5 July 2010
and so on (there are thousands of relevant templates). It's simply not possible to get the full plain text without processing the templates, and the only system that can correctly and completely parse the templates is MediaWiki itself.
Yes it's a huge system entirely written in PHP, but you can make a simple command line parser with it pretty easily (though it took me quite a while to figure out how). The key points are to put something like
$IP = strval(getenv('MW_INSTALL_PATH')) !== ''
? getenv('MW_INSTALL_PATH')
: '/usr/share/mediawiki';
require_once("$IP/maintenance/commandLine.inc");
at the start of it, and then use the Parser class. You get HTML out, but it's simple and well-formed (to get text, start with the top level p tags).
To get it to process templates, get a Wikipedia dump, extract the templates, and use the mwdumper tool to import them into your local MediaWiki database.
I don't know if this is the best or "right" way to do it, but it's the only way I've found that actually works.
- alok-g 14y ago>> To get it to process templates, get a Wikipedia dump, extract the templates, and use the mwdumper tool to import them into your local MediaWiki database. Could you please explain this more? Specifically, what is meant by "extract" the templates? From what I gather from your message, you are proposing using MediaWiki itself to process the templates and output more of a plain text (within the HTML output).
- mkl 14y agoThe dumps of Wikipedia come as big bzip2ed XML files containing all articles, templates, etc., each in a "page" tag. The templates have titles starting with "Template:", so they are easy to detect: <page> <title>Template:Convert</title> ... </page> It's these page tags that need to be copied to a new XML file, along with the header and footer from the original. From what I gather from your message, you are proposing using MediaWiki itself to process the templates and output more of a plain text (within the HTML output). Correct. The MediaWiki parser outputs HTML containing all the text to be displayed, including that generated by templates.
- alok-g 14y ago>> there are thousands of relevant templates Is there a documentation of at least the relevant ones somewhere?
- mkl 14y agoI doubt it - anyone can add new ones any time.
- deleted 14y ago[deleted]