3 ms·
Using Wikipedia as an Impromptu RottenTomatoes API (2021)
- PaulHoule 5y ago(Some lessons learned the hard way from struggling w/ DBpedia, Freebase, WikiData and every other way to suck data out of Wikipedia.) Reading the Wikipedia markup directly is a rabbit hole. It is macros on top of macros and you find that with 20 rules you can extract 80% of the data, with 200 rules you can extract 90% of data, with 2000 rules 95%, ... Try parsing the HTML and you sometimes get 99.5% of the data with 5 rules. The reason the HTML rules is that the feedback loop closes on the HTML output. If the HTML output doesn't look right, editors will fix the markup to make the HTML markup look right. There is no pressure to conform (say automated unit tests) markup that your parser will parse.