3 ms·

> The infoboxes, which is what this guy is scraping, are much easier to scrape from the HTML than from the XML dumps. How is it possible that "give me all the

> The infoboxes, which is what this guy is scraping, are much easier to scrape from the HTML than from the XML dumps. How is it possible that "give me all the infoboxes, please" is more than a single query, download, or even URL at this point?

ceejayoz 5y ago
The problem lies in parsing them. Look at the template for a subway line infobox, for example. https://en.wikipedia.org/wiki/Template:Bakerloo_line_RDT https://en.wikipedia.org/wiki/Template:Bakerloo_line_RDT It's a whole little clever language (https://en.wikipedia.org/wiki/Wikipedia:Route_diagram_template https://en.wikipedia.org/wiki/Wikipedia:Route_diagram_templa...) for making complex diagrams out of rather simple pictograms (https://commons.wikimedia.org/wiki/Template:Bsicon https://commons.wikimedia.org/wiki/Template:Bsicon).
- jancsika 5y ago
  Oh wow. But every other infobox I've seen has key/value pairs where the key was always a string. So what's the spec for an info box? Is it simply to have a starting `<table class="hello_i_am_infobox">` and an ending `</table>`?
  - bawolff 5y ago
    En wikipedia has some standards. Generally though they are user-created tables and its up to the users to make them consistent (if they so desire). En Wikipedia generally does, but its not exactly a hard garuntee. If you want machine readable use wikidata (if you hate rdf you can still scrape the html preview of the data)
cldellow 5y ago
Even just being able to download a tarball of the HTML of the infoboxes would be really powerful, setting aside the difficulty of, say, translating them into a consistent JSON format. That plus a few other key things (categories, opening paragraph, redirects, pageview data) enable a lot of powerful analysis. That actually might be kind of a neat thing to publish. Hmmmm.
- jancsika 5y ago
  Better yet-- what is the set of wikipedia articles which have an info box that cannot be sensibly interpreted as key/value pairs where the key is a simple string?
squeaky-clean 5y ago
The infoboxes aren't standardized at all. The HTML they generate is.
- jancsika 5y ago
  Hehe-- I am going to rankly speculate nearly all of them follow an obvious standard of key/value pairs where the key is a string. And then there are like two or three subcultures on Wikipedia that put rando stuff in there and would troll to the death before being forced to change the their infobox class to "rando_box" or whatever negligible effort it would take them if a standard were to be enforced. Am I anywhere close to being correct?
  - bawolff 5y ago
    I think you'll have to more clearly define what you mean by "key-value" pairs.

ceejayoz 5y ago

The problem lies in parsing them. Look at the template for a subway line infobox, for example. https://en.wikipedia.org/wiki/Template:Bakerloo_line_RDT https://en.wikipedia.org/wiki/Template:Bakerloo_line_RDT It's a whole little clever language (https://en.wikipedia.org/wiki/Wikipedia:Route_diagram_template https://en.wikipedia.org/wiki/Wikipedia:Route_diagram_templa...) for making complex diagrams out of rather simple pictograms (https://commons.wikimedia.org/wiki/Template:Bsicon https://commons.wikimedia.org/wiki/Template:Bsicon).

jancsika 5y ago
Oh wow. But every other infobox I've seen has key/value pairs where the key was always a string. So what's the spec for an info box? Is it simply to have a starting `<table class="hello_i_am_infobox">` and an ending `</table>`?
- bawolff 5y ago
  En wikipedia has some standards. Generally though they are user-created tables and its up to the users to make them consistent (if they so desire). En Wikipedia generally does, but its not exactly a hard garuntee. If you want machine readable use wikidata (if you hate rdf you can still scrape the html preview of the data)

cldellow 5y ago

Even just being able to download a tarball of the HTML of the infoboxes would be really powerful, setting aside the difficulty of, say, translating them into a consistent JSON format. That plus a few other key things (categories, opening paragraph, redirects, pageview data) enable a lot of powerful analysis. That actually might be kind of a neat thing to publish. Hmmmm.

jancsika 5y ago
Better yet-- what is the set of wikipedia articles which have an info box that cannot be sensibly interpreted as key/value pairs where the key is a simple string?

squeaky-clean 5y ago

The infoboxes aren't standardized at all. The HTML they generate is.

jancsika 5y ago
Hehe-- I am going to rankly speculate nearly all of them follow an obvious standard of key/value pairs where the key is a string. And then there are like two or three subcultures on Wikipedia that put rando stuff in there and would troll to the death before being forced to change the their infobox class to "rando_box" or whatever negligible effort it would take them if a standard were to be enforced. Am I anywhere close to being correct?
- bawolff 5y ago
  I think you'll have to more clearly define what you mean by "key-value" pairs.