7 ms·

Please don't scrape raw HTML from Wikipedia. They do a lot of work to make their content accessible in so many machine-readable formats, from the raw XML dumps

Please don't scrape raw HTML from Wikipedia. They do a lot of work to make their content accessible in so many machine-readable formats, from the raw XML dumps (https://dumps.wikimedia.org https://dumps.wikimedia.org) to the fully-featured API with a nice sandbox (https://en.wikipedia.org/wiki/Special:ApiSandbox https://en.wikipedia.org/wiki/Special:ApiSandbox) and Wikidata (https://wikidata.org https://wikidata.org).

whall6 5y ago
Genuine question from a non-programmer: why? Is it because the volume of requests increases load on the servers/costs?
- michaelbuckbee 5y ago
  That's part of it, but also it's typically much more difficult and there's an element of "why are you making this so much harder on yourself".
  - billpg 5y ago
    (Author of original article here.) That's the great thing about HtmlAgilityPack, extracting data from HTML is really easy. I might even say even easier than if I had the page in some table-based data system.
    - SlimyHog 5y ago
      The HTML is more volatile and subject to change than other sources though
      - FalconSensei 5y ago
        Don't remember the last time wikipedia changed the infobox though
  - jcun4128 5y ago
    Can make it even harder, use Puppeteer to take screenshots then pass it to an OCR to get the text.
    - thechao 5y ago
      https://xkcd.com/378/ https://xkcd.com/378/
- nonameiguess 5y ago
  Unlike APIs, html class/tag names or whatever provide no stability guarantees. The site owner can break your parser whenever they want for any reason. They can do that with an API, but usually won't since some guarantee of stability is the point of an API.
  - billpg 5y ago
    True, but the analysis was done on files downloaded over the span of two or three days. If someone had decided to change the CSS class of an infobox during that time, I'd have noticed, investigated and adjusted my code appropriately.
  - pigsonthewing 5y ago
    "html class/tag names or whatever provide no stability guarantees" Not quite. Many Wikipedia infoboxes (and some other templates) use standardised class names from microformats such as hCard: https://en.wikipedia.org/wiki/Wikipedia:Microformats https://en.wikipedia.org/wiki/Wikipedia:Microformats
- matkoniecz 5y ago
  Scrapping, especially on a large scale, can put a noticeable strain on servers. Bulk downloads (database dumps) are much cheaper to serve for someone crawling millions of pages. It gets even more significant if generation of reply is resource intensive (not sure is Wikipedia qualifying for that but complex templates may cause this).
traceroute66 5y ago
IANAL but since the pages are published under Creative Commons Attribution-ShareAlike, if someone wishes to collect the text on the basis of the HTML version then there's not much you can do about it. Wikimedia no doubt have caching, CDNs and all that jazz in place so the likely impact on infrastructure is probably de-minimis in the grand scheme of things (the thousands or millions of humans who visit the site every second).
- learc83 5y ago
  >IANAL but since the pages are published under Creative Commons Attribution-ShareAlike, if someone wishes to collect the text on the basis of the HTML version then there's not much you can do about it. They said please don't, not don't do it or they'll sue you. But content license and site terms of use are different things. From their terms of use you aren’t allowed to > [Disrupt] the services by placing an undue burden on a Project website or the networks or servers connected with a Project website; Wikipedia is also well within their rights to implement scraping countermeasures.
  - traceroute66 5y ago
    > [Disrupt] the services by placing an undue burden on a Project website or the networks or servers connected with a Project website; Two things: 1) The English wikipedia *alone* gets 250 million page views per day ! So you would have to be doing an awful lot to cause "undue burden". 2) The Wikipedia robots.txt page openly implies that crawling (and therefore scraping) is acceptable *as long as* you do it in a rate-limited fashion, e.g.: >Friendly, low-speed bots are welcome viewing article pages, but not dynamically-generated pages please. > There are a lot of pages on this site, and there are some misbehaved spiders out there that go _way_ too fast. >Sorry, wget in its recursive mode is a frequent problem. Please read the man page and use it properly; there is a --wait option you can use to set the delay between hits, for instance.
    - learc83 5y ago
      1. You'd be surprised what kind of traffic scrapers can generate. I've seen scraping companies employing botnets to get around rate limiting that could easily cost enough in extra server fees to cause an "undue burden". At a previous company we had the exact problem that we published all of our content as machine readable xml, but we had scrapers costing us money by insisting on using our search interface to access our content. 2. No one is going to jail for scraping a few thousand or even a few million pages, but just because low speed web crawlers are allowed to index the site, doesn't mean scraping for every possible use is permitted.
cldellow 5y ago
The infoboxes, which is what this guy is scraping, are much easier to scrape from the HTML than from the XML dumps. The reason is that the dumps just have pointers to templates, and you need to understand quite a bit about Wikipedia's bespoke rendering system to know how to fully realize them (or use a constantly-evolving library like wtf_wikipedia [1] to parse them). The rendered HTML, on the other hand, is designed for humans, and so what you see is what you get. [1]: https://github.com/spencermountain/wtf_wikipedia https://github.com/spencermountain/wtf_wikipedia
- lou1306 5y ago
  Still, I guess you could get the dumps and do a local Wikimedia setup based on them, and then crawl that instead?
  - cldellow 5y ago
    You could, and if he was doing this on the entire corpus that'd be the responsible thing to do. But, his project really was very reasonable: - it fetched ~2,400 pages - he cached them after first fetch - Wikipedia aggressively caches anonymous page views (eg the Queen Elizabeth page has a cache age of 82,000 seconds) English Wikipedia does about 250,000,000 pageviews/day. This guy's use was 0.001% of traffic on that day. I get the slippery slope arguments, but to me, it just doesn't apply. As someone who has donated $1,000 to Wikipedia in the past, I'm totally happy to have those funds spent supporting use cases like this, rather than demanding that people who want to benefit from Wikipedia be able to set up a MySQL server, spend hours doing the import, install and configure a PHP server, etc, etc.
    - habibur 5y ago
      > This guy's use was 0.001% of traffic on that day For 1 person consuming from one of the most popular sites on the web, this really reads big.
      - cldellow 5y ago
spoonjim 5y ago
How does scraping raw HTML from Wikipedia hurt them? I'd think they could serve the HTML from cache more likely than the API call.

whall6 5y ago

Genuine question from a non-programmer: why? Is it because the volume of requests increases load on the servers/costs?

michaelbuckbee 5y ago
That's part of it, but also it's typically much more difficult and there's an element of "why are you making this so much harder on yourself".
- billpg 5y ago
  (Author of original article here.) That's the great thing about HtmlAgilityPack, extracting data from HTML is really easy. I might even say even easier than if I had the page in some table-based data system.
  - SlimyHog 5y ago
    The HTML is more volatile and subject to change than other sources though
    - FalconSensei 5y ago
      Don't remember the last time wikipedia changed the infobox though
- jcun4128 5y ago
  Can make it even harder, use Puppeteer to take screenshots then pass it to an OCR to get the text.
  - thechao 5y ago
    https://xkcd.com/378/ https://xkcd.com/378/
nonameiguess 5y ago
Unlike APIs, html class/tag names or whatever provide no stability guarantees. The site owner can break your parser whenever they want for any reason. They can do that with an API, but usually won't since some guarantee of stability is the point of an API.
- billpg 5y ago
  True, but the analysis was done on files downloaded over the span of two or three days. If someone had decided to change the CSS class of an infobox during that time, I'd have noticed, investigated and adjusted my code appropriately.
- pigsonthewing 5y ago
  "html class/tag names or whatever provide no stability guarantees" Not quite. Many Wikipedia infoboxes (and some other templates) use standardised class names from microformats such as hCard: https://en.wikipedia.org/wiki/Wikipedia:Microformats https://en.wikipedia.org/wiki/Wikipedia:Microformats
matkoniecz 5y ago
Scrapping, especially on a large scale, can put a noticeable strain on servers. Bulk downloads (database dumps) are much cheaper to serve for someone crawling millions of pages. It gets even more significant if generation of reply is resource intensive (not sure is Wikipedia qualifying for that but complex templates may cause this).

traceroute66 5y ago

IANAL but since the pages are published under Creative Commons Attribution-ShareAlike, if someone wishes to collect the text on the basis of the HTML version then there's not much you can do about it. Wikimedia no doubt have caching, CDNs and all that jazz in place so the likely impact on infrastructure is probably de-minimis in the grand scheme of things (the thousands or millions of humans who visit the site every second).

learc83 5y ago
>IANAL but since the pages are published under Creative Commons Attribution-ShareAlike, if someone wishes to collect the text on the basis of the HTML version then there's not much you can do about it. They said please don't, not don't do it or they'll sue you. But content license and site terms of use are different things. From their terms of use you aren’t allowed to > [Disrupt] the services by placing an undue burden on a Project website or the networks or servers connected with a Project website; Wikipedia is also well within their rights to implement scraping countermeasures.
- traceroute66 5y ago
  > [Disrupt] the services by placing an undue burden on a Project website or the networks or servers connected with a Project website; Two things: 1) The English wikipedia *alone* gets 250 million page views per day ! So you would have to be doing an awful lot to cause "undue burden". 2) The Wikipedia robots.txt page openly implies that crawling (and therefore scraping) is acceptable *as long as* you do it in a rate-limited fashion, e.g.: >Friendly, low-speed bots are welcome viewing article pages, but not dynamically-generated pages please. > There are a lot of pages on this site, and there are some misbehaved spiders out there that go _way_ too fast. >Sorry, wget in its recursive mode is a frequent problem. Please read the man page and use it properly; there is a --wait option you can use to set the delay between hits, for instance.
  - learc83 5y ago
    1. You'd be surprised what kind of traffic scrapers can generate. I've seen scraping companies employing botnets to get around rate limiting that could easily cost enough in extra server fees to cause an "undue burden". At a previous company we had the exact problem that we published all of our content as machine readable xml, but we had scrapers costing us money by insisting on using our search interface to access our content. 2. No one is going to jail for scraping a few thousand or even a few million pages, but just because low speed web crawlers are allowed to index the site, doesn't mean scraping for every possible use is permitted.

cldellow 5y ago

The infoboxes, which is what this guy is scraping, are much easier to scrape from the HTML than from the XML dumps. The reason is that the dumps just have pointers to templates, and you need to understand quite a bit about Wikipedia's bespoke rendering system to know how to fully realize them (or use a constantly-evolving library like wtf_wikipedia [1] to parse them). The rendered HTML, on the other hand, is designed for humans, and so what you see is what you get. [1]: https://github.com/spencermountain/wtf_wikipedia https://github.com/spencermountain/wtf_wikipedia

lou1306 5y ago
Still, I guess you could get the dumps and do a local Wikimedia setup based on them, and then crawl that instead?
- cldellow 5y ago
  You could, and if he was doing this on the entire corpus that'd be the responsible thing to do. But, his project really was very reasonable: - it fetched ~2,400 pages - he cached them after first fetch - Wikipedia aggressively caches anonymous page views (eg the Queen Elizabeth page has a cache age of 82,000 seconds) English Wikipedia does about 250,000,000 pageviews/day. This guy's use was 0.001% of traffic on that day. I get the slippery slope arguments, but to me, it just doesn't apply. As someone who has donated $1,000 to Wikipedia in the past, I'm totally happy to have those funds spent supporting use cases like this, rather than demanding that people who want to benefit from Wikipedia be able to set up a MySQL server, spend hours doing the import, install and configure a PHP server, etc, etc.
  - habibur 5y ago
    > This guy's use was 0.001% of traffic on that day For 1 person consuming from one of the most popular sites on the web, this really reads big.
    - cldellow 5y ago

spoonjim 5y ago

How does scraping raw HTML from Wikipedia hurt them? I'd think they could serve the HTML from cache more likely than the API call.