3 ms·
I just found a newsgroup meme that pointed me to Wikipedia's policy on crawlers vs. using a Database download: http://en.wikipedia.org/wiki/Wikipedia:Database_
by colortone 18y ago
I just found a newsgroup meme that pointed me to Wikipedia's policy on crawlers vs. using a Database download:
http://en.wikipedia.org/wiki/Wikipedia:Database_download http://en.wikipedia.org/wiki/Wikipedia:Database_download
Can anyone translate this? I understand the part about not letting any crawlers ping Wikipedia more than one time per second, but what does this DB download provide and how often is it updated?
- xirium 18y agoThe full dataset is available as HTML, XML and SQL from http://download.wikipedia.org/ http://download.wikipedia.org/ . Some versions are updated weekly. Some versions are updated less frequently. It doesn't occur on a strict schedule because the volume of data increases and sometimes the snapshot process fails. For general use, the HTML version would be easiest to process. The HTML is compressed at a 14:1 ratio using 7zip, so it vastly reduces Wikipedia's server load and bandwidth utilisation. For diffing edit history, the SQL version would be most appropriate because it mirrors the Wikipedia schema, including previous edits. We were using the HTML dataset for testing our own search, but we've outgrown it.
- colortone 18y agoThanks so much! Seen any cool Wikipedia bots or apps in your travels? ANYthing is interesting to me...