4 ms·
Wikipedia asks people not to crawl it. There are database dumps that you can instead import into your local MySQL and work from there. https://en.wikipedia.org
by nathell 5y ago
Wikipedia asks people not to crawl it. There are database dumps that you can instead import into your local MySQL and work from there.
https://en.wikipedia.org/wiki/Wikipedia:Database_download#Please_do_not_use_a_web_crawler https://en.wikipedia.org/wiki/Wikipedia:Database_download#Pl...
- bawolff 5y agoWikipedia has no objection to crawling a couple thousand pages if you do so at a reasonable speed and set a user-agent with a contact email. If you want to crawl millions of pages please use a dump.