3 ms·
Wikipedia has no objection to crawling a couple thousand pages if you do so at a reasonable speed and set a user-agent with a contact email. If you want to cra
by bawolff 5y ago
Wikipedia has no objection to crawling a couple thousand pages if you do so at a reasonable speed and set a user-agent with a contact email.
If you want to crawl millions of pages please use a dump.