3 ms·
Crawling speed is between 100...1000 pages per second. Stupid question, but you were crawling news.ycombinator.com, right? Its robots.txt (https://news.ycombi
by WalterGR 6y ago
Crawling speed is between 100...1000 pages per second.
Stupid question, but you were crawling news.ycombinator.com, right?
Its robots.txt (https://news.ycombinator.com/robots.txt https://news.ycombinator.com/robots.txt) contains
Crawl-delay: 30
Why did you not follow that?
(I'm not being accusatory. I'm both curious about web crawling in general and have personally been archiving the front page and "new" every 60 seconds or so... (Obviously there's no reason for me to retrieve them more often, but my curiosity persists.))
- wolfgarbe 6y ago>> you were crawling news.ycombinator.com, right? No, for retrieving the Hacker News Posts we were using the public Hacker News API, which returns the posts in JSON format: https://github.com/HackerNews/API https://github.com/HackerNews/API The crawling speed of 100...1000 pages per second refers to crawling the external pages linked from Hacker news posts. As they are from different domains we can achieve a high crawling speed while being a polite crawler with a low crawling rate per domain.