4 ms·
To clearify, I wasn't critiquing your approach, just the phrasing. What you're doing is how it should be done for sources you don't have direct access to. I'd
by klapinat0r 13y ago
To clearify, I wasn't critiquing your approach, just the phrasing.
What you're doing is how it should be done for sources you don't have direct access to.
I'd use 10 minutes though, as http://www.uscourts.gov/robots.txt http://www.uscourts.gov/robots.txt points to a crawl delay of 10. Whether it's checked is the risk you take.
- konklone 13y agoI never thought to check that! Thanks, I may adjust my rate.
- konklone 13y agoRight, now I remember - just double-checked, and Crawl-delay refers to seconds, not minutes. http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl...
- klapinat0r 13y agoOh, my bad. Sorry. Thanks for correcting me.