3 ms·
This is why I put the crucial space between "real" and "time". I cannot hold up to intensive buzzword scrutiny. :) It's every 5 minutes because I don't want th
by konklone 13y ago
This is why I put the crucial space between "real" and "time". I cannot hold up to intensive buzzword scrutiny. :)
It's every 5 minutes because I don't want them to ban me. And in practice, this is an acceptable delay.
- klapinat0r 13y agoTo clearify, I wasn't critiquing your approach, just the phrasing. What you're doing is how it should be done for sources you don't have direct access to. I'd use 10 minutes though, as http://www.uscourts.gov/robots.txt http://www.uscourts.gov/robots.txt points to a crawl delay of 10. Whether it's checked is the risk you take.
- konklone 13y agoI never thought to check that! Thanks, I may adjust my rate.
- konklone 13y agoRight, now I remember - just double-checked, and Crawl-delay refers to seconds, not minutes. http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl...
- klapinat0r 13y agoOh, my bad. Sorry. Thanks for correcting me.