3 ms·
What are the details of crawl-delay? If its supposed to be "number of seconds between hits", for some sites, 1 hit per second may as well be "not crawled at al
by Andys 17y ago
What are the details of crawl-delay? If its supposed to be "number of seconds between hits", for some sites, 1 hit per second may as well be "not crawled at all"..
- ComputerGuru 17y agoRight, but his point is that at least this way you won't get delisted :)