3 ms·
When I mean crawler I mean something that discovers new pages. Refreshing the same URL isn't really crawling. But yes, it may be the best available solution in
by kevincox 4y ago
When I mean crawler I mean something that discovers new pages. Refreshing the same URL isn't really crawling.
But yes, it may be the best available solution in this case, even if I would argue that it isn't really it's main purpose.
- cmatthias 4y agoAfter reading this and your response to a sibling comment I wholeheartedly disagree with you on both the specific definition of the word crawler and what the "main purpose" of robots.txt is, but glad we can agree that Google should be doing more to respect rate limits :)
- ddevault 4y agoWhat you're thinking about, in my opinion, is best referred to as a spider.