3 ms·
It's for crawlers not custom scrapers
by pknerd 1y ago
It's for crawlers not custom scrapers
- n_u 1y agoRespecting robots.txt is a convention not enforced by anything so yes the bot is certainly free to ignore it. But I’m not sure I understand your distinction. A scraper is a crawler regardless of whether it is “custom”or an off the shelf solution. The author also said the bot identifed itself as a crawler > Mozilla/5.0 (compatible; crawler)