2 ms·
Do user agents doing work for users need to respect robots.txt? If yes, does chrome?
by randunel 2y ago
Do user agents doing work for users need to respect robots.txt? If yes, does chrome?
- what 2y agoAny scraper is also a “user agent doing work for users”. Which ones should respect robots.tx?
- randunel 2y agoDoes the user agent fit the definition of a web crawler? If so, then observe robots.txt. This one does not, see https://en.m.wikipedia.org/wiki/Web_crawler https://en.m.wikipedia.org/wiki/Web_crawler