3 ms·
Calling them "scrapers" implies they are doing something nefarious (stealing content). Robots.txt is for law abiding bots.
by dumbfounder 14y ago
Calling them "scrapers" implies they are doing something nefarious (stealing content). Robots.txt is for law abiding bots.
- ig1 14y agoNot really. "Scraping" just refers to extracting data from a site using an automated method, it doesn't have any connotations about the motivation or acceptability of the process.