3 ms·
Yes, but the difference is that the term "scraping" also targets things like automatically generating RSS feeds from HTML pages, which is not covered by robots.
by CaptainFever 2y ago
Yes, but the difference is that the term "scraping" also targets things like automatically generating RSS feeds from HTML pages, which is not covered by robots.txt.
- WesolyKubeczek 2y agoI thought robots.txt covered all automated, programmatic access by third parties where a bot slurps stuff and follows links, without splitting hairs about it. But what do I know, the young whippersnappers will just word lawyer me to death, so I better shut up and go away.