3 ms·

Do you respect robots.txt so administrators can block this tool?

• randunel 2y ago
  Do user agents doing work for users need to respect robots.txt? If yes, does chrome?
  • what 2y ago
    Any scraper is also a “user agent doing work for users”. Which ones should respect robots.tx?
    • randunel 2y ago
      Does the user agent fit the definition of a web crawler? If so, then observe robots.txt. This one does not, see https://en.m.wikipedia.org/wiki/Web_crawler https://en.m.wikipedia.org/wiki/Web_crawler
• canogat 2y ago
  Should I be blocked if I ask Claude Desktop to lower the prices in all of my Craigslist ads by 10%?