3 ms·

TLDR: Perplexity agents allegedly ignored Amazon's robots.txt

• j4kp07 11d ago
  Why should they not? So does my browser when I ask it to open a website.
• Zambyte 11d ago
  robots.txt is a suggestion, not a rule. It's a service to crawlers to help them avoid wasting time. Using robots.txt as a security measure is like trying to stop a foreign invader with a "road closed" sign. Edit: this is obviously assuming they literally meant robots.txt, but from a qyick skim of the site, it doesn't look like that was mentioned at least. I guess you meant it metaphorically :P
  • shermantanktop 11d ago
    > robots.txt is a service to crawlers to help them avoid wasting time I doubt anyone edits robots.txt with a warm fuzzy feeling about how much they are helping crawlers be efficient.
    • Zambyte 11d ago
      Well that's what they're doing.