4 ms·
In my opinion the whole agents.txt thing is bullshit and/or politeness, and should have zero legal significance. If something is openly available on the intern
by stanfordkid 11mo ago
In my opinion the whole agents.txt thing is bullshit and/or politeness, and should have zero legal significance.
If something is openly available on the internet you should be able to crawl it, and it is the server's responsibility to identify, authenticate and/or ban clients that do not adhere to it's requirements. If Amazon states "You must not use agents" in it's website terms of service then it would be the individual perplexity user not perplexity itself that is breaking it's terms of service, since they are the one operating the agent.
All in all lame move by Amazon.
- nickff 11mo agoDoes everything you say apply equally to DDoS attackers? If not, you need to come up with some way to differentiate ‘malicious users’ from benign ones. Amazon’s ‘bright line’ seems reasonable, though I’d personally love to see an even better one.
- postalrat 11mo agoIt does not. And never has.
- shakna 11mo agoPerplexity's AI scraping has regularly passed the DDoS threshold. Sometimes in the thousands of requests per second - for data they should have at least cached! https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/ https://blog.cloudflare.com/perplexity-is-using-stealth-unde...