4 ms·
Respecting robots.txt is not a legal obligation. If you don’t want the traffic, ban the offending IP block or ASN, or put the origin behind Cloudflare for their
by toomuchtodo 3mo ago
Respecting robots.txt is not a legal obligation. If you don’t want the traffic, ban the offending IP block or ASN, or put the origin behind Cloudflare for their crawler blocker.
You could go down the rabbit hole and try to find someone at Amazon to tell their crawler to chill. Don’t expect success but would make a fun blog post.
https://www.cloudflare.com/learning/ai/how-to-block-ai-crawlers/ https://www.cloudflare.com/learning/ai/how-to-block-ai-crawl...
- pera 3mo agoMy concern is more in relation to trying to use an endpoint from non-public source code, it seems negligent to randomly try endpoints like this
- toomuchtodo 3mo agoWhether it’s negligent isn’t terribly relevant. Is it illegal? It is not, they aren’t bypassing access controls. No different than using Shodan, scanning public IPs, crawling open directories, etc. It would be different if they were attempting to brute force credentials to access an endpoint, but they aren’t.
- rovr138 3mo ago> they aren’t bypassing access controls. Weev went to jail for accessing public api's, https://en.wikipedia.org/wiki/Weev#AT&T_data_breach https://en.wikipedia.org/wiki/Weev#AT&T_data_breach > The flaw was part of a publicly-accessible URL, which allowed the group to collect the e-mails without having to break into AT&T's system. It was argued that he didn't circumvent, but it didn't stop them from putting him in jail initially.
- toomuchtodo 3mo agoHis conviction was vacated and Amazon has deep pockets and diffusion of internal liability. The illicit state drug charges did not help his case.