3 ms·
Are they following the sneaker website's robots.txt while doing that? If not, they are probably violating the AWS terms regardless of whether you believe that a
by eli 7y ago
Are they following the sneaker website's robots.txt while doing that? If not, they are probably violating the AWS terms regardless of whether you believe that activity is "malicious."
- ikeboy 7y agoif they're running on AWS, which most crawlers are not When I've run scraping software in the past I used DigitalOcean, which doesn't contain a requirement to abide by robots.txt. As far as I can tell it's both legal and consistent with their ToS to run a program that makes purchases on a website.