if they're running on AWS, which most crawlers are not When I've run scraping software in the past I used DigitalOcean, which doesn't contain a requirement to

if they're running on AWS, which most crawlers are not When I've run scraping software in the past I used DigitalOcean, which doesn't contain a requirement to abide by robots.txt. As far as I can tell it's both legal and consistent with their ToS to run a program that makes purchases on a website.