You obviously should read the robots.txt and follow it, but it's just as much about not just absolutely hammering a site over and over again, 24-7. Your scraper

You obviously should read the robots.txt and follow it, but it's just as much about not just absolutely hammering a site over and over again, 24-7. Your scraper needs to learn about update patterns and adjust it's scraping interval. It needs to back off if the respons times increases. People are bringing up robots.txt because they want to ban AI scrapers, but if those same scrapers didn't constantly pound sites into the ground, it would be less of an issue.