> It almost makes me feel that I am breaking the law when scraping a site, yet web scraping is on of the most basic programming things. Yeah, same for me. Reg

> It almost makes me feel that I am breaking the law when scraping a site, yet web scraping is on of the most basic programming things. Yeah, same for me. Regarding the denylisting, I guess it depends on what is being scraped and how often the scraping happens? I'm maintaining a remote jobs aggregator website and I've never been blocked before (but I'm not scraping more than ~5 times per day the same web page). And with a caching strategy, I think that even a scrape-as-a-service API like the one I'm building in the article should be "kinda" safe (besides edge cases that brute force the cache constantly, like by adding random query-params)?