2 ms·

I'm sympathetic to this. I built a search engine for my senior project and my half baked scraper ended up taking down duke law's site during their registration

I'm sympathetic to this. I built a search engine for my senior project and my half baked scraper ended up taking down duke law's site during their registration period. Ended up getting a not so kindly worded email from them, but honestly this wasn't an especially hard problem to solve. All of my traffic was coming from the cluster that was on my university's subnet, it wouldn't have been that hard to for them to IP address timeouts when my crawler started scraping thousands of pages a second on their site. Not to victim blame, this was totally my fault, but I was a bit surprised that they hadn't experienced this before with how much automated scraping goes on.

brightball 5y ago
I’m honestly more interested in bot detection than anything else at this point. It seems like it should be perfectly legal to detect and then hold the connection open for a long period of time without giving a useful response. Or even send highly compressed gzip responses designed to fill their drives. Legal or not, I can’t see any good reason that we can’t make it painful.
- fjabre 5y ago
  Make it painful if they abuse the site. We all benefit from open data. Polite scrapers are just fine and a natural part of the web ecosystem. Google has been scraping the web all day every day for decades now.