3 ms·
IANAL, but I have done tons of web scraping over the years. My tips: - Keep careful control of the rate you scrape. Every time I have ever heard of someone g
by ltbarcly3 7y ago
IANAL, but I have done tons of web scraping over the years.
My tips:
- Keep careful control of the rate you scrape. Every time I have ever heard of someone getting negative feedback it is because they have scraped pages at a rate that caused an impact on the website they were scraping. If you don't cause a noticeable increase in traffic/load nobody will check to see what is going on, and generally nobody has a reason to care.
- Some sites are notoriously aggressive at going after people, such as craigslist. I wouldn't try to scrape them.
- Use some kind of proxy!
- userbinator 7y ago- Use some kind of proxy! Many proxies, in random order, would be the best. That brings up another curious question: What's the legality of posting a site to something like HN or Slashdot and effectively getting it DDoS'd...?
- kyshoc 7y ago> posting a site to something like HN or Slashdot and effectively getting it DDoS'd I imagine there's some reading of the CFAA that could theoretically land you in hot water for this, but this is silly. Intent is very important. Can one sue or prosecute a popular food critic for writing something about a restaurant, causing lines so long that long-time regulars can't get a seat anymore? On the other hand, you have things like booter services (essentially, DDoS as a service). Continuing the analogy, I imagine if you hired 100 people to physically block the entrance of a restaurant for some reason, you would be on the hook for damages in civil court and something along the lines of "disturbing the peace" in criminal court.