3 ms·
I honestly couldn't really offer you any advice on how to prevent scraping. The best you can do is slow us down. And the best way to do that is the figure out
by Silhouette 15y ago
I honestly couldn't really offer you any advice on how to prevent scraping. The best you can do is slow us down.
And the best way to do that is the figure out who is hitting your pages in such a methodical manner and rate limiting them. If you are smart enough, you might also try to "hellban" us, by serving up totally false data.
Well, no, there are other ways too.
For example, any site behind a paywall probably has your identity, and unless you live in a faraway place with impotent copyright laws -- and there aren't that many of them any more -- there are often staggeringly disproportionate damages for infringement available through the courts these days, certainly enough to justify retaining legal representation to bring a suit in any major jurisdiction. Given a server log showing a pattern of systematic downloading that could only be done by an automated scraper in violation of a site's ToS, and given a credit card in your name linked to the account and an IP address linked to your residence where the downloads went, I imagine it's going to be a fairly short and extremely expensive lawsuit if you upset the wrong site owner.
- lusr 15y agoNot all valuable scrapeable data is copyrightable. I also know of a number of sites I've scraped that don't even bother attempting to restrict your access to their data through T&Cs even though its the basis for their site (not that they'd have much of a legal basis for enforcing that, anyway). Ultimately if you're in the business of selling raw data with no value added, the problem is your business model, not scrapers.
- Silhouette 15y agoNot all valuable scrapeable data is copyrightable. Sure, but a lot of it is, and even the bits that aren't may be protected by other laws such as database rights depending on your jurisdiction. I think anyone who maintains that you can't stop scrapers as a general principle is possibly a little unwise.