5 ms·
There's plenty of arguments that mass scrapers have compute to spare but it seems to me that if Anubis makes it 100x more expensive to scrape then, for any give
by TomatoCo 2mo ago
There's plenty of arguments that mass scrapers have compute to spare but it seems to me that if Anubis makes it 100x more expensive to scrape then, for any given scraping budget, that means you get scraped 100x less. Which is the difference between your server buckling under the load or continuing to serve reliably.
- dullcrisp 2mo agoIf compute isn’t the bottleneck then it wouldn’t make it 100x more expensive to scrape. Of course if it does, then it’ll be effective.
- VladVladikoff 2mo agoIn my experience, not really. Each asset is made by a new proxy, eg some TV somewhere, they send every new request via a new IP address, which is a new machine, and it doesn’t matter how long the response takes, they have already rotated to the next ip immediately and sent another request. Most of these providers have millions of IPs, and it generally doesn’t take millions of requests to scrape a website (unless it’s really big!)
- inigyou 2mo agoAnd each new IP triggers a new Anubis challenge. Isn't it great?
- microtonal 2mo agoSo, once the challenges are solved by the residential proxies (with optimized native implementations [1]). Some unsuspecting TV owner is paying with their electricity bill (not the companies doing the scraping) and we all suffer through challenges. Yes, great outcome! </s> [1] https://lock.cmpxchg8b.com/anubis.html https://lock.cmpxchg8b.com/anubis.html
- inigyou 2mo agoProxy networks just proxy. They don't let their client run compute on the proxy. That isn't even a possibility in socks5 protocol.