2 ms·
I know companies that already solve it.
by ronsor 2y ago
I know companies that already solve it.
- creata 2y agoWhy is spending all that CPU time to scrape the handful of sites that use Anubis worth it to them?
- vhcr 2y agoBecause it's not a lot of CPU, you only have to solve it once per website, and the default policy difficulty of 16 for bots is worthless because you can just change your user agent so you get a difficulty of 4.
- wredcoll 2y agoI mean... knowing how to solve it isn't the trick, it's doing it a million times a minute for your firehose scraper.
- udev4096 2y agoAnubis adds a cookie name `within.website-x-cmd-anubis-auth` which can be used by scrapers for not solving it more than once. Just have a fleet of servers whose sole purpose is to extract the cookie after solving the challenges and make sure all of them stay valid. It's not a big deal
- fc417fc802 2y agoRequests are associated with the cookie meaning you can trace and block or rate limit as necessary. The cost of solving the PoW is the cost of establishing a new session. If you get blocked you have to solve again.