3 ms·

This mistakes the goal. It's not to block the scrapers, but to discourage excessive (and costly) scraping. The Anubis cookies are bound to particular IP addres

This mistakes the goal. It's not to block the scrapers, but to discourage excessive (and costly) scraping. The Anubis cookies are bound to particular IP address. The scrapers are often using a large set of IP addresses, so they'll be paying a far higher cost than this suggests.

ssl-3 2mo ago
There are multiple goals at play. They're easy to find in HN comment sections whenever these topics arise. One goal is to reduce excessive scraping, usually for monetary or performance reasons. This is the goal you mention, and is motivated by a desire keeping the thing working at all. Another goal is to stop bots from ingesting the content, carte blanche. This goal is motivated by a desire to dictate how bots (and by extension, people) may use the information that is otherwise freely-available on the web. These are not the same goals.
cedws 2mo ago
The cost is nothing. The browser implementation is too slow, mobile devices are too slow, and hash algorithms with hardware acceleration are too fast. You can’t balance these three constraints in a way that only keeps out the bad guys. The hashing is just elaborate obfuscation. Anubis uses SHA256 which isn’t ASIC-resistant, and thanks to Bitcoin you could probably buy one off the shelf.
- eqvinox 2mo ago
  and yet, it works.
  - cedws 2mo ago
    The hashing has nothing to do with it. It's just an arbitrary hoop for clients to jump through that filters out the bots that haven't implemented that hoop. So why not just use the client's fingerprint and call it a day? Use their canvas or JA3 fingerprint and I'm willing to bet it would be just as effective.
    - eqvinox 2mo ago
      It's impossible to know without trying, but I suspect those checks would be too easy to bypass/fake in a crawler. You can submit a PR and then we can try…
      - cedws 2mo ago
        Most scrapers are unsophisticated, this is why Anubis "works." For this same reason, basic fingerprinting already employed by Cloudflare et al would have the same efficacy.

ssl-3 2mo ago

There are multiple goals at play. They're easy to find in HN comment sections whenever these topics arise. One goal is to reduce excessive scraping, usually for monetary or performance reasons. This is the goal you mention, and is motivated by a desire keeping the thing working at all. Another goal is to stop bots from ingesting the content, carte blanche. This goal is motivated by a desire to dictate how bots (and by extension, people) may use the information that is otherwise freely-available on the web. These are not the same goals.

cedws 2mo ago

The cost is nothing. The browser implementation is too slow, mobile devices are too slow, and hash algorithms with hardware acceleration are too fast. You can’t balance these three constraints in a way that only keeps out the bad guys. The hashing is just elaborate obfuscation. Anubis uses SHA256 which isn’t ASIC-resistant, and thanks to Bitcoin you could probably buy one off the shelf.

eqvinox 2mo ago
and yet, it works.
- cedws 2mo ago
  The hashing has nothing to do with it. It's just an arbitrary hoop for clients to jump through that filters out the bots that haven't implemented that hoop. So why not just use the client's fingerprint and call it a day? Use their canvas or JA3 fingerprint and I'm willing to bet it would be just as effective.
  - eqvinox 2mo ago
    It's impossible to know without trying, but I suspect those checks would be too easy to bypass/fake in a crawler. You can submit a PR and then we can try…
    - cedws 2mo ago
      Most scrapers are unsophisticated, this is why Anubis "works." For this same reason, basic fingerprinting already employed by Cloudflare et al would have the same efficacy.