6 ms·
It's more likely that the project itself will disappear into irrelevance as soon as AI scrapers bother implementing the PoW (which is trivial for them, as the p
by bakugo 1y ago
It's more likely that the project itself will disappear into irrelevance as soon as AI scrapers bother implementing the PoW (which is trivial for them, as the post explains) or figure out that they can simply remove "Mozilla" from their user-agent to bypass it entirely.
- skydhash 1y agoIt's more about the (intentional?) DDoS from AI scrappers, than preventing them from accessing the content. Bandwidth is not cheap.
- dingnuts 1y ago[flagged]
- shkkmo 1y ago> PoW increases the cost for the bots which is great. But not by any meaningful amount as explained in the article. All it actually does is rely on it's obscurity while interfering with legitimate use.
- nialv7 1y ago> Fuck AI scrapers, and fuck all this copyright infringement at scale. Yes, fuck them. Problem is Anubis here is not doing the job. As the article already explains, currently Anubis is not adding a single cent to the AI scrappers' costs. For Anubis to become effective against scrappers, it will necessarily have to become quite annoying for legitimate users.
- Gibbon1 1y agoBest response to AI scrapers is to poison their models.
- nemomarx 1y agohow well is modern poisoning holding up?
- CursedSilicon 1y agoI'll tell you in a second. First I wanna try adding gasoline to my spaghetti as suggested by Google's search
- snerbles 1y agoA balanced diet of hydrocarbons in your carbohydrates!
- codedokode 1y agoWhat about appealing to ethics, i.e. posting messages about how a poor catgirl ended up on the street because AI took her job? To make AI refuse to reply due to ethical concerns?
- dale_glass 1y agoTo the best of my knowledge, it never really worked. Yes, it probably works in the lab, in carefully picked conditions, but in the wild I've yet to see any effect whatsoever. Nobody in the AI communities seems to be complaining about it, models keep getting better, and people even intentionally trained on poisoned images just to show it can be done. IMO on the long end it's a complete dead end of a strategy. Models are many, poisoning can't target everything at once. Even effective poisoning can be just dealt with by finding the algorithm that doesn't care about it.
- altairprime 1y agoDon’t forget signed attestations from “user probably has skin in the game” cloud providers like iCloud (already live in Safari and accepted by Cloudflare, iirc?) — not because they identify you but because abusive behavior will trigger attestation provider rate limiting and termination of services (which, in Apple’s case, includes potentially a console kill for the associated hardware). It’s not very popular to discuss at HN but I bet Anubis could add support for it regardless :) https://datatracker.ietf.org/wg/privacypass/about/ https://datatracker.ietf.org/wg/privacypass/about/ https://www.w3.org/TR/vc-overview/ https://www.w3.org/TR/vc-overview/
- verteu 1y ago> PoW increases the cost for the bots which is great. Trivial to implement, sure, but that added cost will add up quickly. No, the article estimates it would cost less than a single penny to scrape all pages of 1,000,000 distinct Anubis-guarded websites for an entire month.
- thunderfork 1y agoOnce you've built the system that lets you do that, maybe. You still have to do that, though, so it's still raising the cost floor.
- vmttmv 1y agobut... how? when the author ran the numbers, the rough estimate is solving the challenges at a rate of 10000/5 min, on a single instance of the free tier of google compute. that is an insignificant load at an even more insignificant cost.
- guappa 1y agoThat's insanely slow compared to how fast they normally scrape.
- verteu 1y agoAs mentioned in the article, mining one token gets you unfettered access for 7 days.
- thunderfork 1y ago[dead]
- userbinator 1y agoI thought HN was anti-copyright and anti-imaginary-property, or at least the bulk of its users were. Yet all of a sudden, "but AI!!!!1"? a federal crime The rest of the world doesn't care.
- klabb3 1y ago> I thought HN was anti-copyright Maybe. But what’s happening is ”copyright for thee not for me”, not a universal relaxation of copyright. This loophole exploitation by behemoths doesn’t advance any ideological goals, it only inflames the situation because now you have an adversarial topology. You can see this clearly in practice – more and more resources are going into defense and protection of data than ever before. Fingerprinting, captchas, paywalls, login walls, etc etc.
- debugnik 1y ago> as AI scrapers bother implementing the PoW That's what it's for, isn't it? Make crawling slower and more expensive. Shitty crawlers not being able to run the PoW efficiently or at all is just a plus. Although: > which is trivial for them, as the post explains Sadly the site's being hugged to death right now so I can't really tell if I'm missing part of your argument here. > figure out that they can simply remove "Mozilla" from their user-agent And flag themselves in the logs to get separately blocked or rate limited. Servers win if malicious bots identify themselves again, and forcing them to change the user agent does that.
- shkkmo 1y agoThe explanation of how the estimate is made is more detailed, but here is the referenced conclusion: >> So (11508 websites * 2^16 sha256 operations) / 2^21, that’s about 6 minutes to mine enough tokens for every single Anubis deployment in the world. That means the cost of unrestricted crawler access to the internet for a week is approximately $0. >> In fact, I don’t think we reach a single cent per month in compute costs until several million sites have deployed Anubis.
- debugnik 1y agoThat's a matter of increasing the difficulty isn't it? And if the added cost is really negligible, we can just switch to a "refresh" challenge for the same added latency and without burning energy for no reason.
- deleted 1y ago[deleted]
- unclad5968 1y agoIm not on Firefox or any Firefox derivative and I still get anime cat girls making sure I'm not a bot.
- nemomarx 1y agoMozilla is used in the user agent string of all major browsers for historical reasons, but not necessarily headless ones or so on.
- unclad5968 1y agoOh that's interesting, I had no idea.
- seabrookmx 1y agoThere's some sites[1] that can print your user agent for you. Try it in a few different browsers and you will be surprised. They're honestly unhinged.. I have no idea why we still use this header in 2025! [1]: https://dnschecker.org/user-agent-info.php https://dnschecker.org/user-agent-info.php