10 ms·

A thought on JavaScript "proof of work" anti-scraper systems

194 points by zdw 1y ago

Animats 1y ago
"This page is taking too long to load" on sites with anti-scraper technology. Do Not Want.
- Unmixed0039 1y ago
  [flagged]
  - RamRodification 1y ago
    No, but it also doesn't matter whose fault it is. Having that happen will probably reduce your traffic. It might be worth the trade-off, and it might not. But if people go "Do Not Want!" it is worth paying attention to.
    - alpaca128 1y ago
      The one mentioned in the article was made because LLM scrapers DDOSed a person’s server. If it’s unusable for everyone, making everyone wait a bit to solve the issue is a clear improvement. > it also doesn’t matter whose fault it is People will always blame someone and have the uncanny talent to blame the wrong party. With cookie popups they blame the EU instead of the websites literally selling them out, here they’ll probably turn around and blame the website operators for defending themselves. For corporations running not just the scrapers but also the social networks it’s easy to algorithmically control the narrative.
      - RamRodification 1y ago
        > If it’s unusable for everyone, making everyone wait a bit to solve the issue is a clear improvement. I agree. > People will always blame someone I agree. That's my point.
      - ReptileMan 1y ago
        >With cookie popups they blame the EU instead of the websites literally selling them out EU created the flawed legislation, EU carved the loopholes, EU implementation and enforcement sucked, sites took the most obvious and predictable road. So who is to blame for the worse experience?
    - Disposal8433 1y ago
      > will probably reduce your traffic That's the point. Some web sites are DDOSed by scrapers and they obviously don't want such traffic.
      - RamRodification 1y ago
        I meant 'real' traffic. Maybe that wasn't obvious.
- jchw 1y ago
  In general if you're coming from a "clean" IP your difficulty won't be very high. In the future if these systems coordinate with each other in some way (DHT?) then it should make it possible to drop the baseline difficulty even further.
  - berkes 1y ago
    That's a perfect tool for monopolists to widen their moat even more. In line with how email is technically still federated and distributed, but practically oligopolized by a handfull of big-tech, through "fighting spam".
  - account42 1y ago
    > In general if you're coming from a "clean" IP your difficulty won't be very high. Unless you are using an operating system or browser that isn't the monopoly choice. Fuck off with this idea that some clients are better than others.
    - jchw 1y ago
      That has absolutely fuck all to do with IP reputation, you're mixing up unrelated concepts.
jchw 1y ago
[flagged]
- OutOfHere 1y ago
  Zip bombs are malicious. Captcha is annoying to humans. System or user attestation are pure evil. Proof-of-work has none of these problems.
  - jchw 1y ago
    Zip bombs eat some RAM. I have zero problems serving zip bombs to clients that explicitly disregard "do not enter" signs (and more specifically, actually enter them on purpose.) (Also: it's essentially a honeypot setup, which is nothing terribly novel in concept; people do it for SSH and other things.)
    - jchw 1y ago
      Sigh. I sometimes get the feeling that HN moderation can no longer keep up with actually reviewing the flags.
dannyw 1y ago
This is a poor take. All the major LLM scrapers already run and execute JavaScript, Googlebot has been doing it for probably a decade. Simple limits on runtime atop crypto mining from being too big of a problem.
- TZubiri 1y ago
  "Googlebot has been doing it for probably a decade." This is why Google developed a browser, turns out scraping the web requires one to pretty much develop a V8 engine, so why not publish it as a browser .
  - mschuster91 1y ago
    ... and the fact that even with a browser, content gated behind Macromedia Flash or ActiveX applets was / is not indexable is why Google pushed so hard to expand HTML5 capabilities.
    - chrisco255 1y ago
      Was it really a success though in that regard? HTML5 was great and all, but it never did replace Flash. Websites mainly just became more static. I suspect the lack of mobile integration had more to do with Flash dying than HTML5 getting better. It's a shame in some sense, because Flash was a lot of fun.
  - motoxpro 1y ago
    This is so obvious when you say it, but what an awesome insight.
    - rob_c 1y ago
      You don't work anywhere near the as industry then, people have been grumbling about this for the whole 10 years now
    - nssnsjsjsjs 1y ago
      Except it doesn't make sense. Why not just use Firefox. Or improve the JS engine of Firefox. I reckon they made the browser to control the browser market.
keepamovin 1y ago
Interesting - we dealt with this issue in CloudTabs, a SaaS for BrowserBox remote browser. The way we handle it, is simply to monitor resource usage with a Python script, issue a warning to the user when their tab or all processes are running hot, then when the rules are triggered we just kill the offending processes (that use too much CPU or RAM). Chrome has the nice property that you can kill a render process for a tab and often it just takes that tab down, leaving everything else running fine. This plus warning provides minimal user impact while ensuring resources for all. In the past we experimented with cgroups (both versions) and other mechanisms or limiting but found dynamic monitoring to be the most reliable.
OutOfHere 1y ago
See also http://www.hashcash.org/ http://www.hashcash.org/ which is a famous proof-of-work algorithm. The bigger benefit of proof-of-work is not that it's anti-LLM; it is that it's statelessly anti-DoS. I have been developing a public service, and I intend to use a simple implementation of proof-of-work in it, made to work with a single call without needing back-and-forth information from the server for each request.
- berkes 1y ago
  I've done that as well. The PoC worked, but the statelessness did prove a hurdle. It enforces a pattern in which a client must do the PoW every request. Other difficulties, uncoverd in our PoC were: Not all clients are equal: this punishes an old mobile phone or raspberry-pi much more than a client that runs on a beefy server with GPUs or clients that run on compromised hardware. - I.e. real users are likely punished, while illegitimate users often punished the least. Not all endpoints are equal: We experimented with higher difficulties for e.g. POST/PUT/PATCH/DELETE over GET. and with different difficulties for different endpoints: attempting to match how expensive a call would be for us. That requires back-and-forth to exchange difficulties. It discourages proper HATEOAS or REST, where a client browses through the API by following links and encourages calls that "just include as much as possible in one query". Deminishing our ability to cache, to be flexible and to leverage good HTTP practices.
TZubiri 1y ago
" and LLM scrapers may well have lots of CPU time available through means such as compromised machines." It's not clear whether the author means LLM scrapers in the sense of scrapers that gather training data for Foundational Models, LLM scrapers that browse the web to provide up to date answers, or vibe coders and agents that use browsers at the bequest of the programmer or the user. But in none of those myriad of cases can I imagine compromised machines being relevant. If we are talking about compromised machines, it's irrelevant if an LLM is involved and how, it's a distributed attack completely unrelated to LLMs.
- immibis 1y ago
  You can buy access to proxies in residential networks - with credit card on the open internet - and they may or may not be someone's botnet (probably not, but you don't know that). I'm not aware of anyone selling running code on the device though. It's just an HTTP or SOCKS5 level proxy.
deleted 1y ago
[deleted]
sznio 1y ago
I'd really like this, since it wouldn't impact my scraping stuff. I like to scrape websites and make alternative, personalized frontends for them. Captchas are really painful for me. Proof of work would be painful for a massive scraping operation, but I wouldn't have an issue with spending some CPU time to get the latest posts from a site which doesn't have an RSS feed or an API.
- diggan 1y ago
  Yeah, to me PoW makes a lot of sense in this way too. Captchas are hard for (some) people to solve, and very annoying to fill out, but easy for vision-enabled LLMs to solve (or even use 3rd party services where you pay for N/solves, available for every major captcha service). PoW instead are hard to deal with in a distributed/spammy design, but very easy for any user to just sit and wait a second or two. And all personal scraping tooling just keeps working, just slightly slower. Sounds like an OK solution to a shitty problem that has a bunch of other shitty solutions.
ChocolateGod 1y ago
I'm glad after spending all this time trying to increase power efficiency people have come up with JavaScript that serves no purpose other than to increase power draw. I feel sorry for people with budget phones who now have to battle with these PoW systems and think LLM scrapers will win this one, with everyone else suffering a worse browsing experience.
- jeroenhd 1y ago
  This is rather unfortunate, but the way Anubis works, you will only get the PoW test once. Scrapers, on the other hand, keep throwing out their session cookies (because you could easily limit their access by using cookies if they didn't). They will need to run the PoW workload every page load. If you're visiting loads of different websites, that does suck, but most people won't be affected all that much in practice. There are alternatives, of course. Several attempts at standardising remote attestation have been done. Apple has included remote attestation into Safari years ago. Basically, Apple/Google/Cloudflare give each user a limited set of "tokens" after verifying that they're a real person on real hardware (using TPMs and whatnot), and you exchange those tokens for website visits. Every user gets a load of usable tokens, but bots quickly run out and get denied access. For this approach to work, that means locking out Linux users, people with secure boot disabled, and things like outdated or rooted phones, but in return you don't get PoW walls or Cloudflare CAPTCHAs. In the end, LLM scrapers are why we can't have nice things. The web will only get worse now that these bots are on the loose.
  - ChocolateGod 1y ago
    > Scrapers, on the other hand, keep throwing out their session cookies This isn't very difficult to change. > but the way Anubis works, you will only get the PoW test once. Not if it's on multiple sites, I see the weab girl picture (why?) so much it's embedded into my brain at this point.
    - viraptor 1y ago
      > (why?) So you can pay the developers for the professional version where you can easily change the image. It's a great way of funding the work.
wordofx 1y ago
[flagged]
- seszett 1y ago
  This is about the scraping tools that are used to train LLMs, not the scrapers used by live LLMs.
  - sltkr 1y ago
    Even the ones used live, why couldn't/wouldn't they run some type of headless browser with Javascript enabled, rather than just making an HTTP-request and looking at the HTML code?
    - seszett 1y ago
      That's the point, the scraping that happens live can run through a full browser, performance is not an important issue there The scraping that's taking place to train the models is in a completely different scale and (probably) cannot really bear the added cost of PoW on each page they scrape. That's the scraping that is targeted by these tools.
- darkhelmet 1y ago
  You're missing the point. The approach is block anything that doesn't execute the javascript. The javascript generates a key that allows your queries to work. No javascript => no key => no content for you to scrape. Its a nuclear option. Nobody wins. The site operator can adjust the difficulty as needed to keep the abusive scrapers at bay.
- pjc50 1y ago
  Even before anti-scraping, lots of SPA sites won't give you any content if you don't run Javascript.
- wordofx 1y ago
  [flagged]
- 1y ago
h1fra 1y ago
Most scrapers are not able to monitor each websites performance, what will happen is that it's going to be slower for site to respond and that's it
matt3210 1y ago
The issue isn’t the resource usage as much as the content they’re stealing for reproduction purposes.
- berkes 1y ago
  That's not true for the vast amount of creative-commons, open-source and other permissive licenced content. (Aside: the licenses and distribution advocated by many of the same demography (information wants to be free -folks, jstor protestors, GPL-zealots) that now opposes LLMs using that content. )
  - jsheard 1y ago
    > GPL-zealots I'm sure GPL zealots would be happier about this situation if LLM vendors abided by the spirit of the license by releasing their models under GPL after ingesting GPL data, but we all know that isn't happening.
- reginald78 1y ago
  That is one part, but they are so voracious and aggressive that they are starting to crush hosts of that content and cause things become less open. In a way, they are not only 'stealing' it for themselves but they're also erasing it for humans.
DaSHacka 1y ago
Surprised there hasn't been a fork of Anubis that changes the artificial PoW into a simple Monero mining PoW yet. Would be hilarious to trap scraper bots into endless labyrinths of LLM-generated mediawiki pages, getting them to mine hashes with each progressive article. At least then we would be making money off these rude bots.
- g-b-r 1y ago
  It would be a much bigger incentive to add them with little care for the innocents impacted. Although admittedly millions of sites already ruined themselves with cloudflare without that incentive
- xnorswap 1y ago
  The bots could check if they've hit the jackpot themselves and keep the valid hashes for themselves and only return when they're worthless. Then it's the bots who are making money from work they need to do for the captchas.
  - forty 1y ago
    We need an oblivious crypto currency mining algorithm ^^
  - nssnsjsjsjs 1y ago
    1. The problem is the bot needs to understand the program it is running to do that. Akin to the halting problem. 2. There is no money in mining on the kinda hardware scrapers will run on. Power costs more that they'd earn.
    - immibis 1y ago
      Realistically, the bot owner could notice you're running MoneroAnubis and then would specifically check for MoneroAnubis, for example with a file hash, or a comment saying /* MoneroAnubis 1.0 copyright blah blah GPL license blah */. The bot wouldn't be expected to somehow determine this by itself automatically. Also, the ideal Monero miner is a power-efficient CPU (so probably in-order). There are no Monero ASICs by design.
bmacho 1y ago
Another idea: if your content is ~5kb text, then serve it to whoever asks it. If you don't have the bandwidth, try to make it smaller, static, and put it on the edge, some other people's computers.
- foul 1y ago
  The fun and terrible thing about the web is that the "rockstar in temporary distress" trope can be true and can happen when you expect it the least, like, you know, when you receive a HN kiss of death. You can surely expect that a static content will be static and will not run jpegoptim on a image at any given hit (a dynamic CMS + a sudden visit from ByteDance = your server is DDoSed), but you can't expect that any idiot/idiot hive on this planet will set up a multi-country edge caching servers architecture for a small sized website just in case some blog post will hit a few million visits every ten minutes. That can easily take down a server even for static content. I concur that Anubis is a poor solution, and yet here we are, the UN it's using it to weather down requests.
  - account42 1y ago
    Popularity-based traffic spikes tend to be very temporary and should not be something smaller sites concern themselves with.
    - foul 1y ago
      My example was quite vapid, but you shouldn't concentrate on that use-case. Small doesn't always mean "neglectable infos", while scrapers are always stealing CPU time.
- account42 1y ago
  Exactly, if you have the bandwidth to serve your proof of work scripts and check the results then you also have the bandwidth to simply serve properly optimized content.
forty 1y ago
Can we have proof of work algorithm that compute something actually useful? Like finding large prime numbers or something like this that have distributed computation programs. This way all this power wasted is at least not completely lost.
- g-b-r 1y ago
  Unfortunately useful things usually require much more computation to find a useful result, they require distributing the search, and so can't reliably verify that you performed the work (most searches will not find anything, and you can just pretend to not have found anything without doing any work). If a service had enough concurrent clients to reliably hit useful results quickly, you could verify that most did the work by checking if a hit was found, and either let everyone in or block everyone according to that; but then you're relying on the large majority being honest for your service to work at all, and some dishonest clients would still slip through.
- xena 1y ago
  Anubis author here. I looked into protein folding. The problem is that protein folding requires scientific data, which can easily get into the gigabyte range. That is more data than I want to serve to clients. Unless there's a way to solve the big data problem, a lot of "compute for good" schemes are frankly unworkable.
  - mvid 1y ago
    Zero knowledge proofs are basically arbitrary proof of work models. There is some interesting work being done with MPC and ZK proving, so only a small part needs to live on the client. I wonder if this would make it feasible again
rob_c 1y ago
Seen this already, back in the day the were sites which ran bitcoin hashing on their userbase and there was uproar. If someone dusted of the same tools and managed to get Altmann to buy them a nice car from it, good on them:)
vanschelven 1y ago
TBH most of the talk of "aggressive scraping" has been in the 100K pages/day range (which is ~1 page/s, i.e. "neglegtable). In my mind cloud providers ridiculous egres rates are more to blame here.
- jeroenhd 1y ago
  I've caught Huawei and Tencent IPs scraping the same image over and over again, with different query parameters. Sure, the image was only 260KiB and I don't use Amazon or GCP or Azure so it didn't cost me anything, but it still spammed my logs and caused a constant drain on my servers' resources. The bots keep coming back too, ignoring HTTP status codes, permanent redirects, and what else I can think of to tell them to fuck off. Robots.txt obviously doesn't help. Filtering traffic from data centers didn't help either, because soon after I did that, residential IPs started doing the same thing. I don't know if this is a Chinese ISP abusing their IP ranges or if China just has a massive botnet problem, but either way the traditional ways to get rid of these bots hasn't helped. In the end, I'm now blocking all of China and Singapore. That stops the endless flow of bullshit requests for now, though I see some familiar user agents appearing in other east Asian countries as well.
  - account42 1y ago
    So make sure the image is only available at one canonical URL with proper caching headers? No, obviously the only solution is to install crapware that worsens the experience for regular users.
    - thunderfork 1y ago
      [dead]
- account42 1y ago
  Agreed. Website operators should have a hard look at why their unoptimized crap can't manage such low request rates before contributing to the enshittification of the web by deploying crapware like anubis or buttflare.
bob1029 1y ago
I think this is not a battle that can be won in this way. Scraping content for an LLM is not a hyper time sensitive thing. You don't need to scrape every page every day. Sam Altman does not need a synchronous replica of the internet to achieve his goals.
- CGamesPlay 1y ago
  That is one view of the problem, but the one people are fixing with proof of work systems is the (unintentional) DDoS that LLM scrapers are operating against these sites. Just reducing the amount of traffic to manageable levels lets me get back to the work of doing whatever my site is supposed to be doing. I personally don't care if Sam Altman has a copy of my git server's rendition of the blame of every commit in my open source repo, because he could have just cloned my git repo and gotten the same result.
  - bob1029 1y ago
    I'm a bit confused. Is anyone's website currently being DDOS'd by scrapers for LLMs or is this a hypothetical? If you can't handle the traffic of one request per page per week or month, I think there are bigger problems to solve.
    - heinrich5991 1y ago
      > I'm a bit confused. Is anyone's website currently being DDOS'd by scrapers for LLMs or is this a hypothetical? Yes, there are sites being DDoSed by scrapers for LLMs. > If you can't handle the traffic of one request per page per week or month, I think there are bigger problems to solve. This isn't about one request per week or per month. There were reports from many sites that they're being hit by scrapers that request from many different IP addresses, one request each.
    - 2000UltraDeluxe 1y ago
      25k+ hits/minute here. And that's just the scrapers that doesn't just identify themselves as a browsers. Not sure why you believe massive repeated scraping isn't a problem. It's not like there is just one single actor out there, and ignoring robits.txt seems to be the norm nowadays.
nssnsjsjsjs 1y ago
My violin is microscopic for this problem. It's actually given me ideas!
persnickety 1y ago
> An LLM scraper is operating in a hostile environment [...] because you can't particularly tell a JavaScript proof of work system from JavaScript that does other things. [..] for people who would like to exploit your scraper's CPU to do some cryptocurrency mining, or [...] want to waste as much of your CPU as possible). That's a valid reason to serve JS-based PoW systems scares LLM operators: there's a chance the code might actually be malicious. That's not a valid reason to serve JS-based PoW systems to human users: the entire reason those proofs work against LLMs is the threat that the code is malicious. In other words, PoW works against LLM scrapers not because of PoW, but because they could contain malicious code. Why would you threaten your users with that? And if you can apply the threat only to LLMs, then why don't you cut the PoW garbage start with that instead? I know, it's because it's not so easy. So instead of wielding the Damocles sword of malware, why not standardize on some PoW algorithm that people can honestly apply without the risks?
- pjc50 1y ago
  I don't think this is "malicious" so much as it is "expensive" (in CPU cycles), which is already a problem for ad-heavy sites.
- berkes 1y ago
  > Why would you threaten your users with that? Your users - we, browsing the web - are already threatened with this. Adding a PoW changes nothing here. My browser already has several layers of protection in place. My browser even allows me to improve this protection with addons (ublock etc) and my OSes add even more protection to this. This is enough to allow PoW-thats-legit but block malicious code.
  - account42 1y ago
    Not safety-conscious users who disable javascript.
    - berkes 1y ago
      Those aren't threatened by PoW or malicious versions thereof either.
hardwaresofton 1y ago
Fantastic work by Xe here -- not the first but this seems like the most traction I've seen on a PoW anti-scraper project (with an MIT license to boot!). PoW anti-scraper tools are a good first step, but why don't we just jump straight to the endgame? We're drawing closer to a point where information's value is actually fully realized -- people will stop sharing knowledge for free. It doesn't have to be that way, but it does in a world where people are pressed for economic means, knowledge becomes an obvious thing to convert to capital and attempt to extract rent on. The simple way this happens is just a login wall -- for every website. It doesn't have to be a paid login wall of course (at first), but it's a super simple way to both legally and practically protect from scrapers. I think high quality knowledge, source code (which is basically executable knowledge), being open in general is a miracle/luxury of functioning, economically balanced societies where people feel driven (for many possible reasons) to give back, or have time to think of more than surviving. Don't get me wrong -- the doomer angle is almost always wrong -- every year humanity is almost always better off than we were the previous year on many important metrics, but it's getting harder to see a world where we cartwheel through another technological transformation that this time could possibly impact large percentages of the working population.
- g-b-r 1y ago
  Yes, let's turn the whole web into Facebook, what a bright future
  - Schiendelman 1y ago
    I think we only have two choices here: 1) every webpage requires Facebook login, and then Facebook offers free hosting for the content. 2) every webpage requires some other method of login, but not locked into a single system. I read the GP comment as suggesting we push on the second option there rather than passively waiting for the first option.
    - g-b-r 1y ago
      I hope you see that you shatter anonymity and the open web with that, single system or not
captainmuon 1y ago
As somebody who does some scraping / crawling for legitimate uses, I'm really unhappy with this development. I understand people have valid cases why they don't want their content scraped. Maybe they want to sell it - I can understand that, although I don't like it. Maybe they are opposed to it for fundamental reasons. I for one would like my content to be spread maximally. I want my arguments to be incorporated into AIs, so I can reach more people. But of course that is just me when I'd write certain content, others have different goals. It gets annoying when you have the right to scrape something - either because the owner of the data gave you the OK or because it is openly licensed. But then the webmaster can't be bothered to relax the rate limiter for you, and nobody can give you a nice API. Now people are putting their Open Educational Resources, their open source software, even their freaking essays about openness that they want the world to read behind Anubis. It makes me shake my head. I understand perfectly it is annoying when badly written bots hammer your site. But maybe then HTTP and those bots are the problem. Maybe we should make it easier for site owners to push their content somewhere where we can scrape it easier?
- berkes 1y ago
  Sounds like something IPFS could be nice solution for.
- yladiz 1y ago
  > I understand people have valid cases why they don't want their content scraped. Maybe they want to sell it - I can understand that, although I don't like it. To be frank: it’s not your content, it’s theirs, and it doesn’t matter if you like it or not, they can decide what they want to do with it, you’re not entitled to it. Yes there are some cases that you personally have permission to scrape, or the license explicitly permits it, but this isn’t the norm. The bigger issue isn’t that people don’t want their content to be read it’s that they want it to be read and consumed by a human in most cases, and they want their server resources (network bandwidth, CPU, etc) to be used in a manageable way. If these bots were written to be respectful, then maybe we wouldn’t be in this situation. These bots poisoned the well, and they affect respectful bots because of their actions.
- Analemma_ 1y ago
berkes 1y ago
I doubt any "anti-scraper" system will actually work. But if one is found, it will pave the way for a very dangerous counter-attack: Browser vendors with need for data (i.e. Google) simply using the vast fleet of installed browsers to do this scraping for them. Chrome, Safari, Edge, sending the pages you visit to their data-centers.
- lionkor 1y ago
  This is why we need https://ladybird.org/ https://ladybird.org/
- reginald78 1y ago
  This feels like it already was half happening anyway so it isn't to big of a leap. I also think this is the endgame of things like Recall in windows. Steal the training data right off your PC, no need to wait for the sucker to upload it to the web first.
fewdaysto2025 1y ago
[dead]
dxuh 1y ago
I always thought that JavaScript cryptomining is a better alternative to ads for monetizing websites (as long as people don't depend on those websites and website owners don't take it too far). I'd much rather give you a second of my CPU instead of space in my brain. Why is this so frowned upon? And in the same way I thought Anubis should just mine crypto instead of wasting power.
- thedanbob 1y ago
  > Why is this so frowned upon? Maybe because while ad tech these days is no less shady than crypto mining, the concept of ads is something people understand. Most people don't really understand crypto so it gets lumped in with "hackers" and "viruses". Alternatively, for those who do understand ad tech and crypto, crypto mining still subjectively feels (to me at least) more like you're being stolen from than ads. Same with Anubis, wasting power on PoW "feels" more acceptable to me than mining crypto. One of those quirks of the human psyche I guess.
  - matheusmoreira 1y ago
    Running proof of work on user machines without their consent is theft of their computing and energy resources. Any site doing so for any purpose whatsoever is serving malware and should be treated as such. Advertising is theft of attention which is extremely limited in supply. I'd even say it's mind rape. They forcibly insert their brands and trademarks into our minds without our consent. They deliberately ignore and circumvent any and all attempts to resist. It's all "justified" though, business interests excuse everything.
    - x-complexity 1y ago
      > Advertising is theft of attention which is extremely limited in supply. I'd even say it's mind rape. They forcibly insert their brands and trademarks into our minds without our consent. They deliberately ignore and circumvent any and all attempts to resist. (1): Attention from any given person is fundamentally limited. Said attention has an inherent value. (2): Running *any* website costs money, doubly so for video playback. This is not even mentioning the moderation & copyright mechanisms that a video sharing platform like YouTube has to have in order to keep copyright lawsuits away from YouTube itself. (3): Products do not spawn in with their presence known to the general population. For the product to be successful, people have to know it exists in the first place. Advertising is the consequence of wanting attention to be drawn to (3), and willing to pay for said attention on a given platform (1). (2)'s costs, alongside any payouts to videographers that garner attention to their videos, can be paid for with the money in (1), by placing ads around/before the video itself. You're allowed to not have advertising shown to you, but in exchange, the money to pay for (2) & the people who made the video have to come from somewhere.
myself248 1y ago
If the proof-of-work system is actually a crypto miner, such that visitors end up paying the site for the content they host, have we finally converged on a working implementation of the micropayments-for-websites concepts of decades ago?
- diggan 1y ago
  > If the proof-of-work system is actually a crypto miner, such that visitors end up paying the site for the content they host Unsure how that would work. If the proof you generate could be used for blockchain operations, so that the website operator could be paid by using that proof as generated by the website visitor, why shouldn't the visitor keep that proof to themselves and use it instead? Then they'd get the full amount, and the website operator gets nothing. So then there is no point for it, and the visitor might as well just run a miner locally :)
  - Retr0id 1y ago
    If the user mined it themselves and then paid the site owner before accessing the site, they'd have to pay a transaction fee and wait for a high-latency transaction to commit. The transaction fee could dwarf the actual payment value. Mining on behalf of the site owner negates the need for a transaction entirely.
    - viraptor 1y ago
      (unnecessary)
      - Retr0id 1y ago
        I know this.
        viraptor 1y ago
        Responded to wrong comment, sorry
  - 1y ago
EGreg 1y ago
“ because you can't particularly tell a JavaScript proof of work system from JavaScript that does other things. Letting your scraper run JavaScript means that it can” LLMs can, LOL One of the powerful use cases is that they can catch pretty much EVERY attempt at obfuscation now. Building a marketplace and want to let the participants chat but not take the deal off-site? LLM in the loop. Protecting children? LLM in the loop.
fennecfoxy 1y ago
Hmmmm this seems like something that will be bad for the environment.
- reginald78 1y ago
  AI scrapping, bloated javascript pages that are mostly text, ads, existing captchas, etc also all waste energy for no gain to the end user and we mostly just accept it or maybe run an ad blocker. I see people complain PoW solutions make it hard for low power devices and are a waste of energy, which is true. But that is also true of the status quo which is also an annoying waste of human time and often a privacy nightmare.
avastel 1y ago
Reposting a similar point I made recently about CAPTCHA and scalpers, but it’s even more relevant for scrapers. PoW can help against basic scrapers or DDoS, but it won’t stop anyone serious. Last week I looked into a Binance CAPTCHA solver that didn’t use a browser at all, just a plain HTTP client. https://blog.castle.io/what-a-binance-captcha-solver-tells-us-about-todays-bot-threats/ https://blog.castle.io/what-a-binance-captcha-solver-tells-u... The attacker had fully reverse engineered the signal collection and solved-state flow, including obfuscated parts. They could forge all the expected telemetry. This kind of setup is pretty standard in bot-heavy environments like ticketing or sneaker drops. Scrapers often do the same to cut costs. CAPTCHA and PoW mostly become signal collection protocols, if those signals aren’t tightly coupled to the actual runtime, they get spoofed. And regarding PoW: if you try to make it slow enough to hurt bots, you also hurt users on low-end devices. Someone even ported PerimeterX’s PoW to CUDA to accelerate solving: https://github.com/re-jevi/PerimiterXCudaSolver/blob/main/powsolver.cu https://github.com/re-jevi/PerimiterXCudaSolver/blob/main/po...
benregenspan 1y ago
At a media company, our web performance monitoring tool started flagging long-running clientside XHR requests, which I couldn't reproduce in a real browser. It turned out that an analytics vendor was injecting a script which checked if it looked like the client was a bot. If so, they would then essentially use the client as a worker to perform their own third-party API requests (for data like social share counts). So there's definitely some prior art for this kind of thing.
- apitman 1y ago
  This is really interesting. One naive thought that immediately came to mind is that bots might be capable of making cross site requests. The logical conclusion of this entire arms race is that bots will eventually have no choice but to run actual browsers. Not sure that fact will appreciably reduce their scaping abilities though.
  - benregenspan 1y ago
    > The logical conclusion of this entire arms race is that bots will eventually have no choice but to run actual browsers I think this is almost already the case now. Services like Cloudflare do a pretty good job of classifying primitive bots and if site operators want to block all (or at least vast majority), they can. The only reliable way through is a real browser. (Which does put a floor on resource needs for scraping)
  - dragonwriter 1y ago
    > The logical conclusion of this entire arms race is that bots will eventually have no choice but to run actual browsers. I thought bots using (headless) browsers was an existing workaround for a number of existing issues with simpler bots, so this doesn't seem to be a big change.
akomtu 1y ago
That's DRM essentially: watch, but do not copy. Except that in this case the corporations do piracy.
jameslk 1y ago
What if we move some of the website backend workload to the bots, effectively using them as decentralized cloud infrastructure? web3, we are so back
kbenson 1y ago
With regard to proof of work systems that provide revenue: 1) Making LLM (and other) scrapers pay for the resources they use seems perfectly fine to me. Also, as someone that manages some level of scraping (on the order of low tens of millions of requests a month), I'm fine with this. There's a wide range of scraping that the problem is not some resource cost, but the other side not wanting to deal with setting up APIs or putting so many hurdles on access that it's easier to just bypass it. 2) This seems like it might be an opportunity for Cloudflare. Let customers opt-in to requiring a proof of work when visitors already trip the cloudflare vetting page that runs additional checks to see if you're a bad actor, and apply any revenue to a service credit towards their monthly fee (or if on a free plan, as credit to be used for trying out additional for-pay features). There might be a perverse inventive to toggle on more stringent checking from cloudflare, but ultimately since it's all being paid for that's the site owner's choice on how they want to manage their site.

194 points by zdw 1y ago

Animats 1y ago

"This page is taking too long to load" on sites with anti-scraper technology. Do Not Want.

Unmixed0039 1y ago
[flagged]
- RamRodification 1y ago
  No, but it also doesn't matter whose fault it is. Having that happen will probably reduce your traffic. It might be worth the trade-off, and it might not. But if people go "Do Not Want!" it is worth paying attention to.
  - alpaca128 1y ago
    The one mentioned in the article was made because LLM scrapers DDOSed a person’s server. If it’s unusable for everyone, making everyone wait a bit to solve the issue is a clear improvement. > it also doesn’t matter whose fault it is People will always blame someone and have the uncanny talent to blame the wrong party. With cookie popups they blame the EU instead of the websites literally selling them out, here they’ll probably turn around and blame the website operators for defending themselves. For corporations running not just the scrapers but also the social networks it’s easy to algorithmically control the narrative.
    - RamRodification 1y ago
      > If it’s unusable for everyone, making everyone wait a bit to solve the issue is a clear improvement. I agree. > People will always blame someone I agree. That's my point.
    - ReptileMan 1y ago
      >With cookie popups they blame the EU instead of the websites literally selling them out EU created the flawed legislation, EU carved the loopholes, EU implementation and enforcement sucked, sites took the most obvious and predictable road. So who is to blame for the worse experience?
  - Disposal8433 1y ago
    > will probably reduce your traffic That's the point. Some web sites are DDOSed by scrapers and they obviously don't want such traffic.
    - RamRodification 1y ago
      I meant 'real' traffic. Maybe that wasn't obvious.
jchw 1y ago
In general if you're coming from a "clean" IP your difficulty won't be very high. In the future if these systems coordinate with each other in some way (DHT?) then it should make it possible to drop the baseline difficulty even further.
- berkes 1y ago
  That's a perfect tool for monopolists to widen their moat even more. In line with how email is technically still federated and distributed, but practically oligopolized by a handfull of big-tech, through "fighting spam".
- account42 1y ago
  > In general if you're coming from a "clean" IP your difficulty won't be very high. Unless you are using an operating system or browser that isn't the monopoly choice. Fuck off with this idea that some clients are better than others.
  - jchw 1y ago
    That has absolutely fuck all to do with IP reputation, you're mixing up unrelated concepts.

jchw 1y ago

[flagged]

OutOfHere 1y ago
Zip bombs are malicious. Captcha is annoying to humans. System or user attestation are pure evil. Proof-of-work has none of these problems.
- jchw 1y ago
  Zip bombs eat some RAM. I have zero problems serving zip bombs to clients that explicitly disregard "do not enter" signs (and more specifically, actually enter them on purpose.) (Also: it's essentially a honeypot setup, which is nothing terribly novel in concept; people do it for SSH and other things.)
  - jchw 1y ago
    Sigh. I sometimes get the feeling that HN moderation can no longer keep up with actually reviewing the flags.

dannyw 1y ago

This is a poor take. All the major LLM scrapers already run and execute JavaScript, Googlebot has been doing it for probably a decade. Simple limits on runtime atop crypto mining from being too big of a problem.

TZubiri 1y ago
"Googlebot has been doing it for probably a decade." This is why Google developed a browser, turns out scraping the web requires one to pretty much develop a V8 engine, so why not publish it as a browser .
- mschuster91 1y ago
  ... and the fact that even with a browser, content gated behind Macromedia Flash or ActiveX applets was / is not indexable is why Google pushed so hard to expand HTML5 capabilities.
  - chrisco255 1y ago
    Was it really a success though in that regard? HTML5 was great and all, but it never did replace Flash. Websites mainly just became more static. I suspect the lack of mobile integration had more to do with Flash dying than HTML5 getting better. It's a shame in some sense, because Flash was a lot of fun.
- motoxpro 1y ago
  This is so obvious when you say it, but what an awesome insight.
  - rob_c 1y ago
    You don't work anywhere near the as industry then, people have been grumbling about this for the whole 10 years now
  - nssnsjsjsjs 1y ago
    Except it doesn't make sense. Why not just use Firefox. Or improve the JS engine of Firefox. I reckon they made the browser to control the browser market.

keepamovin 1y ago

Interesting - we dealt with this issue in CloudTabs, a SaaS for BrowserBox remote browser. The way we handle it, is simply to monitor resource usage with a Python script, issue a warning to the user when their tab or all processes are running hot, then when the rules are triggered we just kill the offending processes (that use too much CPU or RAM). Chrome has the nice property that you can kill a render process for a tab and often it just takes that tab down, leaving everything else running fine. This plus warning provides minimal user impact while ensuring resources for all. In the past we experimented with cgroups (both versions) and other mechanisms or limiting but found dynamic monitoring to be the most reliable.

OutOfHere 1y ago

See also http://www.hashcash.org/ http://www.hashcash.org/ which is a famous proof-of-work algorithm. The bigger benefit of proof-of-work is not that it's anti-LLM; it is that it's statelessly anti-DoS. I have been developing a public service, and I intend to use a simple implementation of proof-of-work in it, made to work with a single call without needing back-and-forth information from the server for each request.

berkes 1y ago
I've done that as well. The PoC worked, but the statelessness did prove a hurdle. It enforces a pattern in which a client must do the PoW every request. Other difficulties, uncoverd in our PoC were: Not all clients are equal: this punishes an old mobile phone or raspberry-pi much more than a client that runs on a beefy server with GPUs or clients that run on compromised hardware. - I.e. real users are likely punished, while illegitimate users often punished the least. Not all endpoints are equal: We experimented with higher difficulties for e.g. POST/PUT/PATCH/DELETE over GET. and with different difficulties for different endpoints: attempting to match how expensive a call would be for us. That requires back-and-forth to exchange difficulties. It discourages proper HATEOAS or REST, where a client browses through the API by following links and encourages calls that "just include as much as possible in one query". Deminishing our ability to cache, to be flexible and to leverage good HTTP practices.

TZubiri 1y ago

" and LLM scrapers may well have lots of CPU time available through means such as compromised machines." It's not clear whether the author means LLM scrapers in the sense of scrapers that gather training data for Foundational Models, LLM scrapers that browse the web to provide up to date answers, or vibe coders and agents that use browsers at the bequest of the programmer or the user. But in none of those myriad of cases can I imagine compromised machines being relevant. If we are talking about compromised machines, it's irrelevant if an LLM is involved and how, it's a distributed attack completely unrelated to LLMs.

immibis 1y ago
You can buy access to proxies in residential networks - with credit card on the open internet - and they may or may not be someone's botnet (probably not, but you don't know that). I'm not aware of anyone selling running code on the device though. It's just an HTTP or SOCKS5 level proxy.

deleted 1y ago

[deleted]

sznio 1y ago

I'd really like this, since it wouldn't impact my scraping stuff. I like to scrape websites and make alternative, personalized frontends for them. Captchas are really painful for me. Proof of work would be painful for a massive scraping operation, but I wouldn't have an issue with spending some CPU time to get the latest posts from a site which doesn't have an RSS feed or an API.

diggan 1y ago
Yeah, to me PoW makes a lot of sense in this way too. Captchas are hard for (some) people to solve, and very annoying to fill out, but easy for vision-enabled LLMs to solve (or even use 3rd party services where you pay for N/solves, available for every major captcha service). PoW instead are hard to deal with in a distributed/spammy design, but very easy for any user to just sit and wait a second or two. And all personal scraping tooling just keeps working, just slightly slower. Sounds like an OK solution to a shitty problem that has a bunch of other shitty solutions.

ChocolateGod 1y ago

I'm glad after spending all this time trying to increase power efficiency people have come up with JavaScript that serves no purpose other than to increase power draw. I feel sorry for people with budget phones who now have to battle with these PoW systems and think LLM scrapers will win this one, with everyone else suffering a worse browsing experience.

jeroenhd 1y ago
This is rather unfortunate, but the way Anubis works, you will only get the PoW test once. Scrapers, on the other hand, keep throwing out their session cookies (because you could easily limit their access by using cookies if they didn't). They will need to run the PoW workload every page load. If you're visiting loads of different websites, that does suck, but most people won't be affected all that much in practice. There are alternatives, of course. Several attempts at standardising remote attestation have been done. Apple has included remote attestation into Safari years ago. Basically, Apple/Google/Cloudflare give each user a limited set of "tokens" after verifying that they're a real person on real hardware (using TPMs and whatnot), and you exchange those tokens for website visits. Every user gets a load of usable tokens, but bots quickly run out and get denied access. For this approach to work, that means locking out Linux users, people with secure boot disabled, and things like outdated or rooted phones, but in return you don't get PoW walls or Cloudflare CAPTCHAs. In the end, LLM scrapers are why we can't have nice things. The web will only get worse now that these bots are on the loose.
- ChocolateGod 1y ago
  > Scrapers, on the other hand, keep throwing out their session cookies This isn't very difficult to change. > but the way Anubis works, you will only get the PoW test once. Not if it's on multiple sites, I see the weab girl picture (why?) so much it's embedded into my brain at this point.
  - viraptor 1y ago
    > (why?) So you can pay the developers for the professional version where you can easily change the image. It's a great way of funding the work.

wordofx 1y ago

[flagged]

seszett 1y ago
This is about the scraping tools that are used to train LLMs, not the scrapers used by live LLMs.
- sltkr 1y ago
  Even the ones used live, why couldn't/wouldn't they run some type of headless browser with Javascript enabled, rather than just making an HTTP-request and looking at the HTML code?
  - seszett 1y ago
    That's the point, the scraping that happens live can run through a full browser, performance is not an important issue there The scraping that's taking place to train the models is in a completely different scale and (probably) cannot really bear the added cost of PoW on each page they scrape. That's the scraping that is targeted by these tools.
darkhelmet 1y ago
You're missing the point. The approach is block anything that doesn't execute the javascript. The javascript generates a key that allows your queries to work. No javascript => no key => no content for you to scrape. Its a nuclear option. Nobody wins. The site operator can adjust the difficulty as needed to keep the abusive scrapers at bay.
pjc50 1y ago
Even before anti-scraping, lots of SPA sites won't give you any content if you don't run Javascript.
wordofx 1y ago
[flagged]
1y ago

h1fra 1y ago

Most scrapers are not able to monitor each websites performance, what will happen is that it's going to be slower for site to respond and that's it

matt3210 1y ago

The issue isn’t the resource usage as much as the content they’re stealing for reproduction purposes.

berkes 1y ago
That's not true for the vast amount of creative-commons, open-source and other permissive licenced content. (Aside: the licenses and distribution advocated by many of the same demography (information wants to be free -folks, jstor protestors, GPL-zealots) that now opposes LLMs using that content. )
- jsheard 1y ago
  > GPL-zealots I'm sure GPL zealots would be happier about this situation if LLM vendors abided by the spirit of the license by releasing their models under GPL after ingesting GPL data, but we all know that isn't happening.
reginald78 1y ago
That is one part, but they are so voracious and aggressive that they are starting to crush hosts of that content and cause things become less open. In a way, they are not only 'stealing' it for themselves but they're also erasing it for humans.

DaSHacka 1y ago

Surprised there hasn't been a fork of Anubis that changes the artificial PoW into a simple Monero mining PoW yet. Would be hilarious to trap scraper bots into endless labyrinths of LLM-generated mediawiki pages, getting them to mine hashes with each progressive article. At least then we would be making money off these rude bots.

g-b-r 1y ago
It would be a much bigger incentive to add them with little care for the innocents impacted. Although admittedly millions of sites already ruined themselves with cloudflare without that incentive
xnorswap 1y ago
The bots could check if they've hit the jackpot themselves and keep the valid hashes for themselves and only return when they're worthless. Then it's the bots who are making money from work they need to do for the captchas.
- forty 1y ago
  We need an oblivious crypto currency mining algorithm ^^
- nssnsjsjsjs 1y ago
  1. The problem is the bot needs to understand the program it is running to do that. Akin to the halting problem. 2. There is no money in mining on the kinda hardware scrapers will run on. Power costs more that they'd earn.
  - immibis 1y ago
    Realistically, the bot owner could notice you're running MoneroAnubis and then would specifically check for MoneroAnubis, for example with a file hash, or a comment saying /* MoneroAnubis 1.0 copyright blah blah GPL license blah */. The bot wouldn't be expected to somehow determine this by itself automatically. Also, the ideal Monero miner is a power-efficient CPU (so probably in-order). There are no Monero ASICs by design.

bmacho 1y ago

Another idea: if your content is ~5kb text, then serve it to whoever asks it. If you don't have the bandwidth, try to make it smaller, static, and put it on the edge, some other people's computers.

foul 1y ago
The fun and terrible thing about the web is that the "rockstar in temporary distress" trope can be true and can happen when you expect it the least, like, you know, when you receive a HN kiss of death. You can surely expect that a static content will be static and will not run jpegoptim on a image at any given hit (a dynamic CMS + a sudden visit from ByteDance = your server is DDoSed), but you can't expect that any idiot/idiot hive on this planet will set up a multi-country edge caching servers architecture for a small sized website just in case some blog post will hit a few million visits every ten minutes. That can easily take down a server even for static content. I concur that Anubis is a poor solution, and yet here we are, the UN it's using it to weather down requests.
- account42 1y ago
  Popularity-based traffic spikes tend to be very temporary and should not be something smaller sites concern themselves with.
  - foul 1y ago
    My example was quite vapid, but you shouldn't concentrate on that use-case. Small doesn't always mean "neglectable infos", while scrapers are always stealing CPU time.
account42 1y ago
Exactly, if you have the bandwidth to serve your proof of work scripts and check the results then you also have the bandwidth to simply serve properly optimized content.

forty 1y ago

Can we have proof of work algorithm that compute something actually useful? Like finding large prime numbers or something like this that have distributed computation programs. This way all this power wasted is at least not completely lost.

g-b-r 1y ago
Unfortunately useful things usually require much more computation to find a useful result, they require distributing the search, and so can't reliably verify that you performed the work (most searches will not find anything, and you can just pretend to not have found anything without doing any work). If a service had enough concurrent clients to reliably hit useful results quickly, you could verify that most did the work by checking if a hit was found, and either let everyone in or block everyone according to that; but then you're relying on the large majority being honest for your service to work at all, and some dishonest clients would still slip through.
xena 1y ago
Anubis author here. I looked into protein folding. The problem is that protein folding requires scientific data, which can easily get into the gigabyte range. That is more data than I want to serve to clients. Unless there's a way to solve the big data problem, a lot of "compute for good" schemes are frankly unworkable.
- mvid 1y ago
  Zero knowledge proofs are basically arbitrary proof of work models. There is some interesting work being done with MPC and ZK proving, so only a small part needs to live on the client. I wonder if this would make it feasible again

rob_c 1y ago

Seen this already, back in the day the were sites which ran bitcoin hashing on their userbase and there was uproar. If someone dusted of the same tools and managed to get Altmann to buy them a nice car from it, good on them:)

vanschelven 1y ago

TBH most of the talk of "aggressive scraping" has been in the 100K pages/day range (which is ~1 page/s, i.e. "neglegtable). In my mind cloud providers ridiculous egres rates are more to blame here.

jeroenhd 1y ago
I've caught Huawei and Tencent IPs scraping the same image over and over again, with different query parameters. Sure, the image was only 260KiB and I don't use Amazon or GCP or Azure so it didn't cost me anything, but it still spammed my logs and caused a constant drain on my servers' resources. The bots keep coming back too, ignoring HTTP status codes, permanent redirects, and what else I can think of to tell them to fuck off. Robots.txt obviously doesn't help. Filtering traffic from data centers didn't help either, because soon after I did that, residential IPs started doing the same thing. I don't know if this is a Chinese ISP abusing their IP ranges or if China just has a massive botnet problem, but either way the traditional ways to get rid of these bots hasn't helped. In the end, I'm now blocking all of China and Singapore. That stops the endless flow of bullshit requests for now, though I see some familiar user agents appearing in other east Asian countries as well.
- account42 1y ago
  So make sure the image is only available at one canonical URL with proper caching headers? No, obviously the only solution is to install crapware that worsens the experience for regular users.
  - thunderfork 1y ago
    [dead]
account42 1y ago
Agreed. Website operators should have a hard look at why their unoptimized crap can't manage such low request rates before contributing to the enshittification of the web by deploying crapware like anubis or buttflare.

bob1029 1y ago

I think this is not a battle that can be won in this way. Scraping content for an LLM is not a hyper time sensitive thing. You don't need to scrape every page every day. Sam Altman does not need a synchronous replica of the internet to achieve his goals.

CGamesPlay 1y ago
That is one view of the problem, but the one people are fixing with proof of work systems is the (unintentional) DDoS that LLM scrapers are operating against these sites. Just reducing the amount of traffic to manageable levels lets me get back to the work of doing whatever my site is supposed to be doing. I personally don't care if Sam Altman has a copy of my git server's rendition of the blame of every commit in my open source repo, because he could have just cloned my git repo and gotten the same result.
- bob1029 1y ago
  I'm a bit confused. Is anyone's website currently being DDOS'd by scrapers for LLMs or is this a hypothetical? If you can't handle the traffic of one request per page per week or month, I think there are bigger problems to solve.
  - heinrich5991 1y ago
    > I'm a bit confused. Is anyone's website currently being DDOS'd by scrapers for LLMs or is this a hypothetical? Yes, there are sites being DDoSed by scrapers for LLMs. > If you can't handle the traffic of one request per page per week or month, I think there are bigger problems to solve. This isn't about one request per week or per month. There were reports from many sites that they're being hit by scrapers that request from many different IP addresses, one request each.
  - 2000UltraDeluxe 1y ago
    25k+ hits/minute here. And that's just the scrapers that doesn't just identify themselves as a browsers. Not sure why you believe massive repeated scraping isn't a problem. It's not like there is just one single actor out there, and ignoring robits.txt seems to be the norm nowadays.

nssnsjsjsjs 1y ago

My violin is microscopic for this problem. It's actually given me ideas!

persnickety 1y ago

> An LLM scraper is operating in a hostile environment [...] because you can't particularly tell a JavaScript proof of work system from JavaScript that does other things. [..] for people who would like to exploit your scraper's CPU to do some cryptocurrency mining, or [...] want to waste as much of your CPU as possible). That's a valid reason to serve JS-based PoW systems scares LLM operators: there's a chance the code might actually be malicious. That's not a valid reason to serve JS-based PoW systems to human users: the entire reason those proofs work against LLMs is the threat that the code is malicious. In other words, PoW works against LLM scrapers not because of PoW, but because they could contain malicious code. Why would you threaten your users with that? And if you can apply the threat only to LLMs, then why don't you cut the PoW garbage start with that instead? I know, it's because it's not so easy. So instead of wielding the Damocles sword of malware, why not standardize on some PoW algorithm that people can honestly apply without the risks?

pjc50 1y ago
I don't think this is "malicious" so much as it is "expensive" (in CPU cycles), which is already a problem for ad-heavy sites.
berkes 1y ago
> Why would you threaten your users with that? Your users - we, browsing the web - are already threatened with this. Adding a PoW changes nothing here. My browser already has several layers of protection in place. My browser even allows me to improve this protection with addons (ublock etc) and my OSes add even more protection to this. This is enough to allow PoW-thats-legit but block malicious code.
- account42 1y ago
  Not safety-conscious users who disable javascript.
  - berkes 1y ago
    Those aren't threatened by PoW or malicious versions thereof either.

hardwaresofton 1y ago

Fantastic work by Xe here -- not the first but this seems like the most traction I've seen on a PoW anti-scraper project (with an MIT license to boot!). PoW anti-scraper tools are a good first step, but why don't we just jump straight to the endgame? We're drawing closer to a point where information's value is actually fully realized -- people will stop sharing knowledge for free. It doesn't have to be that way, but it does in a world where people are pressed for economic means, knowledge becomes an obvious thing to convert to capital and attempt to extract rent on. The simple way this happens is just a login wall -- for every website. It doesn't have to be a paid login wall of course (at first), but it's a super simple way to both legally and practically protect from scrapers. I think high quality knowledge, source code (which is basically executable knowledge), being open in general is a miracle/luxury of functioning, economically balanced societies where people feel driven (for many possible reasons) to give back, or have time to think of more than surviving. Don't get me wrong -- the doomer angle is almost always wrong -- every year humanity is almost always better off than we were the previous year on many important metrics, but it's getting harder to see a world where we cartwheel through another technological transformation that this time could possibly impact large percentages of the working population.

g-b-r 1y ago
Yes, let's turn the whole web into Facebook, what a bright future
- Schiendelman 1y ago
  I think we only have two choices here: 1) every webpage requires Facebook login, and then Facebook offers free hosting for the content. 2) every webpage requires some other method of login, but not locked into a single system. I read the GP comment as suggesting we push on the second option there rather than passively waiting for the first option.
  - g-b-r 1y ago
    I hope you see that you shatter anonymity and the open web with that, single system or not

captainmuon 1y ago

As somebody who does some scraping / crawling for legitimate uses, I'm really unhappy with this development. I understand people have valid cases why they don't want their content scraped. Maybe they want to sell it - I can understand that, although I don't like it. Maybe they are opposed to it for fundamental reasons. I for one would like my content to be spread maximally. I want my arguments to be incorporated into AIs, so I can reach more people. But of course that is just me when I'd write certain content, others have different goals. It gets annoying when you have the right to scrape something - either because the owner of the data gave you the OK or because it is openly licensed. But then the webmaster can't be bothered to relax the rate limiter for you, and nobody can give you a nice API. Now people are putting their Open Educational Resources, their open source software, even their freaking essays about openness that they want the world to read behind Anubis. It makes me shake my head. I understand perfectly it is annoying when badly written bots hammer your site. But maybe then HTTP and those bots are the problem. Maybe we should make it easier for site owners to push their content somewhere where we can scrape it easier?

berkes 1y ago
Sounds like something IPFS could be nice solution for.
yladiz 1y ago
> I understand people have valid cases why they don't want their content scraped. Maybe they want to sell it - I can understand that, although I don't like it. To be frank: it’s not your content, it’s theirs, and it doesn’t matter if you like it or not, they can decide what they want to do with it, you’re not entitled to it. Yes there are some cases that you personally have permission to scrape, or the license explicitly permits it, but this isn’t the norm. The bigger issue isn’t that people don’t want their content to be read it’s that they want it to be read and consumed by a human in most cases, and they want their server resources (network bandwidth, CPU, etc) to be used in a manageable way. If these bots were written to be respectful, then maybe we wouldn’t be in this situation. These bots poisoned the well, and they affect respectful bots because of their actions.
Analemma_ 1y ago

berkes 1y ago

I doubt any "anti-scraper" system will actually work. But if one is found, it will pave the way for a very dangerous counter-attack: Browser vendors with need for data (i.e. Google) simply using the vast fleet of installed browsers to do this scraping for them. Chrome, Safari, Edge, sending the pages you visit to their data-centers.

lionkor 1y ago
This is why we need https://ladybird.org/ https://ladybird.org/
reginald78 1y ago
This feels like it already was half happening anyway so it isn't to big of a leap. I also think this is the endgame of things like Recall in windows. Steal the training data right off your PC, no need to wait for the sucker to upload it to the web first.

fewdaysto2025 1y ago

[dead]

dxuh 1y ago

I always thought that JavaScript cryptomining is a better alternative to ads for monetizing websites (as long as people don't depend on those websites and website owners don't take it too far). I'd much rather give you a second of my CPU instead of space in my brain. Why is this so frowned upon? And in the same way I thought Anubis should just mine crypto instead of wasting power.

thedanbob 1y ago
> Why is this so frowned upon? Maybe because while ad tech these days is no less shady than crypto mining, the concept of ads is something people understand. Most people don't really understand crypto so it gets lumped in with "hackers" and "viruses". Alternatively, for those who do understand ad tech and crypto, crypto mining still subjectively feels (to me at least) more like you're being stolen from than ads. Same with Anubis, wasting power on PoW "feels" more acceptable to me than mining crypto. One of those quirks of the human psyche I guess.
- matheusmoreira 1y ago
  Running proof of work on user machines without their consent is theft of their computing and energy resources. Any site doing so for any purpose whatsoever is serving malware and should be treated as such. Advertising is theft of attention which is extremely limited in supply. I'd even say it's mind rape. They forcibly insert their brands and trademarks into our minds without our consent. They deliberately ignore and circumvent any and all attempts to resist. It's all "justified" though, business interests excuse everything.
  - x-complexity 1y ago
    > Advertising is theft of attention which is extremely limited in supply. I'd even say it's mind rape. They forcibly insert their brands and trademarks into our minds without our consent. They deliberately ignore and circumvent any and all attempts to resist. (1): Attention from any given person is fundamentally limited. Said attention has an inherent value. (2): Running *any* website costs money, doubly so for video playback. This is not even mentioning the moderation & copyright mechanisms that a video sharing platform like YouTube has to have in order to keep copyright lawsuits away from YouTube itself. (3): Products do not spawn in with their presence known to the general population. For the product to be successful, people have to know it exists in the first place. Advertising is the consequence of wanting attention to be drawn to (3), and willing to pay for said attention on a given platform (1). (2)'s costs, alongside any payouts to videographers that garner attention to their videos, can be paid for with the money in (1), by placing ads around/before the video itself. You're allowed to not have advertising shown to you, but in exchange, the money to pay for (2) & the people who made the video have to come from somewhere.

myself248 1y ago

If the proof-of-work system is actually a crypto miner, such that visitors end up paying the site for the content they host, have we finally converged on a working implementation of the micropayments-for-websites concepts of decades ago?

diggan 1y ago
> If the proof-of-work system is actually a crypto miner, such that visitors end up paying the site for the content they host Unsure how that would work. If the proof you generate could be used for blockchain operations, so that the website operator could be paid by using that proof as generated by the website visitor, why shouldn't the visitor keep that proof to themselves and use it instead? Then they'd get the full amount, and the website operator gets nothing. So then there is no point for it, and the visitor might as well just run a miner locally :)
- Retr0id 1y ago
  If the user mined it themselves and then paid the site owner before accessing the site, they'd have to pay a transaction fee and wait for a high-latency transaction to commit. The transaction fee could dwarf the actual payment value. Mining on behalf of the site owner negates the need for a transaction entirely.
  - viraptor 1y ago
    (unnecessary)
    - Retr0id 1y ago
      I know this.
      - viraptor 1y ago
        Responded to wrong comment, sorry
- 1y ago

EGreg 1y ago

“ because you can't particularly tell a JavaScript proof of work system from JavaScript that does other things. Letting your scraper run JavaScript means that it can” LLMs can, LOL One of the powerful use cases is that they can catch pretty much EVERY attempt at obfuscation now. Building a marketplace and want to let the participants chat but not take the deal off-site? LLM in the loop. Protecting children? LLM in the loop.

fennecfoxy 1y ago

Hmmmm this seems like something that will be bad for the environment.

reginald78 1y ago
AI scrapping, bloated javascript pages that are mostly text, ads, existing captchas, etc also all waste energy for no gain to the end user and we mostly just accept it or maybe run an ad blocker. I see people complain PoW solutions make it hard for low power devices and are a waste of energy, which is true. But that is also true of the status quo which is also an annoying waste of human time and often a privacy nightmare.

avastel 1y ago

Reposting a similar point I made recently about CAPTCHA and scalpers, but it’s even more relevant for scrapers. PoW can help against basic scrapers or DDoS, but it won’t stop anyone serious. Last week I looked into a Binance CAPTCHA solver that didn’t use a browser at all, just a plain HTTP client. https://blog.castle.io/what-a-binance-captcha-solver-tells-us-about-todays-bot-threats/ https://blog.castle.io/what-a-binance-captcha-solver-tells-u... The attacker had fully reverse engineered the signal collection and solved-state flow, including obfuscated parts. They could forge all the expected telemetry. This kind of setup is pretty standard in bot-heavy environments like ticketing or sneaker drops. Scrapers often do the same to cut costs. CAPTCHA and PoW mostly become signal collection protocols, if those signals aren’t tightly coupled to the actual runtime, they get spoofed. And regarding PoW: if you try to make it slow enough to hurt bots, you also hurt users on low-end devices. Someone even ported PerimeterX’s PoW to CUDA to accelerate solving: https://github.com/re-jevi/PerimiterXCudaSolver/blob/main/powsolver.cu https://github.com/re-jevi/PerimiterXCudaSolver/blob/main/po...

benregenspan 1y ago

At a media company, our web performance monitoring tool started flagging long-running clientside XHR requests, which I couldn't reproduce in a real browser. It turned out that an analytics vendor was injecting a script which checked if it looked like the client was a bot. If so, they would then essentially use the client as a worker to perform their own third-party API requests (for data like social share counts). So there's definitely some prior art for this kind of thing.

apitman 1y ago
This is really interesting. One naive thought that immediately came to mind is that bots might be capable of making cross site requests. The logical conclusion of this entire arms race is that bots will eventually have no choice but to run actual browsers. Not sure that fact will appreciably reduce their scaping abilities though.
- benregenspan 1y ago
  > The logical conclusion of this entire arms race is that bots will eventually have no choice but to run actual browsers I think this is almost already the case now. Services like Cloudflare do a pretty good job of classifying primitive bots and if site operators want to block all (or at least vast majority), they can. The only reliable way through is a real browser. (Which does put a floor on resource needs for scraping)
- dragonwriter 1y ago
  > The logical conclusion of this entire arms race is that bots will eventually have no choice but to run actual browsers. I thought bots using (headless) browsers was an existing workaround for a number of existing issues with simpler bots, so this doesn't seem to be a big change.

akomtu 1y ago

That's DRM essentially: watch, but do not copy. Except that in this case the corporations do piracy.

jameslk 1y ago

What if we move some of the website backend workload to the bots, effectively using them as decentralized cloud infrastructure? web3, we are so back

kbenson 1y ago

With regard to proof of work systems that provide revenue: 1) Making LLM (and other) scrapers pay for the resources they use seems perfectly fine to me. Also, as someone that manages some level of scraping (on the order of low tens of millions of requests a month), I'm fine with this. There's a wide range of scraping that the problem is not some resource cost, but the other side not wanting to deal with setting up APIs or putting so many hurdles on access that it's easier to just bypass it. 2) This seems like it might be an opportunity for Cloudflare. Let customers opt-in to requiring a proof of work when visitors already trip the cloudflare vetting page that runs additional checks to see if you're a bad actor, and apply any revenue to a service credit towards their monthly fee (or if on a free plan, as credit to be used for trying out additional for-pay features). There might be a perverse inventive to toggle on more stringent checking from cloudflare, but ultimately since it's all being paid for that's the site owner's choice on how they want to manage their site.