6 ms·

Please consider installing one of the many PoW schemes such as anubis rather than use these cloudflare "features". I increasingly encounter outright blocks rath

Please consider installing one of the many PoW schemes such as anubis rather than use these cloudflare "features". I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites. Each individual site isn't particularly important to me but it's depressing to watch the process unfold like this. You really are choosing to erode the core basis of the internet if you go this route.

gruez 3mo ago
>I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites. ??? Unless you're browsing around with the googlebot user agent string, you should be getting turnstile challanges at most, not blocks. And if you're getting a turnstile challenge it's unclear how it's different than an anubis challenge. If you're outright blocked, it's probably a site decision (eg. block all VPNs or block everyone not from a given country) rather than cloudflare's.
- gucci-on-fleek 3mo ago
  > And if you're getting a turnstile challenge it's unclear how it's different than an anubis challenge. You are guaranteed to pass an Anubis challenge eventually [0], whereas it's possible to get stuck forever in an infinitely-looping Turnstile challenge. > If you're outright blocked, it's probably a site decision (eg. block all VPNs or block everyone not from a given country) rather than cloudflare's. Cloudflare blocks legitimate users itself sometimes [1]. [0]: Unless you run into a bug, but Anubis is open source, so you can always submit a patch upstream. I've done this myself, and I can confirm that it's relatively straightforward. [1]: https://news.ycombinator.com/item?id=43329320 https://news.ycombinator.com/item?id=43329320
  - gruez 3mo ago
    >You are guaranteed to pass an Anubis challenge eventually [0] That's a double edged sword because bots will eventually get through too, and unlike humans, their time is dirt cheap. >Cloudflare blocks legitimate users itself sometimes [1]. I never ran into this issue despite using seemingly maximally suspicious configs like tor browser. I can't say the same for some other vendors.
    - gucci-on-fleek 3mo ago
      > That's a double edged sword because bots will eventually get through too, and unlike humans, their time is dirt cheap. Yeah, I really have no idea why Anubis works right now: residential proxies are far more expensive than compute, yet the bots seem to have no problem obtaining millions of residential IPs, but they give up on even short-ish Anubis challenges. > I never ran into this issue despite using seemingly maximally suspicious configs like tor browser. I can't say the same for some other vendors. Yeah, I don't like the Cloudflare challenges, but in the past 5 years I've only had it outright block me once, and that fixed itself after 15 minutes. And I use Firefox on Linux with various privacy extensions, so my browser probably appears at least moderately suspicious. Whereas I've been trapped in impossible ReCaptcha loops quite a few times, which is still better than vague error messages that magically go away when I switch to something not running Linux. So I'll begrudgingly accept that Turnstile is the least user-hostile product on the market right now.
      - raincole 3mo ago
        The only reason Anubis works right now is that it's not very commonly used and the bots are not optimized to bypass it (yet).
      - inigyou 3mo ago
        Because the (singular) global ddos adversary is using a very dumb scraper, even dumber than wget --mirror
    - fc417fc802 3mo ago
      > unlike humans, their time is dirt cheap. On the contrary I only want to visit a few pages on a given site and have an entire laptop at my disposal. Meanwhile for bots efficiency is key. A serious scraper (ie the type of actor that actually causes material problems for site operators) is performing tens or hundreds of pages loads per second per core spread across thousands of sites. Making a single page load take even half a second of cpu time is a massive win for the site operator.
    - inigyou 3mo ago
- amatecha 3mo ago
  Try using Firefox with "resist fingerprinting" enabled, on OpenBSD, if you want an idea of how insidious Cloudflare's increasing prevalence is for the openness of the web. Inexplicable 403 responses for entire domains for no apparent reason, no "captcha", just blocked outright. Heck, it happens on Linux too, and even without "resist fingerprinting" enabled.
  - ipaddr 3mo ago
    Happens with older chrome browsers as well. Last supported windows 7 version. Started this month.
- m00dy 3mo ago
  yeah, googlebot user agent thing was a thing maybe around 10 years ago but I don't think it's working today. Maybe if you start the http session from gcloud, it might still work.
- inigyou 3mo ago
  Yesterday Cloudflare told me I was a bot and couldn't access this.
- benhurmarcel 3mo ago
  I’ve been blocked by Turnstile without possibility of challenge, no way to access nor know why. And it was with standard iOS and Windows machines and no VPN or anything. It happens.
matheusmoreira 3mo ago
> Please consider installing one of the many PoW schemes such as anubis Why not go all the way and mine monero instead of just completely wasting the work?
- akersten 3mo ago
  because then someone will complain that you're stealing their CPU cycles or something, on the website they chose to visit and execute
  - matheusmoreira 3mo ago
    "Stealing" CPU cycles is exactly what PoW bot protection is doing. The whole point is to add cost to the bots so they decide it's too expensive and give up. The only difference is the cycles are getting converted into heat now. They could be getting converted into monero instead. It's still heat but at least creator got some money for it.
neya 3mo ago
Adding the link to GitHub here if anyone is curious: https://github.com/techaroHQ/anubis https://github.com/techaroHQ/anubis
prologic 3mo ago
PoW schemes like Anubis don't work. Increasingly bots are using headless browsers and are basically able to solve captchas, proof-of-work(s) and basically bypass all any any attempts to block them. It's becoming impossible to stop bots from hammering your sites/services for unwanted traffic.
- m00dy 3mo ago
  The most recent captchas that involve basic reasoning can block majority of bots today.
  - prologic 3mo ago
    Name one, that an LLM can't solve. The only real mechanism that continues to still work (but doesn't scale) is "Human in the loop".
- fc417fc802 3mo ago
  You are asserting that they don't work without explanation or evidence. Meanwhile it isn't clear why they wouldn't and indeed they appear to accomplish the stated goal of severely rate limiting scrapers.
  - prologic 3mo ago
    Surely I don't need to explain how headless browsers work, bot proxies and distributed crawlers to you do I? If you haven't experienced this first-hand, I can understand. Maybe it warrants a blog post. But I can assure you, these don't work at scale, they might only stop the "less sophisticated" bots and maybe (just maybe) some unwanted spam.
    - fc417fc802 3mo ago
      What do the browsers being headless or the traffic proxied have to do with anything? Each client still has to solve a PoW challenge regardless. That mitigates the (possibly unintentional) DoS by making it exorbitantly expensive.
m00dy 3mo ago
Not sure why Anubis is getting so much hype on HN, but honestly, it is not the solution. A real solution would use behavioral modeling. Most browser fingerprinting issues are already largely solved anyway.
- fc417fc802 3mo ago
  PoW is a reasonable solution as a fallback when other metrics flag a client.
  - microtonal 3mo ago
    How? First, they can solve the Anubis challenge with native code, so they can solve them faster than genuine users. Second, the cost is nothing compared to training LLMs, plus they will just move the work to the residential proxies that they have access to (so, someone is paying through their TV's electricity bill). Anubis only work(s|ed) great for a while when crawlers were not prepared for these challenges. Security through obscurity.
    - fc417fc802 3mo ago
      > they will just move the work to the residential proxies that they have access to The proxies I am familiar with do not offer arbitrary code execution. I think you're thinking of a botnet. Regarding native code, the current crop of solutions seem to work well enough for now. Ultimately a challenge response protocol should be standardized and browsers should ship a native implementation. In the meantime WASM likely gets you close enough to native.
      - microtonal 3mo ago
        Regarding native code, the current crop of solutions seem to work well enough for now. I just quoted a toot in another submission, adding it here since it is relevant: We apologize for a period of extreme slowness today. The army of AI crawlers just leveled up and hit us very badly. [...] It seems like the AI crawlers learned how to solve the Anubis challenges. [...] However, we can confirm that at least Huawei networks now send the challenge responses and they actually do seem to take a few seconds to actually compute the answers. It looks plausible, so we assume that AI crawlers leveled up their computing power to emulate more of real browser behaviour to bypass the diversity of challenges that platform enabled to avoid the bot army. https://social.anoxinon.de/@Codeberg/115033790447125787 https://social.anoxinon.de/@Codeberg/115033790447125787
timpera 3mo ago
Please don't use Anubis, it makes visiting websites very difficult (often multiple minutes wait times) on old hardware and low-end smartphones.
- altairprime 3mo ago
  As things stand right now, the current web may not be possible to maintain for old and low-end phones given the costs imposed by AI training crawlers. Please propose an alternative to both Cloudflare and Anubis, that shields websites against inhuman traffic without frequent operator intervention (or otherwise negates the capacity costs they pay for AI crawling) and is compatible with low-end smartphones. Certainly, I imagine Anubis would be interested in adopting it if it’s effective!
  - expedition32 3mo ago
    Great you now made the internet unusable for half the planet.
    - shimman 3mo ago
      Sounds like something the government should regulate and own then rather than relying on corporate benefactors that all profit off of malware advertising.
      - altairprime 3mo ago
        That will be about as effective as attempts to block soccer streams in Spain are, and I don't think it'll pan out as a viable solution without worldwide enforcement backed by BGP bans of providers that continue to host non-compliant sites after notified. It's technically possible to consider a future where RIPE might withdraw your BGP assignment if you host advertising malware on your website, but ultimately that's just as dystopic as the DMCA is today, only scaled up to worldwide impacts and drama.
    - altairprime 3mo ago

gruez 3mo ago

>I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites. ??? Unless you're browsing around with the googlebot user agent string, you should be getting turnstile challanges at most, not blocks. And if you're getting a turnstile challenge it's unclear how it's different than an anubis challenge. If you're outright blocked, it's probably a site decision (eg. block all VPNs or block everyone not from a given country) rather than cloudflare's.

gucci-on-fleek 3mo ago
> And if you're getting a turnstile challenge it's unclear how it's different than an anubis challenge. You are guaranteed to pass an Anubis challenge eventually [0], whereas it's possible to get stuck forever in an infinitely-looping Turnstile challenge. > If you're outright blocked, it's probably a site decision (eg. block all VPNs or block everyone not from a given country) rather than cloudflare's. Cloudflare blocks legitimate users itself sometimes [1]. [0]: Unless you run into a bug, but Anubis is open source, so you can always submit a patch upstream. I've done this myself, and I can confirm that it's relatively straightforward. [1]: https://news.ycombinator.com/item?id=43329320 https://news.ycombinator.com/item?id=43329320
- gruez 3mo ago
  >You are guaranteed to pass an Anubis challenge eventually [0] That's a double edged sword because bots will eventually get through too, and unlike humans, their time is dirt cheap. >Cloudflare blocks legitimate users itself sometimes [1]. I never ran into this issue despite using seemingly maximally suspicious configs like tor browser. I can't say the same for some other vendors.
  - gucci-on-fleek 3mo ago
    > That's a double edged sword because bots will eventually get through too, and unlike humans, their time is dirt cheap. Yeah, I really have no idea why Anubis works right now: residential proxies are far more expensive than compute, yet the bots seem to have no problem obtaining millions of residential IPs, but they give up on even short-ish Anubis challenges. > I never ran into this issue despite using seemingly maximally suspicious configs like tor browser. I can't say the same for some other vendors. Yeah, I don't like the Cloudflare challenges, but in the past 5 years I've only had it outright block me once, and that fixed itself after 15 minutes. And I use Firefox on Linux with various privacy extensions, so my browser probably appears at least moderately suspicious. Whereas I've been trapped in impossible ReCaptcha loops quite a few times, which is still better than vague error messages that magically go away when I switch to something not running Linux. So I'll begrudgingly accept that Turnstile is the least user-hostile product on the market right now.
    - raincole 3mo ago
      The only reason Anubis works right now is that it's not very commonly used and the bots are not optimized to bypass it (yet).
    - inigyou 3mo ago
      Because the (singular) global ddos adversary is using a very dumb scraper, even dumber than wget --mirror
  - fc417fc802 3mo ago
    > unlike humans, their time is dirt cheap. On the contrary I only want to visit a few pages on a given site and have an entire laptop at my disposal. Meanwhile for bots efficiency is key. A serious scraper (ie the type of actor that actually causes material problems for site operators) is performing tens or hundreds of pages loads per second per core spread across thousands of sites. Making a single page load take even half a second of cpu time is a massive win for the site operator.
  - inigyou 3mo ago
amatecha 3mo ago
Try using Firefox with "resist fingerprinting" enabled, on OpenBSD, if you want an idea of how insidious Cloudflare's increasing prevalence is for the openness of the web. Inexplicable 403 responses for entire domains for no apparent reason, no "captcha", just blocked outright. Heck, it happens on Linux too, and even without "resist fingerprinting" enabled.
- ipaddr 3mo ago
  Happens with older chrome browsers as well. Last supported windows 7 version. Started this month.
m00dy 3mo ago
yeah, googlebot user agent thing was a thing maybe around 10 years ago but I don't think it's working today. Maybe if you start the http session from gcloud, it might still work.
inigyou 3mo ago
Yesterday Cloudflare told me I was a bot and couldn't access this.
benhurmarcel 3mo ago
I’ve been blocked by Turnstile without possibility of challenge, no way to access nor know why. And it was with standard iOS and Windows machines and no VPN or anything. It happens.

matheusmoreira 3mo ago

> Please consider installing one of the many PoW schemes such as anubis Why not go all the way and mine monero instead of just completely wasting the work?

akersten 3mo ago
because then someone will complain that you're stealing their CPU cycles or something, on the website they chose to visit and execute
- matheusmoreira 3mo ago
  "Stealing" CPU cycles is exactly what PoW bot protection is doing. The whole point is to add cost to the bots so they decide it's too expensive and give up. The only difference is the cycles are getting converted into heat now. They could be getting converted into monero instead. It's still heat but at least creator got some money for it.

neya 3mo ago

Adding the link to GitHub here if anyone is curious: https://github.com/techaroHQ/anubis https://github.com/techaroHQ/anubis

prologic 3mo ago

PoW schemes like Anubis don't work. Increasingly bots are using headless browsers and are basically able to solve captchas, proof-of-work(s) and basically bypass all any any attempts to block them. It's becoming impossible to stop bots from hammering your sites/services for unwanted traffic.

m00dy 3mo ago
The most recent captchas that involve basic reasoning can block majority of bots today.
- prologic 3mo ago
  Name one, that an LLM can't solve. The only real mechanism that continues to still work (but doesn't scale) is "Human in the loop".
fc417fc802 3mo ago
You are asserting that they don't work without explanation or evidence. Meanwhile it isn't clear why they wouldn't and indeed they appear to accomplish the stated goal of severely rate limiting scrapers.
- prologic 3mo ago
  Surely I don't need to explain how headless browsers work, bot proxies and distributed crawlers to you do I? If you haven't experienced this first-hand, I can understand. Maybe it warrants a blog post. But I can assure you, these don't work at scale, they might only stop the "less sophisticated" bots and maybe (just maybe) some unwanted spam.
  - fc417fc802 3mo ago
    What do the browsers being headless or the traffic proxied have to do with anything? Each client still has to solve a PoW challenge regardless. That mitigates the (possibly unintentional) DoS by making it exorbitantly expensive.

m00dy 3mo ago

Not sure why Anubis is getting so much hype on HN, but honestly, it is not the solution. A real solution would use behavioral modeling. Most browser fingerprinting issues are already largely solved anyway.

fc417fc802 3mo ago
PoW is a reasonable solution as a fallback when other metrics flag a client.
- microtonal 3mo ago
  How? First, they can solve the Anubis challenge with native code, so they can solve them faster than genuine users. Second, the cost is nothing compared to training LLMs, plus they will just move the work to the residential proxies that they have access to (so, someone is paying through their TV's electricity bill). Anubis only work(s|ed) great for a while when crawlers were not prepared for these challenges. Security through obscurity.
  - fc417fc802 3mo ago
    > they will just move the work to the residential proxies that they have access to The proxies I am familiar with do not offer arbitrary code execution. I think you're thinking of a botnet. Regarding native code, the current crop of solutions seem to work well enough for now. Ultimately a challenge response protocol should be standardized and browsers should ship a native implementation. In the meantime WASM likely gets you close enough to native.
    - microtonal 3mo ago
      Regarding native code, the current crop of solutions seem to work well enough for now. I just quoted a toot in another submission, adding it here since it is relevant: We apologize for a period of extreme slowness today. The army of AI crawlers just leveled up and hit us very badly. [...] It seems like the AI crawlers learned how to solve the Anubis challenges. [...] However, we can confirm that at least Huawei networks now send the challenge responses and they actually do seem to take a few seconds to actually compute the answers. It looks plausible, so we assume that AI crawlers leveled up their computing power to emulate more of real browser behaviour to bypass the diversity of challenges that platform enabled to avoid the bot army. https://social.anoxinon.de/@Codeberg/115033790447125787 https://social.anoxinon.de/@Codeberg/115033790447125787

timpera 3mo ago

Please don't use Anubis, it makes visiting websites very difficult (often multiple minutes wait times) on old hardware and low-end smartphones.

altairprime 3mo ago
As things stand right now, the current web may not be possible to maintain for old and low-end phones given the costs imposed by AI training crawlers. Please propose an alternative to both Cloudflare and Anubis, that shields websites against inhuman traffic without frequent operator intervention (or otherwise negates the capacity costs they pay for AI crawling) and is compatible with low-end smartphones. Certainly, I imagine Anubis would be interested in adopting it if it’s effective!
- expedition32 3mo ago
  Great you now made the internet unusable for half the planet.
  - shimman 3mo ago
    Sounds like something the government should regulate and own then rather than relying on corporate benefactors that all profit off of malware advertising.
    - altairprime 3mo ago
      That will be about as effective as attempts to block soccer streams in Spain are, and I don't think it'll pan out as a viable solution without worldwide enforcement backed by BGP bans of providers that continue to host non-compliant sites after notified. It's technically possible to consider a future where RIPE might withdraw your BGP assignment if you host advertising malware on your website, but ultimately that's just as dystopic as the DMCA is today, only scaled up to worldwide impacts and drama.
  - altairprime 3mo ago