22 ms·

Tell HN: Impassable Cloudflare challenges are ruining my browsing experience

• doubleorseven 2y ago
  I use Whonix quite a lot, Most of the internet is unusable since i get into the "check the box" loop.
• solardev 2y ago
  You're collateral damage in the web's war against bots :( Unfortunately, I think the Cloudflare challenges are designed to filter out users similar to your profile... once you stray far enough from the norm, it just looks like a bot / suspicious traffic to them. Statistically there's not enough users like you (privacy-conscious Linux users on nonstandard browsers) for them to really care enough to do anything about it. Site owners don't care either since you're usually like 1-2% of users at most, and typically also the same ones who block ads, etc., so they don't mind blocking you... it's sad, but I don't think there is really anything you can do about it except conform. It's an ongoing arms race and you're caught in the middle.
  • blakeashleyjr 2y ago
    While you hit the nail on the head, I am still surprised that so many tools targeted at people like me (web hosting, developer tools, etc.) are protected that way.
    • kauegimenes 2y ago
      Its not only about protection, most web developers would use Cloudflare since its a free CDN and would increase the app load time considerably.
      • chrisweekly 2y ago
        increase -> decrease
        • rkagerer 2y ago
          Except for those encountering that dreaded captcha.
      • solardev 2y ago
        You can separately configure (to a large degree) the caching vs protection features, though.
    • warkdarrior 2y ago
      Because if such hosting and developer tools are not protected against bots, the tools end up used for phishing, spamming, etc.
    • rad_gruchalski 2y ago
      They are not targeting people like you. Bots are the target. If you look like a bot, how are they going to distinguish?
      • KronisLV 2y ago
        > If you look like a bot, how are they going to distinguish? Some non-existant system of attesting that I'm person X (possibly through an e-ID card) who has issued a client certificate Y (cert chain, using my e-ID cert to sign) to be used with my device Z (presumably with a device fingerprint or IP range attached to the cert). Of course, this would mean no privacy, but that's not that different from being signed in through Google as an identity provider, we'd just shift the mechanism to be universal (like client certs already are). One of the options that would take more coordination than will probably happen (though very similar to some e-signature solutions in EU, which we already use) but I could see using something like that for a variety of professional/service sites, since signing in with the e-ID card directly is already a thing on some sites here (government sites, banking sites, utilities sites).
  • EGreg 2y ago
    I honestly don't see what's so hard about a bot simulating "the norm" within the margin of error. This cat-and-mouse game is just like a GAN, the end result is indistinguishable even by a bot.
    • nullc 2y ago
      Bot authors are lazy and won't until they have to.. once you do, you can then pretend they aren't bots and include them in the engagement numbers you feed prospective shareholders.
      • tokioyoyo 2y ago
        Agreed. From my past experiences though, a very good chunk of them will give up once there is a resistance. Basically, you want your bot protection to just be a little better than your competitor. Then the bot author will target them instead, because of the path of least resistance.
        • EGreg 2y ago
          Outrun the friend not the bear? Hehe
    • viraptor 2y ago
      It depends on the defences. It starts trivial - just make a http request. Then there's http version, user agent header, other headers, header ordering, cookies, TLS ciphers, session resolution, timing, behaviour for page resources, ... and so many other things. It takes time, even if you order headless chrome.
  • shiomiru 2y ago
    The sad part is that it's trivial to get around CF's bot protection if you're writing a bot (just use curl-impersonate and buy residential IPs), but it's pretty much impossible to bypass as a human if their magical black box doesn't like your browser and/or IP address.
    • solardev 2y ago
      How does it get around captchas?
      • gjsman-1000 2y ago
        You pay contract workers in a third world country a tiny amount of money per day, to spend all day clicking boxes.
      • tedivm 2y ago
        If they don't think you're suspicious they don't make you do the captchas, and as others have mentioned you can always outsource it to captcha farms. There are also AI models which do a fairly decent amount, and since most captchas let you repeat attempts with new patterns you can have a pretty high error rate to get past them. Then there's the ADA, which requires accessibility- many captchas have an audio component as a backup and those are easy to interpret by models.
      • gruez 2y ago
        Cloudflare turnstile isn't even a captcha. The user just has to tick a box. Behind the scenes there's a javascript challenge to make sure you're vaguely a browser and not some script a bazillion requests per minute.
        • xdfgh1112 2y ago
          It's also used for proof of work as many scrapers are using thousands of IPs but only a few CPUs
      • michaelmior 2y ago
        curl-impersonate doesn't solve CAPTCHAs, but the goal is to look enough like a human that Cloudflare doesn't present a CAPTCHA in the first place.
  • modzu 2y ago
    it is discrimination
    • jdironman 2y ago
      Only if enough are discriminated against that it affects the bottom line.
  • devops99 2y ago
    [dead]
  • winrid 2y ago
    I don't buy this because using Chrome is what most bots probably do right? Headless chrome is easy.
  • kevincox 2y ago
    > Site owners don't care either since you're usually like 1-2% of users at most, and typically also the same ones who block ads, etc., so they don't mind blocking you I do believe that it is true that many site owners wouldn't care. But I suspect that in the vast majority of cases they don't actually know. Cloudflare probably shows them a nice dashboard about all of these blocked "threats" and they don't know better than to question it.
• Zaheer 2y ago
  I'd expect this to increase with the proliferation of AI Crawlers and scraping becoming easier with AI.
• kauegimenes 2y ago
  Can't you have a normal firefox profile for such cases? Do you have any javascript filters? I bet the issue must be related to configs messing with the JS runtime.
  • ghjfrdghibt 2y ago
    The issue is scummy companies like cloudflare which are causing these issues. If your software is blocking legitimate users then your software is shit at its job. It's not the users fault.
    • gruez 2y ago
      >The issue is scummy companies like cloudflare which are causing these issues. If your software is blocking legitimate users then your software is shit at its job. It's not the users fault. But if you're going out of your way to look suspicious (ie. "I use a heavily customized Firefox config on Linux"), surely you'd agree at some point it goes from "your software is shit at its job" to "it's your fault for looking suspicious"? If you walk into bank wearing a balaclava and get stopped by security, it's not really "security is shit at its job".
      • ghjfrdghibt 2y ago
        [flagged]
        • gruez 2y ago
          >Everyone should only be allowed to use windows and a chrome browser variant with no ad blocking. Cloudflare 100% should be allowed to arbitrarily block anyone not using this set up because they are suspicious. Seems like a slippery slope argument, but isn't reflective of reality. They still allow Tor browser to pass, of all things.
          • ghjfrdghibt 2y ago
            It wasn't meant to be taken seriously, I was using it to show the ridiculousness of blaming a user for the shortcomings of cloudflare. But if you like: the arbitrarily blocked user if not at fault, cloudflare is at fault.
• blakeashleyjr 2y ago
  What I don't understand is why you have to protect areas that require login so harshly? If I can log in, especially with 2-factor, you can safely assume I am not a bot, or you have a larger problem. If I have entered bad credentials 5+ times, okay, you can start backing me off or challenging me. What am I missing? Fail2ban has been around a long time.
  • gjsman-1000 2y ago
    40% of the internet’s traffic now is bots, with about half of those being malicious. Fail2ban is decent for a very small DDoS, but useless for one with any substance, and also useless against bots scraping data or probing for weaknesses. Also remember, especially on AWS, bandwidth is expensive. A CDN cache + blocking bots = big savings.
  • noprocrasted 2y ago
    Problem is that a significant chunk of the technology industry still relies on "engagement" as its business model. The objective of slapping an overzealous bot protection system isn't to protect high-risk endpoints like logins/etc, it's to ensure a human is "engaging" and human time is being wasted by making even legitimate automated usage impossible. From their perspective, the blocking of power users with unusual setups is actually a happy coincidence, as those are unlikely to "engage" with the product in the desired way (they run ad & spyware blockers, don't fall for dark patterns, and are more likely to fight back if they get defrauded by the corporation).
  • duskwuff 2y ago
    > What am I missing? Fail2ban has been around a long time. Modern threat actors can spread requests out over large pools of source IPs. Rate limiting login attempts by IP isn't an effective means of preventing credential stuffing attacks.
• Terr_ 2y ago
  I'm really afraid of what kind of internet we'll have when these kinds of un-diagnosable un-appealable false-positives are not just transient blips, but become metadata companies use to blindly and permanently kill off accounts on other services. I think it may have been what happened my since-2010 Reddit account was mysteriously killed a couple years ago, and literally the only cause I can think of is that I might've used the wrong public wifi for an evening.
• hulitu 2y ago
  > Cloudflare challenges have made large portions of the web unusable for me. I guess the best web experience is when one filters Cloudfare, Google and Microsoft at the firewall.
• avnfish 2y ago
  I'm experiencing the same issue which is definitely exacerbated by straying from a 'default' configuration e.g. using a custom browser screen reader, browsing from Brazil, using a VPN, using Firefox. I think eventually I'll be completely locked out of the 'mainstream' web
• Animats 2y ago
  > The "unsubscribe" button in Indeed's job notification emails leads me to an impassable Cloudflare challenge. That's a CAN-SPAM act violation. FTC: "Tell recipients how to opt out of receiving future marketing email from you. Your message must include a clear and conspicuous explanation of how the recipient can opt out of getting marketing email from you in the future. Craft the notice in a way that’s easy for an ordinary person to recognize, read, and understand. Creative use of type size, color, and location can improve clarity. Give a return email address or another easy Internet-based way to allow people to communicate their choice to you. You may create a menu to allow a recipient to opt out of certain types of messages, but you must include the option to stop all marketing messages from you. Make sure your spam filter doesn’t block these opt-out requests."[1] Experian was recently fined for making it hard to opt out of their marketing emails. The actual regulation text: § 316.5 Prohibition on charging a fee or imposing other requirements on recipients who wish to opt out. Neither a sender nor any person acting on behalf of a sender may require that any recipient pay any fee, provide any information other than the recipient's electronic mail address and opt-out preferences, or take any other steps except sending a reply electronic mail message or visiting a single Internet Web page, in order to: (a) Use a return electronic mail address or other Internet-based mechanism, required by 15 U.S.C. 7704(a)(3), to submit a request not to receive future commercial electronic mail messages from a sender; or (b) Have such a request honored as required by 15 U.S.C. 7704(a)(3)(B) and (a)(4). That seems to cover it. File a CAN-SPAM act complaint (spam@uce.gov). Send a copy to the legal department of the sender. [1] https://www.ftc.gov/business-guidance/resources/can-spam-act-compliance-guide-business https://www.ftc.gov/business-guidance/resources/can-spam-act...
  • driverdan 2y ago
    Unfortunately the government seems to have given up on enforcing the CAN-SPAM act. If they actually enforced it spam companies like Salesforce would face massive fines.
    • bluGill 2y ago
      You can press charges yourself and get lawyer fees for your efforts. Probably not worth it, but you don't need the government to do this.
• mg 2y ago
  If it is triggered by the customizations you did in Firefox, then running a fresh Firefox in a container might help: docker run -it --rm -e DISPLAY --net=host -v $XAUTHORITY:/root/.Xauthority -v /tmp/.X11-unix:/tmp/.X11-unix debian:12-slim Then inside the container, run: apt update apt install firefox-esr firefox
  • stonogo 2y ago
    what is the advantage here over just running 'firefox -ProfileManager' and making a clean profile?
    • theamk 2y ago
      All host info not accessible via X11 protocol is hidden, for example font list, is replaced with generic one. For even more protection, run VNC server with common resolution in the container and connect to it using VNC viewer. In this case firefox provides a super generic profile (latest debian with mesa GPU), making this browser very hard to distinguish from others. This has some downsides however: First, you cannot resize window. Second, a lot of actual bots use same config, so it might be blocked.
      • veeti 2y ago
        Isn't it suspicious bot-like behavior to only have the bare minimum fonts installed? :-)
        • ghxst 2y ago
          To be fair, Firefox out of the box prevents against font fingerprinting more than Chrome, it's considerably easier to get Firefox to run in a docker container and pass all the client side challenges than Chrome in my experience, you still have a valid point though.
      • homebrewer 2y ago
        mullvad browser is pretty much this, but without messing around with containers. One fingerprint for all users, with the same font list, resolution, canvas behavior, etc. https://mullvad.net/browser https://mullvad.net/browser
• zufallsheld 2y ago
  > - The "unsubscribe" button in Indeed's job notification emails leads me to an impassable Cloudflare challenge. Maybe indeed could be held liable here? From the can spam act (if you're from the US): > You can’t charge a fee, require the recipient to give you any personally identifying information beyond an email address, or make the recipient take any step other than sending a reply email or visiting a single page on an Internet website as a condition for honoring an opt-out request. https://www.ftc.gov/business-guidance/resources/can-spam-act-compliance-guide-business https://www.ftc.gov/business-guidance/resources/can-spam-act...
  • _bin_ 2y ago
    this nevertheless happens all the time. i have an old linkedin account i haven't logged into in years and can't be bothered to dig up the credentials so one of my e-mails gets stupid "network updates". one must log in to disable these and navigate to some obscure settings page in one of the most heinously overcrowded UIs on the web. so i just flagged it all as spam and hoped it hurts their deliverability a little.
    • ToucanLoucan 2y ago
      Honestly I click an unsubscribe link but if it requires me to complete a survey or fill out a form, I just nix the tab and spam filter the email. I'm nobody's fucking admin assistant and my time is valuable: you know my fucking email and could easily add it to the think, or at the most, ask me to type it into a box if you MUST. Anything more than that, if I have to manually opt out of "types" of messages or whatever, nah. Fuck you. I didn't ask for your fucking emails and I sure as shit am not going to do the homework you're assigning me to make them stop.
      • ryandrake 2y ago
        Yep, I just spam filter the E-mails now. If that act adds 0.0001% to that sender having future E-mail deliverability problems, then all the better. If it's commercial or political and I didn't explicitly ask for the sender to E-mail, then it's spam.
• kachapopopow 2y ago
  I have experience bypassing these. The primary cause of this is most likely any kind of 'optimizations' you have in your browser (or missing fingerprints). If you want to 'bypass' these I recommend removing any use of Proxy[1] (via extensions). You should also look into disabling any kind of forced backgrounding. Make sure service workers are working. 1: They catch Proxy usage by using exceptions and analyzing the stacktrace. I assume you know what a javascript proxy is, but incase you don't: It's something that allows you to override any kind of object function such as navigator.hardwareConcurrecy.
  • wewtyflakes 2y ago
    > They catch Proxy usage by using exceptions and analyzing the stacktrace That is really clever, I am guessing this is why various browser automation companies are using custom forks of Chromium.
• dboreham 2y ago
  I ran into this, or something similar recently when our main connection went down (solar powered) and we switched to Starlink. Due to Starlink NAT issues I had tunneled our traffic to to a box colocated in a data center. This broke a number of web sites in weird ways. Became so annoying that I ended up bringing up a tunnel to our office in town to get back to the regular IP we used. Weird problems went away.
• bastard_op 2y ago
  I deal with this fairly commonly, presumably because I use linux, and we all know only botnets use linux. Occasionally with cloudflare I'll just get summary rejection and supposed blocking of my IP, but either it's summary rejection or a pass without challenge. Recently I had to deal with this for alibaba just to look at something, which I usually just use torbrowser with, and finally gave up as I couldn't pass the challenge. I suppose I shouldn't be surprised at that though, they trust me as much as I trust them. The worst is usually adobe and cookielaw with all their related tracking crap, where I can't even get the captcha to render as it's so many layers buried in scripting I can't enable enough sites between ublock, noscript, privacy badger, and firefox strict modes. I treat adobe like malware, but unfortunately things like albertsons.com for groceries and other mega companies love to use it, and their sites literally do not work without allowing their heavy scripting/tracking. There are other usually smaller captcha players that I haven't been human enough to pass with, I forget the names of the stupid to shame, but a few when I see them I recognize to just close the window and forget about whatever it was I was looking for there (like twitter/x). Hooray commerce!
  • krunck 2y ago
    >...when I see them I recognize to just close the window and forget about whatever it was I was looking for there This is the way.
  • TiredOfLife 2y ago
    My main desktop for the past year has been Steamdeck with linux. And don't get any excess Cloudflare challenges.
    • choobacker 2y ago
      Nice idea! How's that working out for you? Stock OS? Bazzite?
      • TiredOfLife 2y ago
        Stock. Browser (Chrome/Firefox) doesn't have hw acceleration for video decode. But other than that it's fine. Fast and silent. VS Code and Jetbrains tools work fine.
• ghjfrdghibt 2y ago
  It seems that if you use Firefox with an adblocker then cloudflare spam is all you see. Though I have experienced this in plain Firefox too. Cloudflare are a scummy company trying to force you to use one browser and view all ads.
  • robhlt 2y ago
    It can't be just that. I use Firefox on Linux with ublock origin, strict tracking protection, and clear cookies on exit, and I've never ever seen a cloudflare challenge. Not even on sites with that "verifying your browser" page enabled.
    • ghjfrdghibt 2y ago
      Maybe you're right, I see it all the time. Assume cloudflare do other dumb stuff too then like up ranges and just being generally crap at their jobs.
• gruez 2y ago
  >I use a heavily customized Firefox config on Linux. This is probably the cause, especially if you're doing stuff like spoofing user agent. It's not cloudflare "cracking down on privacy" or whatever either. Unmodified tor browser passes turnstile challenges just fine.
  • jillyboel 2y ago
    It's up to users to choose their user agent.
    • gruez 2y ago
      And it's up to site owners and website security vendors to choose which user agents to admit.
      • jillyboel 2y ago
        No, that's discrimination.
• dylan604 2y ago
  My local TV station's website refuses to allow my to view their page and instead presents an a modal that cannot be blocked accusing me of using an ad blocker. The funny thing is that only happens on a mobile device using the default browser with no extensions. When I visit the same site on my laptop with uBO, the site is viewable with no blocking modals. Sometimes you miss what you were aiming for I guess
• ugotjelly 2y ago
  What do you mean impassable challenge...? Why isn't it passable? Are you a robot?
  • jillyboel 2y ago
    [flagged]
    • _yb2s 2y ago
      Sadly, we probably all are LLMs/bots on the internet at this point, just talking to one another. The real humans have all become fed up and are now mostly off fishing by a lake.
  • gruez 2y ago
    The challenge is a small javascript program that checks the execution environment is consistent with a real browser. For instance, if your user agent says it's chrome, but it's missing features that'd normally be supported by chrome, it'll fail you. The OP mentioned "heavily customized Firefox config", so he might be doing stuff like this that makes his browser look suspicious.
  • numpad0 2y ago
    From website perspective, yes. GP is likely using extreme ad-blocking and/or coming from regions where tons of bots and/or unwanted traffic are also from. In those cases, some/many human users could be misidentified as bots with little incentives to website admins to rectify. And it's discriminatory, yes.
• viraptor 2y ago
  CrimeFlare is not interested in these problems for the users. If you have access to the hosting side, you can adjust the bot score for specific connections/clients. But consumers don't matter to CF so apart from jumping through their hoops, there's nothing better you can do. Unless you accept the racket of course, start paying them and proxy your traffic through the CF workers https://github.com/pellaeon/cloudflare-worker-proxy https://github.com/pellaeon/cloudflare-worker-proxy and magically most barriers will disappear.
  • gruez 2y ago
    >Unless you accept the racket of course, start paying them and proxy your traffic through the CF workers https://github.com/pellaeon/cloudflare-worker-proxy https://github.com/pellaeon/cloudflare-worker-proxy and magically most barriers will disappear. Source this actually works? ie. that using cloudflare workers allows you to bypass cloudflare protection?
    • viraptor 2y ago
      https://jychp.medium.com/how-to-bypass-cloudflare-bot-protection-1f2c6c0c36fb https://jychp.medium.com/how-to-bypass-cloudflare-bot-protec... and many other posts. Haven't looked into this in a while, so can't tell you exactly how effective it is today. (Definitely corrects the high bot score of your IP though)
      • gruez 2y ago
        Sounds like all it does is make your IP reputation slightly better than tor, which is a pretty low bar to cross. You'd likely get the same effect from using any other VPN service, so it's not exactly evidence that cloudflare is running a "racket" with its worker product. The linked blog post even touts the fact it's free as an advantage. Rackets typically aren't free.
        • viraptor 2y ago
          You also change the headers / TLS signature, because it's their worker doing the connection. That covers quite a lot already. The racket is not in the workers themselves, but rather cloudflare both protecting from internet abuse and protecting sites which sell the abuse services. (For example hosting WebStresser) I meant that by giving them more traffic and accepting that as a workaround, we'd be saying "I'm ok with that".
• deleted 2y ago
  [deleted]
• magic_smoke_ee 2y ago
  I can't use any of the kerbalspaceprogram.com domains because of improper discrimination against IPv6 clients triggered by CloudFlare. Error 1015 Ray ID: .... • xxxx-xx-xx xx:xx:xx UTC You are being rate limited What happened? The owner of this website (wiki.kerbalspaceprogram.com) has banned you temporarily from accessing this website. This sort of monoculture creates an Orwellian SPoF.
  • TiredOfLife 2y ago
    Cloudflare owns kerbalspaceprogram?
    • LeifCarrotson 2y ago
      No, wiki.kerbalspaceprogram.com is a customer of Cloudflare, but the outcome is the same.
      • TiredOfLife 2y ago
        Then ask them to disable Cloudflare.
        • metalliqaz 2y ago
          good luck with that
  • freitasm 2y ago
    I don't think it's an IPv6 problem. IPv6 clients are more static than IPv4, which is usually shared amongst many clients (at home) or at the network level (CGNAT). It could be the address is being reused - is it home, cloud or corporate? Have you tried different browsers? Incognito mode? I have an IPv6 block at home and have no problem accessing that site.
  • duskwuff 2y ago
    That isn't "triggered by Cloudflare". The operator of the web site has deliberately configured it to block your IP range, and Cloudflare is obeying those instructions.
• 015a 2y ago
  I've honestly only experienced the opposite; their captcha is reasonably easy to bypass, and I've successfully automated access to a few sites "protected" by the Cloudflare captcha (behind a VPN, no less). > I use a heavily customized Firefox config on Linux. If you really care about privacy, you should blend in to look like everyone else. Avoiding being tracked raises alarm bells. You have to let them track something; but no one ever said it had to be you.
• antfie 2y ago
  I found a GitHub captcha to be unsolvable. That captcha properly stressed me out.
• SoftTalker 2y ago
  Yes, I run into it from time to time. I just move on. If someone is going to make their website inaccessible to me, I'm not going to bend over backwards to try to work around that. Incidentally, since I configured DNS over HTTPS in Firefox, using Cloudflare's DNS, it seems I see this much less often.
• afh1 2y ago
  Same here, but Cloudflare's captchas in particular are actually the easiest to pass in my experience. Google's ones are the killers. But yeah everything has a captcha if you're using a VPN or Firefox.
• _yb2s 2y ago
  I had similar issues as an (also heavily customized) Firefox user, but was able to fix it by installing Cloudflare's Privacy Pass browser extension. It seems ironic that as a human I can't seem to reliably prove I am a human with a realistic amount of effort via these systems, but having installed a specific automated browser extension does? I am not a fan of Cloudflare and don't like the idea of running their software on my computer, but it seemed like the only options to continue using the internet at all.
  • tempaccount420 2y ago
    I didn't know that extension existed, so after failing to fix it by reinstalling Firefox and removing extensions, I just gave up and installed Chrome.
• 93po 2y ago
  I wish we could popularize some extension that pays a penny per page load or something using some shitcoin both as a means to support our favorite sites but also to validate that I'm not a bot, or at least if I am, I am willing to spend a lot of money in a DDOS that goes directly in your pocket
• shadowgovt 2y ago
  Unfortunately, your setup makes you look like a scraper: no history for Cloudflare to identify, the sort of browser / OS config someone would use to homebrew an automated "I sure am not a bot, look at how authentic my user-agent is!" bot, and so on. If you also have JavaScript disabled and clear your cookies frequently, Cloudflare can't fingerprint your machine to know you passed a trust-check in the past. Maybe keeping a heavily-sandboxed Chrome in a VM for situations where Cloudflare is getting in your way might help? (In the large: this has been an issue a long time coming. Quite a bit of cyberpunk predicts the future where the web bifurcates into the "regular" web that is sanitized, corporate, controlled, and used by most people... And the "everyone else" web that is not, with all the pros and cons that entails. The tech has evolved to the point that companies that want a service provider "keeping the bad guys away" for them can pay to have that done, at the cost of false-positives... But at their scale, the false-positives may not matter to them).
• mikequinlan 2y ago
  If you can't pass the captcha you have to ask yourself, are you really a human being or have you just been programmed to believe that you are?
• PaulHoule 2y ago
  It's ironic but I was having terrible problems accessing archive.today when I was using Cloudflare DNS (1.1.1.1) that cleared up when I switched to either my ISP's provider or Google's 8.8.8.8. I was not the only one https://news.ycombinator.com/item?id=38063548 https://news.ycombinator.com/item?id=38063548 What's funny about it is that as a human I get tormented by those things all the time but I have been writing bots since 1999 and have yet to have had CAPTCHAs affect a webcrawling project in a big way: for instance I have a bot that collected 800,000 images from 4 web sites since last April, at times I thought they had anti-bot countermeasures but I realized that when they were having problems it was because the wheels were coming off their web site (don't blame me, that is 0.03 requests/second and are not parallelized and pipelined like the requests from a web browser.) I'm also prototyping one that can look at an article like https://phys.org/news/2025-01-diversifying-dna-origami-generative-tool.html https://phys.org/news/2025-01-diversifying-dna-origami-gener... see if there are links to journal articles in there, determine if the articles are Open Access and pick out an image for social... so far no problems. But if I want to pay my electric bill there's a CAPTCHA -- I mean, what kind of bot wants to pay my electric bill? (Kinda seems like it is asking for a lawsuit in this day and age if it prevents anyone 'differently abled' from accessing essential services...)
  • bigfatkitten 2y ago
    > I mean, what kind of bot wants to pay my electric bill? None, but they do want to use your electricity company's credit card payment facility to test stolen card numbers.
  • duskwuff 2y ago
    > I was having terrible problems accessing archive.today when I was using Cloudflare DNS (1.1.1.1) That's because that web site returns bad results to Cloudflare DNS, ostensibly because they take issue with the way it handles EDNS0. The fact that it fails to work is a deliberate choice by the site operator; it isn't Cloudflare's fault.
    • johnklos 2y ago
      That's oversimplifying a bit and missing some critical information. Cloudflare wants to "protect" people from exposing even their general region. This has the side effect of making CDNs that aren't Cloudflare work worse. Cloudflare are being dicks because they do to others what they wouldn't want to be done to themselves, or what they themselves don't do to themselves. It's not even that people are choosing to opt in to Cloudflare's bullshit. If you use Firefox in the US (and many other areas, but the US for sure) and you haven't manually configured Firefox or set up a canary domain, all your DNS lookups are going to Cloudflare, and they're using that to make other CDNs work less well. That's definitely shady and definitely bad on Cloudflare's end. I'm glad some people are taking a stand.
• oliwarner 2y ago
  Cloudflare's —and most similar services'— stance here comes from these VPN funnelling not just people like you, but also attackers. It's untrustworthy traffic from their perspective. Use a VPN but use a normal network. VPN back to your home, your office. Your traffic will probably take a throughput and latency hit but it looks like real residential traffic, and that's a lot less sus.
  • Liquix 2y ago
    but then all of your traffic comes from a single IP which is eventually associated with your identity. this defeats one of the core purposes of using a VPN to circumvent surveillance capitalism.
    • oliwarner 2y ago
      I'm not saying you're wrong, but in the context of travel, I would suggest most people use the VPN because they don't trust the networks they're connecting to, more than wanting to avoid surveillance, which would apply without the travel component. I also can't think of one of the popular VPNs that get heavily advertised that I'd trust to actually protect my privacy.
• idop 2y ago
  Yes. I wrote about this on my blog six months ago [1]. CloudFlare has positioned itself as the doorman of the Internet, deciding who gets to visit shitty websites written by AIs and who doesn't. Every time I try to visit a website and get blocked by this company and its unnecessary services, I congratulate myself for avoiding yet another terrible website and move on with my life. [1] https://ido50.net/content/what-chafes-my-groin-9.html https://ido50.net/content/what-chafes-my-groin-9.html
  • squigz 2y ago
    It seems a bit shortsighted to think that CloudFlare only does this for 'shitty websites written by AIs'
    • idop 2y ago
      I thought it was obvious I was being facetious.
      • squigz 2y ago
        It wasn't to me, apologies.
  • gervwyk 2y ago
    The doorman for the internet. well said. Someone need to study how this is likely the most successful marketing campaign ever for a cloud provider.
    • theamk 2y ago
      I don't think they needed much marketing? A lot of website operators want bot/DDoS protection, and cloudfare offers service which works (at least for overwhelming majority of users), and is absolutely free. Offering free stuff which works and that many people want is how internet companies get big.
  • gradschool 2y ago
    Indeed, and cloudflare has also improved search engine effectiveness. If I'm looking for the answer to a technical question and four out of the top five hits are cloudflare captchas, the primary source is readily identifiable.
• focusedone 2y ago
  Exact same situation here. Linux, fairly funky firefox setup, eventually couldn't use half of the internet without hitting CF prompts, often wasn't able to get around them. I wound up removing / reinstalling firefox...same exact setup otherwise. No more cloudflare (or vastly fewer) prompts. The internet is usable again. Hope that helps.
  • tempaccount420 2y ago
    I had to switch to Chrome. Reinstall was not enough for me.
• inetknght 2y ago
  > I use a heavily customized Firefox config on Linux. I also use a (not-so-heavily) customized Firefox config on Linux. I also see repeated abuse of my network activity by Cloudflare.
• sphericalkat 2y ago
  I spent a few days agonizing over this same problem, and the culprit turned out to by my user-agent modifier extension.
• frereubu 2y ago
  People are focusing on your very non-standard setup, but I've experienced this - less than you to be sure - on a standard MacOS setup with Firefox and only uBlock Origin installed. If I switched to Chrome without uBlock Origin it worked. This was on the English National Ballet's ticketing website.
  • throwaway314155 2y ago
    Same problems here. Mac OS with firefox + ublock origin (and a dns based ad blocker) jams me up. Switching to Safari (with dns ad blocker still on) makes it work Has become increasingly more common in the past few months across several sites.
• therealmarv 2y ago
  I do NOT like it at all but I just want to show a way how it works with Cloudflare and to make it painless with them. Basically fully assimilating to them because Resistance is Futile ;) 1) Privacy Pass Extension Install Privacy Pass Client Extension in your browser, here for Chrome https://chromewebstore.google.com/detail/silk-privacy-pass-client/ajhmfdgkijocedmfjonnpjfojldioehi?hl=en https://chromewebstore.google.com/detail/silk-privacy-pass-c... 2) Use Cloudflare Warp (which is a VPN by Cloudflare basically, it's free): https://one.one.one.one/ https://one.one.one.one/
  • stebalien 2y ago
    The privacy pass extension still requires you to pass a cloudlare turnstile which is impossible in some browser configurations. E.g., if you disable browser performance-debugging/timing features (these used to be a vector for Spectre timing attacks).
• casenmgreen 2y ago
  Cloudflare works much, much better than Google - Google captchas for me, on Tor, are flatly impossible, always. They never let get through, no matter whether you get them right or wrong. You always get "try again". The problem I do have with CF is their captchas seem to require human interaction on the page, and this makes getting through them problematic when you open half a dozen tabs, and each loads a CF captcha, and you have to move the mouse around for ten seconds just to get the captcha to load, and loading is not reliable. Often you need to reload the page. It's this type of performance, and poor performance, which is breaking web-pages for me.
  • jeffbee 2y ago
    That sounds like a feature. Tor is for abuse, so you don't want Tor people hanging around on your page.
    • SahAssar 2y ago
      > Tor is for abuse No. Tor is for anonymization. Some might use that for abuse, but that is not it's raison d'être.
    • homebrewer 2y ago
      No it isn't. I discuss politically sensitive topics through it basically every other day, because by doing it directly in my country you will quickly end up in prison. (No, there's no scarecrow of the day involved, just discussing things you take for granted in your liberal democracy.)
  • yuumei 2y ago
    But at least with Google captchas you can use AI to solve them. I use the buster captcha extension to solve them. It moves the mouse around like a human and solves automatically. I pay for captcha solvers for hcaptcha which is worse but cloudflare is just cancer. It’s made the web unusable
  • pixelesque 2y ago
    Ehhh... maybe... Last week I had a run of (legacy) Cloudflare captchas on sites protected by CF to solve of "select all the boxes with motorcycles in", and despite doing it fastidiously and correctly (although I never know how to handle the boxes with like 3 pixels of object in but are otherwise clear), I had to do it like 5 times with different images, until suddenly it was happy.
• omgin 2y ago
  Try creating a cloudflare.com account and stay logged into it. I.e. every few days go into the cloudflare dashboard. Don't know if it will help but they use lots of methods to see if you are hostile, and being logged in and authenticated with them can't harm
• mppm 2y ago
  Amen. Another fun one is logging into bank and government sites while roaming... with sms delivered intermittently and with a 5 minute delay.
• exabrial 2y ago
  Just in time: https://doom-captcha.vercel.app https://doom-captcha.vercel.app
  • idunnoman1222 2y ago
    This is pretty tough on mobile
• deleted 2y ago
  [deleted]
• ravenstine 2y ago
  I've had to give up obfuscating my user agent because Cloudflare becomes nearly impassable as a result, and Cloudflare seems to own most web traffic now.
• tonymet 2y ago
  i recommend everyone test the web with TOR to see how dead the public internet is. Reddit won't respond. Many sites have a 10-minute captcha challenge (e.g. substack). So many sites have deployed countermeasures like Cloudflare, but they aren't actively monitoring the failure mode on those countermeasures. The web is on it's knees and these countermeasures are another nail in the coffin if we don't act fast.
  • Vampiero 2y ago
    Why would anyone care about how a site displays on TOR aside from privacy nerds? The average internet user doesn't even know what TOR is. Though they might have heard the words "Dark Web" once or twice.
    • tonymet 2y ago
      it's a canary
• nicolas_t 2y ago
  I absolutely hate cloudflare for the same reason you have. Besides traveling and using a VPN, I like in Hong Kong, a country that many sites have decided to block completely. It's very frustrating that cloudflare easily enables those kind of blanket bans for no reasons. Cloudflare is the enemy of open web.
• 7e 2y ago
  You’re using a dirty IP and not using Apple Safari, which has solved this via Private Access Tokens. Move out of the sticks.
• klntsky 2y ago
  From the other perspective, I use Cloudflare for DNS and HTTPS certificates. Having an alternative that would cover these two use cases without the need for manually running letsencrypt would be enough for me to switch. I don't want to think about HTTPS, my websites are low risk, mostly static pages (and there are tens of them).
• superasn 2y ago
  I appreciate you bringing up this issue about the Cloudflare challenges making it hard to browse. I had a similar experience where I couldn't access jsfiddle even without using a VPN. As a result, I switched to a different platform for my coding experiments. JsFiddle used to be my favorite for quickly testing out code snippets. It's a shame that due to Cloudflare hurdles, I've stopped using it and don't plan on going back. It may not be much but as more websites and businesses lose genuine web traffic like this, Cloudflare might eventually listen and fix this mess.
  • panic 2y ago
    One concrete thing we can do is to stop seeing Cloudflare as an easy, unproblematic solution. Bring up issues like this when people suggest using it.
• peanut-walrus 2y ago
  The problem is that any solution so far proposed for this is very privacy-unfriendly. For example, Google proposed https://github.com/explainers-by-googlers/Web-Environment-Integrity/blob/main/explainer.md https://github.com/explainers-by-googlers/Web-Environment-In... and this was shot down by privacy advocates (for very good reasons). So basically the choice for website operators is either to fight the bots and accept that their service will be unusable for some subset of their users or not fight the bots, which will lead to their service becoming unusable for everyone. More and more, you see services pushing you very hard towards using their app and the reason is that with the app, they are able to actually verify that you are likely not a bot (or rather, in reality, that at least the app is running on an actual physical device, mobile phone bot farms are unfortunately also a thing). As for Cloudflare - they offer it as a service, so when the website operator has a choice between using them or allocating several engineers for bot-fighting, why would they not just go with Cloudflare? Doing it yourself can be slightly higher fidelity, as you know your customers better, but it is also a lot of effort which could be better spent elsewhere.
  • devops99 2y ago
    Why "fight the bots" anyway? If software that is acting out the will of some humans somewhere is retrieving static contents, what's the big deal?
  • soerxpso 2y ago
    > either to fight the bots and accept that their service will be unusable for some subset of their users or not fight the bots, which will lead to their service becoming unusable for everyone. 2/3 of the issues OP listed would not make the service unusable for anyone if the botcheck were removed. 1. What would be the problem with allowing "bots" to opt out of receiving marketing emails? Why do I need to be a human to tell you to stop spamming me? Who is running such a bot, for what purpose? 2. What would be the problem with allowing a "bot" to log in to an already-verified human account a single time? The only situations where you actually need to confirm that a user "looks human" is for repeated connection attempts in quick enough succession to matter (DDoS prevention), or when they want to do something that someone would actually write a nefarious bot to do (mainly just creating posts/messages visible to other users).
• ryao 2y ago
  The challenges are configurable by cloudflare’s customers. The challenge can either be from turnstile, which is a captcha replacement service that websites use on their own pages, or a cloudflare CDN security setting that will block access to pages until a challenge is passed. It is not clear which one the original poster means. Cloudflare’s customers can largely disable these and rely on other means of detecting bots. In the case of turnstile, it has three modes, two of which are entirely automatic and work by interrogating the web browser, with the other requiring a client: https://developers.cloudflare.com/turnstile/concepts/widget/ https://developers.cloudflare.com/turnstile/concepts/widget/ Cloudflare CDN’s security setting on its free tier also has an essentially off setting that will basically eliminate challenges when browsers accessing pages protected by cloudflare unless there are exceptional circumstances. I believe it can be fully turned off for the enterprise tier. Whenever I configure cloudflare for a website, I always turn off challenges since they are annoying to users. There is an interesting write up about how cloudflare’s bot detection works here: https://blog.capmonster.cloud/en/blog/web-scraping1/how-cloudflare-bot-challenge-and-turnstile-protect-web-traffic https://blog.capmonster.cloud/en/blog/web-scraping1/how-clou... Note that I have yet to use turnstile, so I am speaking from documentation I read rather than from actual experience with it. I have used cloudflare’s CDN and I am speaking from experience with it. Anyway, the website author is the one that should be blamed here.
• aaron695 2y ago
  [dead]
• brunojppb 2y ago
  My workaround for this as a person who travels a lot was to buy 2 raspberry Pi’s and put them at my family houses in different countries and use Tailscale on them as exit nodes, behaving like my own VPN. The residencial IP address makes things a lot easier when connecting from random places.
• rtrgrd 2y ago
  Slightly off topic, but Microsoft ones are even worse - when I tried to sign up to OpenAI/get a new Microsoft account, the captcha were so difficult that it took me 5 minutes to solve (unsuccessfully). As a libre wolf user with very strict settings, I think privacy-aware users bear the externalities of this bot vs server arms race.
• trhway 2y ago
  well, looks like a business opportunity - a service using AI to automatically pass the challenges like this so the people like the original poster could, for a small service subscription fee, use the Internet hassle-free again.
• chrismorgan 2y ago
  AWS WAF is even worse. I recently moved from Australia to India, and quite a few high-profile websites are now completely inaccessible to me because WAF seems to be legitimately broken. Two such sites: https://officeworks.com.au/ https://officeworks.com.au/ and https://centrecom.com.au/ https://centrecom.com.au/. You successfully complete their annoying thingummy, and it redirects you… to the same Human Verification CAPTCHA. This has been the case for at least half a year, so it’s not a recent breakage. If I tunnel via my VPS which is still in Australia, then I can access it. But complete blocks via Cloudflare have also been a problem: I had to do something with VicRoads as part of selling my car, and was blocked outright when I got to the actual form page. Had I not had my VPS in Australia, I don’t know what I would have done. My IP address is massively shared (CGNAT) with plenty of botnet around, so I’m frequently troubled by Cloudflare, but not often outright blocked, and if challenged rather than blocked, I’ve never had any problem with it. Linux, Firefox.
  • bobnamob 2y ago
    As another aussie expat abroad, leaving a box behind at my parents place for an Australian residential IP has got to be one of the most unexpectedly great things I've done. Wireguard/Tailscale and my parents having access to cheap renewable power are the real enablers ofc. To anyone moving abroad in the near future - leave a box behind with your parents/close friends, it's well worth the trouble if they're ok with you occasionally mooching some bandwidth. You absolutely won't regret it
    • imhoguy 2y ago
      And don't be tempted to run any upgrades on it until you come back :)
      • bobnamob 2y ago
        Yeah, I’ll admit there’s some paranoia about losing access after a botched upgrade. I’m considering investing in a https://tinypilotkvm.com/ https://tinypilotkvm.com/, but that can wait till I’ve lost ssh at least once. I’m not hosting aws on the thing so I can afford to play it fast and loose :)
• neilv 2y ago
  Cloudflare is so embedded into so many important services (like some other companies, including Google), that they need to be thinking of their role as having some government-like responsibilities. For example, for starters, Cloudflare and Google need to find ways so that individual people who're wrongly being locked out of services by the company, have some way to get that unlocked. Not "sux2bu we dont do support bro". (Then they can start thinking about the next step, which is due process, and what it means to wrongly lock out someone in the first place.) That said, as an immediate pragmatic matter, one debugging tip with your Firefox is to go to the `about:profiles` URL, and temporarily create a new profile, and without using any Firefox sync feature, and see if Cloudflare lets you through, and then incrementally add back in your extensions and preference customizations, and see if/when CF stops letting you in. (Not that it will necessarily identify the sole and exact trigger, since they might be using scores of multiple factors, but it will be evidence of one thing that pushes it over the edge. And maybe get you to a compromise setup that lets you do your work for now.) Also helpful is to have alternate browsers installed; personally, I keep Chromium installed, as my "violate me every possible way, if you'll just let me access this one page/site I really need right now".
• 1vuio0pswjnm7 2y ago
  Try some popular user agent strings first before concluding that something else like TLS fingerprinting is the problem. Sometimes an acceptable UA string is all it takes.
• ForOldHack 2y ago
  Mess? I got a 8 try guess Ilol to try in 5 tries... in an indistinguishable font. Ooo... Im gonna fail that one... I am good at this stuff, and "Cloudflare challenges have made large portions of the web unusable for me" too.
• mrayycombi 2y ago
  Set up your own vpn on AWS ec2. It will bypass the vpn blocks they have. Problem solved.
• udev4096 2y ago
  I would recommend using FlareSolverr as a proxy in your browser to bypass the clownflare's captcha
• Havoc 2y ago
  Vaguely related, youtube is lately doing a lot of unnecessary forced logouts & reconfirm password. I'm literally on a static IP. On the same computer & browser. With the same cookies. Not accessing anything particularly sensitive. There is no way in hell they don't know precisely who I am & that its me.
  • edmundsauto 2y ago
    Is it possible your account is being attacked in the background?
    • Havoc 2y ago
      Possible I guess. It is yubikeyed though so would need to be a pretty sophisticated attack
• demaga 2y ago
  Now every couple of minutes when scrolling through Reddit, red "Network issue" tab appears. Some comments don't load at all, some are labeled "deleted" even though they aren't. Refreshing the page usually does the trick, but I hate this new experience. I guess they're just protecting themselves from bots, and I look like a bot in their eyes.
• prmoustache 2y ago
  I get this all the time with firefox on linux + ublock origin extension. Often ending up with that blocked ip page. I mostly shrug off and just avoid visiting that kind of sites again. For an unsubscribe challenge I just copy paste the url and visit it using firefox focus on my smartphone on my mobile connection.
• kittikitti 2y ago
  Only the expensive bots with residential IP's and mechanical turks can survive, humans be damned.
• jvaleski2 2y ago
  CF issue, or site programming issue.?
• Spooky23 2y ago
  Stop acting shady.
• johnklos 2y ago
  Cloudflare puts challenges on their abuse contact page and rate limits it to much slower than human speed. It's also still broken after years in that you can't report abusers who register domains through Cloudflare and/or host their DNS using Cloudflare. They really don't want feedback from people who don't pay them.
• citrin_ru 2y ago
  Just tried to disable User-Agent in Firefox by setting general.useragent.override to an empty string and Cloudflare captcha become impenetrable. Cloudflare actively blocks attempts to improve privacy :(
  • bluGill 2y ago
    You don't want that to be an empty string - you would be one of the few people in the world with that value and thus easy to identify. You instead want that string to be exactly the same as everyone else in the world (no matter what the real User-Agent really is). there should be about 50 different contents of the entire header possible for everyone in the world.
• 1oooqooq 2y ago
  you're not welcome. is their message. not a single mention of advertising on all these comments. those captcha are not against bots. bots are only one item in the broader category they block. you, an unmonetizable user, is another. cloudflare et all have the "marketplace conundrum". they need to provide value to both sides, and for the site they do this by blocking hard to monetize traffic. that means traffic that won't generate high yield on ad networks those sites care about.
• superkuh 2y ago
  It used to be just for profit companies web dev's ignorantly putting themselves behind default cloudflare deploys and blocking everyone. But now big academic players like science.org/aaas elsevier and other publishers and individual journals are and I can't even read scientific papers anymore. Even sillier is the RSS/Atom feeds science.org ran have the same cloudflare rules so all actual feed readers were blocked (support told me only real feed readers as a service like Feedly corporation were allowed). It took me months of email back and forth to get them to realize their error and get to someone who could fix it. And that is what I consider a good response. Most just ignore the email.
• PrimaryAlibi 2y ago
  Everyone reading this should start to contact websites/companies who use cloudflare and tell them in simple and few words that it's a problem and link them to a video or article that explains more, maybe even to this HN topic. We are not many, maybe 1-2% of their users/customers I keep reading people saying but I have in the past been able to get big tech companies to change to a friendlier tech. You would be surprised how effective it is to contact them about it. Maybe they have a tech support who already has same opinion as you but they can't make any change until a customer makes a complaint about it, then they happily see it as their opporunity to finally make a change.
• Froedlich 2y ago
  Cloudflare challenges seem to be becoming more and more frequent on my general internet use. Yep, "Cloudflare loop" is a thing. No, I'm not going to download and install a different web browser, dump all my cookies, or whatever other nonsensical "solution" they recommend. I've become to hate Cloudflare with a seething passion.
• zoezoezoezoe 2y ago
  yes yes yes yes yes yes yes. I nearly wrote a borderline hit piece of cloudflare challenges because of this bullshit, but instead I gave into their games and repealed my privacy (only for niche cases mostly), likewise there's no solution for me either and it's just, like you said, some other variant of "too bad, so sad".