9 ms·

You're collateral damage in the web's war against bots :( Unfortunately, I think the Cloudflare challenges are designed to filter out users similar to your pro

You're collateral damage in the web's war against bots :( Unfortunately, I think the Cloudflare challenges are designed to filter out users similar to your profile... once you stray far enough from the norm, it just looks like a bot / suspicious traffic to them. Statistically there's not enough users like you (privacy-conscious Linux users on nonstandard browsers) for them to really care enough to do anything about it. Site owners don't care either since you're usually like 1-2% of users at most, and typically also the same ones who block ads, etc., so they don't mind blocking you... it's sad, but I don't think there is really anything you can do about it except conform. It's an ongoing arms race and you're caught in the middle.

blakeashleyjr 2y ago
While you hit the nail on the head, I am still surprised that so many tools targeted at people like me (web hosting, developer tools, etc.) are protected that way.
- kauegimenes 2y ago
  Its not only about protection, most web developers would use Cloudflare since its a free CDN and would increase the app load time considerably.
  - chrisweekly 2y ago
    increase -> decrease
    - rkagerer 2y ago
      Except for those encountering that dreaded captcha.
  - solardev 2y ago
    You can separately configure (to a large degree) the caching vs protection features, though.
- warkdarrior 2y ago
  Because if such hosting and developer tools are not protected against bots, the tools end up used for phishing, spamming, etc.
- rad_gruchalski 2y ago
  They are not targeting people like you. Bots are the target. If you look like a bot, how are they going to distinguish?
  - KronisLV 2y ago
    > If you look like a bot, how are they going to distinguish? Some non-existant system of attesting that I'm person X (possibly through an e-ID card) who has issued a client certificate Y (cert chain, using my e-ID cert to sign) to be used with my device Z (presumably with a device fingerprint or IP range attached to the cert). Of course, this would mean no privacy, but that's not that different from being signed in through Google as an identity provider, we'd just shift the mechanism to be universal (like client certs already are). One of the options that would take more coordination than will probably happen (though very similar to some e-signature solutions in EU, which we already use) but I could see using something like that for a variety of professional/service sites, since signing in with the e-ID card directly is already a thing on some sites here (government sites, banking sites, utilities sites).
    - rad_gruchalski 2y ago
      Okay. Do that globally. And solve the ddos problem as you’re on it. If you add transparent tls termination, edge, caching, dns… maybe I’ll have a look! I had a guy like that working with me. Blocked every possible tracker, disabled javascript, used some niche browser, proton mail, and then complains that google doesn’t allow him to sign in. I get it, privacy and what not. But the guy was an outlier. Some random blogs, product pages aren’t gov, most likely have no way to opt-in for gov eID (maybe they aren’t based in the EU), and they only care that their service is available fast globally and that they get ddos protection for free (plus some other convenience features).
      - KronisLV 2y ago
        > Do that globally. We already do a simpler version of that with TLS and HTTPS, there are globally trusted root certs that ship with most OSes and browsers. It's just that we haven't extended the same approach to client certs and identity verification, instead having a bunch of walled gardens and governments running legacy methods of figuring out who someone is, as opposed to various eID mechanisms. If I trust news.ycombinator.com because I trust ISRG Root X1, I might similarly trust John Doe's iPhone because I trust the government of France's CA, as a hypothetical, as long as the certification chain is valid there. It's a problem that's technically solvable (say, in 20-50 years), but won't get done because good luck getting a bunch of governments to collaborate on that across the world. It's actually a surprise that we have TLS in the first place.
- luckylion 2y ago
  I'm convinced that's mostly incompetence on the side of the companies that implement that protection. "We have a problem with bots" - "Just create a firewall rule, whatever"
  - rustc 2y ago
    What other way would you suggest to protect a free service from bots? Cloudflare is often the easiest to implement and has a generous limit on their free plan.
    - luckylion 2y ago
      Oh, they absolutely are, I don't disagree -- I use them too. But the immediate response to bots shouldn't be "make everyone go through a captcha". There's lots of nuance that you can tune to deal with your particular situation, but the first thing I'd do is block known bots or ASNs, set up a limit to trigger (bots usually don't make 1 document request a minute), set up higher limits for users who (seem to) have a valid cookie indicating that they are logged in, set up different thresholds for certain countries that are more risky etc etc. What you need to protect your service depends on your situation, it's not a one-size-fits-all solution. E.g. I find that I have no automated contact form spam once I add a simple JS to add some data that isn't standard, but I'm sure that wouldn't hold up if there was enough incentive to try to get past it. But the OP mentioned not just free services, but e.g. webhosting logins. That's just sad, as is Cloudflare's community being behind an aggressive captcha. I'm a user, I'm logged in, I've posted before, I'm in good standing, yet when I go there, I need to solve a captcha. When I then go there again an hour later, guess what, another captcha. Either there's another reason I'm not seeing or it's just lazyness as in "we need to have a forum but we really don't want to spend any resources on it, just put up an aggressive captcha that'll filter out most bots and everyone but the determined users".
      - hombre_fatal 2y ago
        Fwiw, Cloudflare does do a multivariate confidence check which is why it has multiple tiers: no captcha, a one-click captcha, the annoying puzzle captcha once, the annoying puzzle captcha six times in a row. > I'm a user, I'm logged in, I've posted before, I'm in good standing, yet when I go there, I need to solve a captcha. Though consider the fact that taking over someone's account shouldn't give you (a spammer) unlimited access either. The spambots you see on Twitter are mostly cred-stuffed accounts. It's a hard problem. Existing accounts are more dangerous than fresh accounts. Imo, "write your own password" should be a thing of the past. Services should just auto-gen a password or there should be a way to require the OS (like a password manager) to generate one to avoid cred-stuffing. We're letting down the average person by making them come up with unique passwords for every service instead of just helping them. Though I'm way off topic.
- solardev 2y ago
  Most developers I've met were actually similarly lazy... we just use Chrome on Mac, and don't really want to deal with VPNs unless our employers force us to. The last few Firefox holdouts also switched after running into various WebGL/Canvas/etc issues. The same attitude that leads us to focus on "happy path" users and ignore edge cases often also causes us to sheeple into that same basic dev group. Long gone are the days where most devs custom build Linux boxen from scratch and compile custom kernels to our liking... Anyway, I know the "Cloudflare's monopoly gating is killing web openness!" meme is common online, especially on HN, but in real life I've never actually heard anyone else complain about it (either a fellow dev or a customer or a manager). Instead, it's been universal praise for the actual issues Cloudflare exists to solve (CDN, bot protection, serverless, etc)... they are a godsend for small businesses that otherwise get immediately flooded by spam requests, especially from China, Russia, and India. And if you think Cloudflare is bad, it was even worse before they became dominant, with terrible services like Incapsula/Imperva charging way more but providing both worse bot protection AND more false positives, or the really hard early reCAPTCHAs (that Cloudflare was largely able to replace, for users who DO fit within the "norm"). That, or you'd have to fight every random sysadmin with their own lazy rules, like firewall rules that blacklisted entire regional ISPs and took weeks or months to resolve, if they ever even checked their emails. As inconvenient as Cloudflare is for users who take privacy seriously and try to be less trackable, for the other 90% of us who don't care as much and easily fit into their "norm" model, it's much nicer than what came before. Site downtime and slowness are also much less common now, in no small part because of their easy CDN and caching. From the implementation side, I've set up a few Cloudflare accounts in my career, but do take the time to try to configure it to balance security vs accessibility for any given target audience. Sometimes we'd block entire countries, other times we'd minimize security to ensure maximum reach, but usually we'd customize rulesets in the middle for any given company & audience. I never got a complaint about it (our emails were still available and not blocked). This was always a direct response to some business need, usually spambots or DDoS attempts that fail2ban etc. couldn't catch well enough. For the business, it was usually a "shit, our website is down again, what is it this time", and the choice between "for free or $20 we can get it back up again and not have this issue anymore" or "we can spend thousands of dollars and weeks of labor building our own security solution" is pretty easy. "What about that one guy who is proxied behind TOR and three VPNs with a random user agent using a text-only browser he wrote himself?" never really factors into that process =/ There's just not enough users like that out in the wild vs the very real constant threat of bots and malware. It's a shitty situation that the web is like this today, and I wish it weren't the case, but it really is an arms race, and these imperfect weapons are just what most of us have access to...
EGreg 2y ago
I honestly don't see what's so hard about a bot simulating "the norm" within the margin of error. This cat-and-mouse game is just like a GAN, the end result is indistinguishable even by a bot.
- nullc 2y ago
  Bot authors are lazy and won't until they have to.. once you do, you can then pretend they aren't bots and include them in the engagement numbers you feed prospective shareholders.
  - tokioyoyo 2y ago
    Agreed. From my past experiences though, a very good chunk of them will give up once there is a resistance. Basically, you want your bot protection to just be a little better than your competitor. Then the bot author will target them instead, because of the path of least resistance.
    - EGreg 2y ago
      Outrun the friend not the bear? Hehe
- viraptor 2y ago
  It depends on the defences. It starts trivial - just make a http request. Then there's http version, user agent header, other headers, header ordering, cookies, TLS ciphers, session resolution, timing, behaviour for page resources, ... and so many other things. It takes time, even if you order headless chrome.
shiomiru 2y ago
The sad part is that it's trivial to get around CF's bot protection if you're writing a bot (just use curl-impersonate and buy residential IPs), but it's pretty much impossible to bypass as a human if their magical black box doesn't like your browser and/or IP address.
- solardev 2y ago
  How does it get around captchas?
  - gjsman-1000 2y ago
    You pay contract workers in a third world country a tiny amount of money per day, to spend all day clicking boxes.
  - tedivm 2y ago
    If they don't think you're suspicious they don't make you do the captchas, and as others have mentioned you can always outsource it to captcha farms. There are also AI models which do a fairly decent amount, and since most captchas let you repeat attempts with new patterns you can have a pretty high error rate to get past them. Then there's the ADA, which requires accessibility- many captchas have an audio component as a backup and those are easy to interpret by models.
  - gruez 2y ago
    Cloudflare turnstile isn't even a captcha. The user just has to tick a box. Behind the scenes there's a javascript challenge to make sure you're vaguely a browser and not some script a bazillion requests per minute.
    - xdfgh1112 2y ago
      It's also used for proof of work as many scrapers are using thousands of IPs but only a few CPUs
  - michaelmior 2y ago
    curl-impersonate doesn't solve CAPTCHAs, but the goal is to look enough like a human that Cloudflare doesn't present a CAPTCHA in the first place.
modzu 2y ago
it is discrimination
- jdironman 2y ago
  Only if enough are discriminated against that it affects the bottom line.
devops99 2y ago
[dead]
winrid 2y ago
I don't buy this because using Chrome is what most bots probably do right? Headless chrome is easy.
kevincox 2y ago
> Site owners don't care either since you're usually like 1-2% of users at most, and typically also the same ones who block ads, etc., so they don't mind blocking you I do believe that it is true that many site owners wouldn't care. But I suspect that in the vast majority of cases they don't actually know. Cloudflare probably shows them a nice dashboard about all of these blocked "threats" and they don't know better than to question it.

blakeashleyjr 2y ago

While you hit the nail on the head, I am still surprised that so many tools targeted at people like me (web hosting, developer tools, etc.) are protected that way.

kauegimenes 2y ago
Its not only about protection, most web developers would use Cloudflare since its a free CDN and would increase the app load time considerably.
- chrisweekly 2y ago
  increase -> decrease
  - rkagerer 2y ago
    Except for those encountering that dreaded captcha.
- solardev 2y ago
  You can separately configure (to a large degree) the caching vs protection features, though.
warkdarrior 2y ago
Because if such hosting and developer tools are not protected against bots, the tools end up used for phishing, spamming, etc.
rad_gruchalski 2y ago
They are not targeting people like you. Bots are the target. If you look like a bot, how are they going to distinguish?
- KronisLV 2y ago
  > If you look like a bot, how are they going to distinguish? Some non-existant system of attesting that I'm person X (possibly through an e-ID card) who has issued a client certificate Y (cert chain, using my e-ID cert to sign) to be used with my device Z (presumably with a device fingerprint or IP range attached to the cert). Of course, this would mean no privacy, but that's not that different from being signed in through Google as an identity provider, we'd just shift the mechanism to be universal (like client certs already are). One of the options that would take more coordination than will probably happen (though very similar to some e-signature solutions in EU, which we already use) but I could see using something like that for a variety of professional/service sites, since signing in with the e-ID card directly is already a thing on some sites here (government sites, banking sites, utilities sites).
  - rad_gruchalski 2y ago
    Okay. Do that globally. And solve the ddos problem as you’re on it. If you add transparent tls termination, edge, caching, dns… maybe I’ll have a look! I had a guy like that working with me. Blocked every possible tracker, disabled javascript, used some niche browser, proton mail, and then complains that google doesn’t allow him to sign in. I get it, privacy and what not. But the guy was an outlier. Some random blogs, product pages aren’t gov, most likely have no way to opt-in for gov eID (maybe they aren’t based in the EU), and they only care that their service is available fast globally and that they get ddos protection for free (plus some other convenience features).
    - KronisLV 2y ago
      > Do that globally. We already do a simpler version of that with TLS and HTTPS, there are globally trusted root certs that ship with most OSes and browsers. It's just that we haven't extended the same approach to client certs and identity verification, instead having a bunch of walled gardens and governments running legacy methods of figuring out who someone is, as opposed to various eID mechanisms. If I trust news.ycombinator.com because I trust ISRG Root X1, I might similarly trust John Doe's iPhone because I trust the government of France's CA, as a hypothetical, as long as the certification chain is valid there. It's a problem that's technically solvable (say, in 20-50 years), but won't get done because good luck getting a bunch of governments to collaborate on that across the world. It's actually a surprise that we have TLS in the first place.
luckylion 2y ago
I'm convinced that's mostly incompetence on the side of the companies that implement that protection. "We have a problem with bots" - "Just create a firewall rule, whatever"
- rustc 2y ago
  What other way would you suggest to protect a free service from bots? Cloudflare is often the easiest to implement and has a generous limit on their free plan.
  - luckylion 2y ago
    Oh, they absolutely are, I don't disagree -- I use them too. But the immediate response to bots shouldn't be "make everyone go through a captcha". There's lots of nuance that you can tune to deal with your particular situation, but the first thing I'd do is block known bots or ASNs, set up a limit to trigger (bots usually don't make 1 document request a minute), set up higher limits for users who (seem to) have a valid cookie indicating that they are logged in, set up different thresholds for certain countries that are more risky etc etc. What you need to protect your service depends on your situation, it's not a one-size-fits-all solution. E.g. I find that I have no automated contact form spam once I add a simple JS to add some data that isn't standard, but I'm sure that wouldn't hold up if there was enough incentive to try to get past it. But the OP mentioned not just free services, but e.g. webhosting logins. That's just sad, as is Cloudflare's community being behind an aggressive captcha. I'm a user, I'm logged in, I've posted before, I'm in good standing, yet when I go there, I need to solve a captcha. When I then go there again an hour later, guess what, another captcha. Either there's another reason I'm not seeing or it's just lazyness as in "we need to have a forum but we really don't want to spend any resources on it, just put up an aggressive captcha that'll filter out most bots and everyone but the determined users".
    - hombre_fatal 2y ago
      Fwiw, Cloudflare does do a multivariate confidence check which is why it has multiple tiers: no captcha, a one-click captcha, the annoying puzzle captcha once, the annoying puzzle captcha six times in a row. > I'm a user, I'm logged in, I've posted before, I'm in good standing, yet when I go there, I need to solve a captcha. Though consider the fact that taking over someone's account shouldn't give you (a spammer) unlimited access either. The spambots you see on Twitter are mostly cred-stuffed accounts. It's a hard problem. Existing accounts are more dangerous than fresh accounts. Imo, "write your own password" should be a thing of the past. Services should just auto-gen a password or there should be a way to require the OS (like a password manager) to generate one to avoid cred-stuffing. We're letting down the average person by making them come up with unique passwords for every service instead of just helping them. Though I'm way off topic.
solardev 2y ago
Most developers I've met were actually similarly lazy... we just use Chrome on Mac, and don't really want to deal with VPNs unless our employers force us to. The last few Firefox holdouts also switched after running into various WebGL/Canvas/etc issues. The same attitude that leads us to focus on "happy path" users and ignore edge cases often also causes us to sheeple into that same basic dev group. Long gone are the days where most devs custom build Linux boxen from scratch and compile custom kernels to our liking... Anyway, I know the "Cloudflare's monopoly gating is killing web openness!" meme is common online, especially on HN, but in real life I've never actually heard anyone else complain about it (either a fellow dev or a customer or a manager). Instead, it's been universal praise for the actual issues Cloudflare exists to solve (CDN, bot protection, serverless, etc)... they are a godsend for small businesses that otherwise get immediately flooded by spam requests, especially from China, Russia, and India. And if you think Cloudflare is bad, it was even worse before they became dominant, with terrible services like Incapsula/Imperva charging way more but providing both worse bot protection AND more false positives, or the really hard early reCAPTCHAs (that Cloudflare was largely able to replace, for users who DO fit within the "norm"). That, or you'd have to fight every random sysadmin with their own lazy rules, like firewall rules that blacklisted entire regional ISPs and took weeks or months to resolve, if they ever even checked their emails. As inconvenient as Cloudflare is for users who take privacy seriously and try to be less trackable, for the other 90% of us who don't care as much and easily fit into their "norm" model, it's much nicer than what came before. Site downtime and slowness are also much less common now, in no small part because of their easy CDN and caching. From the implementation side, I've set up a few Cloudflare accounts in my career, but do take the time to try to configure it to balance security vs accessibility for any given target audience. Sometimes we'd block entire countries, other times we'd minimize security to ensure maximum reach, but usually we'd customize rulesets in the middle for any given company & audience. I never got a complaint about it (our emails were still available and not blocked). This was always a direct response to some business need, usually spambots or DDoS attempts that fail2ban etc. couldn't catch well enough. For the business, it was usually a "shit, our website is down again, what is it this time", and the choice between "for free or $20 we can get it back up again and not have this issue anymore" or "we can spend thousands of dollars and weeks of labor building our own security solution" is pretty easy. "What about that one guy who is proxied behind TOR and three VPNs with a random user agent using a text-only browser he wrote himself?" never really factors into that process =/ There's just not enough users like that out in the wild vs the very real constant threat of bots and malware. It's a shitty situation that the web is like this today, and I wish it weren't the case, but it really is an arms race, and these imperfect weapons are just what most of us have access to...

EGreg 2y ago

I honestly don't see what's so hard about a bot simulating "the norm" within the margin of error. This cat-and-mouse game is just like a GAN, the end result is indistinguishable even by a bot.

nullc 2y ago
Bot authors are lazy and won't until they have to.. once you do, you can then pretend they aren't bots and include them in the engagement numbers you feed prospective shareholders.
- tokioyoyo 2y ago
  Agreed. From my past experiences though, a very good chunk of them will give up once there is a resistance. Basically, you want your bot protection to just be a little better than your competitor. Then the bot author will target them instead, because of the path of least resistance.
  - EGreg 2y ago
    Outrun the friend not the bear? Hehe
viraptor 2y ago
It depends on the defences. It starts trivial - just make a http request. Then there's http version, user agent header, other headers, header ordering, cookies, TLS ciphers, session resolution, timing, behaviour for page resources, ... and so many other things. It takes time, even if you order headless chrome.

shiomiru 2y ago

The sad part is that it's trivial to get around CF's bot protection if you're writing a bot (just use curl-impersonate and buy residential IPs), but it's pretty much impossible to bypass as a human if their magical black box doesn't like your browser and/or IP address.

solardev 2y ago
How does it get around captchas?
- gjsman-1000 2y ago
  You pay contract workers in a third world country a tiny amount of money per day, to spend all day clicking boxes.
- tedivm 2y ago
  If they don't think you're suspicious they don't make you do the captchas, and as others have mentioned you can always outsource it to captcha farms. There are also AI models which do a fairly decent amount, and since most captchas let you repeat attempts with new patterns you can have a pretty high error rate to get past them. Then there's the ADA, which requires accessibility- many captchas have an audio component as a backup and those are easy to interpret by models.
- gruez 2y ago
  Cloudflare turnstile isn't even a captcha. The user just has to tick a box. Behind the scenes there's a javascript challenge to make sure you're vaguely a browser and not some script a bazillion requests per minute.
  - xdfgh1112 2y ago
    It's also used for proof of work as many scrapers are using thousands of IPs but only a few CPUs
- michaelmior 2y ago
  curl-impersonate doesn't solve CAPTCHAs, but the goal is to look enough like a human that Cloudflare doesn't present a CAPTCHA in the first place.

modzu 2y ago

it is discrimination

jdironman 2y ago
Only if enough are discriminated against that it affects the bottom line.

devops99 2y ago

[dead]

winrid 2y ago

I don't buy this because using Chrome is what most bots probably do right? Headless chrome is easy.

kevincox 2y ago

> Site owners don't care either since you're usually like 1-2% of users at most, and typically also the same ones who block ads, etc., so they don't mind blocking you I do believe that it is true that many site owners wouldn't care. But I suspect that in the vast majority of cases they don't actually know. Cloudflare probably shows them a nice dashboard about all of these blocked "threats" and they don't know better than to question it.