12 ms·
How to bypass Cloudflare bot protection
- luckylion 6y agoAs mentioned in the article, it's easy to mitigate by not only checking that the request is arriving at your endpoint from an official CF IP, but to also check that it does not contain a CF-Worker header (or contains one with your domain). The limits on workers' sub requests (e.g. requests are done by host name only, can't use IPs with a different host name) make it somewhat cumbersome to set up reverse proxy workers (which was my use case), but the tech itself is just great, works very reliably and the delay, while noticeable, is tiny and never bothered us.
- jeroenhd 6y agoThat's a nice trick, with some automation you should be able to register new domain names and Cloudflare accounts to get past the 100k limit as well. Cloudflare's anti bot tooling is terribly annoying while using Tor. I understand why they do it, though. Using Tor is a good way to show how terrifyingly much control Cloudflare has over the internet.
- jchw 6y agoNot to detract from the point but I find Cloudflare to be on the more reasonable side when it comes to blocking Tor. A lot of sites end up effectively banning all Tor IPs, Cloudflare merely requires CAPTCHA. And they do offer Privacy Pass as a sort of approach to make it a little less annoying.
- capableweb 6y ago> Cloudflare merely requires CAPTCHA I do agree that Privacy Pass is in theory a good idea, although to say that CloudFlare "merely" requires captchas to be filled out is a bit disingenuous. You're often required to complete 5-10 captchas, and sometimes even after that, you get denied. Had that happen to me multiple times.
- jchw 6y agoI don’t use Tor for all of my traffic, but sometimes I do use it just because I can. And my experience lately is that I usually get hCaptcha and I usually pass it once and get in. It’s Recaptcha that is a serious problem.
- croutonwagon 6y agoGoogle owned domains, namely Youtube, are even worse. They often redirect to different domains for blocks so just changing the circuit does nothing.
- akalsz 6y agoYeah after years of suffering from reCAPTCHA I'm somewhat thankful for hCaptcha. It is still annoying (especially Cloudflare's integration which seems barely compatible with the Tor Browser's cookie and circuit management), but at least I don't have to switch exit nodes twenty times just to have a chance of my solution being accepted (like with reCAPTCHA).
- mikequinlan 6y agoIf you can't pass a CAPTCHA, you need to stop and ask yourself -- are you really a human being, or have you just been programed to believe that you are?
- capableweb 6y agoActually a good point as I never even considered that. Maybe I'm the one who's at wrong here? Seemingly I am passing the captchas, they are not giving me any errors. Simply seems to accept my input, reload the page and ask me to do more.
- iaml 6y agoNot a valid question when google literally has a patent on unsolvable captcha. https://patents.google.com/patent/US9407661 https://patents.google.com/patent/US9407661
- grishka 6y agoDiscriminating IP addresses is a terrible idea to begin with. You're breaking the internet by doing it. Please treat all IPs equally.
- pepemon 6y agoNo, we won't. Many of those IPs are the origins for spam, botnets, crawlers, DDoS, exploitation engines, etc. The only feasible solution even with all modern heuristics and ML is still good ol' IP scoring and/or banning.
- catblast01 6y agoIt’s just silly to suggest this when quite simply not all IPs behave equally and the differences are not subtle. It’s annoying when “bad” IPs get reassigned and change, but that is a relatively infrequent event - and certainly CloudFlare is at least adaptive. It’s just not a principal aligned with reality.
- grishka 6y agoThe problem that no one seems to acknowledge: one IP doesn't necessarily mean one person or entity behind it. Residential ISPs often use a single public IP as an exit point for many subscribers simultaneously. So if one, just one, of these subscribers does something that our internet overlord Cloudflare considers nefarious, everyone else sharing this IP gets punished for doing nothing wrong.
- jbluepolarbear 6y agoThis specific problem happens all the time. That’s the ISPs problem, not cloudflare. The ISP has a user negatively impacting other users, best course of action would be remove that user and notify cloudflare after.
- grishka 6y agoDo ISPs actually do any of this? Does any ISP ever do any of this? I don't think so. Also, no one knows what was the particular thing that pissed off Cloudflare, because it's a black box. I'll say it again: centralizing the internet around a single company is a terrible idea. Especially so when said company terminates TLS and thus has access to unencrypted traffic.
- ffpip 6y ago> Cloudflare merely requires CAPTCHA. Cloudflare doesn't "require" captchas for TOR users. By default, they treat TOR IPs it like any other IP. However since a lot of traffic comes out of a TOR IP, it looks like bot traffic
- Sebb767 6y agoThey at least monitor TOR IPs. See this blog post [0] from 2016. https://blog.cloudflare.com/the-trouble-with-tor/ https://blog.cloudflare.com/the-trouble-with-tor/
- t0astbread 6y agoAs far as I know, Cloudflare transparently redirects some Tor users to their own hidden services and then uses the circuit ID for rate limiting: https://blog.cloudflare.com/cloudflare-onion-service/ https://blog.cloudflare.com/cloudflare-onion-service/ "Some" because they seem to have internal heuristics for detecting Tor Browser users before they enable that feature for a connection, so it doesn't apply to all connections made through Tor. YMMV I guess but I haven't seen a Cloudflare CAPTCHA over Tor in a long time.
- akalsz 6y agoI still wonder how that detection thing works. My custom Firefox setup with requests proxied through Tor passes as the TBB, but copying the same request as a curl command somehow doesn't.
- t0astbread 6y agoYeah, it's pretty weird and apparently it's being updated as well. Like, a year ago or so I wrote a browser extension to trick that detection mechanism when I'm on stock Firefox. Just when writing that comment I discovered something has changed and that's not needed anymore.
- m-p-3 6y agoAnd I found hCaptcha to be way less annoying than ReCaptcha when using Tor.
- rattray 6y agoI hadn't heard of "Privacy Pass" before. https://blog.cloudflare.com/cloudflare-supports-privacy-pass/ https://blog.cloudflare.com/cloudflare-supports-privacy-pass... Sounds like it allows you to browse without tracking in a way that more reliably signals you're a human, which seems very useful. Why would a human browsing the internet with Tor not want to use Privacy Pass?
- dotancohen 6y ago> And they do offer Privacy Pass as a sort of > approach to make it a little less annoying. The Privacy Pass extension requires, as it mentions when installing it, access to all the data on all websites that you visit. So in order to stop getting so many Captchas, one has to give this company unfettered access to all sites that one visits? Why even bother with an HTTPS connection when the browser is potentially leaking the information away with explicit user consent? And I don't even use Tor. But every few weeks I'll have a two or three day spat where every webpage that I visit requires at least two captchas to be solved. Maybe another user from my ISP is scrapping, but I'm a good citizen on the 'net.
- deanclatworthy 6y agoTry signing up to gitlab and then signing in. It’s completely broke with TOR because of cloudflares crappy redirect and nobody has done anything about it. The captcha also breaks after you spend about thirty mins trying to find a relay which doesn’t trigger the redirect page protection loop.
- xwdv 6y agoAlso worth noting it’s a good time to buy Cloudflare stock, it’s down from recent highs and may be on its way back up in the next month. This is a good long term hold.
- flas9sd 6y agonot only Tor, but other network resources that are legit, but can be abused because of their no-cost nature like Hurricane Electrics ipv6-in-ipv4 tunnel broker. Not sure how I feel about cloudflares role as arbiter. I think instead of relying on a service, a good nginx module that has ratelimiting abilities easily surfaced with some grasp of ASN origin could satisfy the need of most sites out there.
- hedora 6y ago> I understand why they do it Care to elaborate? Why block access to static pages by default? Tor’s aggregate bandwidth is tiny compared to CF’s aggregate bandwidth, so DOS attacks surely aren’t the reason.
- jchw 6y agoIt’s not bandwidth/DoS attacks, but it is malicious traffic. Even for websites that don’t explicitly block Tor exit nodes, they quickly end up banning most Tor exit nodes just by virtue of blocking malicious traffic that crops up. Malicious traffic takes many forms. As an attacker trying to attack a website, you are probably not going to come in from a residential IP address, unless it’s one you have access to illegally. It is much more likely you will use Tor. And because of the nature of Tor, you literally, full stop, cannot ban individual Tor users to any meaningful degree. So if you offer services not requiring sign up, or services where the sign up is unobtrusive, users abusing the service via Tor are difficult to stop. Of course there are counter points... such as [1]. But let’s be honest, from the perspective of a tired sysadmin trying to stop ongoing vandalism, your options are limited, especially if you don’t want an intrusive sign up option. As a counterpoint to the counterpoint, if Tor users were treated like any other user malicious parties might use the Tor network even more often than they already do. Edit: also, though it is not necessarily malicious per-se, there are some types of traffic like scraping that may be undesirable from the PoV of the website. I am not going to make a value judgment regarding the merit of blocking scrapers, but it is nonetheless a goal where you can see the reasoning behind blanket banning Tor. [1]: https://blog.torproject.org/the-value-of-anonymous-contributions-wikipedia https://blog.torproject.org/the-value-of-anonymous-contribut...
- BlueTemplar 6y agoSearch engines use scraping. I would guess that most websites want to be featured on the search engines?
- jchw 6y agoNot all websites want to be scraped even by search engines, and some only want to be scraped by search engines. Unwanted scrapers may refuse to follow robots.txt or do things with the scraped data that is either undesired to their business model or possibly even illegal, though that is complex. The point is, for better or worse, as far as I know, websites have the right to block scrapers while allowing search engines, and as long as it remains viable to prevent scraping, it remains likely that sites intending to do this will end up blocking or obstructing Tor traffic. Is it a good thing? I would argue it’s too abstract to judge in such a generalized fashion. Whether or not it is a good thing or a bad thing, though, is not important. It is the world we live in right now.
- foolmeonce 6y agoI'd wondered about the openness of the workers do connect to unrelated resources myself.. But I agree with CF, they don't inherently have a security problem. They have the normal situation of a proxy again. If a lot of new domains show up to bot proxy then the Baysian for an unknown worker domain is going to become bot..
- oefrha 6y ago> As you can see, X-FORWARDED-FOR can be set to an arbitrary value, so you can bypass IP limitation requests during scrapping or more dangerous IP verification during a login procedure … The origin IP is not forwarded to the website, so the only way to block this kind of request on your server is to filter on CF-WORKER header. X-Forwarded-For should always be treated as forged at least for security purposes (except the hops set by reverse proxies under your control). If you do IP whitelisting based on untrusted X-Forwarded-For you’re doing it terribly wrong.
- daurnimator 6y agoA lot of people just verify that X-Forwarded-For matches a Cloudflare-owned IP. As this post demonstrates, you need to verify against the cloudflare origin pull CA (see https://support.cloudflare.com/hc/en-us/articles/204899617-Authenticated-Origin-Pulls https://support.cloudflare.com/hc/en-us/articles/204899617-A... )
- tgsovlerkhgsel 6y agoI suspect that the idea here is that XFF is supposed to be trusted, because it is supposed to come from Cloudflare.
- userbinator 6y agoFor a very long time, XFF-spoofing was a trick to get past journal subscriptions and other paywalls. I don't feel so guilty about disclosing this here now, as a lot of them have (unfortunately) blocked that route. No doubt a few popular bookz/journalz sites owe at least some of their content to this technique. But it's something to try and might still work on a few of them...
- RiceAndWaters 6y agoHey, I saw one of your past comments where you mentioned being (briefly) active in the bookz scene in the past. I really want to know more about this scene. If it's not too much work for you, please contact me at Bibliohoarding@tutanota.com Thank you!
- xPaw 6y agoSo is there any reason not to block any requests on your server if they contain the CF-WORKER header?
- oefrha 6y agoI’ve built legit CORS proxies for web-based community tools using CF Workers when the first party doesn’t set CORS headers.
- minitech 6y agoYep – any legitimate tool that makes requests on behalf of a user could do so using Cloudflare workers. (A webpage translation service, for example.) It’s the same as any other provider. You could block based on the value of the CF-Worker header if you find it’s being abused.
- minitech 6y agoThis just sounds like a way to make a proxy with Cloudflare workers, which aren’t considered by Cloudflare’s filters to be as suspicious as Tor yet. You could do the same with any new VPS/cloud function provider. Am I missing some actual bypass? If the X-Forwarded-For value from the request is forwarded by Cloudflare’s proxies unaltered, giving special privileges to Cloudflare workers, that’s a problem, but it’s not clear whether “a site who display your headers” is protected by Cloudflare and displaying an unaltered X-Forwarded-For. I don’t expect Cloudflare to have made this oversight, and the e-mail response from them matches: “All workers subrequests go through Cloudflare again, and therefore you won’t be able to bypass any restrictions directly.”
- spzb 6y ago> Am I missing some actual bypass? No, you're not. Which is why Cloudflare were correct to say this isn't a security issue.
- bassdropvroom 6y ago> You could do the same with any new VPS/cloud function provider. I'm not so sure this is true. Not long ago Cloudflare's CEO was making fun of people trying (and failing) to bypass Cloudflare's anti-bot measures. This allows to bypass this because as far as Cloudflare is concerned, the requests are coming from a trusted source. Setting up a proxy with a provider that isn't Cloudflare, means the requests aren't coming from a trusted source any more.
- judge2020 6y agoThe thing is that it isn't being seen as a trusted source. The same global rate limits apply, and the `Cf-Connecting-Ip` (which is the one you should use for security) request header still can't be forged and is passed to the end website correctly.
- tyingq 6y agoI have seen workers code that appears to remove Cf-Connecting-Ip prior to a fetch(), though no idea if that works. Maybe it just gets replaced with the workers ip? Like: router.route('/proxy/:url', '*', async (event) => { event.request.headers.delete('cf-connecting-ip'); event.request.headers.delete('x-real-ip'); return await fetch(event.parameters.url,event.request); Edit: Tried several variations of this. It doesn't work. The cf-connecting-ip header is always in the fetch request with the right data. It's a bit confusing because some of the Cloudflare tools don't show you all the headers.
- klaudius 6y agoThey say there are multiple ways to do IP spoofing. Does anyone know some other ways to do it?
- Tepix 6y agoFind a provider that doesn't prevent you from doing it.
- SergeAx 6y ago> At this point you can request any site using Cloudflare I don't see why I can't request any site on the internet whatsoever, having legit Cloudflare IP address. Is there some limitation of Cloudflare workers?
- judge2020 6y ago1. They limit subrequests to 50 per workers request 2. they have abuse alarms for obvious "spray requests by utilizing Workers" behavior (although we don’t know how well these might work) 2. The Cf-Connecting-Ip header is passed with the correct originating request IP, so if a website at least partially trusts CF (without directly using CF's proxy) they can match based on this header if the REMOTE_ADDR is Cloudflare. They could also block requests with the header Cf-Worker[0]. 3. You can't forge the HOST header on CF, so if a website does use CF you can't bypass the actual CF proxy firewall by fetching the IP and setting the HOST header yourself (you can do it within your zone though)[1]. 0: https://community.cloudflare.com/t/is-the-cf-worker-header-from-worker-fetch-reliable/185560/2?u=judge https://community.cloudflare.com/t/is-the-cf-worker-header-f... 1: https://developers.cloudflare.com/workers/runtime-apis/request#requestinitcfproperties https://developers.cloudflare.com/workers/runtime-apis/reque... (resolveOverride)
- toredash 6y agoI believe you can with Enterprise plan, override the Host header that is.
- winrid 6y agoTheir bot protection is so annoying for integrations, I may just use this...
- casefields 6y agoAt first I didn't mind, but it seems to be popping up more and more. Like a little vampire sucking up to 5 seconds of life each time, that I'll never get back.
- winrid 6y agoEven worse: Lot more than 5mins to get whitelisting etc setup. The same person in the org setting up the integration probably doesn't have cloudflare account access, so it becomes a source of customers dropping off.
- MaheshC 6y agoIf somebody from cloudflare is readying this. This is great. Please do not fix it, or if you do, provide a way for legitimate limited bypassing when, for instance, using puppeteer from an AWS instance.
- freedomben 6y agoI'm glad I'm not the only one that can't type Cloudflare reliably. Couldflare is an important part of my architecture. Same with Cloudlfare. No CloudFalre or Cloudfalre yet but there's still time ;-)
- p2detar 6y agoCool. I’d be interested in any hints about bypassing Akamai’s protection bot. They use some annoying js machinery to detect browser sensors and stuff. Stumbled on that some days ago. Quite annoying and there doesn’t seem to be a easy hack.
- kentonv 6y agoHi, I'm the tech lead of Cloudflare Workers. This article contains several misunderstandings. If a Cloudflare customer has configured their origin server to respond only to Cloudflare IPs, then they MUST also verify that the "Host" header on any request actually matches their domain name. If they do not verify the Host header, then anyone can sign up for Cloudflare and simply configure their DNS to point to the victim's origin IP address, and requests will be routed there -- but will have the attacker's domain in the Host header. Workers is not needed for such an attack. This attack has always been possible and is common to basically all CDNs. Fundamentally, the CDN has no way of knowing if the origin server that a user has configured really belongs to them -- the CDN can only tell the origin (via the Host header) what customer it thinks it is serving, and expect the origin not to accept requests that were on behalf of a different customer. Instead of IP-based authentication, we strongly recommend using mTLS-based authenticated origin pulls (with a zone-specific key pair) or Argo Tunnel, as these methods are much more secure. As long as the origin server is verifying via one of the above means (IP+host header, AOP, or Argo Tunnel) that the request was processed by Cloudflare on behalf of the customer's zone, then the attack described in the article doesn't accomplish anything. If a Worker makes a request to a hostname that is on Cloudflare, then all of the target host's Cloudflare security settings will apply to that request the same as if the request came from an external client. Additionally, as mentioned in the article, any requests coming from a Worker will have the CF-Worker header identifying the zone which sent the request. If a customer suspects abusive requests coming from Workers, they should report it to us. ~~The article claims that the X-Forwarded-For header can be forged, but this is not true if the target host is itself a Cloudflare customer.~~ EDIT: What I said about X-Forwarded-For seems to be incorrect. CF-Connecting-IP has always been the header we use to identify the client IP, and it cannot be forged. X-Forwarded-For seems to have more complicated and subtle behavior that I'm not familiar with. I'm investigating.
- Craighead 6y agoMedium using Cloudflare with articles explaining how to abuse Cloudflare is interesting. Hopefully now that the position is tested, Cloudflare won't pressure medium to take down this article, even if it does contain misrepresentations.
- 6y ago
- dcow 6y agoTL;DR: somebody learned how proxies work and is naively mad at Cloudflare because they offer a cloud worker product that lets you proxy requests and their “bug” report is actually a feature.
- deleted 6y ago[deleted]
- CFA178B 6y agoDisclaimer: Throwaway account. There is a way to bypass CF bot protection, which mostly uses basic HTTP features, nothing fancy (and especially not a security issue).
- userbinator 6y agoMore precisely, "bot and minority browser/system" protection... when I come across a site that has it, I will just go somewhere else, or use Google's cache if absolutely necessary.