11 ms·

How to Bypass Cloudflare: A Comprehensive Guide

260 points by jakobdabo 4y ago

Tiberium 4y ago
The actual "easiest" way (at least for me) to bypass Cloudflare is to find the actual IP of the web-server running behind it. Of course in a lot of cases it's not possible, for example when the web admin correctly limits the webserver to only respond to Cloudflare IP ranges, or if https://developers.cloudflare.com/ssl/origin-configuration/origin-ca/ https://developers.cloudflare.com/ssl/origin-configuration/o... is used. Most useful services for that are https://shodan.io/ https://shodan.io/ and https://search.censys.io/ https://search.censys.io/. I've had decent successes with Censys on finding real IP addresses of websites behind Cloudflare. Of course you might also have success by checking history of DNS records for a particular domain.
- discreditable 4y ago
  This is what I expected the article to be about. I would wager a lot of shops don't to the whitelisting. If they wanted to be really intense they could do authenticated origin pulls.
  - akira2501 4y ago
    AWS CloudFront with S3 recommends that you just set your S3 to require a specific 'Referer' header variable and you set CloudFront to send that custom 'Referer' with each origin request. Seems to work great when you use something like a GUID, and no need for IP whitelisting.
- PigiVinci83 4y ago
  Companies spend thousands of dollars on these anti-bot solutions and then they are so misconfigured that using a specific user agent or faking browsing via mobile, bypasses them. Real life stories.
  - grogenaut 4y ago
    Often this is because you are hamstrung by old mobile apps or TV apps that can't be updated forcibly and so you break users. So your making a trade-off of user pain and bot deflection. So many times this is actually known and on purpose. Botters hitting that loophole helps prioritize closing that loophole in an agile customer experience and makes it easier for engineering and product to prioritize. Real life stories
    - PigiVinci83 4y ago
      Totally agree.
    - PigiVinci83 4y ago
      Just shared some of these stories here https://twitter.com/pigivinciguerra/status/1571543794348089345 https://twitter.com/pigivinciguerra/status/15715437943480893...
- gingerlime 4y ago
  > or if https://developers.cloudflare.com/ssl/origin-configuration/o https://developers.cloudflare.com/ssl/origin-configuration/o... is used. How is using CF’s origin CA preventing the connection to the real backend in order to bypass Cloudflare? you cam just ignore the SSL error couldn’t you?
  - temp0826 4y ago
    In addition, I think CF provides a list of IPs to whitelist to only allow access from their servers.
  - kevincox 4y ago
    They probably meant to link https://developers.cloudflare.com/ssl/origin-configuration/authenticated-origin-pull/ https://developers.cloudflare.com/ssl/origin-configuration/a... where Cloudflare uses a client TLS certificate to pull from the origin and the origin should be configured to reject requests without a client certificate.
- nlh 4y ago
  Just want to say THANK YOU for this insight. It never occurred to me, and I just checked out one of the big sites that I scrape for a side project and, lo and behold, you are 100% correct. Found their origin in Censys in about 30 seconds and I've never been able to crunch through their pages more easily. To others out there who explore this: As with all scraping, be gentle! If you start pounding on someone's origin server directly, you're much more likely to be noticed than if you're pounding on something behind a CloudFlare cache. Set rate limits, scrape during off-peak hours, etc. Be a good scraping citizen.
  - lostmsu 4y ago
    How did you do it? Asking for a friend who would like to get to the webservers behind CDNs that front Russian websites.
- flicker-rate 4y ago
  One more technique to find the backend IP address of the web-server is to lookup the DNS record history of the domain in question. Usually, the admins will test their website with with SSL on a real domain to ensure it works. Only afterward, they will protect it with Cloudflare/Akamai/Cloudfront, etc, which means you'll have a DNS history trail to lookup. I've had good luck with services such as SecurityTrails, Virustotal and CompleteDNS for DNS history. Once you find the original A record IP address, you can put it in your hosts file, pointing to the domain. Then, you will be able to access the site, bypassing the CDN without issue.
dizhn 4y ago
Use zenrows. Got it. It's clickbait but it does provide a good summary of how cloudflare's anti bot stuff works.
- yjftsjthsd-h 4y ago
  I'd call it an ad, not clickbait. An ad with some useful content, but still. Edit: I think the modern term is content native advertising, although I'm perfectly happy to keep using the word infomercial.
  - dizhn 4y ago
    I called it clickbait because the article does not contain the thing promised in the title.
  - throwaway81523 4y ago
    I'd call it clickbait since its actual nature is not revealed til the very end.
    - tzs 4y ago
      The article is ~32 pages long if printed using the default settings in my browser. The first 31 pages pretty extensively provide information on bypassing Cloudflare, which would be very useful to and save a lot of time for someone who is tasked with implementing Cloudflare bypass software. This is followed by 1 page that talks about their product for doing this. In other words it delivers pretty much what the title said it would, and then follows that with a small mention of their product. That's not clickbait by any even remotely reasonable definition.
      - dizhn 4y ago
        The bulk of the article is about the protections in place and there is no direct answer to how to bypass them.
        4y ago
nothasan 4y ago
Some impressive documentation on how to get around this BM solution.
donutshop 4y ago
Are there other products out there that offers a similar feature set at this price point?
yjftsjthsd-h 4y ago
The frustrating thing to me is that CF is that invasive and still can't distinguish bots from people; it usually eventually lets me through, but I've spent enough time staring at the "are you sure you're not a not?" screen to laugh off their claims about human/not traffic ratios.
alokjnv10 4y ago
I hate cloudflare. I had a really hard time making a web scraper.
- vntok 4y ago
  That's... the whole point.
  - unixbane 4y ago
    And it's an invalid point. Scraping prevention is the most stupid thing Cloudflare has ever done, and that's after a very long list.
    - simondotau 4y ago
      It’s not an invalid point. Setting aside Government and business services, you aren’t morally entitled to clean, uninterrupted access to any random website. If a webmaster chooses to make your life difficult for any reason, that’s entirely their prerogative.
      - unixbane 4y ago
        I'm saying anti-scraping is a misconception and if you buy it (or get one for free) you just paid for a placebo and it's also worse than nothing. Is there anything confusing about this?
        simondotau 4y ago
        If it's a misconception, why did you complain that Cloudflare caused you "a really hard time making a web scraper"?
        unixbane 4y ago
        You just confused me with someone else. Cloudflare gives me a hard time as a user only. When you view a page with a proxy or shared IP address, you have to solve a captcha or enable JS to see any text with @ in it, and such insanities. Scraping is a user use case. Web crap is meant to be automateable and automatically navigible (though as a cattle user who sees it as an interactive one time experience, like someone walking into a shop, I can see how you see otherwise). Anyway it's easy to scrape even with Cloudflare, it just causes more bumps and adds to your fingerprint and makes the internet as a whole more insecure.
cj 4y ago
There are legitimate use cases for bypassing cloudflare's bot protection. I discovered our company's help documentation (and integration guides), hosted by readme.com, were completely de-indexed from Google for the past 3 months. Our Readme docs were formerly our #1 source of organic (free) leads. After investigating, Cloudflare (as configured by Readme) was blocking Googlebot when using Cloudflare Workers. Cloudflare was returning a 403 for Googlebot, but returning pages as usual for regular users. The cause: we were using Workers to rewrite some URLs at the edge (replacing Readme's default images with optimized + compressed images, using Cloudflare's own image optimization service). By using Workers to do this, it resulted in Readme's Cloudflare account receiving requests from our domain with "googlebot" useragent, but from an IP that wasn't verified as a googlebot IP address (I assume the Worker was requesting the Readme site using the Googlebot user agent but with whatever IP address is used when using CF Workers). I emailed Cloudflare support but it was clear it would take a lot of time to get them to understand the issue (and probably longer to fix it). So, we had to spend a lot of time figuring out how to allow Googlebot requests past Cloudflare's "fake bot" firewall rule. In our own Cloudflare account, we have all security settings at the lowest sensitivity possible (or turned off completely). We serve over 500 billion requests a month (10+ TB of bandwidth), and the amount of blocked traffic to seemingly legitimate clients was surprisingly high. I love Cloudflare (and own quite a bit of their stock) but I'm beginning to rethink my stance on their service. They make it extremely easy to enable powerful features with little visibility or control over the details of how those features work. Another SEO nightmare is their "Crawler Hints" service. I highly recommend no one uses this if you are ever the target of automated security scanners (e.g. ones used by bug bounty white hat hackers). With "crawler hints" enabled and with a white hat hacker running a scan of your site hitting random URLs... results in bingbot, yandex, and other search engines attempting to index every single one of the URLs hit by the security scanners used by hackers. Basically, it's a mess, and the only way to really fix it is to bypass cloudflare or spend a lot of time and money with Cloudflare debugging. Next quarter I'm faced with the decision of either doubling down of Cloudflare and getting an Enterprise plan with them ($20k+) or just ripping them out of our stack and going back to our old AWS Cloudfront set up which has fewer POPs, but was much less of a hassle.
- pbowyer 4y ago
Anunayj 4y ago
I would also like to mention FlareSolverr [1] here, which just uses a headless browser to solve the challenges, which might be acceptable in some situations (that don't need high request rate) 1. https://github.com/FlareSolverr/FlareSolverr https://github.com/FlareSolverr/FlareSolverr
urtom 4y ago
If I just need to make plain GET requests in my web scraping, I've found the easiest way to bypass Cloudflare on most sites is to make the requests via the Internet Archive. That has some rate limiting, but it can be worked around by using several source IP addresses in parallel.
midislack 4y ago
All this and it's just an ad for some SAAS? Fuck I got gypped.
unixbane 4y ago
- userbinator 4y ago
  Never underestimate the power of corporate interests and what they'll do if they feel threatened. and so you will only be able to browse websites in a certified way, like using your bare IP address or a big 4 browser ...and that browser will have to be running on "trusted" software and hardware, which is very much the definition of user-hostility. There were a few articles recently about the imminent threat of remote attestation that got buried in a similar way. They first went after RMS, managed to get their user-hostile antifeatures into Linux, and are slowing locking things down and going after the dissenters. All in the name of "security" --- to secure their interests, not yours.
  - bogomipz 4y ago
    >"They first went after RMS, managed to get their user-hostile antifeatures into Linux,..." What is RMS here?
    - userbinator 4y ago
      Richard Matthew Stallman
- bogomipz 4y ago
  Your observation in your edit is indeed a phenomenon. I have noticed this as well. Cloudflare seems to use HN as their exclusive marketing channel. Their CTO seems to submit almost every one of their blog posts or product updates here. Then many of the company's employees will join in on the thread and it becomes almost like a customer forum. I don't think this mixed with the knee jerk reactions to anyone with a critical voice is healthy.
  - deleted 4y ago
    [deleted]
  - 4y ago
userbinator 4y ago
If you use even a slightly customised user-agent and/or OS setup, it's likely you'll be blocked. They'll of course say it's for your "security", but we all know what that really means: use only the software and hardware configurations that we approve. Of all the ways to "herd the sheeple", this is the most insidious because it punishes those who don't want to submit to their whims. Meanwhile the actual attackers are going to still have enough motivation to find ways around it, similar to how DRM has encouraged piracy.
Ralo 4y ago
In the past I've always found that the easiest way to bypass Cloudflare was looking up DNS history of their domain. Majority of servers will continue to respond off their IP directly.
1vuio0pswjnm7 4y ago
Correct me if I have missed something but this elaborate fingerprinting exercise called "bot protection" cannot distinguish conclusively whether a person is giving commands to a computer in real-time or from if the computer is reading from a script of commands. It only serves to distinguish what OS, client, IP address, etc. It is collecting tracking data. Of course, those trying to profit from online advertising services seek to collect the same (fingerprinting) data. Do Cloudflare terms of service/privacy policy allow Cloudflare to do anything they want with this data, or are there limits.
throwaway892238 4y ago
IceWreck 4y ago
Well this is ironic https://i.imgur.com/93CAN9f.png https://i.imgur.com/93CAN9f.png

260 points by jakobdabo 4y ago

The actual "easiest" way (at least for me) to bypass Cloudflare is to find the actual IP of the web-server running behind it. Of course in a lot of cases it's not possible, for example when the web admin correctly limits the webserver to only respond to Cloudflare IP ranges, or if https://developers.cloudflare.com/ssl/origin-configuration/origin-ca/ https://developers.cloudflare.com/ssl/origin-configuration/o... is used. Most useful services for that are https://shodan.io/ https://shodan.io/ and https://search.censys.io/ https://search.censys.io/. I've had decent successes with Censys on finding real IP addresses of websites behind Cloudflare. Of course you might also have success by checking history of DNS records for a particular domain.

discreditable 4y ago
This is what I expected the article to be about. I would wager a lot of shops don't to the whitelisting. If they wanted to be really intense they could do authenticated origin pulls.
- akira2501 4y ago
  AWS CloudFront with S3 recommends that you just set your S3 to require a specific 'Referer' header variable and you set CloudFront to send that custom 'Referer' with each origin request. Seems to work great when you use something like a GUID, and no need for IP whitelisting.
PigiVinci83 4y ago
Companies spend thousands of dollars on these anti-bot solutions and then they are so misconfigured that using a specific user agent or faking browsing via mobile, bypasses them. Real life stories.
- grogenaut 4y ago
  Often this is because you are hamstrung by old mobile apps or TV apps that can't be updated forcibly and so you break users. So your making a trade-off of user pain and bot deflection. So many times this is actually known and on purpose. Botters hitting that loophole helps prioritize closing that loophole in an agile customer experience and makes it easier for engineering and product to prioritize. Real life stories
  - PigiVinci83 4y ago
    Totally agree.
  - PigiVinci83 4y ago
    Just shared some of these stories here https://twitter.com/pigivinciguerra/status/1571543794348089345 https://twitter.com/pigivinciguerra/status/15715437943480893...
gingerlime 4y ago
> or if https://developers.cloudflare.com/ssl/origin-configuration/o https://developers.cloudflare.com/ssl/origin-configuration/o... is used. How is using CF’s origin CA preventing the connection to the real backend in order to bypass Cloudflare? you cam just ignore the SSL error couldn’t you?
- temp0826 4y ago
  In addition, I think CF provides a list of IPs to whitelist to only allow access from their servers.
- kevincox 4y ago
  They probably meant to link https://developers.cloudflare.com/ssl/origin-configuration/authenticated-origin-pull/ https://developers.cloudflare.com/ssl/origin-configuration/a... where Cloudflare uses a client TLS certificate to pull from the origin and the origin should be configured to reject requests without a client certificate.
nlh 4y ago
Just want to say THANK YOU for this insight. It never occurred to me, and I just checked out one of the big sites that I scrape for a side project and, lo and behold, you are 100% correct. Found their origin in Censys in about 30 seconds and I've never been able to crunch through their pages more easily. To others out there who explore this: As with all scraping, be gentle! If you start pounding on someone's origin server directly, you're much more likely to be noticed than if you're pounding on something behind a CloudFlare cache. Set rate limits, scrape during off-peak hours, etc. Be a good scraping citizen.
- lostmsu 4y ago
  How did you do it? Asking for a friend who would like to get to the webservers behind CDNs that front Russian websites.
flicker-rate 4y ago
One more technique to find the backend IP address of the web-server is to lookup the DNS record history of the domain in question. Usually, the admins will test their website with with SSL on a real domain to ensure it works. Only afterward, they will protect it with Cloudflare/Akamai/Cloudfront, etc, which means you'll have a DNS history trail to lookup. I've had good luck with services such as SecurityTrails, Virustotal and CompleteDNS for DNS history. Once you find the original A record IP address, you can put it in your hosts file, pointing to the domain. Then, you will be able to access the site, bypassing the CDN without issue.

dizhn 4y ago

Use zenrows. Got it. It's clickbait but it does provide a good summary of how cloudflare's anti bot stuff works.

yjftsjthsd-h 4y ago
I'd call it an ad, not clickbait. An ad with some useful content, but still. Edit: I think the modern term is content native advertising, although I'm perfectly happy to keep using the word infomercial.
- dizhn 4y ago
  I called it clickbait because the article does not contain the thing promised in the title.
- throwaway81523 4y ago
  I'd call it clickbait since its actual nature is not revealed til the very end.
  - tzs 4y ago
    The article is ~32 pages long if printed using the default settings in my browser. The first 31 pages pretty extensively provide information on bypassing Cloudflare, which would be very useful to and save a lot of time for someone who is tasked with implementing Cloudflare bypass software. This is followed by 1 page that talks about their product for doing this. In other words it delivers pretty much what the title said it would, and then follows that with a small mention of their product. That's not clickbait by any even remotely reasonable definition.
    - dizhn 4y ago
      The bulk of the article is about the protections in place and there is no direct answer to how to bypass them.
      - 4y ago

nothasan 4y ago

Some impressive documentation on how to get around this BM solution.

donutshop 4y ago

Are there other products out there that offers a similar feature set at this price point?

yjftsjthsd-h 4y ago

The frustrating thing to me is that CF is that invasive and still can't distinguish bots from people; it usually eventually lets me through, but I've spent enough time staring at the "are you sure you're not a not?" screen to laugh off their claims about human/not traffic ratios.

alokjnv10 4y ago

I hate cloudflare. I had a really hard time making a web scraper.

vntok 4y ago
That's... the whole point.
- unixbane 4y ago
  And it's an invalid point. Scraping prevention is the most stupid thing Cloudflare has ever done, and that's after a very long list.
  - simondotau 4y ago
    It’s not an invalid point. Setting aside Government and business services, you aren’t morally entitled to clean, uninterrupted access to any random website. If a webmaster chooses to make your life difficult for any reason, that’s entirely their prerogative.
    - unixbane 4y ago
      I'm saying anti-scraping is a misconception and if you buy it (or get one for free) you just paid for a placebo and it's also worse than nothing. Is there anything confusing about this?
      - simondotau 4y ago
        If it's a misconception, why did you complain that Cloudflare caused you "a really hard time making a web scraper"?
        unixbane 4y ago
        You just confused me with someone else. Cloudflare gives me a hard time as a user only. When you view a page with a proxy or shared IP address, you have to solve a captcha or enable JS to see any text with @ in it, and such insanities. Scraping is a user use case. Web crap is meant to be automateable and automatically navigible (though as a cattle user who sees it as an interactive one time experience, like someone walking into a shop, I can see how you see otherwise). Anyway it's easy to scrape even with Cloudflare, it just causes more bumps and adds to your fingerprint and makes the internet as a whole more insecure.

cj 4y ago

There are legitimate use cases for bypassing cloudflare's bot protection. I discovered our company's help documentation (and integration guides), hosted by readme.com, were completely de-indexed from Google for the past 3 months. Our Readme docs were formerly our #1 source of organic (free) leads. After investigating, Cloudflare (as configured by Readme) was blocking Googlebot when using Cloudflare Workers. Cloudflare was returning a 403 for Googlebot, but returning pages as usual for regular users. The cause: we were using Workers to rewrite some URLs at the edge (replacing Readme's default images with optimized + compressed images, using Cloudflare's own image optimization service). By using Workers to do this, it resulted in Readme's Cloudflare account receiving requests from our domain with "googlebot" useragent, but from an IP that wasn't verified as a googlebot IP address (I assume the Worker was requesting the Readme site using the Googlebot user agent but with whatever IP address is used when using CF Workers). I emailed Cloudflare support but it was clear it would take a lot of time to get them to understand the issue (and probably longer to fix it). So, we had to spend a lot of time figuring out how to allow Googlebot requests past Cloudflare's "fake bot" firewall rule. In our own Cloudflare account, we have all security settings at the lowest sensitivity possible (or turned off completely). We serve over 500 billion requests a month (10+ TB of bandwidth), and the amount of blocked traffic to seemingly legitimate clients was surprisingly high. I love Cloudflare (and own quite a bit of their stock) but I'm beginning to rethink my stance on their service. They make it extremely easy to enable powerful features with little visibility or control over the details of how those features work. Another SEO nightmare is their "Crawler Hints" service. I highly recommend no one uses this if you are ever the target of automated security scanners (e.g. ones used by bug bounty white hat hackers). With "crawler hints" enabled and with a white hat hacker running a scan of your site hitting random URLs... results in bingbot, yandex, and other search engines attempting to index every single one of the URLs hit by the security scanners used by hackers. Basically, it's a mess, and the only way to really fix it is to bypass cloudflare or spend a lot of time and money with Cloudflare debugging. Next quarter I'm faced with the decision of either doubling down of Cloudflare and getting an Enterprise plan with them ($20k+) or just ripping them out of our stack and going back to our old AWS Cloudfront set up which has fewer POPs, but was much less of a hassle.

pbowyer 4y ago

Anunayj 4y ago

I would also like to mention FlareSolverr [1] here, which just uses a headless browser to solve the challenges, which might be acceptable in some situations (that don't need high request rate) 1. https://github.com/FlareSolverr/FlareSolverr https://github.com/FlareSolverr/FlareSolverr

urtom 4y ago

If I just need to make plain GET requests in my web scraping, I've found the easiest way to bypass Cloudflare on most sites is to make the requests via the Internet Archive. That has some rate limiting, but it can be worked around by using several source IP addresses in parallel.

midislack 4y ago

All this and it's just an ad for some SAAS? Fuck I got gypped.

unixbane 4y ago

userbinator 4y ago
Never underestimate the power of corporate interests and what they'll do if they feel threatened. and so you will only be able to browse websites in a certified way, like using your bare IP address or a big 4 browser ...and that browser will have to be running on "trusted" software and hardware, which is very much the definition of user-hostility. There were a few articles recently about the imminent threat of remote attestation that got buried in a similar way. They first went after RMS, managed to get their user-hostile antifeatures into Linux, and are slowing locking things down and going after the dissenters. All in the name of "security" --- to secure their interests, not yours.
- bogomipz 4y ago
  >"They first went after RMS, managed to get their user-hostile antifeatures into Linux,..." What is RMS here?
  - userbinator 4y ago
    Richard Matthew Stallman
bogomipz 4y ago
Your observation in your edit is indeed a phenomenon. I have noticed this as well. Cloudflare seems to use HN as their exclusive marketing channel. Their CTO seems to submit almost every one of their blog posts or product updates here. Then many of the company's employees will join in on the thread and it becomes almost like a customer forum. I don't think this mixed with the knee jerk reactions to anyone with a critical voice is healthy.
- deleted 4y ago
  [deleted]
- 4y ago

userbinator 4y ago

If you use even a slightly customised user-agent and/or OS setup, it's likely you'll be blocked. They'll of course say it's for your "security", but we all know what that really means: use only the software and hardware configurations that we approve. Of all the ways to "herd the sheeple", this is the most insidious because it punishes those who don't want to submit to their whims. Meanwhile the actual attackers are going to still have enough motivation to find ways around it, similar to how DRM has encouraged piracy.

Ralo 4y ago

In the past I've always found that the easiest way to bypass Cloudflare was looking up DNS history of their domain. Majority of servers will continue to respond off their IP directly.

1vuio0pswjnm7 4y ago

Correct me if I have missed something but this elaborate fingerprinting exercise called "bot protection" cannot distinguish conclusively whether a person is giving commands to a computer in real-time or from if the computer is reading from a script of commands. It only serves to distinguish what OS, client, IP address, etc. It is collecting tracking data. Of course, those trying to profit from online advertising services seek to collect the same (fingerprinting) data. Do Cloudflare terms of service/privacy policy allow Cloudflare to do anything they want with this data, or are there limits.

throwaway892238 4y ago

IceWreck 4y ago

Well this is ironic https://i.imgur.com/93CAN9f.png https://i.imgur.com/93CAN9f.png