5 ms·

The actual "easiest" way (at least for me) to bypass Cloudflare is to find the actual IP of the web-server running behind it. Of course in a lot of cases it's n

The actual "easiest" way (at least for me) to bypass Cloudflare is to find the actual IP of the web-server running behind it. Of course in a lot of cases it's not possible, for example when the web admin correctly limits the webserver to only respond to Cloudflare IP ranges, or if https://developers.cloudflare.com/ssl/origin-configuration/origin-ca/ https://developers.cloudflare.com/ssl/origin-configuration/o... is used. Most useful services for that are https://shodan.io/ https://shodan.io/ and https://search.censys.io/ https://search.censys.io/. I've had decent successes with Censys on finding real IP addresses of websites behind Cloudflare. Of course you might also have success by checking history of DNS records for a particular domain.

discreditable 4y ago
This is what I expected the article to be about. I would wager a lot of shops don't to the whitelisting. If they wanted to be really intense they could do authenticated origin pulls.
- akira2501 4y ago
  AWS CloudFront with S3 recommends that you just set your S3 to require a specific 'Referer' header variable and you set CloudFront to send that custom 'Referer' with each origin request. Seems to work great when you use something like a GUID, and no need for IP whitelisting.
PigiVinci83 4y ago
Companies spend thousands of dollars on these anti-bot solutions and then they are so misconfigured that using a specific user agent or faking browsing via mobile, bypasses them. Real life stories.
- grogenaut 4y ago
  Often this is because you are hamstrung by old mobile apps or TV apps that can't be updated forcibly and so you break users. So your making a trade-off of user pain and bot deflection. So many times this is actually known and on purpose. Botters hitting that loophole helps prioritize closing that loophole in an agile customer experience and makes it easier for engineering and product to prioritize. Real life stories
  - PigiVinci83 4y ago
    Totally agree.
  - PigiVinci83 4y ago
    Just shared some of these stories here https://twitter.com/pigivinciguerra/status/1571543794348089345 https://twitter.com/pigivinciguerra/status/15715437943480893...
gingerlime 4y ago
> or if https://developers.cloudflare.com/ssl/origin-configuration/o https://developers.cloudflare.com/ssl/origin-configuration/o... is used. How is using CF’s origin CA preventing the connection to the real backend in order to bypass Cloudflare? you cam just ignore the SSL error couldn’t you?
- temp0826 4y ago
  In addition, I think CF provides a list of IPs to whitelist to only allow access from their servers.
- kevincox 4y ago
  They probably meant to link https://developers.cloudflare.com/ssl/origin-configuration/authenticated-origin-pull/ https://developers.cloudflare.com/ssl/origin-configuration/a... where Cloudflare uses a client TLS certificate to pull from the origin and the origin should be configured to reject requests without a client certificate.
nlh 4y ago
Just want to say THANK YOU for this insight. It never occurred to me, and I just checked out one of the big sites that I scrape for a side project and, lo and behold, you are 100% correct. Found their origin in Censys in about 30 seconds and I've never been able to crunch through their pages more easily. To others out there who explore this: As with all scraping, be gentle! If you start pounding on someone's origin server directly, you're much more likely to be noticed than if you're pounding on something behind a CloudFlare cache. Set rate limits, scrape during off-peak hours, etc. Be a good scraping citizen.
- lostmsu 4y ago
  How did you do it? Asking for a friend who would like to get to the webservers behind CDNs that front Russian websites.
flicker-rate 4y ago
One more technique to find the backend IP address of the web-server is to lookup the DNS record history of the domain in question. Usually, the admins will test their website with with SSL on a real domain to ensure it works. Only afterward, they will protect it with Cloudflare/Akamai/Cloudfront, etc, which means you'll have a DNS history trail to lookup. I've had good luck with services such as SecurityTrails, Virustotal and CompleteDNS for DNS history. Once you find the original A record IP address, you can put it in your hosts file, pointing to the domain. Then, you will be able to access the site, bypassing the CDN without issue.

discreditable 4y ago

This is what I expected the article to be about. I would wager a lot of shops don't to the whitelisting. If they wanted to be really intense they could do authenticated origin pulls.

akira2501 4y ago
AWS CloudFront with S3 recommends that you just set your S3 to require a specific 'Referer' header variable and you set CloudFront to send that custom 'Referer' with each origin request. Seems to work great when you use something like a GUID, and no need for IP whitelisting.

PigiVinci83 4y ago

Companies spend thousands of dollars on these anti-bot solutions and then they are so misconfigured that using a specific user agent or faking browsing via mobile, bypasses them. Real life stories.

grogenaut 4y ago
Often this is because you are hamstrung by old mobile apps or TV apps that can't be updated forcibly and so you break users. So your making a trade-off of user pain and bot deflection. So many times this is actually known and on purpose. Botters hitting that loophole helps prioritize closing that loophole in an agile customer experience and makes it easier for engineering and product to prioritize. Real life stories
- PigiVinci83 4y ago
  Totally agree.
- PigiVinci83 4y ago
  Just shared some of these stories here https://twitter.com/pigivinciguerra/status/1571543794348089345 https://twitter.com/pigivinciguerra/status/15715437943480893...

gingerlime 4y ago

> or if https://developers.cloudflare.com/ssl/origin-configuration/o https://developers.cloudflare.com/ssl/origin-configuration/o... is used. How is using CF’s origin CA preventing the connection to the real backend in order to bypass Cloudflare? you cam just ignore the SSL error couldn’t you?

temp0826 4y ago
In addition, I think CF provides a list of IPs to whitelist to only allow access from their servers.
kevincox 4y ago
They probably meant to link https://developers.cloudflare.com/ssl/origin-configuration/authenticated-origin-pull/ https://developers.cloudflare.com/ssl/origin-configuration/a... where Cloudflare uses a client TLS certificate to pull from the origin and the origin should be configured to reject requests without a client certificate.

nlh 4y ago

Just want to say THANK YOU for this insight. It never occurred to me, and I just checked out one of the big sites that I scrape for a side project and, lo and behold, you are 100% correct. Found their origin in Censys in about 30 seconds and I've never been able to crunch through their pages more easily. To others out there who explore this: As with all scraping, be gentle! If you start pounding on someone's origin server directly, you're much more likely to be noticed than if you're pounding on something behind a CloudFlare cache. Set rate limits, scrape during off-peak hours, etc. Be a good scraping citizen.

lostmsu 4y ago
How did you do it? Asking for a friend who would like to get to the webservers behind CDNs that front Russian websites.

flicker-rate 4y ago

One more technique to find the backend IP address of the web-server is to lookup the DNS record history of the domain in question. Usually, the admins will test their website with with SSL on a real domain to ensure it works. Only afterward, they will protect it with Cloudflare/Akamai/Cloudfront, etc, which means you'll have a DNS history trail to lookup. I've had good luck with services such as SecurityTrails, Virustotal and CompleteDNS for DNS history. Once you find the original A record IP address, you can put it in your hosts file, pointing to the domain. Then, you will be able to access the site, bypassing the CDN without issue.