5 ms·

Hi, I'm the tech lead of Cloudflare Workers. This article contains several misunderstandings. If a Cloudflare customer has configured their origin server to r

Hi, I'm the tech lead of Cloudflare Workers. This article contains several misunderstandings. If a Cloudflare customer has configured their origin server to respond only to Cloudflare IPs, then they MUST also verify that the "Host" header on any request actually matches their domain name. If they do not verify the Host header, then anyone can sign up for Cloudflare and simply configure their DNS to point to the victim's origin IP address, and requests will be routed there -- but will have the attacker's domain in the Host header. Workers is not needed for such an attack. This attack has always been possible and is common to basically all CDNs. Fundamentally, the CDN has no way of knowing if the origin server that a user has configured really belongs to them -- the CDN can only tell the origin (via the Host header) what customer it thinks it is serving, and expect the origin not to accept requests that were on behalf of a different customer. Instead of IP-based authentication, we strongly recommend using mTLS-based authenticated origin pulls (with a zone-specific key pair) or Argo Tunnel, as these methods are much more secure. As long as the origin server is verifying via one of the above means (IP+host header, AOP, or Argo Tunnel) that the request was processed by Cloudflare on behalf of the customer's zone, then the attack described in the article doesn't accomplish anything. If a Worker makes a request to a hostname that is on Cloudflare, then all of the target host's Cloudflare security settings will apply to that request the same as if the request came from an external client. Additionally, as mentioned in the article, any requests coming from a Worker will have the CF-Worker header identifying the zone which sent the request. If a customer suspects abusive requests coming from Workers, they should report it to us. ~~The article claims that the X-Forwarded-For header can be forged, but this is not true if the target host is itself a Cloudflare customer.~~ EDIT: What I said about X-Forwarded-For seems to be incorrect. CF-Connecting-IP has always been the header we use to identify the client IP, and it cannot be forged. X-Forwarded-For seems to have more complicated and subtle behavior that I'm not familiar with. I'm investigating.

Craighead 6y ago
Medium using Cloudflare with articles explaining how to abuse Cloudflare is interesting. Hopefully now that the position is tested, Cloudflare won't pressure medium to take down this article, even if it does contain misrepresentations.
orf 6y ago
> If a Cloudflare customer has configured their origin server to respond only to Cloudflare IPs, then they MUST also verify that the "Host" header on any request actually matches their domain name. Out of interest do you provide any automatic checks for this? It seems like it would be trivial to add some kind of verification step that simulates a request with a bogus host header and checks the status code of the response.
- EB66 6y ago
  What would CloudFlare automatically check the Host header against? The Host header would be set to the attacker's domain (which is in CloudFlare) and, per OP, CloudFlare does not verify ownership of the origin server IP. It would be a nice feature if CloudFlare customers could opt-in for origin server IP verification. Once the origin IP is verified, it would prevent other CloudFlare users from using the IP as their own origin. However, I understand why CloudFlare doesn't verify origin IPs -- it would require a verification process that doesn't rely solely on uploading a verification file to a web accessible directory. It'd be a bit difficult to keep the verification process simple for end-users if they aren't running a web server. Or maybe I'm wrong and non-HTTP use cases don't need to be considered and a verification file uploaded to a web accessible folder would work fine.
  - orf 6y ago
    Somewhere in the interface you’d have a button that when pressed would send a request to the specified origin server with a Host header set to “foobar.com”. If the status code is 200 you’d display a warning saying “perhaps you should re-read our documentation about verifying host headers”. I can think of a whole bunch of corner cases to this, but that’s the general jist.
    - EB66 6y ago
      But in order to safely enable that check for all requests that go from CloudFlare to the origin IP, you would first need to verify ownership of the origin IP. You wouldn't want users who don't own the origin IP to be able to turn on that sort of check and potentially cause a service disruption. It could be the case that the legitimate owner already has a Host header mismatch in their HTTP requests/responses.
kentonv 6y ago
A couple other notes: * The IP address 2a06:98c0:3600::103, mentioned in the article as appearing in the CF-Connecting-IP header, is a special IP address that is used for all cross-zone requests that come from Workers. This is not the actual address of any Cloudflare machine. Workers fundamentally don't have distinct IP addresses. Instead, the CF-Worker header needs to be used to identify which worker sent the request. We intentionally do not identify the original client's IP here because the Worker itself could be working against the client, and could have maliciously modified their request to be something entirely different. Hence, the request can't be considered to have come from the original client. * Any cloud service that lets you make HTTP requests can, obviously, be used to set up a proxy that hides your IP address. Workers is actually less useful for this compared to many other cloud hosts since we tag all outgoing HTTP requests from a Worker with the sender's domain name (in the CF-Worker header), which makes it relatively easier to track, report, and block abuse.
abiro 6y ago
> If a Cloudflare customer has configured their origin server to respond only to Cloudflare IPs, then they MUST also verify that the "Host" header on any request actually matches their domain name. Where is this documented?
- toredash 6y ago
  I'm 100% sure it isn't. Cloudflare docs are not great, and as a Enterprise customer I spend more time with support than needed as a result.
aeyes 6y ago
> If a Cloudflare customer has configured their origin server to respond only to Cloudflare IPs, then they MUST also verify that the "Host" header on any request actually matches their domain name. The worker code sends the correct host name. https://github.com/jychp/cloudflare-bypass/blob/dd7c1cfc2e6b43d729333ed6b3736925cbc4aea1/cfproxy.py#L33 https://github.com/jychp/cloudflare-bypass/blob/dd7c1cfc2e6b...
- kentonv 6y ago
  Right. If the hostname requested by the worker is a host that is itself protected by Cloudflare, then all of that host's usual security settings apply. Nothing is bypassed. The author of the blog post seems to have been confused by the fact that different behavior happens if the target host is on Cloudflare vs. not on Cloudflare.
r1ch 6y ago
> If a Cloudflare customer has configured their origin server to respond only to Cloudflare IPs, then they MUST also verify that the "Host" header on any request actually matches their domain name. This could be improved considerably if Cloudflare removed support for insecure TLS options. "Full" SSL bypasses certificate hostname verification on the origin server which could prevent such an attack at the TLS level. Every time I've added a new zone to Cloudflare, Full or Flexible (ie, no SSL) has been the default. The only safe option is Full (Strict) but I'd bet good money that the majority of Cloudflare customers aren't using that.
MuffinFlavored 6y ago
From the article: > X-FORWARDED-FOR: 1.2.3.4 (yes, we can override this header with whatever we want !) Why aren't you guys disallowing that field from having any external value?
- modoc 6y ago
  Because it's legitimate to have values before the request gets to CloudFlare, for instance if it was routed out through a corporate proxy, etc..
  - MuffinFlavored 6y ago
    Ouch. I wonder if it's time to cryptographically sign those values or add a checksum or something.
    - acdha 6y ago
      Basically this is why you want to go full zero trust: if you use mutual authentication over TLS, this becomes much less of a concern since you’re not trying to craft policies based on IP addresses with multiple intermediaries.
  - ec109685 6y ago
    Only possible if non-TLS or the company is man in the middling TLS (and certificate pinning isn’t enabled).
    - allending 6y ago
      So... completely possible then.
- kentonv 6y ago
  So we're supposed to be appending the client's IP to whatever the client sent for X-Forwarded-For, documented here: https://support.cloudflare.com/hc/en-us/articles/200170986-How-does-Cloudflare-handle-HTTP-Request-headers- https://support.cloudflare.com/hc/en-us/articles/200170986-H... For better or worse, this is how X-Forwarded-For works, by convention: each forwarder is supposed to append a value. This means that, even in a correct implementation, an attacker can specify an arbitrary prefix of values. The consumer must be very careful about which part of the chain they trust. But the report here seems to be suggesting that we're mishandling this appending in some cases. We're investigating and if true will fix ASAP. Because X-Forwarded-For is surprisingly complicated, we recommend using CF-Connecting-IP instead. As the docs say: "To restore original visitor IP addresses at your origin web server, Cloudflare recommends your logs or applications look at CF-Connecting-IP or True-Client-IP instead of X-Forwarded-For since CF-Connecting-IP and True-Client-IP have a consistent format containing only one IP."
zamalek 6y ago
> This attack has always been possible and is common to basically all CDNs. I strongly recommend you play around with Azure CDN, because this is not entirely true.
njbooher 6y ago
Once upon a time: https://njbooher.github.io/blog/cloudflare-workers-ip-spoofing https://njbooher.github.io/blog/cloudflare-workers-ip-spoofi...
- kentonv 6y ago
  Yep, I worked on that one. Thank you for reporting it. Ironically, the purported problem in this case is the opposite: Because we don't attribute the request as coming from the original client (in order to avoid the security problem you reported), a client that has bad reputation associated with their IP can proxy through a worker to hide that. The answer is to assign reputation to the Worker itself.
  - njbooher 6y ago
    Yeah, I don't envy you. Finding loopholes is more fun than trying to account for everything upfront.
  - ec109685 6y ago
    You see the request on “both sides” of the worker, so shouldn’t you be able to evaluate the “connecting client to the worker’s reputation” when allowing or denying the upstream request?
    - kentonv 6y ago
      You mean by somehow evaluating whether the incoming request and the outgoing one are sufficiently "the same", in which case the outgoing request can be attributed to the original client? This seems hard. Let's say the request is the very simplest "GET /" request. The worker does nothing except rewrite the hostname, in order to proxy it to a different server. Is it "the same" request? Well, imagine the new destination host is a company-internal dashboard that authenticates users based on IP address. Now this "GET /" request which was proxied will receive a response that contains secrets. The worker that did the proxying can intercept the response and learn the secrets. So clearly if the request's host is rewritten, it cannot be "the same" request and it cannot be considered to have come from the original client. But the only case where we care about any of this is when a worker makes a request to a different domain. If a request is to the same domain, we trust it, because in that case the attacker can only attack themselves anyway. So the only case that even matters here is when the host has been rewritten, and in that case, we've established that the new request definitely cannot be attributed to the original client. So it seems to me there's nothing interesting we can do here. A request coming out of a worker (going to a different domain) must always be attributed to the worker itself, not to the client that triggered the worker.
thesis 6y ago
This seems like the same kind of reason we're not using Cloudflare Access to protect some internal resources. We'd love to use Argo Tunnel but unfortunately it breaks with an OPTIONS request for us. We've tried opening tickets with Cloudflare.
meowface 6y ago
>If a Cloudflare customer has configured their origin server to respond only to Cloudflare IPs, then they MUST also verify that the "Host" header on any request actually matches their domain name. If they do not verify the Host header, then anyone can sign up for Cloudflare and simply configure their DNS to point to the victim's origin IP address, and requests will be routed there -- but will have the attacker's domain in the Host header. What exactly could such a misconfiguration enable? Just that the attacker could configure lower levels of Cloudflare security settings for their domain, to bypass the origin's intended Cloudflare security settings? Would this also bypass I'm Under Attack Mode?
btown 6y ago
> any requests coming from a Worker will have the CF-Worker header identifying the zone which sent the request. If a customer suspects abusive requests coming from Workers, they should report it to us. What happens if you receive a report of abusive behavior from a Worker? Would you rate limit the specific Worker, or would there be any impact on the larger Cloudflare account of the Worker's owner? For instance, if we have a Worker that connects to an arbitrary endpoint based on user-generated parameters, or some vulnerability is found (in the JS code for the Worker) that allows this to happen, and it ends up being used for abusive behavior, would we have recourse to fix the vulnerability, or would action be taken immediately? Would our larger account, or our unrelated Workers, be at risk of being taken offline?
- kentonv 6y ago
  These issues are handled on a case-by-case basis. I can't make any hard promises here (it's not my department), but generally, if your worker is not obviously malicious, we will talk to you before taking any drastic action.
maxgashkov 6y ago
It seems to me that author of the article implies that Cloudflare treats Workers' IPs as having much higher reputation (threat score in CF's terms) than any other IP on the Internet by default. This allows worker to bypass bot protection as it does not trigger at all or only when zone is configured to trigger on all requests (under attack mode). Could you comment on that?
- lilyball 6y ago
  It looks to me like it’s just about how you can use Cloudflare as a reverse proxy, to hide the fact that your origin is Tor. And the bit about being able to change X-FORWARDED-FOR seems to just be about defeating IP-based rate limiting on sites that trust Cloudflare sufficiently to respect the X-FORWARDED-FOR header.
- kentonv 6y ago
  To be honest, I don't know anything about Cloudflare's bot management product. That isn't my department. I would speculate that proxying through any cloud service (Workers, AWS, Digital Ocean, whatever) can indeed be a tool to hide bot activity, and that any bots product needs to think about how to combat that. I would also speculate that originating IP address is not the only signal the product uses to determine threat. Moreover, I would speculate that this is all an arms race, there will be a constant stream of new techniques developed by bot authors and new techniques deployed to thwart them. You may be right that the blog author intended to report a problem specifically in bot management, and I have no idea whether that specific problem is real. However, the blog post is not very clear, and many readers have misinterpreted it as reporting a way to totally bypass all Cloudflare protections, such as WAF, geo firewall, rate limiting, etc. That's the misunderstanding I meant to address. Sorry, I could probably have been clearer about that.

Craighead 6y ago

Medium using Cloudflare with articles explaining how to abuse Cloudflare is interesting. Hopefully now that the position is tested, Cloudflare won't pressure medium to take down this article, even if it does contain misrepresentations.

orf 6y ago

> If a Cloudflare customer has configured their origin server to respond only to Cloudflare IPs, then they MUST also verify that the "Host" header on any request actually matches their domain name. Out of interest do you provide any automatic checks for this? It seems like it would be trivial to add some kind of verification step that simulates a request with a bogus host header and checks the status code of the response.

EB66 6y ago
What would CloudFlare automatically check the Host header against? The Host header would be set to the attacker's domain (which is in CloudFlare) and, per OP, CloudFlare does not verify ownership of the origin server IP. It would be a nice feature if CloudFlare customers could opt-in for origin server IP verification. Once the origin IP is verified, it would prevent other CloudFlare users from using the IP as their own origin. However, I understand why CloudFlare doesn't verify origin IPs -- it would require a verification process that doesn't rely solely on uploading a verification file to a web accessible directory. It'd be a bit difficult to keep the verification process simple for end-users if they aren't running a web server. Or maybe I'm wrong and non-HTTP use cases don't need to be considered and a verification file uploaded to a web accessible folder would work fine.
- orf 6y ago
  Somewhere in the interface you’d have a button that when pressed would send a request to the specified origin server with a Host header set to “foobar.com”. If the status code is 200 you’d display a warning saying “perhaps you should re-read our documentation about verifying host headers”. I can think of a whole bunch of corner cases to this, but that’s the general jist.
  - EB66 6y ago
    But in order to safely enable that check for all requests that go from CloudFlare to the origin IP, you would first need to verify ownership of the origin IP. You wouldn't want users who don't own the origin IP to be able to turn on that sort of check and potentially cause a service disruption. It could be the case that the legitimate owner already has a Host header mismatch in their HTTP requests/responses.

kentonv 6y ago

A couple other notes: * The IP address 2a06:98c0:3600::103, mentioned in the article as appearing in the CF-Connecting-IP header, is a special IP address that is used for all cross-zone requests that come from Workers. This is not the actual address of any Cloudflare machine. Workers fundamentally don't have distinct IP addresses. Instead, the CF-Worker header needs to be used to identify which worker sent the request. We intentionally do not identify the original client's IP here because the Worker itself could be working against the client, and could have maliciously modified their request to be something entirely different. Hence, the request can't be considered to have come from the original client. * Any cloud service that lets you make HTTP requests can, obviously, be used to set up a proxy that hides your IP address. Workers is actually less useful for this compared to many other cloud hosts since we tag all outgoing HTTP requests from a Worker with the sender's domain name (in the CF-Worker header), which makes it relatively easier to track, report, and block abuse.

abiro 6y ago

toredash 6y ago
I'm 100% sure it isn't. Cloudflare docs are not great, and as a Enterprise customer I spend more time with support than needed as a result.

aeyes 6y ago

> If a Cloudflare customer has configured their origin server to respond only to Cloudflare IPs, then they MUST also verify that the "Host" header on any request actually matches their domain name. The worker code sends the correct host name. https://github.com/jychp/cloudflare-bypass/blob/dd7c1cfc2e6b43d729333ed6b3736925cbc4aea1/cfproxy.py#L33 https://github.com/jychp/cloudflare-bypass/blob/dd7c1cfc2e6b...

kentonv 6y ago
Right. If the hostname requested by the worker is a host that is itself protected by Cloudflare, then all of that host's usual security settings apply. Nothing is bypassed. The author of the blog post seems to have been confused by the fact that different behavior happens if the target host is on Cloudflare vs. not on Cloudflare.

r1ch 6y ago

> If a Cloudflare customer has configured their origin server to respond only to Cloudflare IPs, then they MUST also verify that the "Host" header on any request actually matches their domain name. This could be improved considerably if Cloudflare removed support for insecure TLS options. "Full" SSL bypasses certificate hostname verification on the origin server which could prevent such an attack at the TLS level. Every time I've added a new zone to Cloudflare, Full or Flexible (ie, no SSL) has been the default. The only safe option is Full (Strict) but I'd bet good money that the majority of Cloudflare customers aren't using that.

MuffinFlavored 6y ago

From the article: > X-FORWARDED-FOR: 1.2.3.4 (yes, we can override this header with whatever we want !) Why aren't you guys disallowing that field from having any external value?

modoc 6y ago
Because it's legitimate to have values before the request gets to CloudFlare, for instance if it was routed out through a corporate proxy, etc..
- MuffinFlavored 6y ago
  Ouch. I wonder if it's time to cryptographically sign those values or add a checksum or something.
  - acdha 6y ago
    Basically this is why you want to go full zero trust: if you use mutual authentication over TLS, this becomes much less of a concern since you’re not trying to craft policies based on IP addresses with multiple intermediaries.
- ec109685 6y ago
  Only possible if non-TLS or the company is man in the middling TLS (and certificate pinning isn’t enabled).
  - allending 6y ago
    So... completely possible then.
kentonv 6y ago
So we're supposed to be appending the client's IP to whatever the client sent for X-Forwarded-For, documented here: https://support.cloudflare.com/hc/en-us/articles/200170986-How-does-Cloudflare-handle-HTTP-Request-headers- https://support.cloudflare.com/hc/en-us/articles/200170986-H... For better or worse, this is how X-Forwarded-For works, by convention: each forwarder is supposed to append a value. This means that, even in a correct implementation, an attacker can specify an arbitrary prefix of values. The consumer must be very careful about which part of the chain they trust. But the report here seems to be suggesting that we're mishandling this appending in some cases. We're investigating and if true will fix ASAP. Because X-Forwarded-For is surprisingly complicated, we recommend using CF-Connecting-IP instead. As the docs say: "To restore original visitor IP addresses at your origin web server, Cloudflare recommends your logs or applications look at CF-Connecting-IP or True-Client-IP instead of X-Forwarded-For since CF-Connecting-IP and True-Client-IP have a consistent format containing only one IP."

zamalek 6y ago

> This attack has always been possible and is common to basically all CDNs. I strongly recommend you play around with Azure CDN, because this is not entirely true.

njbooher 6y ago

Once upon a time: https://njbooher.github.io/blog/cloudflare-workers-ip-spoofing https://njbooher.github.io/blog/cloudflare-workers-ip-spoofi...

kentonv 6y ago
Yep, I worked on that one. Thank you for reporting it. Ironically, the purported problem in this case is the opposite: Because we don't attribute the request as coming from the original client (in order to avoid the security problem you reported), a client that has bad reputation associated with their IP can proxy through a worker to hide that. The answer is to assign reputation to the Worker itself.
- njbooher 6y ago
  Yeah, I don't envy you. Finding loopholes is more fun than trying to account for everything upfront.
- ec109685 6y ago
  You see the request on “both sides” of the worker, so shouldn’t you be able to evaluate the “connecting client to the worker’s reputation” when allowing or denying the upstream request?
  - kentonv 6y ago
    You mean by somehow evaluating whether the incoming request and the outgoing one are sufficiently "the same", in which case the outgoing request can be attributed to the original client? This seems hard. Let's say the request is the very simplest "GET /" request. The worker does nothing except rewrite the hostname, in order to proxy it to a different server. Is it "the same" request? Well, imagine the new destination host is a company-internal dashboard that authenticates users based on IP address. Now this "GET /" request which was proxied will receive a response that contains secrets. The worker that did the proxying can intercept the response and learn the secrets. So clearly if the request's host is rewritten, it cannot be "the same" request and it cannot be considered to have come from the original client. But the only case where we care about any of this is when a worker makes a request to a different domain. If a request is to the same domain, we trust it, because in that case the attacker can only attack themselves anyway. So the only case that even matters here is when the host has been rewritten, and in that case, we've established that the new request definitely cannot be attributed to the original client. So it seems to me there's nothing interesting we can do here. A request coming out of a worker (going to a different domain) must always be attributed to the worker itself, not to the client that triggered the worker.

thesis 6y ago

This seems like the same kind of reason we're not using Cloudflare Access to protect some internal resources. We'd love to use Argo Tunnel but unfortunately it breaks with an OPTIONS request for us. We've tried opening tickets with Cloudflare.

meowface 6y ago

>If a Cloudflare customer has configured their origin server to respond only to Cloudflare IPs, then they MUST also verify that the "Host" header on any request actually matches their domain name. If they do not verify the Host header, then anyone can sign up for Cloudflare and simply configure their DNS to point to the victim's origin IP address, and requests will be routed there -- but will have the attacker's domain in the Host header. What exactly could such a misconfiguration enable? Just that the attacker could configure lower levels of Cloudflare security settings for their domain, to bypass the origin's intended Cloudflare security settings? Would this also bypass I'm Under Attack Mode?

btown 6y ago

> any requests coming from a Worker will have the CF-Worker header identifying the zone which sent the request. If a customer suspects abusive requests coming from Workers, they should report it to us. What happens if you receive a report of abusive behavior from a Worker? Would you rate limit the specific Worker, or would there be any impact on the larger Cloudflare account of the Worker's owner? For instance, if we have a Worker that connects to an arbitrary endpoint based on user-generated parameters, or some vulnerability is found (in the JS code for the Worker) that allows this to happen, and it ends up being used for abusive behavior, would we have recourse to fix the vulnerability, or would action be taken immediately? Would our larger account, or our unrelated Workers, be at risk of being taken offline?

kentonv 6y ago
These issues are handled on a case-by-case basis. I can't make any hard promises here (it's not my department), but generally, if your worker is not obviously malicious, we will talk to you before taking any drastic action.

maxgashkov 6y ago

It seems to me that author of the article implies that Cloudflare treats Workers' IPs as having much higher reputation (threat score in CF's terms) than any other IP on the Internet by default. This allows worker to bypass bot protection as it does not trigger at all or only when zone is configured to trigger on all requests (under attack mode). Could you comment on that?

lilyball 6y ago
It looks to me like it’s just about how you can use Cloudflare as a reverse proxy, to hide the fact that your origin is Tor. And the bit about being able to change X-FORWARDED-FOR seems to just be about defeating IP-based rate limiting on sites that trust Cloudflare sufficiently to respect the X-FORWARDED-FOR header.
kentonv 6y ago
To be honest, I don't know anything about Cloudflare's bot management product. That isn't my department. I would speculate that proxying through any cloud service (Workers, AWS, Digital Ocean, whatever) can indeed be a tool to hide bot activity, and that any bots product needs to think about how to combat that. I would also speculate that originating IP address is not the only signal the product uses to determine threat. Moreover, I would speculate that this is all an arms race, there will be a constant stream of new techniques developed by bot authors and new techniques deployed to thwart them. You may be right that the blog author intended to report a problem specifically in bot management, and I have no idea whether that specific problem is real. However, the blog post is not very clear, and many readers have misinterpreted it as reporting a way to totally bypass all Cloudflare protections, such as WAF, geo firewall, rate limiting, etc. That's the misunderstanding I meant to address. Sorry, I could probably have been clearer about that.