6 ms·

Tunnel Vision: CloudflareD AbuseD in the Wild

51 points by redm 3y ago

ehhthing 3y ago
I don't understand why this is news for anybody. This is just the cloudflare version of ngrok...
- deleted 3y ago
  [deleted]
- TechBro8615 3y ago
  It's a bit more pernicious than ngrok just because it's more difficult to isolate and block only malicious usage of the feature. But yeah I agree it's not breaking news that implants will find ways to tunnel out of your infrastructure when exfiltrating data...
- neodymiumphish 3y ago
  Doesn't ngrok require a local config? This technique allows attackers to maintain a nonfunctional tunnel and enable functionality only when they want to engage with the victim machine, similar to TA569's SocGholish campaigns across news sites last year.
Daviey 3y ago
As an enterprise cloudflare customer, we were interested in using this product for legitimate purposes but also block internal non-legitimate access and asked them for advice how to do this.. Support wasn't able to offer any guidance. EDIT: Toned down the language.
- sam-at-cf 3y ago
  Hi from the Cloudflare Product team - blank stares shouldn't be the response there. Can you send me a note, srhea AT cloudflare, and I'll make sure you get a response?
  - leetrout 3y ago
    I vouched for your comment. You might want to email hacker news mods so they can make sure your account is in good standing.
    - mike_d 3y ago
      I'm actually glad HN is putting some effort into combating the Cloudflare employee vote brigading.
      - leetrout 3y ago
        I was unaware there was an issue. any prior info about this?
  - Daviey 3y ago
    Hi Sam, thank you for the constructive reply. I've toned down what I said. I'm not able to talk about the specific example, but are you able to provide guidance for this generic issue of how organisations could block entirely this service at the egress edge and/or make it selective for approved/unapproved usage? Thanks
Cyykratahk 3y ago
I feel like this is just another case of "It rather involved being on the other side of this airtight hatchway" [0] If an attacker is capable of installing apps on your server... you've already lost. 0. https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31283 https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31...
- mschuster91 3y ago
  > If an attacker is capable of installing apps on your server... you've already lost. You're right, but cloudflared is a whole lot easier to get running than previous RATs, and attackers don't need to operate their own c&c infrastructure - you can't defend yourself from such attacks by blanket banning Chinese, Russian, Iranian and North Korean IP addresses (which I honestly advise everyone to do if they don't have business in that country), and you can't easily block outbound to Cloudflare either as half the Internet is hiding behind them. Basically, cloudflared lowers the price and effort for attackers dramatically, while the effort to defend against this threat model has now risen significantly. If Cloudflare were to actually be willing to do something against being used by scammers, they'd put the ingress IPs for the C&C infrastructure on dedicated IP ranges and publish these in a machine-readable format so every reasonable person that does not use Cloudflare tunnels can ban them. (Side note: "zero trust" needs to die)
  - josephcsible 3y ago
    > If Cloudflare were to actually be willing to do something against being used by scammers, they'd put the ingress IPs for the C&C infrastructure on dedicated IP ranges and publish these in a machine-readable format so every reasonable person that does not use Cloudflare tunnels can ban them. I don't like the idea of making it easier to block certain services, because it goes both ways: it'd also be easier for bad guys to block good guys from using said services.
    - eigenmeister 3y ago
      I'm not seeing how the GP's idea could be used by bad guys to block access to services - it'd be something you could add to your firewall, if you don't use this service, to prevent it's use in gaining persistence. If the bad guys can modify your firewall config, you already have a problem.
filleokus 3y ago
Hmm, risking sounding like the dropbox-is-only-ftp-and-svn guy but shouldn't any threat actor worth their salt be able to use any array of open source tools to do this? Especially if you already have access on the machine and the possibility to place binaries there? The biggest risk I see is if the target is already using this for legit use cases, since I guess it would be really difficult to discern between the two.
- j16sdiz 3y ago
  > I guess it would be really difficult to discern between the two I guess renaming the malware binary to chrome .exe would have the same risk?
  - filleokus 3y ago
    The article mentions other signals, like port numbers and DNS lookups for outgoing connections to Cloudflare when setting up the tunnel. If you use this Cloudflare feature for other stuff, you obviously can’t use those as high SNR indicators.
redm 3y ago
I find that more and more CloudFlare is so ubiquitous that we have to use CloudFlare tools to protect ourselves from other people attacking us via CloudFlare. Here's an example of a rule we had to put in place to block people using CloudFlare workers to scrape pages and bypass security. CloudFlare doesn't seem to care about this kind of abuse (or maybe they do but aren't talking about it publicly). (!cf.bot_management.verified_bot) and (cf.bot_management.score lt 10 ) and len(cf.worker.upstream_zone) gt 0 and not cf.worker.upstream_zone in {"<zone>"} and (not ip.geoip.asnum eq <exception as>)
- kentonv 3y ago
  We definitely care about abuse generated by Workers. A request coming from someone else's Worker should not "bypass" any security -- it should be treated the same as if they were running their bot on any other cloud provider and making requests across the internet. If you find otherwise, please file it as a security vulnerability: https://hackerone.com/cloudflare https://hackerone.com/cloudflare (Actually, it should be slightly harder to run this kind of abuse from Workers, because any request coming from a Worker has the cf-worker header identifying the domain that owns the worker. It looks like your block rule is actually taking advantage of this. Other cloud providers usually cannot inject headers like this so it's harder to trace and block abuse from them.) (I'm the tech lead for Workers.)
  - redm 3y ago
    Thank you for your response. Since I have your ear, I'll provide some direct feedback. "We definitely care about abuse generated by Workers." That may be the case internally but it's not as evident through support. I base this on their ability to answer security-related questions, diagnose odd behavior, or mitigate problems. FYI, we are paying for your highest tier of support. "it should be treated as if they were running their bot on any other cloud provider and making requests across the internet." Part of this is how Site Analytics and WAF analytics represent worker data (or don't). Even though the worker modifies the contents, the IP is passed through as the end-user IP (at least the way we use them). These analytics systems need to do a better job of identifying anything worker or tunnels related. If this were being done through AWS it would be evident. We can use various mechanisms to mitigate this, but I wanted to clarify why abuse through Workers is different for our implementation. Also, the process to block non-zone workers was undocumented, and it took a couple of weeks for someone to dig it up and only after several trials and errors. Support wasn't sure it would work, and there was no other documentation (at the time). We have yet to get an answer about the high volume of tunnel requests from specific goes. Combine this with the opaque stack you have that causes odd situations. For example, I added a worker that automatically retries 500 errors from the origin. Simple enough. This breaks Zero Trust, though, and causes a redirect loop. There may be documentation somewhere, but there isn't a good breakdown of how Zero Trust fits into a customer-defined worker, and a redirect loop is undoubtedly a poor failure mode. There is a world of difference between the information you have and may provide as the Technical Lead of Workers, and what I'm getting via support or raw documentation. I could provide you more information offline if you want. Edit: Let me just add the other recent thread [1] regarding redirect loops in verification is par for the course. We had the same issue and tried for weeks for progress or resolution. The issue was never resolved, nor could we get information on why the failure mode was so poor, etc. That ambivalence adds up to expecting general ambivalence or inability to make meaningful progress. At some point, you say to yourself, "Why waste my time talking to support and putting in much effort and getting the run-around." [1] https://news.ycombinator.com/item?id=37049016 https://news.ycombinator.com/item?id=37049016
oefrha 3y ago
Well, I would say that’s a pretty good ad for legitimate uses of Cloudflare Tunnel.
mike_d 3y ago
(Context: part of my job is breaking into stuff all day) This is a technique commonly referred to as "living off the land" where an attacker makes use of a tool like cloudflared to conduct an action that would otherwise be blocked by security tools. It makes the defenders job so much harder because you now need to differentiate between your devops team being cool and a legitimate threat inside your network by looking at the exact same indicators generated by the two. Looking for things like unsigned applications making outbound network connections are removed from the defenders toolbox. Yes, cloudflared does the same thing as ngrok. You'll also find that ngrok is blocked in most corporate environments as well for posing an equal risk. As an attacker, you have a good chance of setting off alarms that (should) specifically detect ngrok. I think the point of this post it to highlight that cloudflare tunnels need to be block by default as well and only allowed when there are specific approved use cases.
- flangola7 3y ago
  Your environment shouldn't be able to run unsigned applications period. On Mac and mobile devices this is already the default behavior.
rafaelturk 3y ago
If an attacker is capable of installing apps on your server... than anything is possible. Don't know why mention Cloudflared other than clickbait. O Overall I'm huge fan of CF Zero Trust and tunnels. I wish documenation and examples were clear, but form a security stand point CF is one of the best security solutions we use.
- strangescript 3y ago
  Yes, their docs, everywhere, leave something to be desired. For example, don't put worker code snippets with little context. Always post an entire file that the user can expand and collapse. Snippets assume the user knows things that they might not.
- neodymiumphish 3y ago
  Two reasons I conducted this research. Firstly, because we saw its use in the wild after exploitation occurred. Second, because it poses a significant risk for data exfiltration. Many tools can be used for this effect, but the fact that the configuration can be easily modified attacker-side through the cloudflare dashboard, and only the token for the tunnel is exposed "client" - side poses a serious challenge for defenders, especially IR teams conducting post-breach forensics.
neodymiumphish 3y ago
I'd like to just say that I (the blog author) am not saying Cloudflared is bad. In fact my research on it makes me really want to test out its usefulness for some of my personal projects. But it is important to demonstrate the ways in which this seemingly benign tool can be (and has been) used to conduct nefarious activity if not properly detected and defended against.
ocdtrekkie 3y ago
We have a blanket block on QUIC traffic at work, and it continues to pay off.

51 points by redm 3y ago

ehhthing 3y ago

I don't understand why this is news for anybody. This is just the cloudflare version of ngrok...

deleted 3y ago
[deleted]
TechBro8615 3y ago
It's a bit more pernicious than ngrok just because it's more difficult to isolate and block only malicious usage of the feature. But yeah I agree it's not breaking news that implants will find ways to tunnel out of your infrastructure when exfiltrating data...
neodymiumphish 3y ago
Doesn't ngrok require a local config? This technique allows attackers to maintain a nonfunctional tunnel and enable functionality only when they want to engage with the victim machine, similar to TA569's SocGholish campaigns across news sites last year.

Daviey 3y ago

As an enterprise cloudflare customer, we were interested in using this product for legitimate purposes but also block internal non-legitimate access and asked them for advice how to do this.. Support wasn't able to offer any guidance. EDIT: Toned down the language.

sam-at-cf 3y ago
Hi from the Cloudflare Product team - blank stares shouldn't be the response there. Can you send me a note, srhea AT cloudflare, and I'll make sure you get a response?
- leetrout 3y ago
  I vouched for your comment. You might want to email hacker news mods so they can make sure your account is in good standing.
  - mike_d 3y ago
    I'm actually glad HN is putting some effort into combating the Cloudflare employee vote brigading.
    - leetrout 3y ago
      I was unaware there was an issue. any prior info about this?
- Daviey 3y ago
  Hi Sam, thank you for the constructive reply. I've toned down what I said. I'm not able to talk about the specific example, but are you able to provide guidance for this generic issue of how organisations could block entirely this service at the egress edge and/or make it selective for approved/unapproved usage? Thanks

Cyykratahk 3y ago

I feel like this is just another case of "It rather involved being on the other side of this airtight hatchway" [0] If an attacker is capable of installing apps on your server... you've already lost. 0. https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31283 https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31...

mschuster91 3y ago
> If an attacker is capable of installing apps on your server... you've already lost. You're right, but cloudflared is a whole lot easier to get running than previous RATs, and attackers don't need to operate their own c&c infrastructure - you can't defend yourself from such attacks by blanket banning Chinese, Russian, Iranian and North Korean IP addresses (which I honestly advise everyone to do if they don't have business in that country), and you can't easily block outbound to Cloudflare either as half the Internet is hiding behind them. Basically, cloudflared lowers the price and effort for attackers dramatically, while the effort to defend against this threat model has now risen significantly. If Cloudflare were to actually be willing to do something against being used by scammers, they'd put the ingress IPs for the C&C infrastructure on dedicated IP ranges and publish these in a machine-readable format so every reasonable person that does not use Cloudflare tunnels can ban them. (Side note: "zero trust" needs to die)
- josephcsible 3y ago
  > If Cloudflare were to actually be willing to do something against being used by scammers, they'd put the ingress IPs for the C&C infrastructure on dedicated IP ranges and publish these in a machine-readable format so every reasonable person that does not use Cloudflare tunnels can ban them. I don't like the idea of making it easier to block certain services, because it goes both ways: it'd also be easier for bad guys to block good guys from using said services.
  - eigenmeister 3y ago
    I'm not seeing how the GP's idea could be used by bad guys to block access to services - it'd be something you could add to your firewall, if you don't use this service, to prevent it's use in gaining persistence. If the bad guys can modify your firewall config, you already have a problem.

filleokus 3y ago

Hmm, risking sounding like the dropbox-is-only-ftp-and-svn guy but shouldn't any threat actor worth their salt be able to use any array of open source tools to do this? Especially if you already have access on the machine and the possibility to place binaries there? The biggest risk I see is if the target is already using this for legit use cases, since I guess it would be really difficult to discern between the two.

j16sdiz 3y ago
> I guess it would be really difficult to discern between the two I guess renaming the malware binary to chrome .exe would have the same risk?
- filleokus 3y ago
  The article mentions other signals, like port numbers and DNS lookups for outgoing connections to Cloudflare when setting up the tunnel. If you use this Cloudflare feature for other stuff, you obviously can’t use those as high SNR indicators.

redm 3y ago

I find that more and more CloudFlare is so ubiquitous that we have to use CloudFlare tools to protect ourselves from other people attacking us via CloudFlare. Here's an example of a rule we had to put in place to block people using CloudFlare workers to scrape pages and bypass security. CloudFlare doesn't seem to care about this kind of abuse (or maybe they do but aren't talking about it publicly). (!cf.bot_management.verified_bot) and (cf.bot_management.score lt 10 ) and len(cf.worker.upstream_zone) gt 0 and not cf.worker.upstream_zone in {"<zone>"} and (not ip.geoip.asnum eq <exception as>)

kentonv 3y ago
We definitely care about abuse generated by Workers. A request coming from someone else's Worker should not "bypass" any security -- it should be treated the same as if they were running their bot on any other cloud provider and making requests across the internet. If you find otherwise, please file it as a security vulnerability: https://hackerone.com/cloudflare https://hackerone.com/cloudflare (Actually, it should be slightly harder to run this kind of abuse from Workers, because any request coming from a Worker has the cf-worker header identifying the domain that owns the worker. It looks like your block rule is actually taking advantage of this. Other cloud providers usually cannot inject headers like this so it's harder to trace and block abuse from them.) (I'm the tech lead for Workers.)
- redm 3y ago
  Thank you for your response. Since I have your ear, I'll provide some direct feedback. "We definitely care about abuse generated by Workers." That may be the case internally but it's not as evident through support. I base this on their ability to answer security-related questions, diagnose odd behavior, or mitigate problems. FYI, we are paying for your highest tier of support. "it should be treated as if they were running their bot on any other cloud provider and making requests across the internet." Part of this is how Site Analytics and WAF analytics represent worker data (or don't). Even though the worker modifies the contents, the IP is passed through as the end-user IP (at least the way we use them). These analytics systems need to do a better job of identifying anything worker or tunnels related. If this were being done through AWS it would be evident. We can use various mechanisms to mitigate this, but I wanted to clarify why abuse through Workers is different for our implementation. Also, the process to block non-zone workers was undocumented, and it took a couple of weeks for someone to dig it up and only after several trials and errors. Support wasn't sure it would work, and there was no other documentation (at the time). We have yet to get an answer about the high volume of tunnel requests from specific goes. Combine this with the opaque stack you have that causes odd situations. For example, I added a worker that automatically retries 500 errors from the origin. Simple enough. This breaks Zero Trust, though, and causes a redirect loop. There may be documentation somewhere, but there isn't a good breakdown of how Zero Trust fits into a customer-defined worker, and a redirect loop is undoubtedly a poor failure mode. There is a world of difference between the information you have and may provide as the Technical Lead of Workers, and what I'm getting via support or raw documentation. I could provide you more information offline if you want. Edit: Let me just add the other recent thread [1] regarding redirect loops in verification is par for the course. We had the same issue and tried for weeks for progress or resolution. The issue was never resolved, nor could we get information on why the failure mode was so poor, etc. That ambivalence adds up to expecting general ambivalence or inability to make meaningful progress. At some point, you say to yourself, "Why waste my time talking to support and putting in much effort and getting the run-around." [1] https://news.ycombinator.com/item?id=37049016 https://news.ycombinator.com/item?id=37049016

oefrha 3y ago

Well, I would say that’s a pretty good ad for legitimate uses of Cloudflare Tunnel.

mike_d 3y ago

(Context: part of my job is breaking into stuff all day) This is a technique commonly referred to as "living off the land" where an attacker makes use of a tool like cloudflared to conduct an action that would otherwise be blocked by security tools. It makes the defenders job so much harder because you now need to differentiate between your devops team being cool and a legitimate threat inside your network by looking at the exact same indicators generated by the two. Looking for things like unsigned applications making outbound network connections are removed from the defenders toolbox. Yes, cloudflared does the same thing as ngrok. You'll also find that ngrok is blocked in most corporate environments as well for posing an equal risk. As an attacker, you have a good chance of setting off alarms that (should) specifically detect ngrok. I think the point of this post it to highlight that cloudflare tunnels need to be block by default as well and only allowed when there are specific approved use cases.

flangola7 3y ago
Your environment shouldn't be able to run unsigned applications period. On Mac and mobile devices this is already the default behavior.

rafaelturk 3y ago

If an attacker is capable of installing apps on your server... than anything is possible. Don't know why mention Cloudflared other than clickbait. O Overall I'm huge fan of CF Zero Trust and tunnels. I wish documenation and examples were clear, but form a security stand point CF is one of the best security solutions we use.

strangescript 3y ago
Yes, their docs, everywhere, leave something to be desired. For example, don't put worker code snippets with little context. Always post an entire file that the user can expand and collapse. Snippets assume the user knows things that they might not.
neodymiumphish 3y ago
Two reasons I conducted this research. Firstly, because we saw its use in the wild after exploitation occurred. Second, because it poses a significant risk for data exfiltration. Many tools can be used for this effect, but the fact that the configuration can be easily modified attacker-side through the cloudflare dashboard, and only the token for the tunnel is exposed "client" - side poses a serious challenge for defenders, especially IR teams conducting post-breach forensics.

neodymiumphish 3y ago

I'd like to just say that I (the blog author) am not saying Cloudflared is bad. In fact my research on it makes me really want to test out its usefulness for some of my personal projects. But it is important to demonstrate the ways in which this seemingly benign tool can be (and has been) used to conduct nefarious activity if not properly detected and defended against.

ocdtrekkie 3y ago

We have a blanket block on QUIC traffic at work, and it continues to pay off.