9 ms·

Kubernetes clusters being hijacked to mine cryptocurrencies

183 points by igama 8y ago

WrtCdEvrydy 8y ago
JSON file is still available (http://192.99.142.232:8220/222.json http://192.99.142.232:8220/222.json)
- gammateam 8y ago
  > "algo": "cryptonight", Nice, Monero mining Cryptocurrencies makes the bug bounty market A LOT more efficient than companies, legislation or HackerOne ever could.
nineteen999 8y ago
This is one of the side-effects of products having enormous hype in this industry. Far too many people are adopting Docker/Kubernetes as they have been the hot new product for the last couple of years, often regardless of whether they are actually the best or most appropriate tool for the job. A lot of the people who get sucked into the hype are often inexperienced programmers, devops or admin types who are in positions of power or influence in companies that they probably shouldn't be, IMHO. As a result, they don't have the Linux or networking experience to be able to know when they are deploying these complex products securely or not, and they are putting their employers businesses at risk.
- cobookman 8y ago
  Why do you assume that other platforms are not at similar risk? VMs also have zero-days that have been exploited for cryptomining.
  - balgan 8y ago
    CEO of BinaryEdge here, ur 100% right. If I show you the queue of posts we have you'd see similar posts to this one just with different technologies that we have seen being infected or misused(etcd, docker, and about 10 or 20 more types of DB's).
  - nineteen999 8y ago
    I don't assume that at all. I mentioned Docker explicitly and people are pulling Docker containers from untrusted sources with malware pre-installed, because they lack the experience that would tell them that pulling untrusted Docker containers and running them is a bad idea. https://threatpost.com/malicious-docker-containers-earn-crypto-miners-90000/132816/ https://threatpost.com/malicious-docker-containers-earn-cryp... From the article itself, although they mention the CVE at the top, the real point they are making is that people are deploying the products with poor defaults: "as is typical with our findings, lots of companies are exposing their Kubernetes API with no authentication; inside the Kubernetes cluster" Not to mention a bunch of NoSQL type db's you can easily search on Shodan if you wanted to have some fun. So yes - the problem here is experience, or lack thereof, and not Kubernetes itself. The CVE can be patched. You can't patch inexperience - except with experience I suppose. All I am saying is that there a lot of people who are downloading and deploying these products because of hype, who are unable or unwilling to secure them.
    - cbzbc 8y ago
      Leaving aside NoSQL db's - there's also a ton of normal SQL databases wide open, I don't think hype is necessarily the issue there.
      - nineteen999 8y ago
        Sure, maybe your average garden variety Postgres or MySQL instances, and probably some MS-SQL as well. Companies that have a large investment in commercial RDBMS (eg. Oracle, DB2, etc) tend not to be so careless in my experience.
- justicezyx 8y ago
  Note K8s is designed to control a lot of machines, sometime the entire fleet of smbs. So itself should be more sensitive than other infrastructure pieces. And I think op meant to say that, not that k8s is particularly bad in security in general. Or k8s is less experienced in security. The down vote is not warranted.
  - outworlder 8y ago
    The "hype" part is pretty subjective and may have warranted down votes. It's not hype if it solves a lot of organizations pain points.
    - justicezyx 8y ago
      Let's be honest, pretty much every new tech got hyped initially. K8s is no doubt hype, otherwise it won't enjoy the explosive growth. That's not subjective, at least IMHO
      - bonesss 8y ago
        It's the delineation of 'hype' and 'excitment' that is tricky. If magic CPUs that were 10x better showed up tomorrow we'd all be justified in being very excited. But running with hype around the next Zune? ... that's not excitement backed with meaning.
        justicezyx 8y ago
        Then we should agree to disagree.
- andrewstuart2 8y ago
  I disagree that hype leads to the problem you describe. Kubernetes is good at its job, and therefore it's popular, and therefore it's used by people who may not understand it. You could say the exact same thing about Linux, Cisco, Dell, or pretty much any of the popular FOSS projects. Popular things, regardless of their complexity, get chosen by people of all experience levels. Inexperienced people are less likely to properly configure something, regardless of its popularity or hype. If anything, having a few attractive projects tends to be beneficial (or at least neutral) for security as there are so many more people scrutinizing it, and many more people learning how to properly use it.
- ram_rar 8y ago
  >A lot of the people who get sucked into the hype are often inexperienced programmers, devops or admin types who are in positions of power or influence in companies that they probably shouldn't be, IMHO. I cannot agree more. Many times, I feel you da easily do away with ansible and terraform to setup VMs / docker. you dont quite need k8s. Just cuz K8s are cool.. people feel the need to use it.
  - ownagefool 8y ago
    It's more complicated that that. Whilst some people are probably jumping on kubernetes for the hype, there's a lot of things it makes really easy, especially for less experience teams. For example: - You want to spin up ephemeral environments to test PRs end2end. Sure, create a namespace, deploy your charts and run your tests. You want to do that with ansible, sure you can, but it's harder. - You org is running apps via a multi-cloud and on-prem strategy? Okay, lets just write lots of tooling per cloud and another for on-prem, or we could abstract that away via kubernetes and only worry about tooling for kube itself. - You want to do have rolling-upgrades. Sure, you build them with ansible then, or you could just use kubes. Further to that, kubernetes is guiding reasonble abstractions, seperating infrastructure from code. Sure, it comes with complexity, but so does most things when you start throwing in scaling and auto-recovery. For example, deploy terraform from your laptop? The device you probably browse porn on has becomes an attack surface. Move this to Jenkins, the CI is the attack surface. Put your code on Bitbucket? Bitbucket and the Jenkinsfile becomes the attack surface. Pretty much everything we do has complexity and attack surface _problems_ and using a managed k8s service will allow you some easy wins so you can actually think about those other problems, and those solutions will work on all platforms you can run k8s on.
    - nineteen999 8y ago
      Can I ask because I'm genuinely interested - what on earth do you do for third-party applications (for eg. closed source) that have to be integrated into your environment that don't come pre-packaged in a convenient container? Do you containerize these yourselves, whether or not the vendor says that will support that? Or does it get pushed to some other team that manages whole VM's/AWS instances that are not container hosts. Or is this a scenario that just doesn't happen in your environment? Genuinely curious. Also: > using a managed k8s service will allow you some easy wins so you can actually think about those other problems, and those solutions will work on all platforms you can run k8s on None of which matters one jot, if one cannot properly manage ingress/egress filtering on one's API endpoints, or a reasonable level of password/credential security. One will be used for cryptomining or worse, as per the fine article. In that instance, one needs to go back and get some basic UNIX/Linux/network and security training before one starts playing with complicated software on publicly connected clouds. Or hire some people who actually know what they are doing with respect to that.
gipmon 8y ago
These guys are amazing. They have a lot of data and an excellent app with a lot of potential!
tetha 8y ago
Ugh. I mean, I recently got in an argument if anything but a hard firewall could or should be exposed to a WAN interface on the internet and we kinda agreed to not agree for now. But, popular services, on default ports, with default APIs enabled, without hard authentication on a WAN interface? That should be a paddling. That doesn't fly. Or, well it does, except not for the guy paying the power.
- kevin_nisbet 8y ago
  To be fair, kubernetes itself and most distributions are quite secure by default. So with kubernetes it's not the same as it was NoSQL databases that didn't have authentication that were bound to the internet. I'm not familiar with enough distributions to know if there is a popular distribution that totally disabled authentication by default, but in my companies distribution, kubeadm clusters, and I suspect all managed clusters (GKE/EKS/AKS/etc), the vector outlined in the article would only work if an admin specifically disabled the authentication. In gravity (my companies distribution), we even disable anonymous-auth, so someone would have to do real work to allow API access to the internet.
  - tetha 8y ago
    How do you provide your initial credentials, though? Providing decently secure default initial credentials is possible, but tricky. And that's where I'll turn around 180 degrees and say: If you can't give me a hard reason why you'll be a hard target on the internet, you shouldn't have a public address. Default authentication isn't enough. I dislike trusting my edge firewall, but it gives me time to handle weak internal systems.
    - TheDong 8y ago
      Kubernetes only accepts very limited forms of auth by default. Typically, it's limited to client certificates that have been signed by the private key the apiserver has access to. Client cert auth over tls is pretty damn secure. I expose my kubernetes cluster's apiserver to the internet and have, to my knowledge, had no issues yet.
unstatusthequo 8y ago
Heading continued: “... thieves make off with $4.50”
whalesalad 8y ago
Got in a pretty heated debate with a colleague once about this. We had a really great infrastructure setup with a VPN bastion host that would get you into our VPC. You couldn't reach any of our kube nodes externally. Your Google account was your VPN account. It was pretty solid. When this engineer redid things they opted to go the public internet route where the master runs a public api and auth is done via a certificate. The logic here was so that external 3rd party stuff (CI) could control our master. To my knowledge this setup is still running and chances are these machines are vulnerable to this issue. Contrast to the prior setup where, immediately upon being offboarded from the company your VPN access became automatically terminated (thank you LDAP and Foxpass!)
- deleted 8y ago
  [deleted]
- honkycat 8y ago
  I can't imagine a good reason to expose ANY of my services to the public internet. Aside from a rest-api that drives our application, where that is the feature of course. With software like google IAP, and many similar products, it just seems silly.
  - TheDong 8y ago
    May I recommend reading up on beyondcorp [0]? Google has moved its internal stuff to the beyondcorp model, and it honestly seems like a better approach if you really care about security and have a big enough security team to make it work. [0]: https://www.beyondcorp.com/ https://www.beyondcorp.com/
    - raesene9 8y ago
      Beyondcorp is a great model IF you can afford to manage it correctly. Google have a) huge resources and b) a threat model which means they're subject to a lot of high-end attacks all the time. for many corp's the idea of exposing all their services and endpoints to the general internet without firewalls or VPNs would ... end poorly...
clubm8 8y ago
Is anyone else a little tired of "X used to mine crypto" stories? Yes - if it has a CPU and access to the public internet, someone will hack it and make it mine "cypto". Let's stop pretending we aren't aware that the internet of things exists and writing breathless stories every time a toaster, router, or adult toy starts churning out Monero.
- roguecoder 8y ago
  This is an important vulnerability in widely-used software. Crypto is relevant because the inherent design of crypto makes hacks like this more profitable, but it's not the main thing about the article.
  - igama 8y ago
    Exactly, the main story is Kubernetes being exploited in the wild and in large numbers, Crypto mining is just one of the "attacks" tacking place.
voltagex_ 8y ago
At least cryptocurrency has removed most of the creativity from script kiddies - there's so many more interesting things you could do than just mine coins.
- blazespin 8y ago
  Yeah, exactly. It’s almost like a bounty for find a vuln. It seems to be a mostly harmless attack that doesn’t cause global internet grief like a DDoS or something.
  - gammateam 8y ago
    WannaCry’s very public global ransom brought attention to a Monero mining botnet which was using the same exploit for weeks beforehand, it was making $40,000 per day. It made much more than WannaCry and its operators are still unknown and would have been able to cash out Script kiddies are just annoying and their actions resulted in the patch killing that silent mining botnet as well.
conanthe 8y ago
Is kubernetes a mongodb of orchestrators?
igama 8y ago
CTO Binaryedge here. For those wondering, We have detected more than 15k Kubernetes APIs with Auth. This post focuses on ~1.5k found without Auth, that are fully open. It's not just a Kubernetes Problem. Like many have posted, many databases, other types of clusters, shares, are accessible without Auth for those that know how to look for them (not that hard now days), mainly malicious actors.

183 points by igama 8y ago

WrtCdEvrydy 8y ago

JSON file is still available (http://192.99.142.232:8220/222.json http://192.99.142.232:8220/222.json)

gammateam 8y ago
> "algo": "cryptonight", Nice, Monero mining Cryptocurrencies makes the bug bounty market A LOT more efficient than companies, legislation or HackerOne ever could.

nineteen999 8y ago

This is one of the side-effects of products having enormous hype in this industry. Far too many people are adopting Docker/Kubernetes as they have been the hot new product for the last couple of years, often regardless of whether they are actually the best or most appropriate tool for the job. A lot of the people who get sucked into the hype are often inexperienced programmers, devops or admin types who are in positions of power or influence in companies that they probably shouldn't be, IMHO. As a result, they don't have the Linux or networking experience to be able to know when they are deploying these complex products securely or not, and they are putting their employers businesses at risk.

cobookman 8y ago
Why do you assume that other platforms are not at similar risk? VMs also have zero-days that have been exploited for cryptomining.
- balgan 8y ago
  CEO of BinaryEdge here, ur 100% right. If I show you the queue of posts we have you'd see similar posts to this one just with different technologies that we have seen being infected or misused(etcd, docker, and about 10 or 20 more types of DB's).
- nineteen999 8y ago
  I don't assume that at all. I mentioned Docker explicitly and people are pulling Docker containers from untrusted sources with malware pre-installed, because they lack the experience that would tell them that pulling untrusted Docker containers and running them is a bad idea. https://threatpost.com/malicious-docker-containers-earn-crypto-miners-90000/132816/ https://threatpost.com/malicious-docker-containers-earn-cryp... From the article itself, although they mention the CVE at the top, the real point they are making is that people are deploying the products with poor defaults: "as is typical with our findings, lots of companies are exposing their Kubernetes API with no authentication; inside the Kubernetes cluster" Not to mention a bunch of NoSQL type db's you can easily search on Shodan if you wanted to have some fun. So yes - the problem here is experience, or lack thereof, and not Kubernetes itself. The CVE can be patched. You can't patch inexperience - except with experience I suppose. All I am saying is that there a lot of people who are downloading and deploying these products because of hype, who are unable or unwilling to secure them.
  - cbzbc 8y ago
    Leaving aside NoSQL db's - there's also a ton of normal SQL databases wide open, I don't think hype is necessarily the issue there.
    - nineteen999 8y ago
      Sure, maybe your average garden variety Postgres or MySQL instances, and probably some MS-SQL as well. Companies that have a large investment in commercial RDBMS (eg. Oracle, DB2, etc) tend not to be so careless in my experience.
justicezyx 8y ago
Note K8s is designed to control a lot of machines, sometime the entire fleet of smbs. So itself should be more sensitive than other infrastructure pieces. And I think op meant to say that, not that k8s is particularly bad in security in general. Or k8s is less experienced in security. The down vote is not warranted.
- outworlder 8y ago
  The "hype" part is pretty subjective and may have warranted down votes. It's not hype if it solves a lot of organizations pain points.
  - justicezyx 8y ago
    Let's be honest, pretty much every new tech got hyped initially. K8s is no doubt hype, otherwise it won't enjoy the explosive growth. That's not subjective, at least IMHO
    - bonesss 8y ago
      It's the delineation of 'hype' and 'excitment' that is tricky. If magic CPUs that were 10x better showed up tomorrow we'd all be justified in being very excited. But running with hype around the next Zune? ... that's not excitement backed with meaning.
      - justicezyx 8y ago
        Then we should agree to disagree.
andrewstuart2 8y ago
I disagree that hype leads to the problem you describe. Kubernetes is good at its job, and therefore it's popular, and therefore it's used by people who may not understand it. You could say the exact same thing about Linux, Cisco, Dell, or pretty much any of the popular FOSS projects. Popular things, regardless of their complexity, get chosen by people of all experience levels. Inexperienced people are less likely to properly configure something, regardless of its popularity or hype. If anything, having a few attractive projects tends to be beneficial (or at least neutral) for security as there are so many more people scrutinizing it, and many more people learning how to properly use it.
ram_rar 8y ago
>A lot of the people who get sucked into the hype are often inexperienced programmers, devops or admin types who are in positions of power or influence in companies that they probably shouldn't be, IMHO. I cannot agree more. Many times, I feel you da easily do away with ansible and terraform to setup VMs / docker. you dont quite need k8s. Just cuz K8s are cool.. people feel the need to use it.
- ownagefool 8y ago
  It's more complicated that that. Whilst some people are probably jumping on kubernetes for the hype, there's a lot of things it makes really easy, especially for less experience teams. For example: - You want to spin up ephemeral environments to test PRs end2end. Sure, create a namespace, deploy your charts and run your tests. You want to do that with ansible, sure you can, but it's harder. - You org is running apps via a multi-cloud and on-prem strategy? Okay, lets just write lots of tooling per cloud and another for on-prem, or we could abstract that away via kubernetes and only worry about tooling for kube itself. - You want to do have rolling-upgrades. Sure, you build them with ansible then, or you could just use kubes. Further to that, kubernetes is guiding reasonble abstractions, seperating infrastructure from code. Sure, it comes with complexity, but so does most things when you start throwing in scaling and auto-recovery. For example, deploy terraform from your laptop? The device you probably browse porn on has becomes an attack surface. Move this to Jenkins, the CI is the attack surface. Put your code on Bitbucket? Bitbucket and the Jenkinsfile becomes the attack surface. Pretty much everything we do has complexity and attack surface _problems_ and using a managed k8s service will allow you some easy wins so you can actually think about those other problems, and those solutions will work on all platforms you can run k8s on.
  - nineteen999 8y ago
    Can I ask because I'm genuinely interested - what on earth do you do for third-party applications (for eg. closed source) that have to be integrated into your environment that don't come pre-packaged in a convenient container? Do you containerize these yourselves, whether or not the vendor says that will support that? Or does it get pushed to some other team that manages whole VM's/AWS instances that are not container hosts. Or is this a scenario that just doesn't happen in your environment? Genuinely curious. Also: > using a managed k8s service will allow you some easy wins so you can actually think about those other problems, and those solutions will work on all platforms you can run k8s on None of which matters one jot, if one cannot properly manage ingress/egress filtering on one's API endpoints, or a reasonable level of password/credential security. One will be used for cryptomining or worse, as per the fine article. In that instance, one needs to go back and get some basic UNIX/Linux/network and security training before one starts playing with complicated software on publicly connected clouds. Or hire some people who actually know what they are doing with respect to that.

gipmon 8y ago

These guys are amazing. They have a lot of data and an excellent app with a lot of potential!

tetha 8y ago

Ugh. I mean, I recently got in an argument if anything but a hard firewall could or should be exposed to a WAN interface on the internet and we kinda agreed to not agree for now. But, popular services, on default ports, with default APIs enabled, without hard authentication on a WAN interface? That should be a paddling. That doesn't fly. Or, well it does, except not for the guy paying the power.

kevin_nisbet 8y ago
To be fair, kubernetes itself and most distributions are quite secure by default. So with kubernetes it's not the same as it was NoSQL databases that didn't have authentication that were bound to the internet. I'm not familiar with enough distributions to know if there is a popular distribution that totally disabled authentication by default, but in my companies distribution, kubeadm clusters, and I suspect all managed clusters (GKE/EKS/AKS/etc), the vector outlined in the article would only work if an admin specifically disabled the authentication. In gravity (my companies distribution), we even disable anonymous-auth, so someone would have to do real work to allow API access to the internet.
- tetha 8y ago
  How do you provide your initial credentials, though? Providing decently secure default initial credentials is possible, but tricky. And that's where I'll turn around 180 degrees and say: If you can't give me a hard reason why you'll be a hard target on the internet, you shouldn't have a public address. Default authentication isn't enough. I dislike trusting my edge firewall, but it gives me time to handle weak internal systems.
  - TheDong 8y ago
    Kubernetes only accepts very limited forms of auth by default. Typically, it's limited to client certificates that have been signed by the private key the apiserver has access to. Client cert auth over tls is pretty damn secure. I expose my kubernetes cluster's apiserver to the internet and have, to my knowledge, had no issues yet.

unstatusthequo 8y ago

Heading continued: “... thieves make off with $4.50”

whalesalad 8y ago

Got in a pretty heated debate with a colleague once about this. We had a really great infrastructure setup with a VPN bastion host that would get you into our VPC. You couldn't reach any of our kube nodes externally. Your Google account was your VPN account. It was pretty solid. When this engineer redid things they opted to go the public internet route where the master runs a public api and auth is done via a certificate. The logic here was so that external 3rd party stuff (CI) could control our master. To my knowledge this setup is still running and chances are these machines are vulnerable to this issue. Contrast to the prior setup where, immediately upon being offboarded from the company your VPN access became automatically terminated (thank you LDAP and Foxpass!)

deleted 8y ago
[deleted]
honkycat 8y ago
I can't imagine a good reason to expose ANY of my services to the public internet. Aside from a rest-api that drives our application, where that is the feature of course. With software like google IAP, and many similar products, it just seems silly.
- TheDong 8y ago
  May I recommend reading up on beyondcorp [0]? Google has moved its internal stuff to the beyondcorp model, and it honestly seems like a better approach if you really care about security and have a big enough security team to make it work. [0]: https://www.beyondcorp.com/ https://www.beyondcorp.com/
  - raesene9 8y ago
    Beyondcorp is a great model IF you can afford to manage it correctly. Google have a) huge resources and b) a threat model which means they're subject to a lot of high-end attacks all the time. for many corp's the idea of exposing all their services and endpoints to the general internet without firewalls or VPNs would ... end poorly...

clubm8 8y ago

Is anyone else a little tired of "X used to mine crypto" stories? Yes - if it has a CPU and access to the public internet, someone will hack it and make it mine "cypto". Let's stop pretending we aren't aware that the internet of things exists and writing breathless stories every time a toaster, router, or adult toy starts churning out Monero.

roguecoder 8y ago
This is an important vulnerability in widely-used software. Crypto is relevant because the inherent design of crypto makes hacks like this more profitable, but it's not the main thing about the article.
- igama 8y ago
  Exactly, the main story is Kubernetes being exploited in the wild and in large numbers, Crypto mining is just one of the "attacks" tacking place.

voltagex_ 8y ago

At least cryptocurrency has removed most of the creativity from script kiddies - there's so many more interesting things you could do than just mine coins.

blazespin 8y ago
Yeah, exactly. It’s almost like a bounty for find a vuln. It seems to be a mostly harmless attack that doesn’t cause global internet grief like a DDoS or something.
- gammateam 8y ago
  WannaCry’s very public global ransom brought attention to a Monero mining botnet which was using the same exploit for weeks beforehand, it was making $40,000 per day. It made much more than WannaCry and its operators are still unknown and would have been able to cash out Script kiddies are just annoying and their actions resulted in the patch killing that silent mining botnet as well.

conanthe 8y ago

Is kubernetes a mongodb of orchestrators?

igama 8y ago

CTO Binaryedge here. For those wondering, We have detected more than 15k Kubernetes APIs with Auth. This post focuses on ~1.5k found without Auth, that are fully open. It's not just a Kubernetes Problem. Like many have posted, many databases, other types of clusters, shares, are accessible without Auth for those that know how to look for them (not that hard now days), mainly malicious actors.