9 ms·

SaltStack Mining Attack

157 points by photon-torpedo 6y ago

alexandercrohde 6y ago
>>> If you're here, chances are you're already compromised. WTF does that mean? Our salt implementation is on an entirely private network, so why would I be more likely than not to be compromised already? ---- Edit: Re-downvotes -- This is a sincere question. Is there some evidence that the majority of salt implementations are compromised, or some mechanism by which this hits private networks? Or is that line just for dramatic effect?
- jkingsman 6y ago
  With you on this one... a coworker put it well "[this tone is] at a 9, it needs to be at a 3".
- busterarm 6y ago
  99% of companies I've encountered running SaltStack are masterless anyway.
  - saltyguy22 6y ago
    Depends on their use case. IMO, Salts best feature is probably running with the Master / Minion setup because it connects out to the master. Masterless is handy for using in conjunction with Vagrant, Packer, or another provisioning tool not so much for managing a the lifecycle of a server / OS.
    - busterarm 6y ago
      I'm not clueless. I know this. This doesn't detract at all from what I said. In production, an outsized number of companies are using SaltStack a certain way. The reason is because of the way we all evaluate business risk. I could have said that out of all of the companies I've worked with using Salt that have compliance requirements, 100% of them are using it Masterless, but then some smart-ass would have piped up with a "not me" comment.
      - Ascetik 6y ago
        not me :D
      - kapilvt 6y ago
        How would you compare it to ansible in that usage model? I don't do much config mgmt these days, so curious for a master(less) mindset, whats the comparison for salt vs. ansible?
        busterarm 6y ago
        Ansible gets a ton more use. Its push model is highly favored in IT organizations. Salt's event/reactor system has tricks Ansible can't do.
      - rexarex 6y ago
        Not me
  - devonkim 6y ago
    Salt is a bit of a rarity among ops people but the funny thing is my experience is the opposite of yours - 66%+ of companies I see with Salt run in master/minion mode. I prefer it over Ansible when my ssh settings are super awkward (MFA, multiple bastions, etc) to integrate into an Ansible inventory file. One place with 900+ hosts and 300+ random IP ranges with tons of (badly written) compliance I spent about 3 weeks trying to get an inventory file hacked up and I gave up and deployed Salt within a day for the basic stuff I needed to do for basic running of shell commands from a single point of control.
    - busterarm 6y ago
      Most of the places I've seen it are banks. Oddly. Your use case makes absolute perfect sense though -- nice!
- twic 6y ago
  I assumed that the problem was that if a master is accessible on your intranet, it could be hit with some sort of XSS attack from browsers inside the firewall. But apparently there are 6000 people just straight up exposing their masters to the internet: https://gbhackers.com/saltstack-salt/ https://gbhackers.com/saltstack-salt/
  - NikolaeVarius 6y ago
    > This whole "don't have your salt master exposed to the internet" thing has me annoyed. The whole point of salt is to manage boxes all over the place. I manage around 500 machines. Most of them are behind the firewalls of incompetent admins who have spent hours in the past trying to set up port forwards when salt-minion crashed so I could access the box again. I'm about to test binding salt-master to localhost and salt-minion to localhost and then setting up spiped to wrap the traffic... Some companies need better DevOps apparently
    - dijit 6y ago
      Nitpick: this is purely an operations/sysadmin problem. The DevOps grouping doesn't apply too much, Operations-minded staff should be focusing on keeping things locked down. Unless, you know, you're not hiring those people and instead are hoping that developers take the Ops burden. ;)
      - deleted 6y ago
        [deleted]
      - dijit 6y ago
        People downvoting: I guess I hit a nerve, but could you explain why? Development and Operations are different disciplines and the idea was to remove silos, not make one person responsible for both.
  - Spivak 6y ago
    But that's like half the point of SaltStack. It's supposed be secure enough to run on the public internet to manage road-warrior endpoints. "I found this site called Alexa that lists a bunch of companies that just straight up expose their web servers to the internet. Crazy." And I have some bad news about how many companies are exposing their VPN servers to the internet too.
- TheDong 6y ago
  >>>> If you're here, chances are you're already compromised. > WTF does that mean? The website assumes that if you dropped by or googled the vulnerability, it's because you had saltstack exposed to the public internet. You don't have to react so harshly to that turn of phrase.
  - alexandercrohde 6y ago
    My main impression was that this site was blurring the line between professional "Bulletin" and scare-mongering. > Even if you didn't notice any unexpected symptoms, please: nuke and restart. If I were writing this, I'd probably write something like "On May x, a remote-code-execution in (all versions?) public salt-masters (not minions?), was unveiled. Shortly thereafter, actual exploitation in the wild is being used. If your salt installation uses a salt-master, and it's internet-reachable, it may already be compromised, along with much your infrastructure. Section 2 is how to see if you are infected, and Section 3 is how to remove the infection."
    - TheDong 6y ago
      > Section 2 is how to see if you are infected and Section 3 is how to remove the infection That's much worse. That implies there is a reliable way to detect and remove the infection. That's not the case. This website included some known attacks and such. There's a high chance that there were attacks of this vulnerability with additional payloads. There's no way to know if you were hit with a rootkit that persists itself in the bootloader or other parts of your system. There's no way to know if you were infected or not. The only way to be sure is to nuke the machine, as they said. Security advice should always err on the right side for a naive reader.
formersaltuser 6y ago
SaltStack has a long history of home brew protocols, internally written encryption, security issues, and bugs. I'm not surprised they would end up being used as a vector for attacks.
- riffic 6y ago
  Something about Salt has always rubbed me the wrong way. I'd rather stick with Puppet, Chef, or CF Engine if I was considering this tool.
  - xenophonf 6y ago
    I'm all about IT automation and like the tech a lot, but after running SaltStack for 5 years, I decided to transition to plain old written documentation. When you consider everything that underlies the "infrastructure as code" tech stack, it ends up being an extremely steep learning curve. I was very productive with SaltStack but turned into the only person who could write or edit our deployment scripts. It was a bad situation. Now, nine months after dropping SaltStack, my colleagues are editing the docs I've created, and more importantly, they're also writing their own. That's a huge win. What I've lost in terms of personal productivity, I've regained in terms of wider participation in our standardization and documentation efforts. I might slowly re-introduce IT automation technologies, maybe something more popular like Ansible or Docker, but only after making sure the rest of my team has a solid understanding of general IT automation concepts. I think we're on the right track. I had a colleague today ask me about what Python programming certification they should get, so I pointed them toward Google's Python-based IT automation course on Coursera. I supported another colleague's efforts to create a library of standard server images based on our internal deployment standards for one of our private clouds, and I'm encouraging them to expand that work to our other data centers. And so on. The team as a whole works better together, so that's where I'm staying focused.
    - chucky_z 6y ago
      So you went from full automation to... Copying and pasting by hand?
docsapp_io 6y ago
This impacted ghost.org hosting really bad https://status.ghost.org/ https://status.ghost.org/
- _vafj 6y ago
  Impacted LineageOS too - https://twitter.com/LineageAndroid/status/1256821056100163584 https://twitter.com/LineageAndroid/status/125682105610016358...
  - twic 6y ago
    Oh god, the responses. Half of them are completely unrelated, asking Lineage to support particular phones. And then there's someone who saw "salt" and tried to help: https://twitter.com/mvonwi/status/1256989787321438209 https://twitter.com/mvonwi/status/1256989787321438209
rhakyr 6y ago
Debian hasn't fixed this yet in their packaging so you might want to work around that if you are using that as a salt-master server. https://security-tracker.debian.org/tracker/CVE-2020-11651 https://security-tracker.debian.org/tracker/CVE-2020-11651
thesz 6y ago
How does things like these affect Monero's reputation? I saw Monero mining in JS libraries, now it is in virus form. Thus, why Monero? Does Monero somehow discourage these "practices"?
- jackhalford 6y ago
  Maybe it's to do with the fact that monero tokens are non-fungible, unlike some other crypto-currencies.
  - 3456j34jht 6y ago
    I think you mean 'fungible'. Non-fungible means 1 XMR is not equivalent to another 1 XMR. The use of ring signatures, key images and transaction amount hiding gives Monero its fungible properties. This makes it computationally difficult to trace a particular transaction graph. This provides fungibility beyond that of Nakamoto style coins, in that you cannot easily block/identify addresses or transactions.
  - thinkmassive 6y ago
    Monero is likely the absolute most fungible cryptocurrency.
- colatkinson 6y ago
  IIRC one of Monero's design goals is that it isn't easily GPU/ASIC-mineable, so CPU mining is competitive. It's also designed to be difficult to trace. Both of these properties make it a relatively safe and profitable avenue for malware authors.
- asymptotically2 6y ago
  Mining with JavaScript or WebAssembly is no longer viable as being competitive now requires using a JIT compiler. Nothing can really be done about it. Pool operators can ban the wallet addresses of known bot masters from connecting to their pool, but you can't really ban someone from participating in an anonymous decentralised network.
scblzn 6y ago
Algolia got impacted too, apparently their Salt masters were opened to the whole internet: https://blog.algolia.com/salt-incident-may-3rd-2020-retrospective-and-update/ https://blog.algolia.com/salt-incident-may-3rd-2020-retrospe...
- jlgaddis 6y ago
  One of DigiCert's Certificate Transparency logs was likewise open to the entire Internet -- and likewise compromised [0]: > I'm sad to report that we discovered today that CT Log 2's key used to sign SCTs was compromised last night at 7 pm via the Salt vulnerability. Several other "high-profile" sites have been compromised as well, including LineageOS and Ghost. I expect we'll hear of many more in the next few days. --- [0]: https://groups.google.com/a/chromium.org/forum/m/#!topic/ct-policy/aKNbZuJzwfM https://groups.google.com/a/chromium.org/forum/m/#!topic/ct-...
lol768 6y ago
This hit RamNode too. > This message is to customers with VPSs on our legacy SolusVM system. > At approximately 20:34 eastern (GMT -4) on May 2, recently published SaltStack vulnerabilities (CVE-2020-11651, CVE-2020-11652) were used to launch cryptocurrency miners on our SolusVM host nodes. The attack disrupted various services in order to allocate as much CPU as possible to the miners. SSH and QEMU processes were killed on some of our CentOS 6 KVM hosts, causing extended downtime in certain cases. > Upon detecting the disruption, we quickly began to re-enable SSH, disable and remove Salt, kill related processes, and boot shutdown KVM guests. After careful analysis of the exploit used, we do not believe any data was compromised. > RamNode was not specifically targeted, but rather anyone running SaltStack versions prior to the one released a few days ago (April 29).
- londons_explore 6y ago
  > After careful analysis of the exploit used, we do not believe any data was compromised. If someone had code running as root on their machine, they can't say that statement with any confidence whatsoever.
  - lol768 6y ago
    > If someone had code running as root on their machine, they can't say that statement with any confidence whatsoever. Indeed, I think you need to assume all guests running on the affected nodes are now compromised.
circlingthesun 6y ago
We got hit by this on Saturday night :|
nerdbaggy 6y ago
We all got lucky on this one. The outcome could of been much worse, like secret leaking and rm -rf. We record everything that saltstack does and the scripts didn’t even upload anything from the infected servers.
- willjp 6y ago
  You may have been lucky, but that was not the case with everyone. Some of the entries appear to be missing, but last night someone setup a honeypot and was posting what they saw. Adding keys to /root/.ssh/authorized_keys, scp'ing sshkeys, flushing iptables. I am of course one of the idiots that had it exposed to the internet. I wouldn't do it with a database, but I didn't think twice about salt. Where was my head!
  - mrits 6y ago
    Those things you listed are easily fixed which is what I assume they meant by lucky.
tormeh 6y ago
Former SaltStack user here. If you're using it, just stop. Switch to Ansible, a Kubernetes/Helm/Flux solution or experiment with Chef Habitat if you're feeling futuristic. SaltStack is just bad. It's buggy, clunky and has really bad error messages. Its only saving grace is that it came before Ansible, but that's only relevant in a historical context. Just no.
- lykr0n 6y ago
  None of those are compatible solutions. Ansible is very inflexible when it comes to configuration and how you structure your playbooks. Kubernetes/Helm/Flux only works if you're inside k8s. I don't know why you would be using Salt for that. Chef Habitat looks nice, but again. I'm doing more then just management of applications. The only real alternative to Salt is Puppet. So, No. Just no.
- NikolaeVarius 6y ago
  Packer with Salt Masterless has been fine though.
- dijit 6y ago
  SaltStack is definitely the underdog when comparing puppet/chef or ansible. But I can chime in on why we use it: 1) First class windows support. (or, as first class as it comes) with no need to enable winrm (which, is much more difficult to do than just installing a salt-minion on first boot) 2) It scales really well; I can run hundreds of thousands of commands in parallel, ansible can't do that; it gets very starved on CPU. 3) It's push based (unlike chef, which you have to run on the client node itself, causing people to cron-job it. 4) it's easily extended; you can write custom modules and it's very easy and pleasant. --- Now, saying all of that, Salt has warts. Like, hundreds, maybe thousands. It's not uncommon that I find something that's just broken or bugged. Often upgrading fixes 1 bug but makes 3 more. I feel like this might be caused by being an underdog. But it's a real drawback that makes me cautious when recommending it.
lndarj 6y ago
Our server were affected. They added a cronjob which ran every minute wgetting a .sh file and executing it. Most of the time the server which the file was in was either offline or returned 404, but every once in a while it returned the malicious script. When this script got executed, it killed our nginx server, which is how we noticed something odd was happening. If it wasn't for nginx dying, we might have not even noticed we were infected. Here's the contents of the sh script for the curious. https://pastebin.com/CbupwQMG https://pastebin.com/CbupwQMG
- DeepYogurt 6y ago
  Very helpful. Thank you for sharing.
- nikanj 6y ago
  Looks like it goes through a massive amount of effort to eliminate competing miners and malware. Should neuter that script and run it periodically on boxes :)
willjp 6y ago
I'm so grateful this happened over the weekend, when I had time to respond. My girlfriend woke me up and told me the CPU-fan was going crazy in the living room. I realize this isn't the case for everyone. I extend my deepest consolations to those affected.
georgianar 6y ago
Email this great herbal doctor who cured me from herpes virus and also brought my lover back via his email robinsonbucler@gmail. com ...........

>>> If you're here, chances are you're already compromised. WTF does that mean? Our salt implementation is on an entirely private network, so why would I be more likely than not to be compromised already? ---- Edit: Re-downvotes -- This is a sincere question. Is there some evidence that the majority of salt implementations are compromised, or some mechanism by which this hits private networks? Or is that line just for dramatic effect?

jkingsman 6y ago
With you on this one... a coworker put it well "[this tone is] at a 9, it needs to be at a 3".
busterarm 6y ago
99% of companies I've encountered running SaltStack are masterless anyway.
- saltyguy22 6y ago
  Depends on their use case. IMO, Salts best feature is probably running with the Master / Minion setup because it connects out to the master. Masterless is handy for using in conjunction with Vagrant, Packer, or another provisioning tool not so much for managing a the lifecycle of a server / OS.
  - busterarm 6y ago
    I'm not clueless. I know this. This doesn't detract at all from what I said. In production, an outsized number of companies are using SaltStack a certain way. The reason is because of the way we all evaluate business risk. I could have said that out of all of the companies I've worked with using Salt that have compliance requirements, 100% of them are using it Masterless, but then some smart-ass would have piped up with a "not me" comment.
    - Ascetik 6y ago
      not me :D
    - kapilvt 6y ago
      How would you compare it to ansible in that usage model? I don't do much config mgmt these days, so curious for a master(less) mindset, whats the comparison for salt vs. ansible?
      - busterarm 6y ago
        Ansible gets a ton more use. Its push model is highly favored in IT organizations. Salt's event/reactor system has tricks Ansible can't do.
    - rexarex 6y ago
      Not me
- devonkim 6y ago
  Salt is a bit of a rarity among ops people but the funny thing is my experience is the opposite of yours - 66%+ of companies I see with Salt run in master/minion mode. I prefer it over Ansible when my ssh settings are super awkward (MFA, multiple bastions, etc) to integrate into an Ansible inventory file. One place with 900+ hosts and 300+ random IP ranges with tons of (badly written) compliance I spent about 3 weeks trying to get an inventory file hacked up and I gave up and deployed Salt within a day for the basic stuff I needed to do for basic running of shell commands from a single point of control.
  - busterarm 6y ago
    Most of the places I've seen it are banks. Oddly. Your use case makes absolute perfect sense though -- nice!
twic 6y ago
I assumed that the problem was that if a master is accessible on your intranet, it could be hit with some sort of XSS attack from browsers inside the firewall. But apparently there are 6000 people just straight up exposing their masters to the internet: https://gbhackers.com/saltstack-salt/ https://gbhackers.com/saltstack-salt/
- NikolaeVarius 6y ago
  > This whole "don't have your salt master exposed to the internet" thing has me annoyed. The whole point of salt is to manage boxes all over the place. I manage around 500 machines. Most of them are behind the firewalls of incompetent admins who have spent hours in the past trying to set up port forwards when salt-minion crashed so I could access the box again. I'm about to test binding salt-master to localhost and salt-minion to localhost and then setting up spiped to wrap the traffic... Some companies need better DevOps apparently
  - dijit 6y ago
    Nitpick: this is purely an operations/sysadmin problem. The DevOps grouping doesn't apply too much, Operations-minded staff should be focusing on keeping things locked down. Unless, you know, you're not hiring those people and instead are hoping that developers take the Ops burden. ;)
    - deleted 6y ago
      [deleted]
    - dijit 6y ago
      People downvoting: I guess I hit a nerve, but could you explain why? Development and Operations are different disciplines and the idea was to remove silos, not make one person responsible for both.
- Spivak 6y ago
  But that's like half the point of SaltStack. It's supposed be secure enough to run on the public internet to manage road-warrior endpoints. "I found this site called Alexa that lists a bunch of companies that just straight up expose their web servers to the internet. Crazy." And I have some bad news about how many companies are exposing their VPN servers to the internet too.
TheDong 6y ago
>>>> If you're here, chances are you're already compromised. > WTF does that mean? The website assumes that if you dropped by or googled the vulnerability, it's because you had saltstack exposed to the public internet. You don't have to react so harshly to that turn of phrase.
- alexandercrohde 6y ago
  My main impression was that this site was blurring the line between professional "Bulletin" and scare-mongering. > Even if you didn't notice any unexpected symptoms, please: nuke and restart. If I were writing this, I'd probably write something like "On May x, a remote-code-execution in (all versions?) public salt-masters (not minions?), was unveiled. Shortly thereafter, actual exploitation in the wild is being used. If your salt installation uses a salt-master, and it's internet-reachable, it may already be compromised, along with much your infrastructure. Section 2 is how to see if you are infected, and Section 3 is how to remove the infection."
  - TheDong 6y ago
    > Section 2 is how to see if you are infected and Section 3 is how to remove the infection That's much worse. That implies there is a reliable way to detect and remove the infection. That's not the case. This website included some known attacks and such. There's a high chance that there were attacks of this vulnerability with additional payloads. There's no way to know if you were hit with a rootkit that persists itself in the bootloader or other parts of your system. There's no way to know if you were infected or not. The only way to be sure is to nuke the machine, as they said. Security advice should always err on the right side for a naive reader.

formersaltuser 6y ago

SaltStack has a long history of home brew protocols, internally written encryption, security issues, and bugs. I'm not surprised they would end up being used as a vector for attacks.

riffic 6y ago
Something about Salt has always rubbed me the wrong way. I'd rather stick with Puppet, Chef, or CF Engine if I was considering this tool.
- xenophonf 6y ago
  I'm all about IT automation and like the tech a lot, but after running SaltStack for 5 years, I decided to transition to plain old written documentation. When you consider everything that underlies the "infrastructure as code" tech stack, it ends up being an extremely steep learning curve. I was very productive with SaltStack but turned into the only person who could write or edit our deployment scripts. It was a bad situation. Now, nine months after dropping SaltStack, my colleagues are editing the docs I've created, and more importantly, they're also writing their own. That's a huge win. What I've lost in terms of personal productivity, I've regained in terms of wider participation in our standardization and documentation efforts. I might slowly re-introduce IT automation technologies, maybe something more popular like Ansible or Docker, but only after making sure the rest of my team has a solid understanding of general IT automation concepts. I think we're on the right track. I had a colleague today ask me about what Python programming certification they should get, so I pointed them toward Google's Python-based IT automation course on Coursera. I supported another colleague's efforts to create a library of standard server images based on our internal deployment standards for one of our private clouds, and I'm encouraging them to expand that work to our other data centers. And so on. The team as a whole works better together, so that's where I'm staying focused.
  - chucky_z 6y ago
    So you went from full automation to... Copying and pasting by hand?

docsapp_io 6y ago

This impacted ghost.org hosting really bad https://status.ghost.org/ https://status.ghost.org/

_vafj 6y ago
Impacted LineageOS too - https://twitter.com/LineageAndroid/status/1256821056100163584 https://twitter.com/LineageAndroid/status/125682105610016358...
- twic 6y ago
  Oh god, the responses. Half of them are completely unrelated, asking Lineage to support particular phones. And then there's someone who saw "salt" and tried to help: https://twitter.com/mvonwi/status/1256989787321438209 https://twitter.com/mvonwi/status/1256989787321438209

rhakyr 6y ago

Debian hasn't fixed this yet in their packaging so you might want to work around that if you are using that as a salt-master server. https://security-tracker.debian.org/tracker/CVE-2020-11651 https://security-tracker.debian.org/tracker/CVE-2020-11651

thesz 6y ago

How does things like these affect Monero's reputation? I saw Monero mining in JS libraries, now it is in virus form. Thus, why Monero? Does Monero somehow discourage these "practices"?

jackhalford 6y ago
Maybe it's to do with the fact that monero tokens are non-fungible, unlike some other crypto-currencies.
- 3456j34jht 6y ago
  I think you mean 'fungible'. Non-fungible means 1 XMR is not equivalent to another 1 XMR. The use of ring signatures, key images and transaction amount hiding gives Monero its fungible properties. This makes it computationally difficult to trace a particular transaction graph. This provides fungibility beyond that of Nakamoto style coins, in that you cannot easily block/identify addresses or transactions.
- thinkmassive 6y ago
  Monero is likely the absolute most fungible cryptocurrency.
colatkinson 6y ago
IIRC one of Monero's design goals is that it isn't easily GPU/ASIC-mineable, so CPU mining is competitive. It's also designed to be difficult to trace. Both of these properties make it a relatively safe and profitable avenue for malware authors.
asymptotically2 6y ago
Mining with JavaScript or WebAssembly is no longer viable as being competitive now requires using a JIT compiler. Nothing can really be done about it. Pool operators can ban the wallet addresses of known bot masters from connecting to their pool, but you can't really ban someone from participating in an anonymous decentralised network.

scblzn 6y ago

Algolia got impacted too, apparently their Salt masters were opened to the whole internet: https://blog.algolia.com/salt-incident-may-3rd-2020-retrospective-and-update/ https://blog.algolia.com/salt-incident-may-3rd-2020-retrospe...

jlgaddis 6y ago
One of DigiCert's Certificate Transparency logs was likewise open to the entire Internet -- and likewise compromised [0]: > I'm sad to report that we discovered today that CT Log 2's key used to sign SCTs was compromised last night at 7 pm via the Salt vulnerability. Several other "high-profile" sites have been compromised as well, including LineageOS and Ghost. I expect we'll hear of many more in the next few days. --- [0]: https://groups.google.com/a/chromium.org/forum/m/#!topic/ct-policy/aKNbZuJzwfM https://groups.google.com/a/chromium.org/forum/m/#!topic/ct-...

lol768 6y ago

This hit RamNode too. > This message is to customers with VPSs on our legacy SolusVM system. > At approximately 20:34 eastern (GMT -4) on May 2, recently published SaltStack vulnerabilities (CVE-2020-11651, CVE-2020-11652) were used to launch cryptocurrency miners on our SolusVM host nodes. The attack disrupted various services in order to allocate as much CPU as possible to the miners. SSH and QEMU processes were killed on some of our CentOS 6 KVM hosts, causing extended downtime in certain cases. > Upon detecting the disruption, we quickly began to re-enable SSH, disable and remove Salt, kill related processes, and boot shutdown KVM guests. After careful analysis of the exploit used, we do not believe any data was compromised. > RamNode was not specifically targeted, but rather anyone running SaltStack versions prior to the one released a few days ago (April 29).

londons_explore 6y ago
> After careful analysis of the exploit used, we do not believe any data was compromised. If someone had code running as root on their machine, they can't say that statement with any confidence whatsoever.
- lol768 6y ago
  > If someone had code running as root on their machine, they can't say that statement with any confidence whatsoever. Indeed, I think you need to assume all guests running on the affected nodes are now compromised.

circlingthesun 6y ago

We got hit by this on Saturday night :|

nerdbaggy 6y ago

We all got lucky on this one. The outcome could of been much worse, like secret leaking and rm -rf. We record everything that saltstack does and the scripts didn’t even upload anything from the infected servers.

willjp 6y ago
You may have been lucky, but that was not the case with everyone. Some of the entries appear to be missing, but last night someone setup a honeypot and was posting what they saw. Adding keys to /root/.ssh/authorized_keys, scp'ing sshkeys, flushing iptables. I am of course one of the idiots that had it exposed to the internet. I wouldn't do it with a database, but I didn't think twice about salt. Where was my head!
- mrits 6y ago
  Those things you listed are easily fixed which is what I assume they meant by lucky.

tormeh 6y ago

Former SaltStack user here. If you're using it, just stop. Switch to Ansible, a Kubernetes/Helm/Flux solution or experiment with Chef Habitat if you're feeling futuristic. SaltStack is just bad. It's buggy, clunky and has really bad error messages. Its only saving grace is that it came before Ansible, but that's only relevant in a historical context. Just no.

lykr0n 6y ago
None of those are compatible solutions. Ansible is very inflexible when it comes to configuration and how you structure your playbooks. Kubernetes/Helm/Flux only works if you're inside k8s. I don't know why you would be using Salt for that. Chef Habitat looks nice, but again. I'm doing more then just management of applications. The only real alternative to Salt is Puppet. So, No. Just no.
NikolaeVarius 6y ago
Packer with Salt Masterless has been fine though.
dijit 6y ago
SaltStack is definitely the underdog when comparing puppet/chef or ansible. But I can chime in on why we use it: 1) First class windows support. (or, as first class as it comes) with no need to enable winrm (which, is much more difficult to do than just installing a salt-minion on first boot) 2) It scales really well; I can run hundreds of thousands of commands in parallel, ansible can't do that; it gets very starved on CPU. 3) It's push based (unlike chef, which you have to run on the client node itself, causing people to cron-job it. 4) it's easily extended; you can write custom modules and it's very easy and pleasant. --- Now, saying all of that, Salt has warts. Like, hundreds, maybe thousands. It's not uncommon that I find something that's just broken or bugged. Often upgrading fixes 1 bug but makes 3 more. I feel like this might be caused by being an underdog. But it's a real drawback that makes me cautious when recommending it.

lndarj 6y ago

Our server were affected. They added a cronjob which ran every minute wgetting a .sh file and executing it. Most of the time the server which the file was in was either offline or returned 404, but every once in a while it returned the malicious script. When this script got executed, it killed our nginx server, which is how we noticed something odd was happening. If it wasn't for nginx dying, we might have not even noticed we were infected. Here's the contents of the sh script for the curious. https://pastebin.com/CbupwQMG https://pastebin.com/CbupwQMG

DeepYogurt 6y ago
Very helpful. Thank you for sharing.
nikanj 6y ago
Looks like it goes through a massive amount of effort to eliminate competing miners and malware. Should neuter that script and run it periodically on boxes :)

willjp 6y ago

I'm so grateful this happened over the weekend, when I had time to respond. My girlfriend woke me up and told me the CPU-fan was going crazy in the living room. I realize this isn't the case for everyone. I extend my deepest consolations to those affected.

georgianar 6y ago

Email this great herbal doctor who cured me from herpes virus and also brought my lover back via his email robinsonbucler@gmail. com ...........