6 ms·

This whole salt-stack incident could've been handled a lot better by salt themselves: - the notification was a week ago to a small mailing list, which is tucke

by cetra3 6y ago

This whole salt-stack incident could've been handled a lot better by salt themselves: - the notification was a week ago to a small mailing list, which is tucked away on their site - no notification to the registry to when you go to download salt (at least I never received an email, but still get plenty of marketing spam) - no posts on social media as far as I can tell, I couldn't find a tweet, anything on reddit, or anything on hn. - they only blogged about it on their official site yesterday, way after damage had been done - one week's notice between the initial announcement and the patch coming out. The patch being released is basically a disclosure of the vulnerability - the patch was released late Thursday early Friday depending on your timezone, giving attackers the weekend head start - the official salt docker images were only patched yesterday - You can't get a patch for older versions without filling out a form and supplying details - Ubuntu and other repositories are still vulnerable

mtam 6y ago
+1, however, from what I read, the vulnerability can only be exploited if the attacker has network access to the salt masters port, which should never occur. The people that got compromised had Salt exposed to the Internet, which is obviously ridiculous. Not trying to downplay the critical nature of the vulnerability but the ones that were compromised by this issue have deeper security issues to deal with.
- cetra3 6y ago
  If you look at their current `hardening` document it still has pretty unclear language about what is acceptable and what isn't. > Use a hardened bastion server or a VPN to restrict direct access to the Salt master from the internet Is this SSH access or is this access to the salt master from minions? Or just access in general?
  - brians 6y ago
    Apparently it includes minion-master interaction. If that’s to be “hardened” over SSH, what’s the point of all the salt keys?
    - deleted 6y ago
      [deleted]
  - user5994461 6y ago
    SSH bastions and VPN are two standard ways to allow external clients access into an internal network, meaning salt is never exposed publicly. I read this as a guideline that the salt master must not be exposed to the internet. Albeit could be better worded for a developer audience who doesn't understand bastions or VPN well.
- mike_d 6y ago
  > has network access to the salt masters port, which should never occur You seem to prescribe to the "hard shell soft gooey center" network security philosophy. Should people expose an Oracle server to the internet? Absolutely not. Does moving it behind a firewall change the fact that every mildly skilled exploit developer is sitting on an Oracle 0day? Absolutely not. People have legitimate reasons for exposing Salt to the internet. I do. It's how I bootstrap random VMs and bare metal from the internet. But in my case the attack was mitigated by the fact that Salt cascades changes in a bunch of other systems and re-masters minions to a host only reachable over a tunnel. I blew away the internet master, restored from a backup, and patched. > the ones that were compromised by this issue have deeper security issues to deal with Or it was just another Monday. When you become sufficiently large you deal with incidents on a daily basis. Kudos to the people who publicly postmortem and talk about what went well and what didn't. (For the record, I've already been working for a few months on a move to Ansible for non-security reasons)
  - hajhatten 6y ago
    You piqued my interest, what made you move? I personally haven't touched Saltstack in years, but was super happy with it when I did.
    - mike_d 6y ago
      Twice I encountered breaking changes between versions that required manually upgrading minions. I also got the overall feeling Salt was built by developers, Ansible by sysadmins - and I fit into the latter bucket.
      - Conan_Kudo 6y ago
        Ansible (originally known as "Fedora Unified Network Controller" or "func") was made for solving the problems automating Fedora Infrastructure. Puppet did not make Fedora Infrastructure administrators happy. So func was designed around solving their problems, and expanded its scope as people found it useful. Then it was renamed to Ansible, the developers left Red Hat to create AnsibleWorks, and the rest is history! So yes, it was made for sysadmins. :)
- tasssko 6y ago
  +1 agree but exposing salt to the internet is not the problem. A simple ip whitelist ingress firewall rule on the salt master port would have helped, blocking access is also possible on this port. With cloud services it has become trivial to group server resources so that when they belong to the same group they can communicate with each other. I don’t use salt however i am not a proponent of network isolation as a form of security.
  - techslave 6y ago
    that’s just plain negligent. of course you use network isolation as one of your defenses in depth.
- StreamBright 6y ago
  Coconut security is not great either. Hard shell, soft internals. Not exposing the ports to the internet is just one layer.
- CJefferson 6y ago
  If they wrote software which should never be visible to the internet, they should have made that clearer. It's far too easy to make something internet-visible. They could have set up a simple check to see if the service is internet, and refused to work if it was.
- Shish2k 6y ago
  I was in this situation; I went with “salt master exposed to the internet” because it’s the only service on that box - if I’d wrapped it in a VPN, then I’m replacing one exposed service with a different exposed service, and VPNs aren’t immune to exploits either (plus an extra layer of configuration means an extra layer of things that can go wrong)
isodude 6y ago
Weird that something with this title got buried here on HN: https://news.ycombinator.com/item?id=23041528 https://news.ycombinator.com/item?id=23041528
bawolff 6y ago
> one week's notice between the initial announcement and the patch coming out. The patch being released is basically a disclosure of the vulnerability While your other points may be valid, one week should be plenty of time between announcement and patch. Any longer and i would call the time table problematic.
- aneutron 6y ago
  You have clearly never worked at a large enough OLD corporation. One week is nothing compared to what it would take to upgrade your configuration management system.
  - bawolff 6y ago
    That sounds like an old corporation problem. The world should not pay the price of old corporations inflexibility. If someone hacks your system you certainly wont have a week to respond. The longer a vendor sits on a vuln, the more likely it is to leak or to be rediscovered by a malicious party.
section_me 6y ago
It was posted, just no traction on it (eg. https://news.ycombinator.com/item?id=22972100 https://news.ycombinator.com/item?id=22972100 posted 11 days ago). But yeah supprising the lack of posts about it.
cat199 6y ago
> - Ubuntu and other repositories are still vulnerable isn't really salt's problem though.. same could be said for relying on any distro-provided package

by cetra3 6y ago

mtam 6y ago

+1, however, from what I read, the vulnerability can only be exploited if the attacker has network access to the salt masters port, which should never occur. The people that got compromised had Salt exposed to the Internet, which is obviously ridiculous. Not trying to downplay the critical nature of the vulnerability but the ones that were compromised by this issue have deeper security issues to deal with.

cetra3 6y ago
If you look at their current `hardening` document it still has pretty unclear language about what is acceptable and what isn't. > Use a hardened bastion server or a VPN to restrict direct access to the Salt master from the internet Is this SSH access or is this access to the salt master from minions? Or just access in general?
- brians 6y ago
  Apparently it includes minion-master interaction. If that’s to be “hardened” over SSH, what’s the point of all the salt keys?
  - deleted 6y ago
    [deleted]
- user5994461 6y ago
  SSH bastions and VPN are two standard ways to allow external clients access into an internal network, meaning salt is never exposed publicly. I read this as a guideline that the salt master must not be exposed to the internet. Albeit could be better worded for a developer audience who doesn't understand bastions or VPN well.
mike_d 6y ago
> has network access to the salt masters port, which should never occur You seem to prescribe to the "hard shell soft gooey center" network security philosophy. Should people expose an Oracle server to the internet? Absolutely not. Does moving it behind a firewall change the fact that every mildly skilled exploit developer is sitting on an Oracle 0day? Absolutely not. People have legitimate reasons for exposing Salt to the internet. I do. It's how I bootstrap random VMs and bare metal from the internet. But in my case the attack was mitigated by the fact that Salt cascades changes in a bunch of other systems and re-masters minions to a host only reachable over a tunnel. I blew away the internet master, restored from a backup, and patched. > the ones that were compromised by this issue have deeper security issues to deal with Or it was just another Monday. When you become sufficiently large you deal with incidents on a daily basis. Kudos to the people who publicly postmortem and talk about what went well and what didn't. (For the record, I've already been working for a few months on a move to Ansible for non-security reasons)
- hajhatten 6y ago
  You piqued my interest, what made you move? I personally haven't touched Saltstack in years, but was super happy with it when I did.
  - mike_d 6y ago
    Twice I encountered breaking changes between versions that required manually upgrading minions. I also got the overall feeling Salt was built by developers, Ansible by sysadmins - and I fit into the latter bucket.
    - Conan_Kudo 6y ago
      Ansible (originally known as "Fedora Unified Network Controller" or "func") was made for solving the problems automating Fedora Infrastructure. Puppet did not make Fedora Infrastructure administrators happy. So func was designed around solving their problems, and expanded its scope as people found it useful. Then it was renamed to Ansible, the developers left Red Hat to create AnsibleWorks, and the rest is history! So yes, it was made for sysadmins. :)
tasssko 6y ago
+1 agree but exposing salt to the internet is not the problem. A simple ip whitelist ingress firewall rule on the salt master port would have helped, blocking access is also possible on this port. With cloud services it has become trivial to group server resources so that when they belong to the same group they can communicate with each other. I don’t use salt however i am not a proponent of network isolation as a form of security.
- techslave 6y ago
  that’s just plain negligent. of course you use network isolation as one of your defenses in depth.
StreamBright 6y ago
Coconut security is not great either. Hard shell, soft internals. Not exposing the ports to the internet is just one layer.
CJefferson 6y ago
If they wrote software which should never be visible to the internet, they should have made that clearer. It's far too easy to make something internet-visible. They could have set up a simple check to see if the service is internet, and refused to work if it was.
Shish2k 6y ago
I was in this situation; I went with “salt master exposed to the internet” because it’s the only service on that box - if I’d wrapped it in a VPN, then I’m replacing one exposed service with a different exposed service, and VPNs aren’t immune to exploits either (plus an extra layer of configuration means an extra layer of things that can go wrong)

isodude 6y ago

Weird that something with this title got buried here on HN: https://news.ycombinator.com/item?id=23041528 https://news.ycombinator.com/item?id=23041528

bawolff 6y ago

> one week's notice between the initial announcement and the patch coming out. The patch being released is basically a disclosure of the vulnerability While your other points may be valid, one week should be plenty of time between announcement and patch. Any longer and i would call the time table problematic.

aneutron 6y ago
You have clearly never worked at a large enough OLD corporation. One week is nothing compared to what it would take to upgrade your configuration management system.
- bawolff 6y ago
  That sounds like an old corporation problem. The world should not pay the price of old corporations inflexibility. If someone hacks your system you certainly wont have a week to respond. The longer a vendor sits on a vuln, the more likely it is to leak or to be rediscovered by a malicious party.

section_me 6y ago

It was posted, just no traction on it (eg. https://news.ycombinator.com/item?id=22972100 https://news.ycombinator.com/item?id=22972100 posted 11 days ago). But yeah supprising the lack of posts about it.

cat199 6y ago

> - Ubuntu and other repositories are still vulnerable isn't really salt's problem though.. same could be said for relying on any distro-provided package