5 ms·

+1, however, from what I read, the vulnerability can only be exploited if the attacker has network access to the salt masters port, which should never occur. Th

by mtam 6y ago

+1, however, from what I read, the vulnerability can only be exploited if the attacker has network access to the salt masters port, which should never occur. The people that got compromised had Salt exposed to the Internet, which is obviously ridiculous. Not trying to downplay the critical nature of the vulnerability but the ones that were compromised by this issue have deeper security issues to deal with.

cetra3 6y ago
If you look at their current `hardening` document it still has pretty unclear language about what is acceptable and what isn't. > Use a hardened bastion server or a VPN to restrict direct access to the Salt master from the internet Is this SSH access or is this access to the salt master from minions? Or just access in general?
- brians 6y ago
  Apparently it includes minion-master interaction. If that’s to be “hardened” over SSH, what’s the point of all the salt keys?
  - deleted 6y ago
    [deleted]
- user5994461 6y ago
  SSH bastions and VPN are two standard ways to allow external clients access into an internal network, meaning salt is never exposed publicly. I read this as a guideline that the salt master must not be exposed to the internet. Albeit could be better worded for a developer audience who doesn't understand bastions or VPN well.
mike_d 6y ago
> has network access to the salt masters port, which should never occur You seem to prescribe to the "hard shell soft gooey center" network security philosophy. Should people expose an Oracle server to the internet? Absolutely not. Does moving it behind a firewall change the fact that every mildly skilled exploit developer is sitting on an Oracle 0day? Absolutely not. People have legitimate reasons for exposing Salt to the internet. I do. It's how I bootstrap random VMs and bare metal from the internet. But in my case the attack was mitigated by the fact that Salt cascades changes in a bunch of other systems and re-masters minions to a host only reachable over a tunnel. I blew away the internet master, restored from a backup, and patched. > the ones that were compromised by this issue have deeper security issues to deal with Or it was just another Monday. When you become sufficiently large you deal with incidents on a daily basis. Kudos to the people who publicly postmortem and talk about what went well and what didn't. (For the record, I've already been working for a few months on a move to Ansible for non-security reasons)
- hajhatten 6y ago
  You piqued my interest, what made you move? I personally haven't touched Saltstack in years, but was super happy with it when I did.
  - mike_d 6y ago
    Twice I encountered breaking changes between versions that required manually upgrading minions. I also got the overall feeling Salt was built by developers, Ansible by sysadmins - and I fit into the latter bucket.
    - Conan_Kudo 6y ago
      Ansible (originally known as "Fedora Unified Network Controller" or "func") was made for solving the problems automating Fedora Infrastructure. Puppet did not make Fedora Infrastructure administrators happy. So func was designed around solving their problems, and expanded its scope as people found it useful. Then it was renamed to Ansible, the developers left Red Hat to create AnsibleWorks, and the rest is history! So yes, it was made for sysadmins. :)
- empath75 6y ago
  > People have legitimate reasons for exposing Salt to the internet. I do. It's how I bootstrap random VMs and bare metal from the internet. I question that that is a legitimate reason to expose it to the internet. Defense in depth is a thing and putting the keys to the kingdom at layer 0 doesn’t seem wise even if a vpn or bastion doesn’t offer perfect protection.
  - mike_d 6y ago
    Read the sentence after the ones you quoted. The internet connected salt master is used to provision accepted hosts in to the tunneled (VPN) network where the real master lives.
- mixedCase 6y ago
  > It's how I bootstrap random VMs and bare metal from the internet. What's stopping you from bootstrapping a bastion server through a simpler script and then bootstrapping with Salt from there?
tasssko 6y ago
+1 agree but exposing salt to the internet is not the problem. A simple ip whitelist ingress firewall rule on the salt master port would have helped, blocking access is also possible on this port. With cloud services it has become trivial to group server resources so that when they belong to the same group they can communicate with each other. I don’t use salt however i am not a proponent of network isolation as a form of security.
- techslave 6y ago
  that’s just plain negligent. of course you use network isolation as one of your defenses in depth.
StreamBright 6y ago
Coconut security is not great either. Hard shell, soft internals. Not exposing the ports to the internet is just one layer.
CJefferson 6y ago
If they wrote software which should never be visible to the internet, they should have made that clearer. It's far too easy to make something internet-visible. They could have set up a simple check to see if the service is internet, and refused to work if it was.
Shish2k 6y ago
I was in this situation; I went with “salt master exposed to the internet” because it’s the only service on that box - if I’d wrapped it in a VPN, then I’m replacing one exposed service with a different exposed service, and VPNs aren’t immune to exploits either (plus an extra layer of configuration means an extra layer of things that can go wrong)

by mtam 6y ago

cetra3 6y ago

If you look at their current `hardening` document it still has pretty unclear language about what is acceptable and what isn't. > Use a hardened bastion server or a VPN to restrict direct access to the Salt master from the internet Is this SSH access or is this access to the salt master from minions? Or just access in general?

brians 6y ago
Apparently it includes minion-master interaction. If that’s to be “hardened” over SSH, what’s the point of all the salt keys?
- deleted 6y ago
  [deleted]
user5994461 6y ago
SSH bastions and VPN are two standard ways to allow external clients access into an internal network, meaning salt is never exposed publicly. I read this as a guideline that the salt master must not be exposed to the internet. Albeit could be better worded for a developer audience who doesn't understand bastions or VPN well.

mike_d 6y ago

> has network access to the salt masters port, which should never occur You seem to prescribe to the "hard shell soft gooey center" network security philosophy. Should people expose an Oracle server to the internet? Absolutely not. Does moving it behind a firewall change the fact that every mildly skilled exploit developer is sitting on an Oracle 0day? Absolutely not. People have legitimate reasons for exposing Salt to the internet. I do. It's how I bootstrap random VMs and bare metal from the internet. But in my case the attack was mitigated by the fact that Salt cascades changes in a bunch of other systems and re-masters minions to a host only reachable over a tunnel. I blew away the internet master, restored from a backup, and patched. > the ones that were compromised by this issue have deeper security issues to deal with Or it was just another Monday. When you become sufficiently large you deal with incidents on a daily basis. Kudos to the people who publicly postmortem and talk about what went well and what didn't. (For the record, I've already been working for a few months on a move to Ansible for non-security reasons)

hajhatten 6y ago
You piqued my interest, what made you move? I personally haven't touched Saltstack in years, but was super happy with it when I did.
- mike_d 6y ago
  Twice I encountered breaking changes between versions that required manually upgrading minions. I also got the overall feeling Salt was built by developers, Ansible by sysadmins - and I fit into the latter bucket.
  - Conan_Kudo 6y ago
    Ansible (originally known as "Fedora Unified Network Controller" or "func") was made for solving the problems automating Fedora Infrastructure. Puppet did not make Fedora Infrastructure administrators happy. So func was designed around solving their problems, and expanded its scope as people found it useful. Then it was renamed to Ansible, the developers left Red Hat to create AnsibleWorks, and the rest is history! So yes, it was made for sysadmins. :)
empath75 6y ago
> People have legitimate reasons for exposing Salt to the internet. I do. It's how I bootstrap random VMs and bare metal from the internet. I question that that is a legitimate reason to expose it to the internet. Defense in depth is a thing and putting the keys to the kingdom at layer 0 doesn’t seem wise even if a vpn or bastion doesn’t offer perfect protection.
- mike_d 6y ago
  Read the sentence after the ones you quoted. The internet connected salt master is used to provision accepted hosts in to the tunneled (VPN) network where the real master lives.
mixedCase 6y ago
> It's how I bootstrap random VMs and bare metal from the internet. What's stopping you from bootstrapping a bastion server through a simpler script and then bootstrapping with Salt from there?

tasssko 6y ago

+1 agree but exposing salt to the internet is not the problem. A simple ip whitelist ingress firewall rule on the salt master port would have helped, blocking access is also possible on this port. With cloud services it has become trivial to group server resources so that when they belong to the same group they can communicate with each other. I don’t use salt however i am not a proponent of network isolation as a form of security.

techslave 6y ago
that’s just plain negligent. of course you use network isolation as one of your defenses in depth.

StreamBright 6y ago

Coconut security is not great either. Hard shell, soft internals. Not exposing the ports to the internet is just one layer.

CJefferson 6y ago

If they wrote software which should never be visible to the internet, they should have made that clearer. It's far too easy to make something internet-visible. They could have set up a simple check to see if the service is internet, and refused to work if it was.

Shish2k 6y ago

I was in this situation; I went with “salt master exposed to the internet” because it’s the only service on that box - if I’d wrapped it in a VPN, then I’m replacing one exposed service with a different exposed service, and VPNs aren’t immune to exploits either (plus an extra layer of configuration means an extra layer of things that can go wrong)