5 ms·

Two main lessons for me: 1. Always run services (e.g. ElasticSearch) with a unique user dedicated to that service and nothing else. 2. You're never as secure

Two main lessons for me: 1. Always run services (e.g. ElasticSearch) with a unique user dedicated to that service and nothing else. 2. You're never as secure as you think.

BinaryIdiot 10y ago
Ah the fabled one user per service. I try then something weird happens with permissions then I run it as root "temporarily" for development and then 3 years later...oops. Good thing I don't do dev ops except for my personal projects.
- droopyEyelids 10y ago
  Ahh! this is such fun and accessible problem to troubleshoot
- akx 10y ago
  At that point it's well worth figuring out what has gone awry with permissions rather than executing as root. If nothing else, you'll have added valuable information to your internal knowledge base about that particular permission problem and know how to fix it when it inevitably crops up again.
  - BinaryIdiot 10y ago
    Sure but that's much easier to say than to actually do. Sometimes it's not necessarily a permission issue but something missing in a user account. When you're under a tight deadline, especially at a start-up, it's not always easy to say "well I'm just going to keep working on the problem until I fix it" when you can also say "I can fix this in 3 seconds for now and hopefully revisit later". It's simply reality. I don't enjoy it. Fortunately I've never done that in any type of production environment (just development). But I don't entirely blame people. Many of the infrastructure pieces that need to be deployed can get very complex very quickly.
    - mootothemax 10y ago
      >Sure but that's much easier to say than to actually do. Sometimes it's not necessarily a permission issue but something missing in a user account. When you're under a tight deadline, especially at a start-up, it's not always easy to say "well I'm just going to keep working on the problem until I fix it" when you can also say "I can fix this in 3 seconds for now and hopefully revisit later". I used to be in the same boat as you until I had my lightbulb moment: if you're constantly coming across permission errors like this, it's a sign that you're just doing things wrong. I don't mean this as an insult; it's more that often when you download, install and provision a new piece of kit, it's easy to do so without taking a few minutes to read the docs and find out the best way of doing things. Especially if you're glancing over e.g. Ubuntu instructions when installing on CentOS. (in fact, this rule applies to a lot of IT. Spending hours trying to get a CI tool to do X, Y and Z? Chances are it's just not meant to do it that way. God knows how much time I've lost to this.)
      - throwaway7767 10y ago
        Agreed. After a few years of doing these sysadmin tasks "the right way", it doesn't take any longer than the insecure way, as you gain a solid mental model of the various permission structures and you've learned all of the diagnostic tools available to you. > I don't mean this as an insult; it's more that often when you download, install and provision a new piece of kit, it's easy to do so without taking a few minutes to read the docs and find out the best way of doing things. Especially if you're glancing over e.g. Ubuntu instructions when installing on CentOS. As a sysadmin, I find software documentation to often be the worst place for deployment advice. It's often written by developers with the same mindset as the GP - just get it working, best-practices can come later (which they never do). One app per user (and the corollary, proper segmentation between system users) is something you just have to train yourself to do, screw what the developer had in mind with his deployment instructions.
      - BinaryIdiot 10y ago
        Oh I certainly agree and fortunately this hasn't happened much to me. But it has happened to myself and many developers I know hence my original comment :) Though to be fair I've also run into weird bugs with OpenShift, Mongo and a few others over the years so it's not necessarily something wrong that the developer is doing.
voidlogic 10y ago
>Always run services (e.g. ElasticSearch) with a unique user dedicated to that service and nothing else. Quick tip; If you do this, you can also make your iptables rules be per user. For example, "webserverUser" can only accept inbound connections on 80/443 and only have outbound connections that are related to established inbound ones. If an attacker gains execution as this user, they cannot download new code, etc or even do DNS lookups for that matter.
- breakingcups 10y ago
  Wow, that's great, I never considered doing that. Thanks!
  - startling 10y ago
    It's only marginally helpful: it doesn't actually prevent attackers from uploading code to your server. Instead of having 'nc -l 8080 | bash' or whatever as your payload, an attacker can just run code instead. "pwd > /var/www/html/exfiltration.html". If they absolutely need a shell, they could e.g. alter nginx or its config files to run `bash` on POSTs to a hidden route. This does make it a little trickier, and potentially a little easier to detect. But it certainly doesn't make it so that "they cannot download new code".
    - voidlogic 10y ago
      >This does make it a little trickier, and potentially a little easier to detect. Correct, I should have said, it eliminates many easy ways to download code. Defense is depth is all about making the attackers job harder and increasing their likelihood of being detected.
- startling 10y ago
  This is totally wrong. Once you have RCE, you can upload code the same way you're executing code.
jwcrux 10y ago
Always run Elasticsearch behind an auth proxy like nginx or their Shield offering.

Two main lessons for me: 1. Always run services (e.g. ElasticSearch) with a unique user dedicated to that service and nothing else. 2. You're never as secure as you think.

BinaryIdiot 10y ago

Ah the fabled one user per service. I try then something weird happens with permissions then I run it as root "temporarily" for development and then 3 years later...oops. Good thing I don't do dev ops except for my personal projects.

droopyEyelids 10y ago
Ahh! this is such fun and accessible problem to troubleshoot
akx 10y ago
At that point it's well worth figuring out what has gone awry with permissions rather than executing as root. If nothing else, you'll have added valuable information to your internal knowledge base about that particular permission problem and know how to fix it when it inevitably crops up again.
- BinaryIdiot 10y ago
  Sure but that's much easier to say than to actually do. Sometimes it's not necessarily a permission issue but something missing in a user account. When you're under a tight deadline, especially at a start-up, it's not always easy to say "well I'm just going to keep working on the problem until I fix it" when you can also say "I can fix this in 3 seconds for now and hopefully revisit later". It's simply reality. I don't enjoy it. Fortunately I've never done that in any type of production environment (just development). But I don't entirely blame people. Many of the infrastructure pieces that need to be deployed can get very complex very quickly.
  - mootothemax 10y ago
    >Sure but that's much easier to say than to actually do. Sometimes it's not necessarily a permission issue but something missing in a user account. When you're under a tight deadline, especially at a start-up, it's not always easy to say "well I'm just going to keep working on the problem until I fix it" when you can also say "I can fix this in 3 seconds for now and hopefully revisit later". I used to be in the same boat as you until I had my lightbulb moment: if you're constantly coming across permission errors like this, it's a sign that you're just doing things wrong. I don't mean this as an insult; it's more that often when you download, install and provision a new piece of kit, it's easy to do so without taking a few minutes to read the docs and find out the best way of doing things. Especially if you're glancing over e.g. Ubuntu instructions when installing on CentOS. (in fact, this rule applies to a lot of IT. Spending hours trying to get a CI tool to do X, Y and Z? Chances are it's just not meant to do it that way. God knows how much time I've lost to this.)
    - throwaway7767 10y ago
      Agreed. After a few years of doing these sysadmin tasks "the right way", it doesn't take any longer than the insecure way, as you gain a solid mental model of the various permission structures and you've learned all of the diagnostic tools available to you. > I don't mean this as an insult; it's more that often when you download, install and provision a new piece of kit, it's easy to do so without taking a few minutes to read the docs and find out the best way of doing things. Especially if you're glancing over e.g. Ubuntu instructions when installing on CentOS. As a sysadmin, I find software documentation to often be the worst place for deployment advice. It's often written by developers with the same mindset as the GP - just get it working, best-practices can come later (which they never do). One app per user (and the corollary, proper segmentation between system users) is something you just have to train yourself to do, screw what the developer had in mind with his deployment instructions.
    - BinaryIdiot 10y ago
      Oh I certainly agree and fortunately this hasn't happened much to me. But it has happened to myself and many developers I know hence my original comment :) Though to be fair I've also run into weird bugs with OpenShift, Mongo and a few others over the years so it's not necessarily something wrong that the developer is doing.

voidlogic 10y ago

>Always run services (e.g. ElasticSearch) with a unique user dedicated to that service and nothing else. Quick tip; If you do this, you can also make your iptables rules be per user. For example, "webserverUser" can only accept inbound connections on 80/443 and only have outbound connections that are related to established inbound ones. If an attacker gains execution as this user, they cannot download new code, etc or even do DNS lookups for that matter.

breakingcups 10y ago
Wow, that's great, I never considered doing that. Thanks!
- startling 10y ago
  It's only marginally helpful: it doesn't actually prevent attackers from uploading code to your server. Instead of having 'nc -l 8080 | bash' or whatever as your payload, an attacker can just run code instead. "pwd > /var/www/html/exfiltration.html". If they absolutely need a shell, they could e.g. alter nginx or its config files to run `bash` on POSTs to a hidden route. This does make it a little trickier, and potentially a little easier to detect. But it certainly doesn't make it so that "they cannot download new code".
  - voidlogic 10y ago
    >This does make it a little trickier, and potentially a little easier to detect. Correct, I should have said, it eliminates many easy ways to download code. Defense is depth is all about making the attackers job harder and increasing their likelihood of being detected.
startling 10y ago
This is totally wrong. Once you have RCE, you can upload code the same way you're executing code.

jwcrux 10y ago

Always run Elasticsearch behind an auth proxy like nginx or their Shield offering.