8 ms·

The premise of this thing is not good advice. 1) Your first couple minutes on a server should be used to install a configuration management client, if your boo

The premise of this thing is not good advice. 1) Your first couple minutes on a server should be used to install a configuration management client, if your bootstrap policies somehow don't already install one. 2) Everything else listed in this document should be configured by a configuration management system. 3) "User account sync tools" should have no place in a modern infrastructure, you should use your configuration management tool to (at the bare minimum) deploy /etc/passwd and /etc/sudoers across your infrastructure. 4) You should not use shared/role accounts. The "incremental cost" is paid back immediately when someone leaves your organization; having to update everyone of a changed password or having a password change have any negative impact at all should not be a thing your company does. This stuff isn't hard. It's worth doing right.

mef 14y ago
For #4, wouldn't the only change when someone leaves the organization be to remove their key from authorized_keys for the shared account? Why would anyone else have to be updated?
- mbreese 14y ago
  But they still would know the 'deploy' password needed for sudo access. And while you could be relatively sure that they couldn't get access, you still couldn't be completely sure since they did have sudo access to begin with. So, the best thing would be to change the shared password. That could be avoided with non-shared accounts.
  - luser001 14y ago
    Are you insinuating that the user could have used sudo access to install a backdoor of some sort? If so, changing the password won't stop them either. Am I missing something?
- joelesalas 14y ago
  It's a basic principle of security. Each account represents one person so that you have a full audit of who did what by watching the activity of a given user account. If everything is run as "devops" user for example, you have no idea who actually performed a given task. Was it Bill, or was it an automated job? PCI-DSS requirements also affect your model for user accounts (hint: shared users are often not compliant). From the perspective of a sysadmin, this article has a lot of issues and it's inadvisable to follow its recommendations. Who doesn't use a hardware firewall? Who exposes ssh to the internet (requiring fail2ban) when a VPN server is much more secure and easier to use? Setting up an LDAP server is really easy and costs nothing. There's no excuse for shared accounts.
  - luser001 14y ago
    Why do you say a VPN server is more secure? Which one? I, for one, trust ssh more than any other software wrt security, especially with password login disabled. Disclaimer: I am not a security expert.
    - deleted 14y ago
      [deleted]
    - rdl 14y ago
      When someone advocates using a VPN, that doesn't mean not using SSH too. VPN + firewall just restricts who has the potential to try to SSH to you, and provides additional protection and central access control/management.
    - mryan 14y ago
      I, also, trust SSH more than any other software. But it is still worth adding an additional layer of security in front of SSH to help protect from exploits. Let's say that, hypothetically, a 0-day exploit was discovered in SSH which allowed remote code execution. A script kiddie begins trawling the internet for publicly accessible SSH servers to attack. Your servers allow SSH from anywhere on the internet, and are eventually discovered and exploited. Mine, which will only allow SSH connections from my VPN bastion host, are effectively invisible to the attacker and will not get exploited (by this particular script kiddie, at least). Adding a VPN server in front of SSH won't protect you from an APT, but it will protect you from 99% of the random, automated attacks that take place.
      - lmm 14y ago
        Your outermost server is the one where you should be most worried about having vulnerabilities - if you have a VPN as the outer layer that means the VPN server must be exposed to the public internet, and anyone who compromises it is in a pretty good position. And I'd rate the odds of a 0-day being found at higher for most VPN software than for SSH.
        14y ago
- vidarh 14y ago
  If they have had access to a shared account, how much work are you willing to put in to verify that there's nothing in that shared account that will get executed by another user later that will quietly reinstate a key? A reason to have separate accounts is that not only do you terminate access, you also have an easier time ensuring that less of what that person had access to could have been compromised. (This of course goes right out the window if said person has sudo/su access, in which case you have a much harder time, but even then giving them individual accounts means your opportunity to audit becomes so much greater) After all, it's not the honest guy who'll never try to log in again you're primarily trying to protect against (in fact: for the honest people, a good security policy protects them by making them less likely to become potential suspects if/when something happens - it's in your own interest when you leave an organisation to ensure you get locked out), but the guy who might decide to try to do something later, or who might even be thinking about doing something before they leave.
  - Nursie 14y ago
    But if you give people any access to any system surely this is a concern. As a software dev, maybe I've inserted something into one of the build scripts that quietly re-opens my backdoor to the source control server... I haven't, but if you assume actually malicious users you're probably going to end up with something so locked down it's useless. Aren't you?
    - jackalope 14y ago
      Quite the contrary: You're probably going to end up with documented procedures for deploying software that are simplified, follow existing standards/best practices, and don't rely on complex stone soup build/init scripts concocted by inexperienced developers (and I've seen some doozies). A developer is more likely to create better and more easily maintainable software if the target audience is assumed to be an ordinary user with no special system privileges. In my experience, when a developer has root and assumes everyone else does, deployment becomes a nightmare.
      - Nursie 14y ago
wvenable 14y ago
Is there a free and recommended configuration management system that does all this?
- davidbanham 14y ago
  We use Chef, many people run Puppet. http://www.opscode.com/chef/ http://www.opscode.com/chef/ https://puppetlabs.com/ https://puppetlabs.com/
- chrisbolt 14y ago
  Puppet, chef, cfengine, ansible, and salt are a few.
  - elktea 14y ago
    Salt is wonderful.
    - plasticgun 14y ago
      I just ready your post out of context. Made me chuckle.
- shizcakes 14y ago
  Chef or Puppet are common choices. There are many others, but those two are modern, have large communities, and decent documentation. There's a decent chance someone has already open sourced a cookbook/module for each of many of these items!
  - deleted 14y ago
    [deleted]
- Axsuul 14y ago
  Highly recommend Chef due to its Ruby DSL.
- grey-area 14y ago
  On linode, which he is using, there is one built in - they allow shell scripts to run on first boot which could replay his commands as they are without many changes: http://www.linode.com/stackscripts/ http://www.linode.com/stackscripts/
raverbashing 14y ago
You are right, this is a better path But sometimes, and especially for servers that will be delivered to the customer afterwards, it's not practical to use a configuration management tool. Also, millions of servers were deployed before Chef/Puppet appeared. You can't tell they did wrong. Also, Chef/Puppet type solutions may be overkill for some tasks, fabric takes care of the easier cases for example.
- vidarh 14y ago
  It's always practical to use a configuration management tool. It's not always practical to use specific tools like Chef/Puppet. Even a shell script that automates your standard install scripts is better than doing it by hand, because they can ensure you don't forget any steps and verify the state afterwards and ensure you don't forget any of the verification steps either.
  - raverbashing 14y ago
    Ok, great, that's the spirit You can put all his recommendations in a shell script (as the easiest solution) then run it (and if you ever did this more than ONE time you see the value in it)
    - kelnos 14y ago
      The problem is that most of the recommendations are bad. Having an up-to-date system and only accepting security updates is a good policy. Fail2ban is a good tool (but it's a starting point; you should be doing other things to detect suspicious behavior). The rest is just bad advice. Having everyone log in using a single user account is a terrible idea. You can't audit who did what, ever. You have to remember to remove people from authorized_keys when they leave, and also make sure that they haven't left themselves a backdoor -- a cron job that reinstates the key, an extra user account, even just changing the root password to something else (how often do you actually check the root password on your boxes?). User account management is a pain, so that's why we have things like LDAP. Everyone has their own user account. You can audit who does what on every machine, and for stuff that requires root, sudo will log the things people do (of course, if you let people have root shells, that's harder). The only people who get access to a local account (and/or root, but I still think root should just have a random password that no one knows) are a few sysadmins. When someone leaves, you kill their account in the LDAP server. Even better, if this is a possibility, put up a VPN, and only allow ssh access via the VPN (using a firewall). Tie the VPN login to LDAP (and don't let non-VPN-admins ssh directly to the VPN server), and then you can be sure that without a user account in LDAP, no one can log into your servers. Blind-updating systems in production is a terrible idea. Things break in the open source world all the time when you do this. Never ever use unattended-upgrades. You just need to be on top of security updates. Period. No excuses. You should never even have a "my first five minutes on a server" type thing anyway. Rolling out a new server should be fully automatically operationalized. The first time you log into the server, it should be completely ready to go. It should be ready to go without you needing to log into it at all. This takes a small amount of up-front effort, and will pay off immediately when you bring up your second server.
dhimes 14y ago
I would love to see a link to a counter-post of how to do it your way. Not generalities, but a specific guide.
matthuggins 14y ago
I'm a good developer, but setting up systems is something I do so infrequently that I always have to relearn it, and I am positive I could be doing things better. Do you have a link to an article that explains how to go about performing some of these ideas you mention?
jasonlotito 14y ago
> This stuff isn't hard. It's worth doing right. Can you provide an article as equally succinct as the OP's that provides this information? Your list is painfully devoid of anything of true value. Since it's not hard, and worth doing right, I imagine something should already be written.
- shizcakes 14y ago
  This is an excellent request. Reading through all the comments here, it seems like a lot of people are feeling frustrated with the variety of information available and no clear way to discern what is "good". I don't typically publish writings, but this seems like a good place to start. I'll write something up and post it here for the same critique that we've given Bryan :-) In the meantime, a decent source of generalized (not succinct) Ops-type knowledge can be found here: http://www.opsschool.org/en/latest/ http://www.opsschool.org/en/latest/
  - jasonlotito 14y ago
    Thanks! I really appreciate it. All to often we are quick to criticize someone for missing information without any willingness to back it up with better, correct information. The jokes goes that the best way to get an answer is to publish the wrong one. =)
  - epo 14y ago
    That site is useless, sample pages I looked contained brief overview paragraphs and no real content. Many contained "todo" items. So you criticised the original article, but when asked to provide information or advice of your own merely came up with something entirely content free. The original article provided succint, useful advice, something you have failed to do.
- npsimons 14y ago
  If Bastille was still in working order, I'd recommend it as a very good starting point for locking down a configuration. Actual configuration deployment setups I don't put much stock in because I'm not managing enough machines for it to be worthwhile. Would love to hear from someone who manages a large farm/cluster/VM hosting on a very clean and straightforward way to manage configurations. Last I looked, most setups were custom, or some hobbled together packages that didn't "hang" together very well.

mef 14y ago

For #4, wouldn't the only change when someone leaves the organization be to remove their key from authorized_keys for the shared account? Why would anyone else have to be updated?

mbreese 14y ago
But they still would know the 'deploy' password needed for sudo access. And while you could be relatively sure that they couldn't get access, you still couldn't be completely sure since they did have sudo access to begin with. So, the best thing would be to change the shared password. That could be avoided with non-shared accounts.
- luser001 14y ago
  Are you insinuating that the user could have used sudo access to install a backdoor of some sort? If so, changing the password won't stop them either. Am I missing something?
joelesalas 14y ago
It's a basic principle of security. Each account represents one person so that you have a full audit of who did what by watching the activity of a given user account. If everything is run as "devops" user for example, you have no idea who actually performed a given task. Was it Bill, or was it an automated job? PCI-DSS requirements also affect your model for user accounts (hint: shared users are often not compliant). From the perspective of a sysadmin, this article has a lot of issues and it's inadvisable to follow its recommendations. Who doesn't use a hardware firewall? Who exposes ssh to the internet (requiring fail2ban) when a VPN server is much more secure and easier to use? Setting up an LDAP server is really easy and costs nothing. There's no excuse for shared accounts.
- luser001 14y ago
  Why do you say a VPN server is more secure? Which one? I, for one, trust ssh more than any other software wrt security, especially with password login disabled. Disclaimer: I am not a security expert.
  - deleted 14y ago
    [deleted]
  - rdl 14y ago
    When someone advocates using a VPN, that doesn't mean not using SSH too. VPN + firewall just restricts who has the potential to try to SSH to you, and provides additional protection and central access control/management.
  - mryan 14y ago
    I, also, trust SSH more than any other software. But it is still worth adding an additional layer of security in front of SSH to help protect from exploits. Let's say that, hypothetically, a 0-day exploit was discovered in SSH which allowed remote code execution. A script kiddie begins trawling the internet for publicly accessible SSH servers to attack. Your servers allow SSH from anywhere on the internet, and are eventually discovered and exploited. Mine, which will only allow SSH connections from my VPN bastion host, are effectively invisible to the attacker and will not get exploited (by this particular script kiddie, at least). Adding a VPN server in front of SSH won't protect you from an APT, but it will protect you from 99% of the random, automated attacks that take place.
    - lmm 14y ago
      Your outermost server is the one where you should be most worried about having vulnerabilities - if you have a VPN as the outer layer that means the VPN server must be exposed to the public internet, and anyone who compromises it is in a pretty good position. And I'd rate the odds of a 0-day being found at higher for most VPN software than for SSH.
      - 14y ago
vidarh 14y ago
If they have had access to a shared account, how much work are you willing to put in to verify that there's nothing in that shared account that will get executed by another user later that will quietly reinstate a key? A reason to have separate accounts is that not only do you terminate access, you also have an easier time ensuring that less of what that person had access to could have been compromised. (This of course goes right out the window if said person has sudo/su access, in which case you have a much harder time, but even then giving them individual accounts means your opportunity to audit becomes so much greater) After all, it's not the honest guy who'll never try to log in again you're primarily trying to protect against (in fact: for the honest people, a good security policy protects them by making them less likely to become potential suspects if/when something happens - it's in your own interest when you leave an organisation to ensure you get locked out), but the guy who might decide to try to do something later, or who might even be thinking about doing something before they leave.
- Nursie 14y ago
  But if you give people any access to any system surely this is a concern. As a software dev, maybe I've inserted something into one of the build scripts that quietly re-opens my backdoor to the source control server... I haven't, but if you assume actually malicious users you're probably going to end up with something so locked down it's useless. Aren't you?
  - jackalope 14y ago
    Quite the contrary: You're probably going to end up with documented procedures for deploying software that are simplified, follow existing standards/best practices, and don't rely on complex stone soup build/init scripts concocted by inexperienced developers (and I've seen some doozies). A developer is more likely to create better and more easily maintainable software if the target audience is assumed to be an ordinary user with no special system privileges. In my experience, when a developer has root and assumes everyone else does, deployment becomes a nightmare.
    - Nursie 14y ago

wvenable 14y ago

Is there a free and recommended configuration management system that does all this?

davidbanham 14y ago
We use Chef, many people run Puppet. http://www.opscode.com/chef/ http://www.opscode.com/chef/ https://puppetlabs.com/ https://puppetlabs.com/
chrisbolt 14y ago
Puppet, chef, cfengine, ansible, and salt are a few.
- elktea 14y ago
  Salt is wonderful.
  - plasticgun 14y ago
    I just ready your post out of context. Made me chuckle.
shizcakes 14y ago
Chef or Puppet are common choices. There are many others, but those two are modern, have large communities, and decent documentation. There's a decent chance someone has already open sourced a cookbook/module for each of many of these items!
- deleted 14y ago
  [deleted]
Axsuul 14y ago
Highly recommend Chef due to its Ruby DSL.
grey-area 14y ago
On linode, which he is using, there is one built in - they allow shell scripts to run on first boot which could replay his commands as they are without many changes: http://www.linode.com/stackscripts/ http://www.linode.com/stackscripts/

raverbashing 14y ago

You are right, this is a better path But sometimes, and especially for servers that will be delivered to the customer afterwards, it's not practical to use a configuration management tool. Also, millions of servers were deployed before Chef/Puppet appeared. You can't tell they did wrong. Also, Chef/Puppet type solutions may be overkill for some tasks, fabric takes care of the easier cases for example.

vidarh 14y ago
It's always practical to use a configuration management tool. It's not always practical to use specific tools like Chef/Puppet. Even a shell script that automates your standard install scripts is better than doing it by hand, because they can ensure you don't forget any steps and verify the state afterwards and ensure you don't forget any of the verification steps either.
- raverbashing 14y ago
  Ok, great, that's the spirit You can put all his recommendations in a shell script (as the easiest solution) then run it (and if you ever did this more than ONE time you see the value in it)
  - kelnos 14y ago
    The problem is that most of the recommendations are bad. Having an up-to-date system and only accepting security updates is a good policy. Fail2ban is a good tool (but it's a starting point; you should be doing other things to detect suspicious behavior). The rest is just bad advice. Having everyone log in using a single user account is a terrible idea. You can't audit who did what, ever. You have to remember to remove people from authorized_keys when they leave, and also make sure that they haven't left themselves a backdoor -- a cron job that reinstates the key, an extra user account, even just changing the root password to something else (how often do you actually check the root password on your boxes?). User account management is a pain, so that's why we have things like LDAP. Everyone has their own user account. You can audit who does what on every machine, and for stuff that requires root, sudo will log the things people do (of course, if you let people have root shells, that's harder). The only people who get access to a local account (and/or root, but I still think root should just have a random password that no one knows) are a few sysadmins. When someone leaves, you kill their account in the LDAP server. Even better, if this is a possibility, put up a VPN, and only allow ssh access via the VPN (using a firewall). Tie the VPN login to LDAP (and don't let non-VPN-admins ssh directly to the VPN server), and then you can be sure that without a user account in LDAP, no one can log into your servers. Blind-updating systems in production is a terrible idea. Things break in the open source world all the time when you do this. Never ever use unattended-upgrades. You just need to be on top of security updates. Period. No excuses. You should never even have a "my first five minutes on a server" type thing anyway. Rolling out a new server should be fully automatically operationalized. The first time you log into the server, it should be completely ready to go. It should be ready to go without you needing to log into it at all. This takes a small amount of up-front effort, and will pay off immediately when you bring up your second server.

dhimes 14y ago

I would love to see a link to a counter-post of how to do it your way. Not generalities, but a specific guide.

matthuggins 14y ago

I'm a good developer, but setting up systems is something I do so infrequently that I always have to relearn it, and I am positive I could be doing things better. Do you have a link to an article that explains how to go about performing some of these ideas you mention?

jasonlotito 14y ago

> This stuff isn't hard. It's worth doing right. Can you provide an article as equally succinct as the OP's that provides this information? Your list is painfully devoid of anything of true value. Since it's not hard, and worth doing right, I imagine something should already be written.

shizcakes 14y ago
This is an excellent request. Reading through all the comments here, it seems like a lot of people are feeling frustrated with the variety of information available and no clear way to discern what is "good". I don't typically publish writings, but this seems like a good place to start. I'll write something up and post it here for the same critique that we've given Bryan :-) In the meantime, a decent source of generalized (not succinct) Ops-type knowledge can be found here: http://www.opsschool.org/en/latest/ http://www.opsschool.org/en/latest/
- jasonlotito 14y ago
  Thanks! I really appreciate it. All to often we are quick to criticize someone for missing information without any willingness to back it up with better, correct information. The jokes goes that the best way to get an answer is to publish the wrong one. =)
- epo 14y ago
  That site is useless, sample pages I looked contained brief overview paragraphs and no real content. Many contained "todo" items. So you criticised the original article, but when asked to provide information or advice of your own merely came up with something entirely content free. The original article provided succint, useful advice, something you have failed to do.
npsimons 14y ago
If Bastille was still in working order, I'd recommend it as a very good starting point for locking down a configuration. Actual configuration deployment setups I don't put much stock in because I'm not managing enough machines for it to be worthwhile. Would love to hear from someone who manages a large farm/cluster/VM hosting on a very clean and straightforward way to manage configurations. Last I looked, most setups were custom, or some hobbled together packages that didn't "hang" together very well.