4 ms·

As a developer, you should want as little power on as few servers as needed to do your work. With the unlimited access to do whatever you want, you also get lot

As a developer, you should want as little power on as few servers as needed to do your work. With the unlimited access to do whatever you want, you also get lots of responsibility, which means you need to develop your security-consciousness to near paranoid levels, and that's not fun. If it's possible for you to log into a server and just become root (hopefully there's at least sudo so you don't log in directly as root), you have to make sure your personal laptop or workstation is encrypted, your SSH key is passphrase-protected and you use an SSH agent (or even an external physical device) to store your private key. Certainly you should not be using passwords to log in anywhere. You should do these things anyway, but the more direct power you have, the more important it becomes. This is especially true if you have programmatic access to a public cloud somewhere: you do not want any more power than necessary for any longer than necessary because administrator access to eg. an AWS account means that when it leaks your infra can be automatically compromised in less time that you can react in and you can't even put firewalls in front because it's accessible from the internet. Leave the operations that require real power to CI systems and automated processes where you have some testing and review in the middle. Get your logs and memory dumps shipped out of the system into tools that actually help you make use of them; if you need to debug a live system as root, that should be an exceptional situation that raises all kinds of alarms so that it can be formally accepted. I'm a guy whose job involves maintaining hundreds of systems and I'm always happy when I can do it with less power.

jcrites 5y ago
> (hopefully there's at least sudo so you don't log in directly as root Correct, at both FAANG I logged in as myself and had passwordless sudo. (Passwordless sudo being preferable to minimize the number of machines that have user password hashes at the company, since these can be brute-forced when employees set bad passwords). > your SSH key is passphrase-protected and you use an SSH agent I don't have long-lived SSH keys. The FAANGs where I've work use short-lived (~24h) SSH client certificates, issued by Single Sign On and a server-trusted SSH Certificate Authority, where the SSO authenticates the user via an MFA (typically FIDO2 or variants with a Yubikey press). (Yes, you can indeed employ CAs with SSH, and generate short-lived certificates [1]). I don't generally type my password after signing into my client machine, or possibly once more to authenticate into the corporate web framework (w/ MFA). After that, every SSH action requires a individual MFA approval. It doesn't require a password or any SSH key protected by a password – I think long-lived SSH keys, like you'd protect by a password, are an anti-pattern because when employees leave the company or change teams you have to remember to revoke them or update their keys. With the SSH CA approach, you simply change which certs it's willing to issue to an employee for which servers based on policy; and then people who leave the company can't access that system altogether. You get the right behavior by default without anyone remembering to take action during off-boarding. No long-lived SSH keys with passwords. Short-lived SSH certificates don't need their own passwords. As a developer operating systems on a suite of servers, it's simply a business need for me to have SSH into the (virtual machines that I own) running on those servers (not necessarily the underlying hypervisors – those are an entirely different beast). If you were in the cloud it's equivalent to being able to obtain SSH certificate that allows you to log onto e.g. EC2 instances. Or if I'm a team that's operating Bare Metal instances, I can log into those. I don't accept that root is less than necessary to do my job. When things go wrong there are too many directions that debugging can go that you cannot explore without root. Like I mentioned, I can't take a machine out of service, and attach a debugger to it, and invoke the failing functionality. I can't change machine parameters that may fix the problem like memory limits. You can't even examine the contents of a number of crucial logs without it. I agree with the principle of least access and employ it in my personal systems as well. At work, I think POLA results in you having root to every machine you're responsible for. [1] https://engineering.fb.com/2016/09/12/security/scalable-and-secure-access-with-ssh/ https://engineering.fb.com/2016/09/12/security/scalable-and-... (Disclosure: the FAANG I currently work for is Facebook)
- chousuke 5y ago
  SSH certificates are a good solution, though I think you could still use longer-lived SSH identities as part of the authentication process to the system that signs your certificate with MFA on top. I personally really just don't like passwords for authentication at all, and SSH is super convenient to use on top of being secure. Having a solution like that in place goes a long way towards reducing the immediate power you have available at any one time, so I don't think we disagree much at all.

jcrites 5y ago

> (hopefully there's at least sudo so you don't log in directly as root Correct, at both FAANG I logged in as myself and had passwordless sudo. (Passwordless sudo being preferable to minimize the number of machines that have user password hashes at the company, since these can be brute-forced when employees set bad passwords). > your SSH key is passphrase-protected and you use an SSH agent I don't have long-lived SSH keys. The FAANGs where I've work use short-lived (~24h) SSH client certificates, issued by Single Sign On and a server-trusted SSH Certificate Authority, where the SSO authenticates the user via an MFA (typically FIDO2 or variants with a Yubikey press). (Yes, you can indeed employ CAs with SSH, and generate short-lived certificates [1]). I don't generally type my password after signing into my client machine, or possibly once more to authenticate into the corporate web framework (w/ MFA). After that, every SSH action requires a individual MFA approval. It doesn't require a password or any SSH key protected by a password – I think long-lived SSH keys, like you'd protect by a password, are an anti-pattern because when employees leave the company or change teams you have to remember to revoke them or update their keys. With the SSH CA approach, you simply change which certs it's willing to issue to an employee for which servers based on policy; and then people who leave the company can't access that system altogether. You get the right behavior by default without anyone remembering to take action during off-boarding. No long-lived SSH keys with passwords. Short-lived SSH certificates don't need their own passwords. As a developer operating systems on a suite of servers, it's simply a business need for me to have SSH into the (virtual machines that I own) running on those servers (not necessarily the underlying hypervisors – those are an entirely different beast). If you were in the cloud it's equivalent to being able to obtain SSH certificate that allows you to log onto e.g. EC2 instances. Or if I'm a team that's operating Bare Metal instances, I can log into those. I don't accept that root is less than necessary to do my job. When things go wrong there are too many directions that debugging can go that you cannot explore without root. Like I mentioned, I can't take a machine out of service, and attach a debugger to it, and invoke the failing functionality. I can't change machine parameters that may fix the problem like memory limits. You can't even examine the contents of a number of crucial logs without it. I agree with the principle of least access and employ it in my personal systems as well. At work, I think POLA results in you having root to every machine you're responsible for. [1] https://engineering.fb.com/2016/09/12/security/scalable-and-secure-access-with-ssh/ https://engineering.fb.com/2016/09/12/security/scalable-and-... (Disclosure: the FAANG I currently work for is Facebook)

chousuke 5y ago
SSH certificates are a good solution, though I think you could still use longer-lived SSH identities as part of the authentication process to the system that signs your certificate with MFA on top. I personally really just don't like passwords for authentication at all, and SSH is super convenient to use on top of being secure. Having a solution like that in place goes a long way towards reducing the immediate power you have available at any one time, so I don't think we disagree much at all.