4 ms·

I disagree with most of this, to be perfectly honest. For one, devs don't need root in prod in the vast majority of shops, which aren't FAANG scale. Assuming a

by hug 5y ago

I disagree with most of this, to be perfectly honest. For one, devs don't need root in prod in the vast majority of shops, which aren't FAANG scale. Assuming a nice sane environment where your deploys to pre-prod mirror your prod environment, I can't think of many cases where having root on prod makes much difference. It's the inverse of your statement, where places that aren't massive should find it feasible to 1:1 their prod environment for the devs to noodle about in. For two, statements like this one make me wary of a blanket policy of devs getting root: > if your team runs public web servers, then you need to listen on ports like 80 and 443 while requiring root. Have you heard of setcap and CAP_NET_BIND_SERVICE, perhaps? I know this is only one example of many where you may "need" root, but in a lot of cases devs just don't have the ops knowledge to know what they actually need.

jcrites 5y ago
Yes, I am aware of these capabilities. However, you as the developer still need to be able to write software that runs as root on the box, both to (likely) install new software version and to drop the capability either shortly before executing the primary application (preferred); or the application drops it shortly after launching (seems to be more common – e.g. Postfix Architecture [1] – probably because the App knows what minimal permissions it needs and this is less work for the user). Meanwhile, as a developer you still need the ability to deploy applications unless some kind of "deployment layer" handles depositing new software versions onto the machine. I'm also concerned about the lack of root for being able to inspect what's going wrong with software on the machine (`ps` and `lsof` and socket-specific equivalents). How do I take a heap dump or stack trace from erroneous software on a prod machine without root? How can I step through the software in a debugger? (You could argue that I shouldn't do this but it's a tradeoff between velocity and security, the right answer which may depend on the business we're in.) Containers are kind-of this but I wouldn't use containers as security sandbox, since they're not designed for it. User namespaces get close, but I prefer to rely on the hypervisor for this kind of security boundary, not namespaces, cgroups, or containers – because the kernel API is massive and security vulnerabilities are found in it regularly. (Root exploits less often.) How would you build an operate a server in production without root? I honestly don't have experience. All of my experience has been running installation scripts as root, and having personal root access for inspecting/debugging (potentially even live debugger attaching and stepping); unless the `sudo` commands were very carefully managed I wouldn't be able to use the majority of my standard toolkit for fixing problems, and I wouldn't be able to fix the problem if I found them. Am I running out of file descriptors because the limit is set too low? OK, bump up the limit. That's a code change but I can initiate an emergency recovery by locally changing it on all machines in the fleet. We should also acknowledge that there are different kinds of software systems. If I was working at Coinbase or on Payments at Stripe I might have a different perspective. I previously worked at AWS and now currently work on WhatsApp so I am still incredibly security-conscious, but the level of security required by a product is a negotiable product specification, not a hard-and-fast-rule like the necessary strength of a bridge in civil engineering. [1] Postfix Architecture: http://www.postfix.org/OVERVIEW.html http://www.postfix.org/OVERVIEW.html
- jcims 5y ago
  >We should also acknowledge that there are different kinds of software systems. If I was working at Coinbase or on Payments at Stripe I might have a different perspective. It all boils down to this at the end of the day. There is clearly some level of tradeoff between control and velocity, and there are valid business cases for setting the dial just about anywhere on the spectrum.
- chousuke 5y ago
  As a developer, you should want as little power on as few servers as needed to do your work. With the unlimited access to do whatever you want, you also get lots of responsibility, which means you need to develop your security-consciousness to near paranoid levels, and that's not fun. If it's possible for you to log into a server and just become root (hopefully there's at least sudo so you don't log in directly as root), you have to make sure your personal laptop or workstation is encrypted, your SSH key is passphrase-protected and you use an SSH agent (or even an external physical device) to store your private key. Certainly you should not be using passwords to log in anywhere. You should do these things anyway, but the more direct power you have, the more important it becomes. This is especially true if you have programmatic access to a public cloud somewhere: you do not want any more power than necessary for any longer than necessary because administrator access to eg. an AWS account means that when it leaks your infra can be automatically compromised in less time that you can react in and you can't even put firewalls in front because it's accessible from the internet. Leave the operations that require real power to CI systems and automated processes where you have some testing and review in the middle. Get your logs and memory dumps shipped out of the system into tools that actually help you make use of them; if you need to debug a live system as root, that should be an exceptional situation that raises all kinds of alarms so that it can be formally accepted. I'm a guy whose job involves maintaining hundreds of systems and I'm always happy when I can do it with less power.
  - jcrites 5y ago
    > (hopefully there's at least sudo so you don't log in directly as root Correct, at both FAANG I logged in as myself and had passwordless sudo. (Passwordless sudo being preferable to minimize the number of machines that have user password hashes at the company, since these can be brute-forced when employees set bad passwords). > your SSH key is passphrase-protected and you use an SSH agent I don't have long-lived SSH keys. The FAANGs where I've work use short-lived (~24h) SSH client certificates, issued by Single Sign On and a server-trusted SSH Certificate Authority, where the SSO authenticates the user via an MFA (typically FIDO2 or variants with a Yubikey press). (Yes, you can indeed employ CAs with SSH, and generate short-lived certificates [1]). I don't generally type my password after signing into my client machine, or possibly once more to authenticate into the corporate web framework (w/ MFA). After that, every SSH action requires a individual MFA approval. It doesn't require a password or any SSH key protected by a password – I think long-lived SSH keys, like you'd protect by a password, are an anti-pattern because when employees leave the company or change teams you have to remember to revoke them or update their keys. With the SSH CA approach, you simply change which certs it's willing to issue to an employee for which servers based on policy; and then people who leave the company can't access that system altogether. You get the right behavior by default without anyone remembering to take action during off-boarding. No long-lived SSH keys with passwords. Short-lived SSH certificates don't need their own passwords. As a developer operating systems on a suite of servers, it's simply a business need for me to have SSH into the (virtual machines that I own) running on those servers (not necessarily the underlying hypervisors – those are an entirely different beast). If you were in the cloud it's equivalent to being able to obtain SSH certificate that allows you to log onto e.g. EC2 instances. Or if I'm a team that's operating Bare Metal instances, I can log into those. I don't accept that root is less than necessary to do my job. When things go wrong there are too many directions that debugging can go that you cannot explore without root. Like I mentioned, I can't take a machine out of service, and attach a debugger to it, and invoke the failing functionality. I can't change machine parameters that may fix the problem like memory limits. You can't even examine the contents of a number of crucial logs without it. I agree with the principle of least access and employ it in my personal systems as well. At work, I think POLA results in you having root to every machine you're responsible for. [1] https://engineering.fb.com/2016/09/12/security/scalable-and-secure-access-with-ssh/ https://engineering.fb.com/2016/09/12/security/scalable-and-... (Disclosure: the FAANG I currently work for is Facebook)
lmm 5y ago
> For one, devs don't need root in prod in the vast majority of shops, which aren't FAANG scale. Assuming a nice sane environment where your deploys to pre-prod mirror your prod environment, I can't think of many cases where having root on prod makes much difference. It's the inverse of your statement, where places that aren't massive should find it feasible to 1:1 their prod environment for the devs to noodle about in. You can and should mirror the environment, but it's rarely going to be perfect (particularly in terms of traffic), and while it's often worth putting time into better test environments, being able to just hop onto a prod machine and properly trace your programs' behaviour (which often means you need BPF probes or similar) is a gamechanger for fixing production problems quickly. > Have you heard of setcap and CAP_NET_BIND_SERVICE, perhaps? I know this is only one example of many where you may "need" root, but in a lot of cases devs just don't have the ops knowledge to know what they actually need. We absolutely don't. Would learning it actually be a good use of time though? Linux user accounts are not generally seen as a hard security boundary, local root exploits are ten-a-penny, and the user account that's running the service necessarily has full access to the actually important stuff (user data) anyway. Particularly in this age of disposable VMs for single processes, I'd think the cost-benefit favours running everything as root, giving developers root, and structuring your security model so that your servers don't trust each other.

by hug 5y ago

jcrites 5y ago

Yes, I am aware of these capabilities. However, you as the developer still need to be able to write software that runs as root on the box, both to (likely) install new software version and to drop the capability either shortly before executing the primary application (preferred); or the application drops it shortly after launching (seems to be more common – e.g. Postfix Architecture [1] – probably because the App knows what minimal permissions it needs and this is less work for the user). Meanwhile, as a developer you still need the ability to deploy applications unless some kind of "deployment layer" handles depositing new software versions onto the machine. I'm also concerned about the lack of root for being able to inspect what's going wrong with software on the machine (`ps` and `lsof` and socket-specific equivalents). How do I take a heap dump or stack trace from erroneous software on a prod machine without root? How can I step through the software in a debugger? (You could argue that I shouldn't do this but it's a tradeoff between velocity and security, the right answer which may depend on the business we're in.) Containers are kind-of this but I wouldn't use containers as security sandbox, since they're not designed for it. User namespaces get close, but I prefer to rely on the hypervisor for this kind of security boundary, not namespaces, cgroups, or containers – because the kernel API is massive and security vulnerabilities are found in it regularly. (Root exploits less often.) How would you build an operate a server in production without root? I honestly don't have experience. All of my experience has been running installation scripts as root, and having personal root access for inspecting/debugging (potentially even live debugger attaching and stepping); unless the `sudo` commands were very carefully managed I wouldn't be able to use the majority of my standard toolkit for fixing problems, and I wouldn't be able to fix the problem if I found them. Am I running out of file descriptors because the limit is set too low? OK, bump up the limit. That's a code change but I can initiate an emergency recovery by locally changing it on all machines in the fleet. We should also acknowledge that there are different kinds of software systems. If I was working at Coinbase or on Payments at Stripe I might have a different perspective. I previously worked at AWS and now currently work on WhatsApp so I am still incredibly security-conscious, but the level of security required by a product is a negotiable product specification, not a hard-and-fast-rule like the necessary strength of a bridge in civil engineering. [1] Postfix Architecture: http://www.postfix.org/OVERVIEW.html http://www.postfix.org/OVERVIEW.html

jcims 5y ago
>We should also acknowledge that there are different kinds of software systems. If I was working at Coinbase or on Payments at Stripe I might have a different perspective. It all boils down to this at the end of the day. There is clearly some level of tradeoff between control and velocity, and there are valid business cases for setting the dial just about anywhere on the spectrum.
chousuke 5y ago
As a developer, you should want as little power on as few servers as needed to do your work. With the unlimited access to do whatever you want, you also get lots of responsibility, which means you need to develop your security-consciousness to near paranoid levels, and that's not fun. If it's possible for you to log into a server and just become root (hopefully there's at least sudo so you don't log in directly as root), you have to make sure your personal laptop or workstation is encrypted, your SSH key is passphrase-protected and you use an SSH agent (or even an external physical device) to store your private key. Certainly you should not be using passwords to log in anywhere. You should do these things anyway, but the more direct power you have, the more important it becomes. This is especially true if you have programmatic access to a public cloud somewhere: you do not want any more power than necessary for any longer than necessary because administrator access to eg. an AWS account means that when it leaks your infra can be automatically compromised in less time that you can react in and you can't even put firewalls in front because it's accessible from the internet. Leave the operations that require real power to CI systems and automated processes where you have some testing and review in the middle. Get your logs and memory dumps shipped out of the system into tools that actually help you make use of them; if you need to debug a live system as root, that should be an exceptional situation that raises all kinds of alarms so that it can be formally accepted. I'm a guy whose job involves maintaining hundreds of systems and I'm always happy when I can do it with less power.
- jcrites 5y ago
  > (hopefully there's at least sudo so you don't log in directly as root Correct, at both FAANG I logged in as myself and had passwordless sudo. (Passwordless sudo being preferable to minimize the number of machines that have user password hashes at the company, since these can be brute-forced when employees set bad passwords). > your SSH key is passphrase-protected and you use an SSH agent I don't have long-lived SSH keys. The FAANGs where I've work use short-lived (~24h) SSH client certificates, issued by Single Sign On and a server-trusted SSH Certificate Authority, where the SSO authenticates the user via an MFA (typically FIDO2 or variants with a Yubikey press). (Yes, you can indeed employ CAs with SSH, and generate short-lived certificates [1]). I don't generally type my password after signing into my client machine, or possibly once more to authenticate into the corporate web framework (w/ MFA). After that, every SSH action requires a individual MFA approval. It doesn't require a password or any SSH key protected by a password – I think long-lived SSH keys, like you'd protect by a password, are an anti-pattern because when employees leave the company or change teams you have to remember to revoke them or update their keys. With the SSH CA approach, you simply change which certs it's willing to issue to an employee for which servers based on policy; and then people who leave the company can't access that system altogether. You get the right behavior by default without anyone remembering to take action during off-boarding. No long-lived SSH keys with passwords. Short-lived SSH certificates don't need their own passwords. As a developer operating systems on a suite of servers, it's simply a business need for me to have SSH into the (virtual machines that I own) running on those servers (not necessarily the underlying hypervisors – those are an entirely different beast). If you were in the cloud it's equivalent to being able to obtain SSH certificate that allows you to log onto e.g. EC2 instances. Or if I'm a team that's operating Bare Metal instances, I can log into those. I don't accept that root is less than necessary to do my job. When things go wrong there are too many directions that debugging can go that you cannot explore without root. Like I mentioned, I can't take a machine out of service, and attach a debugger to it, and invoke the failing functionality. I can't change machine parameters that may fix the problem like memory limits. You can't even examine the contents of a number of crucial logs without it. I agree with the principle of least access and employ it in my personal systems as well. At work, I think POLA results in you having root to every machine you're responsible for. [1] https://engineering.fb.com/2016/09/12/security/scalable-and-secure-access-with-ssh/ https://engineering.fb.com/2016/09/12/security/scalable-and-... (Disclosure: the FAANG I currently work for is Facebook)

lmm 5y ago

> For one, devs don't need root in prod in the vast majority of shops, which aren't FAANG scale. Assuming a nice sane environment where your deploys to pre-prod mirror your prod environment, I can't think of many cases where having root on prod makes much difference. It's the inverse of your statement, where places that aren't massive should find it feasible to 1:1 their prod environment for the devs to noodle about in. You can and should mirror the environment, but it's rarely going to be perfect (particularly in terms of traffic), and while it's often worth putting time into better test environments, being able to just hop onto a prod machine and properly trace your programs' behaviour (which often means you need BPF probes or similar) is a gamechanger for fixing production problems quickly. > Have you heard of setcap and CAP_NET_BIND_SERVICE, perhaps? I know this is only one example of many where you may "need" root, but in a lot of cases devs just don't have the ops knowledge to know what they actually need. We absolutely don't. Would learning it actually be a good use of time though? Linux user accounts are not generally seen as a hard security boundary, local root exploits are ten-a-penny, and the user account that's running the service necessarily has full access to the actually important stuff (user data) anyway. Particularly in this age of disposable VMs for single processes, I'd think the cost-benefit favours running everything as root, giving developers root, and structuring your security model so that your servers don't trust each other.