5 ms·

I've worked for two of the FAANG companies together about 15 years, and I've always had local administrator access and root access to (1) my personal client mac

I've worked for two of the FAANG companies together about 15 years, and I've always had local administrator access and root access to (1) my personal client machines (laptop, desktop on all platforms) (2) my personal development machines on the corporate network (3) my teams' production machines. I'm a software developer and wouldn't have it any other way. Software developers need root/admin access if you expect any development velocity and ability to solve problems in production with expedience. There are too many diagnostic operations that depend on having root/admin, and especially fixes to errors. One exception might be if the production fleet is handled by a separate operations team – but I'm not in favor of this model, which has its own negative impact on velocity/delivery speed due to the coordination involved, and the tension between "software developers want to ship" and "operations team wants software to remain unchanging and stable". I'm not in favor of operations teams until the software team is overwhelmed with support requests and their velocity is slowing down. In that case, I'd start with beefing up the support team; and then finally at tremendous scale with high operational burden, hire an operations team / SRE team to operate the system. The solution is to make software responsible for owning/operating their own systems until they are extremely large scale (Amazon S3 scale) – as scale so large that rare spontaneous faults would eat up a significant percentage of the overall software team's time. I'm not opposed to having frontline customer support in front of the developers, potentially multiple tiers (since a lot of customers need to Read The Fine Manual); but once the escalation hits a developer, to expeditiously solve the problem they need root/admin. Your software developers with root/admin access should still be logged such that central security team can observe what's going on on every machine at the company, via configuration of SSHD, and regular inspection of that config – e.g. a shell that logs off-box command execution in real-time by default, either for all commands or commands root; and this will capture what they're doing unless they go very far out of the way to conceal and obfuscate their activities. Even then, the initial steps to begin the concealment will show up in the logs, unless very well obfuscated.) Lastly, if your team runs public web servers, then you need to listen on ports like 80 and 443 while requiring root. Well-written servers drop root immediately after opening the socket, but custom written servers may run on those ports and not drop capabilities and run in a container. For example, Faceook Messenger Platform's onboarding instructions and example code –https://developers.facebook.com/docs/messenger-platform/getting-started/webhook-setup https://developers.facebook.com/docs/messenger-platform/gett... – recommends binding a TCP port >1024: ``` app.listen(process.env.PORT || 1337, () => console.log('webhook is listening')); ``` Aside: Nice and professional port number there. If I'd written that documentation I would have used 8081 or 8443, and included a discussion of the implications of running a web server on HTTPS (443) or HTTP (80). I understand that having software developers having root/admin on their machine is very different than non-tech jobs, but for any jobs in the engineering family it's essential. The builders and operators of a system should always have root access to it, because (1) it's needed for development throughput (2) they can trivially get root access anyway through a variety of different means, such as incorporating a grant of root access into the code they're shipping, with minor obfuscation. At my last company I was also involved in a major effort to make easier to not run software as root. However, for "typical" services running on single-purpose machines or Virtual Machines that don't do anything but receive or process commands from any other software, and don't run any other software (besides company Infra), I think the risk is low. "root" and "nobody" are marginally different on a machine that runs a single application because an attacker who breaks in can access all of its data anyway. It's true that with root an attacker can implant rootkits and more firmly maintain a grasp of the server, but competent FAANG companies (1) require MFA in order to log in to servers besides the initial hardware client in front of the user (2) will audit machines for unexpected changes like what I'm describing above. I'm guessing only highly sophisticated attackers like APTs would be able to conceal from those scans, and there will always be the record of initial penetration to review. Incident response will quickly fan out to all machines the pwned machines has communicated with. One of the FAANGs had complete network logs for machines like this and would have been able to inspect the network traffic. Lastly, sophisticated corps employ defense-in-depth and multiple layers of redundancy like (1) you can't attempt to log into machines on the network without both being on the VPN and having authenticated certificates from the enterprise root authority, and (2) Bastion machines which can log commands being sent to machines even if the machines themselves don't, and so on.

hug 5y ago
I disagree with most of this, to be perfectly honest. For one, devs don't need root in prod in the vast majority of shops, which aren't FAANG scale. Assuming a nice sane environment where your deploys to pre-prod mirror your prod environment, I can't think of many cases where having root on prod makes much difference. It's the inverse of your statement, where places that aren't massive should find it feasible to 1:1 their prod environment for the devs to noodle about in. For two, statements like this one make me wary of a blanket policy of devs getting root: > if your team runs public web servers, then you need to listen on ports like 80 and 443 while requiring root. Have you heard of setcap and CAP_NET_BIND_SERVICE, perhaps? I know this is only one example of many where you may "need" root, but in a lot of cases devs just don't have the ops knowledge to know what they actually need.
- jcrites 5y ago
  Yes, I am aware of these capabilities. However, you as the developer still need to be able to write software that runs as root on the box, both to (likely) install new software version and to drop the capability either shortly before executing the primary application (preferred); or the application drops it shortly after launching (seems to be more common – e.g. Postfix Architecture [1] – probably because the App knows what minimal permissions it needs and this is less work for the user). Meanwhile, as a developer you still need the ability to deploy applications unless some kind of "deployment layer" handles depositing new software versions onto the machine. I'm also concerned about the lack of root for being able to inspect what's going wrong with software on the machine (`ps` and `lsof` and socket-specific equivalents). How do I take a heap dump or stack trace from erroneous software on a prod machine without root? How can I step through the software in a debugger? (You could argue that I shouldn't do this but it's a tradeoff between velocity and security, the right answer which may depend on the business we're in.) Containers are kind-of this but I wouldn't use containers as security sandbox, since they're not designed for it. User namespaces get close, but I prefer to rely on the hypervisor for this kind of security boundary, not namespaces, cgroups, or containers – because the kernel API is massive and security vulnerabilities are found in it regularly. (Root exploits less often.) How would you build an operate a server in production without root? I honestly don't have experience. All of my experience has been running installation scripts as root, and having personal root access for inspecting/debugging (potentially even live debugger attaching and stepping); unless the `sudo` commands were very carefully managed I wouldn't be able to use the majority of my standard toolkit for fixing problems, and I wouldn't be able to fix the problem if I found them. Am I running out of file descriptors because the limit is set too low? OK, bump up the limit. That's a code change but I can initiate an emergency recovery by locally changing it on all machines in the fleet. We should also acknowledge that there are different kinds of software systems. If I was working at Coinbase or on Payments at Stripe I might have a different perspective. I previously worked at AWS and now currently work on WhatsApp so I am still incredibly security-conscious, but the level of security required by a product is a negotiable product specification, not a hard-and-fast-rule like the necessary strength of a bridge in civil engineering. [1] Postfix Architecture: http://www.postfix.org/OVERVIEW.html http://www.postfix.org/OVERVIEW.html

hug 5y ago

I disagree with most of this, to be perfectly honest. For one, devs don't need root in prod in the vast majority of shops, which aren't FAANG scale. Assuming a nice sane environment where your deploys to pre-prod mirror your prod environment, I can't think of many cases where having root on prod makes much difference. It's the inverse of your statement, where places that aren't massive should find it feasible to 1:1 their prod environment for the devs to noodle about in. For two, statements like this one make me wary of a blanket policy of devs getting root: > if your team runs public web servers, then you need to listen on ports like 80 and 443 while requiring root. Have you heard of setcap and CAP_NET_BIND_SERVICE, perhaps? I know this is only one example of many where you may "need" root, but in a lot of cases devs just don't have the ops knowledge to know what they actually need.

jcrites 5y ago
Yes, I am aware of these capabilities. However, you as the developer still need to be able to write software that runs as root on the box, both to (likely) install new software version and to drop the capability either shortly before executing the primary application (preferred); or the application drops it shortly after launching (seems to be more common – e.g. Postfix Architecture [1] – probably because the App knows what minimal permissions it needs and this is less work for the user). Meanwhile, as a developer you still need the ability to deploy applications unless some kind of "deployment layer" handles depositing new software versions onto the machine. I'm also concerned about the lack of root for being able to inspect what's going wrong with software on the machine (`ps` and `lsof` and socket-specific equivalents). How do I take a heap dump or stack trace from erroneous software on a prod machine without root? How can I step through the software in a debugger? (You could argue that I shouldn't do this but it's a tradeoff between velocity and security, the right answer which may depend on the business we're in.) Containers are kind-of this but I wouldn't use containers as security sandbox, since they're not designed for it. User namespaces get close, but I prefer to rely on the hypervisor for this kind of security boundary, not namespaces, cgroups, or containers – because the kernel API is massive and security vulnerabilities are found in it regularly. (Root exploits less often.) How would you build an operate a server in production without root? I honestly don't have experience. All of my experience has been running installation scripts as root, and having personal root access for inspecting/debugging (potentially even live debugger attaching and stepping); unless the `sudo` commands were very carefully managed I wouldn't be able to use the majority of my standard toolkit for fixing problems, and I wouldn't be able to fix the problem if I found them. Am I running out of file descriptors because the limit is set too low? OK, bump up the limit. That's a code change but I can initiate an emergency recovery by locally changing it on all machines in the fleet. We should also acknowledge that there are different kinds of software systems. If I was working at Coinbase or on Payments at Stripe I might have a different perspective. I previously worked at AWS and now currently work on WhatsApp so I am still incredibly security-conscious, but the level of security required by a product is a negotiable product specification, not a hard-and-fast-rule like the necessary strength of a bridge in civil engineering. [1] Postfix Architecture: http://www.postfix.org/OVERVIEW.html http://www.postfix.org/OVERVIEW.html