4 ms·

3. Never grand direct SSH to access the the staging/production environment. Automate the build and deployment environment on a system that developers don't hav

3. Never grand direct SSH to access the the staging/production environment. Automate the build and deployment environment on a system that developers don't have direct access. Instead of pushing changes, the bot pulls a specified release branch, builds, tests, and deploys the code. All without human interaction. If malicious code were somehow introduced from a developer's environment, it would be recorded and reflected in the commit history. AFAIK, that's how GitHub manages deployments via HubBot. See https://www.youtube.com/watch?v=NST3u-GjjFw https://www.youtube.com/watch?v=NST3u-GjjFw. ----- To take things a step further, public-facing environments should be made immutable wherever possible. With the entire system being built and released as a whole. Docker alleviates some of the complexity and overhead but I think this space is where Unikernels have a lot of potential to shine. There's a very good talk about how the Wunderlist team used chaos and frequent destruction of their envronments to overcome fear and uncertainty here. https://www.youtube.com/watch?v=RrX_28s70ww&app=desktop https://www.youtube.com/watch?v=RrX_28s70ww&app=desktop. Emphasis being placed on the the frequent disposal and recreation of environments rather than building long-running persistent environments. This setup probably won't work for long-lived systems (ex databases). In those cases, access via a transient environment like a USB bootable OS would be ideal to prevent persistent viruses/trojans. Ironically, the best current options are security-focused distros like KaliLinux that put a special emphasis on avoiding persistent state. Maybe one day soon we'll see an admin-focused OS that better fits this role.

jvehent 11y ago
> 3. Never grand direct SSH to access the the staging/production environment. Most complex systems fail in unexpected ways, and the best way to debug is to grant the devs access to production machines. It can be temporary, monitored and through a secure channel, but it's still something most organizations can hardly live without.
- shawn-furyan 11y ago
  I think that this issue is potentially moot if you've transitioned to immutable deploys. Under such a system, arbitrage steps would tend to change from the standard have an SA/Dev ssh in and muck around until the issue is found. Instead, first step may be just redeploying (in case of transient/intermittent issues that are disrupting service), and then checking out the production system locally to do root cause analysis. So yeah, most organizations couldn't do this today, because they don't have the deploy process in place. But when an organization has made the switch, then it is much more realistic to assume that it can get along without granting ad hoc ssh access to production systems.
  - EvanPlaice 11y ago
    Exactly. It's also possible to capture stack traces remotely from a running system and/or after a crash. http://techblog.netflix.com/2015/12/debugging-nodejs-in-production.html http://techblog.netflix.com/2015/12/debugging-nodejs-in-prod.... Hopefully, this practice -- and the tools required to make it happen -- get better with time.

jvehent 11y ago

> 3. Never grand direct SSH to access the the staging/production environment. Most complex systems fail in unexpected ways, and the best way to debug is to grant the devs access to production machines. It can be temporary, monitored and through a secure channel, but it's still something most organizations can hardly live without.

shawn-furyan 11y ago
I think that this issue is potentially moot if you've transitioned to immutable deploys. Under such a system, arbitrage steps would tend to change from the standard have an SA/Dev ssh in and muck around until the issue is found. Instead, first step may be just redeploying (in case of transient/intermittent issues that are disrupting service), and then checking out the production system locally to do root cause analysis. So yeah, most organizations couldn't do this today, because they don't have the deploy process in place. But when an organization has made the switch, then it is much more realistic to assume that it can get along without granting ad hoc ssh access to production systems.
- EvanPlaice 11y ago
  Exactly. It's also possible to capture stack traces remotely from a running system and/or after a crash. http://techblog.netflix.com/2015/12/debugging-nodejs-in-production.html http://techblog.netflix.com/2015/12/debugging-nodejs-in-prod.... Hopefully, this practice -- and the tools required to make it happen -- get better with time.