8 ms·

Immutable Infrastructure: No SSH

89 points by axelfontaine 11y ago

huslage 11y ago
Docker?
- bojo 11y ago
  Still requires an underlying OS which may need to be patched or upgraded.
georgebarnett 11y ago
Removing ssh ties the hands of your operations team when an outage hits because you have removed their debugging console. With this change, the team loses access to all the standard OS tools which are useful in the event of a service outage for debugging. What usually ends up happening then is that the standard tools are replaced with substandard and patchy implementations. If your infrastructure is drifting then a better solution is to restart it more regularly. If you find staff are still making untracked changes then that's not a technology problem - it's a people problem.
- StavrosK 11y ago
  A better alternative is to kill entire servers every once in a while and reprovision them. That way, console access effectively becomes read-only, since any changes will soon be lost, but you can still debug stuff.
  - nailer 11y ago
    +1 netflix made Chaos Monkey specifically for that (not that anyone on HN couldn't code an equivalent, just that it's a technique used at scale).
  - deleted 11y ago
    [deleted]
    - atsaloli 11y ago
      Was the talk recorded?
      - georgerobinson 11y ago
        Sorry, it doesn't look like it! I did find the same presentation but from another event: https://www.youtube.com/watch?v=d6pU4C4PVoY https://www.youtube.com/watch?v=d6pU4C4PVoY However, I cannot recall whether this was a subject of the talk or whether I asked it in Q&A. I think it was the latter.
    - nasalgoat 11y ago
      I hope that's a manual process, because I lose SSH sessions all the time and I'd hate for it to fire off while I was debugging.
  - georgebarnett 11y ago
    Absolutely correct. It's not clear from my reply however my intention is that servers should reset to a known config on restart. Edit; spelling
  - ninkendo 11y ago
    For that matter, you could just take away the "server" abstraction altogether and run isolated processes in production instead, each having an ephemeral filesystem in the form of a container. SSH access would be more like "heroku run bash" where you don't log into an existing instance, you log into a fresh ephemeral instance that just runs sshd and gets destroyed on logout. Debugging running instances is more interesting though, and I think is possible if each container has an SSH daemon with ephemeral keys, and a central authority they register with that you can hop through to get shell on any instance. Ops is a really fun job to be in when you stop thinking of servers as "things that people log into".
- mpdehaan2 11y ago
  This. It's simply "don't perform install, maintaince, or upgrade mechanisms over manual means ever". Do NOT cut off your SSH channel. You'll be kicking yourself when you want to sftp a file off of a box pretty quickly. And you might need it in an emergency. Focus on good automation - whether immutable systems or otherwise, and be sure you can rebuild your infrastructure completely from source control. Do this by refusing to make manual changes, and knowing any manual changes will be often clobbered by the system.
  - aalbertson 11y ago
    While I agree (my boss and I are actually having this conversation presently), I would argue that if you're in any sort of cloud environment (which this article has set as a point of narrative), then just having an Ops s3 bucket or similar to send the files works. Although that of course still means SSH to get there "most likely". But could be accomplished without. FTR, I'm in favor of restricting SSH/RDP, perhaps even disabling from the security groups (AWS), on a default basis, and enabling as needed for ops troubleshooting. If we have logging and monitoring in place cleanly enough, then we should be able to troubleshoot the majority of issues off instance. However that is almost NEVER the case (even if you have the tools, you still have too many edge cases). Case in point, I've been troubleshooting a Windows CPU usage issue recently. Our monitoring wasn't catching all of the disparate processes, so I couldn't see what was eating it up. Plus I would pull the node from the ELB prior to investigating, which then caused it to no longer peg CPU. Had I not been able to login while "live", I may not have ever seen what it was. Obviously in order to perform such a function, one needs incredible reliability in your monitoring/logging tools. Sadly this is a more difficult function than it should be....
socketpuppet 11y ago
We do something similar with Nix (https://www.nixos.org https://www.nixos.org). We have a CI server (Hydra) which creates a closure containing our software and all its dependencies, which we then upload to a network of AWS machines that are created with a few python scripts. This works fairly well, but from experience we do need to keep an SSH setup on the machines. Just last week we had a load spike on a production server which was caused by a software bug, triggered by a usage pattern that was unexpected and not covered in our tests. If we did not have SSH (and access to the CLI tools on that machine) we would not have been able to debug this unexpected problem. I guess what I'm saying is that as long as software has bugs and hardware has glitches, we'll sometimes need access to low-level tools which can help us figure out the cause of these unexpected scenarios.
- ris 11y ago
  Exactly what I was thinking as I read this. Just use nix. Forget the dogma.
rotten 11y ago
Somewhere in the infrastructure there has to be mutable servers - databases and log servers at the least. I can't think of many architectures where everything can be a closed appliance. Are they suggesting that every time you need to apply a security patch, you'll have to build a new server and cut your traffic over to it? That sounds like a lot of DNS and Load Balancer config updates (mutable!) even if the newly patched server builds are 99% automated. The idea of black-box appliances has been around a long time, and it has its place in the modern infrastructure, but I'm not sure it really solves the problems they are trying to solve (which sound more like change management issues).
- axelfontaine 11y ago
  This type of image is primarily designed for 12-factor apps where all persistent state is kept outside the instance in some geo-redundant highly available system like Amazon RDS or S3.
  - mugsie 11y ago
    RDS is not "geo-redundant". It is an multi-AZ DB, which is very different to multi-region. (Availability Zones (AZs) are within a DC, regions are separate DCs)
    - axelfontaine 11y ago
      "Availability Zones (AZs) are within a DC" => not really See this Re:Invent 2014 talk about AWS Innovation At Scale http://mvdirona.com/jrh/talksandpapers/JamesHamiltonReInvent2014.pdf http://mvdirona.com/jrh/talksandpapers/JamesHamiltonReInvent... => Slide 9: "Each AZ is 1 or more DC, No data center is in two AZs, Some AZs have as many as 6 DCs"
      - mugsie 11y ago
        Sorry - you are right, with the scale of Amazon they are within a region. But that is still not Geo-Redundant. If a hurricane takes out Virgina, you are still going to loose your data.
eyko 11y ago
> Trouble starts the minute you start relying on commands like this: sudo apt-get install mypkg So what's wrong with `sudo apt-get install mypkg=<version>`?
- yebyen 11y ago
  Do you do this every time? Must be cumbersome, there's no equivalent of Gemfile.lock and bundler for apt-get, is there? (Or is there? Serious question)
  - oblio 11y ago
    https://help.ubuntu.com/community/PinningHowto https://help.ubuntu.com/community/PinningHowto My guess is that apt pinning/holding predates Gemfile.lock by quite a few years :)
    - yebyen 11y ago
      This and Ansible are non-answers to me. The functionality of bundle install and bundle update are not replicated by apt pinning or holding. How do you roll back a bad update to your pinned packages? Put back the old Gemfile.lock and bundle (again)? Does that really work? I guess I don't really use Ansible so I don't know, but it's very easy to roll back a bundle update if you keep your Gemfile.lock in revision control too. I've used apt pinning before though, and I'm quite sure rollback of a package update that has a dependency conflict with the old version is not as easy.
      - icebraining 11y ago
        I've used apt pinning before though, and I'm quite sure rollback of a package update that has a dependency conflict with the old version is not as easy. Aptitude has always handled that just fine for me, downgrading the dependencies as well if needed.
        yebyen 11y ago
therealmarv 11y ago
What about logs? What about looking into the server to find out what is going on on the server?
- axelfontaine 11y ago
  Application logs should be shipped over the network to either your own ELK infrastructure or some hosted solution like Loggly, Logentries or Papertrail. Boot logs can be obtained from your infrastructure (like EC2 instance logs).
  - spacecowboy_lon 11y ago
    And when that stops working for some reason? of course the real solution to have out of band managemnet
    - jopython 11y ago
      I have a seen a decent answer for this question when people talk about 'immutable' hosts.
    - liveoneggs 11y ago
      the network is 100% reliable, always. The internet, doubly so (it's so big, you know). Maybe you missed the memo.
      - spacecowboy_lon 11y ago
        Well as a "circuit switched bigot" Vint must have left me off the mailing list :-) I used to to do OSI international interconnect support an testing for my sins many moons ago.
deleted 11y ago
[deleted]
dschiptsov 11y ago
May I suggest another "innovation" - statically link everything into a single blob and run it under systemd. The building scripts should be written in JavaScript, of course. Package it as a startup. Become millionaire.
mugsie 11y ago
"Vulnerabilities like ShellShock simply vanish" - eh .... not really. Shellshock was not just from SSH, it was any program that used environment variables to pass data around. Immutable infrastructure is a great ideal, but has various downsides. sometimes people need to access the raw system to do debugging. While shoot the node, and boot a new one has advantages, it does just move the problem down the line. The BBC solution is a nice balance.
viraptor 11y ago
It's quite silly. I mean they mix up two completely separate issues. Deployment problem and people problem. Image deployments are not new - computing clusters have done it for ages. Immutable deployments aren't new either - actual immutable filesystems booted over network were there for many many years. Yes, that's going to prevent drift at first. No, it's not going to prevent people from doing silly stuff. At least ssh allows them to do silly stuff in a "normal" and visible way and doesn't stop you from debugging issues when you actually need it. If you enforce no-login without teaching people, they'll just reinvent it... badly - for example by making executable configurations, or pseudo-web-shells. And there's one big issue with images which will cause a lot of friction in real teams - they're slow to build. And even slower to test. If you're doing it properly it means first testing locally on some approximate environment, then handing the package over to a system which will test the whole deployment, which takes ages to boot up all components. (time scales with system complexity - it's not ages for a single webserver of course) Anyone developing something related to integrating services will get additional frustration unfortunately. > Enforcing immutability [...] You need to prevent log in. No... If you need immutability, you make your system actually immutable and then audit logins when logins are needed. Preventing logins is just punishing people and slowing them down when they need it the most, because you don't trust them. > Vulnerabilities like ShellShock simply vanish Said a person who does not understand why ShellShock was an issue and why it doesn't matter if a user who already has local access uses shellshock.
- mugsie 11y ago
  As someone who worked in a company that didn't always have access to the customer machine that ran our software, I can confirm. There might have been a magic page in the software that allowed raw SQL queries...
- shangxiao 11y ago
  > Vulnerabilities like ShellShock simply vanish I'm actually surprised at the naivety of whoever wrote this. I'm not an op at all and I understood enough about ShellShock to know that this isn't correct at all.
amalcon 11y ago
> Vulnerabilities like ShellShock simply vanish. What? No, they don't! ShellShock was about executing unintended code in a bash process where you control an environment variable but not stdin. If you control stdin, there's no need to sneak a command into an environment variable: you can just type it into the shell. Apart from this, the article makes no argument for removing SSH: it only argues for restricting access to privileged userids. That is to say, the standard best practice for decades. Root should be reserved for special situations, not used by default.
exarch 11y ago
This is what happens when dev teams hate ops teams. Troubleshooting a production bug without a terminal is a recipe for lengthy outages.
- jacques_chester 11y ago
  I worked on Cloud Foundry buildpacks for 7 months. I was working at the Pivotal Labs offices in NYC and I can tell you this for free: devs want SSH access too. When a box dies in staging, it's nice to learn why.
- lamontcg 11y ago
  Yes, exactly. This is an old-school ops team that learned about "devops" and "immutable infrastructure" and figured now they have all the necessary arguments to kick the devs off the servers once and for all. Not progress.
asymmetric 11y ago
This is not even a problem if you're using Ansible (and possibly others): you just say you want the latest version of the package installed, and ansible will to an `apt-get update` for you as a preliminary step, and install/update the package if needed. So basically, all the machines you're running ansible on will have the latest version (if that's what you want, of course)
stephen 11y ago
Interesting; so, boxfuse is one of these (..."micro" kernels? What's the name, of when you run in a hypervisor without a real OS, so no user isolation, no process isolation, etc.) OS/image-creation tools... I didn't realize they were production ready yet, but sounds pretty spiffy. What's interesting is that I think Boxfuse is spinning what most people would call a weakness ("if stuff breaks, I can't SSH into the shell and fix it (because, not only is that a 'bad idea', there just isn't a Linux OS there to SSH into anyway)" and calling it a strength. Nice spin. :-) If I tried boxfuse, I'd probably look at embedding the Apache Java-based SSHD server, not for app setup, but debugging ... assuming its terminal session would provide useful information in the Boxfuse "there is no OS" environment. (The boxfuse site uses the term "Secure Micro OS, Few MBs", so if I'm misinterpreting what their platform does, someone please correct me.)
kazinator 11y ago
"Sign up for your Boxfuse account." What? And disturb the tranquil immutability of boxfuse.com?
- waspleg 11y ago
  I admit I lol'd.
- kaonashi 11y ago
  It actually gets swapped with a new Boxfuse.com on write.
grhmc 11y ago
So I've considered this before, and I know it is coming from a good place. It seems great for the good times. At Clarify.io we use immutable infrastructure and we don't log in to boxes. Except when there are problems. And there are problems. Always. You don't want to have to fight your way in in the middle of a disaster. At Clarify.io we've considered having an on-login event fire when an admin uses SSH to log in to the box. This would schedule the server for termination within 24 hours. This gets you best of all worlds: - Gives tools to debug and restore service when there is a failure - Forces admins to not depend on SSH to bring a system up - Replaces servers when they are potentially drifting - Encourages a "cattle" mentality about the servers
- aalbertson 11y ago
  I really like this idea. Would be keen to see what your orchestration code looks like to handle that.
  - grhmc 11y ago
    We already use JanitorMonkey. It would likely involve tagging the instance in AWS with a tainted tag, and adding code to JanitorMonkey to do the normal mark-and-sweep.
    - aalbertson 11y ago
      That's awesome! I don't know how I've missed JanitorMonkey amongst their tools, but that looks great. Definitely giving it a whirl.
HugoDaniel 11y ago
On *BSD kernels you can use securelevel to achieve proper immutability to some extent (like unchangeable pf rules, read-only raw disk devices, etc...). Disabling ssh only gives you the guarantee that ssh is going to be disabled, and that is quite different from immutability.
tgeek 11y ago
I find the better pattern here is to limit and discourage SSH and then monitor and log the hell out of it. There are numerous tools out there that can centralize any actions being taken on a host and sending it to a log that can be centralized. Outright removing all SSH puts you in a rough spot if things go south with some piece of software that your system monitoring/centralized logging don't cover 100%, and makes it way hard to do things like strace on a process.
- grhmc 11y ago
  Or if one of the problems of the server caused logging to not correctly start :)
  - aalbertson 11y ago
    Which is certainly a valid case, but I would argue that if that IS the case, and it cannot autorecover, then that is a good candidate for the instance to self terminate.
  - Sanddancer 11y ago
    If logging never started, then your service heartbeat script should have caught it and raised an alert. Logging being down is Kinda Important.
- sneak 11y ago
  Also, make it part of the process that each time ssh is used, logging or monitoring is set up to catch what it was used for, much the same way that a test is added when a bug is discovered.
mwcampbell 11y ago
A common response on this thread has been that if you take away SSH, you can't log into individual machines to debug. But you know the other situation where you can't do that? When your software is running on your users' machines rather than yours, i.e. mobile and desktop apps. I agree with the position of the OP, that it's better to make the software robust by design and through testing.
- icebraining 11y ago
  it's better to make the software robust by design and through testing. But that's a false dichotomy; having SSH in no way prevents you from designing and testing your software just as well.
  - juliangregorian 11y ago
    I don't know, I think there's a legitimate argument to be made that some tools can become a crutch and a detriment. At one of my consults all the developers lean heavily on IDE debugging at the expense of developing well-defined contracts and interfaces. Nobody blinks twice at having a ton of threads mutating global variables because they have all these great debugging and tracing tools.
- breischl 11y ago
  If you're unable to debug due to the nature of your software then that sucks but it's life. And notice that lot of devs in that situation try to fix it with various remote debugging tools. The article seems to advocate putting yourself into that situation when you don't have to, which is silly.
  - mwcampbell 11y ago
    The hard reality of no remote debugging in software deployed to end-user machines leads a wise developer to make potentially inconvenient trade-offs to compensate, e.g. reducing the chance of failure by minimizing dynamism. I believe, and I think the OP would agree, that applying the same discipline to server-side software leads to more robust applications, whereas if you know you have the crutch of remote debugging (via SSH or otherwise) when something goes wrong, you may well be more lax about preventing failures in the first place.
peterwwillis 11y ago
What is drift? Local edits, updates not applied, literally ntpd/ntpdate not updating the time. Your host is misconfigured; it's broken. How do you deal with systems breaking? By monitoring for them. It also helps to make them 'disposable', but this is not a replacement for monitoring. Good monitoring will tell you when time is off, when disks are about to fill up, when permissions are bad and when packages aren't up to date. Removing ssh does not give you monitoring. It does not prevent system state from changing. It really doesn't do anything but remove a very secure and simple method of communication and file transfer. Minimal/disposable images? That's fine. It won't make your servers or services immutable. One day your hardware's going to catch on fire and your first clue won't be the smoke billowing into the neighboring rack, it will be the errors in your logs.
deleted 11y ago
[deleted]
Sleaker 11y ago
Uhh... you now how we handle drift? We manage the update repository ourselves. New packages don't get released until we are ready to release them.... You don't need to limit ssh to help handle state drift. Article writer doesn't seem to understand the difference between user access, and software deployment.
njharman 11y ago
I ssh into machines for many reasons other than to change them. In fact most changes are done remotely with Ansible.
lamontcg 11y ago
This is just dumb as nails. Now I can't login to the production server in order to strace the process to figure out why production is misbehaving. And don't say you should always replicate the issue in preprod, sometimes you simply can't. Some issues only emerge at prod scale+load+latency that you don't have the financial resources or technical ability to be able to exactly replicate in a testbed. You are drunk on too much koolaid, go home.
- juliangregorian 11y ago
  That's great if strace et. al. can actually tell you why production is misbehaving. Of course, strace doesn't exist in vacuum, it changes the way your code executes. Everything is going to be slower, and race conditions and locks may not present themselves so readily. Then you also have the added stress of screwing around with a live server, what if you do something that messes it up worse? Sometimes it makes more sense just to kill that server and add a fresh one to the pool. Better yet if that process is automated. Good logging can actually be more valuable than mucking about in the live environment (or as I call it, panic mode).
siliconc0w 11y ago
I'm a fan of immutable infrastructure but there is still a need to get access to a container now and again to troubleshoot things. We don't run SSH inside the containers but it's easy enough to write some tooling to find a host running an instance and use docker exec to get a shell on it.
OriPekelman 11y ago
This is so the wrong question. Immutable infrastructure does not require nor warrant restricting the form of access you have to your services. Having everything build-oriented (basically building the builder on every deployment, than using it to build the service) is what prevents drift. Running on a read-only file system is what guarantees immutability and prevents drift. Not allowing SSH is plugging a single hole which guarantees nothing. Can the app-server write on its own disk? Can it modify configuration files? Maybe even code? As a developer I want to be able to write applications that are not specific to any one deployment method. I want to be able to change my mind to host this on different kind of infrastructure. These kind of limits force you to create a strong adherence between your application and its tooling and the deployment method. Of course, the Read/Write domain has to be explicit and limited, but this is good practice anyway. Running on a distributed storage grid based on something like CEPH can allow you to address this domain not only explicitly but also in a fault-tolerant and scalable way. When you cut off SSH you are cutting off half of the tooling we have and you make everything overly complicated. I want to be able to tail any log on any host. I want to be able to scp for inspection any file to my dev machine. At platform.sh (a PaaS running on a distributed grid of micro-containers) we run a build oriented immutable infrastructure so there is no, there can be no infrastructure drift. We proxy SSH so security stays the responsibility of the orchestration layer not any single host or service. This means we can filter by role any connection to any service, but developers can still just use the tools that work.

89 points by axelfontaine 11y ago

huslage 11y ago

Docker?

bojo 11y ago
Still requires an underlying OS which may need to be patched or upgraded.

georgebarnett 11y ago

Removing ssh ties the hands of your operations team when an outage hits because you have removed their debugging console. With this change, the team loses access to all the standard OS tools which are useful in the event of a service outage for debugging. What usually ends up happening then is that the standard tools are replaced with substandard and patchy implementations. If your infrastructure is drifting then a better solution is to restart it more regularly. If you find staff are still making untracked changes then that's not a technology problem - it's a people problem.

StavrosK 11y ago
A better alternative is to kill entire servers every once in a while and reprovision them. That way, console access effectively becomes read-only, since any changes will soon be lost, but you can still debug stuff.
- nailer 11y ago
  +1 netflix made Chaos Monkey specifically for that (not that anyone on HN couldn't code an equivalent, just that it's a technique used at scale).
- deleted 11y ago
  [deleted]
  - atsaloli 11y ago
    Was the talk recorded?
    - georgerobinson 11y ago
      Sorry, it doesn't look like it! I did find the same presentation but from another event: https://www.youtube.com/watch?v=d6pU4C4PVoY https://www.youtube.com/watch?v=d6pU4C4PVoY However, I cannot recall whether this was a subject of the talk or whether I asked it in Q&A. I think it was the latter.
  - nasalgoat 11y ago
    I hope that's a manual process, because I lose SSH sessions all the time and I'd hate for it to fire off while I was debugging.
- georgebarnett 11y ago
  Absolutely correct. It's not clear from my reply however my intention is that servers should reset to a known config on restart. Edit; spelling
- ninkendo 11y ago
  For that matter, you could just take away the "server" abstraction altogether and run isolated processes in production instead, each having an ephemeral filesystem in the form of a container. SSH access would be more like "heroku run bash" where you don't log into an existing instance, you log into a fresh ephemeral instance that just runs sshd and gets destroyed on logout. Debugging running instances is more interesting though, and I think is possible if each container has an SSH daemon with ephemeral keys, and a central authority they register with that you can hop through to get shell on any instance. Ops is a really fun job to be in when you stop thinking of servers as "things that people log into".
mpdehaan2 11y ago
This. It's simply "don't perform install, maintaince, or upgrade mechanisms over manual means ever". Do NOT cut off your SSH channel. You'll be kicking yourself when you want to sftp a file off of a box pretty quickly. And you might need it in an emergency. Focus on good automation - whether immutable systems or otherwise, and be sure you can rebuild your infrastructure completely from source control. Do this by refusing to make manual changes, and knowing any manual changes will be often clobbered by the system.
- aalbertson 11y ago
  While I agree (my boss and I are actually having this conversation presently), I would argue that if you're in any sort of cloud environment (which this article has set as a point of narrative), then just having an Ops s3 bucket or similar to send the files works. Although that of course still means SSH to get there "most likely". But could be accomplished without. FTR, I'm in favor of restricting SSH/RDP, perhaps even disabling from the security groups (AWS), on a default basis, and enabling as needed for ops troubleshooting. If we have logging and monitoring in place cleanly enough, then we should be able to troubleshoot the majority of issues off instance. However that is almost NEVER the case (even if you have the tools, you still have too many edge cases). Case in point, I've been troubleshooting a Windows CPU usage issue recently. Our monitoring wasn't catching all of the disparate processes, so I couldn't see what was eating it up. Plus I would pull the node from the ELB prior to investigating, which then caused it to no longer peg CPU. Had I not been able to login while "live", I may not have ever seen what it was. Obviously in order to perform such a function, one needs incredible reliability in your monitoring/logging tools. Sadly this is a more difficult function than it should be....

socketpuppet 11y ago

We do something similar with Nix (https://www.nixos.org https://www.nixos.org). We have a CI server (Hydra) which creates a closure containing our software and all its dependencies, which we then upload to a network of AWS machines that are created with a few python scripts. This works fairly well, but from experience we do need to keep an SSH setup on the machines. Just last week we had a load spike on a production server which was caused by a software bug, triggered by a usage pattern that was unexpected and not covered in our tests. If we did not have SSH (and access to the CLI tools on that machine) we would not have been able to debug this unexpected problem. I guess what I'm saying is that as long as software has bugs and hardware has glitches, we'll sometimes need access to low-level tools which can help us figure out the cause of these unexpected scenarios.

ris 11y ago
Exactly what I was thinking as I read this. Just use nix. Forget the dogma.

rotten 11y ago

Somewhere in the infrastructure there has to be mutable servers - databases and log servers at the least. I can't think of many architectures where everything can be a closed appliance. Are they suggesting that every time you need to apply a security patch, you'll have to build a new server and cut your traffic over to it? That sounds like a lot of DNS and Load Balancer config updates (mutable!) even if the newly patched server builds are 99% automated. The idea of black-box appliances has been around a long time, and it has its place in the modern infrastructure, but I'm not sure it really solves the problems they are trying to solve (which sound more like change management issues).

axelfontaine 11y ago
This type of image is primarily designed for 12-factor apps where all persistent state is kept outside the instance in some geo-redundant highly available system like Amazon RDS or S3.
- mugsie 11y ago
  RDS is not "geo-redundant". It is an multi-AZ DB, which is very different to multi-region. (Availability Zones (AZs) are within a DC, regions are separate DCs)
  - axelfontaine 11y ago
    "Availability Zones (AZs) are within a DC" => not really See this Re:Invent 2014 talk about AWS Innovation At Scale http://mvdirona.com/jrh/talksandpapers/JamesHamiltonReInvent2014.pdf http://mvdirona.com/jrh/talksandpapers/JamesHamiltonReInvent... => Slide 9: "Each AZ is 1 or more DC, No data center is in two AZs, Some AZs have as many as 6 DCs"
    - mugsie 11y ago
      Sorry - you are right, with the scale of Amazon they are within a region. But that is still not Geo-Redundant. If a hurricane takes out Virgina, you are still going to loose your data.

eyko 11y ago

> Trouble starts the minute you start relying on commands like this: sudo apt-get install mypkg So what's wrong with `sudo apt-get install mypkg=<version>`?

yebyen 11y ago
Do you do this every time? Must be cumbersome, there's no equivalent of Gemfile.lock and bundler for apt-get, is there? (Or is there? Serious question)
- oblio 11y ago
  https://help.ubuntu.com/community/PinningHowto https://help.ubuntu.com/community/PinningHowto My guess is that apt pinning/holding predates Gemfile.lock by quite a few years :)
  - yebyen 11y ago
    This and Ansible are non-answers to me. The functionality of bundle install and bundle update are not replicated by apt pinning or holding. How do you roll back a bad update to your pinned packages? Put back the old Gemfile.lock and bundle (again)? Does that really work? I guess I don't really use Ansible so I don't know, but it's very easy to roll back a bundle update if you keep your Gemfile.lock in revision control too. I've used apt pinning before though, and I'm quite sure rollback of a package update that has a dependency conflict with the old version is not as easy.
    - icebraining 11y ago
      I've used apt pinning before though, and I'm quite sure rollback of a package update that has a dependency conflict with the old version is not as easy. Aptitude has always handled that just fine for me, downgrading the dependencies as well if needed.
      - yebyen 11y ago

therealmarv 11y ago

What about logs? What about looking into the server to find out what is going on on the server?

axelfontaine 11y ago
Application logs should be shipped over the network to either your own ELK infrastructure or some hosted solution like Loggly, Logentries or Papertrail. Boot logs can be obtained from your infrastructure (like EC2 instance logs).
- spacecowboy_lon 11y ago
  And when that stops working for some reason? of course the real solution to have out of band managemnet
  - jopython 11y ago
    I have a seen a decent answer for this question when people talk about 'immutable' hosts.
  - liveoneggs 11y ago
    the network is 100% reliable, always. The internet, doubly so (it's so big, you know). Maybe you missed the memo.
    - spacecowboy_lon 11y ago
      Well as a "circuit switched bigot" Vint must have left me off the mailing list :-) I used to to do OSI international interconnect support an testing for my sins many moons ago.

deleted 11y ago

[deleted]

dschiptsov 11y ago

May I suggest another "innovation" - statically link everything into a single blob and run it under systemd. The building scripts should be written in JavaScript, of course. Package it as a startup. Become millionaire.

mugsie 11y ago

"Vulnerabilities like ShellShock simply vanish" - eh .... not really. Shellshock was not just from SSH, it was any program that used environment variables to pass data around. Immutable infrastructure is a great ideal, but has various downsides. sometimes people need to access the raw system to do debugging. While shoot the node, and boot a new one has advantages, it does just move the problem down the line. The BBC solution is a nice balance.

viraptor 11y ago

It's quite silly. I mean they mix up two completely separate issues. Deployment problem and people problem. Image deployments are not new - computing clusters have done it for ages. Immutable deployments aren't new either - actual immutable filesystems booted over network were there for many many years. Yes, that's going to prevent drift at first. No, it's not going to prevent people from doing silly stuff. At least ssh allows them to do silly stuff in a "normal" and visible way and doesn't stop you from debugging issues when you actually need it. If you enforce no-login without teaching people, they'll just reinvent it... badly - for example by making executable configurations, or pseudo-web-shells. And there's one big issue with images which will cause a lot of friction in real teams - they're slow to build. And even slower to test. If you're doing it properly it means first testing locally on some approximate environment, then handing the package over to a system which will test the whole deployment, which takes ages to boot up all components. (time scales with system complexity - it's not ages for a single webserver of course) Anyone developing something related to integrating services will get additional frustration unfortunately. > Enforcing immutability [...] You need to prevent log in. No... If you need immutability, you make your system actually immutable and then audit logins when logins are needed. Preventing logins is just punishing people and slowing them down when they need it the most, because you don't trust them. > Vulnerabilities like ShellShock simply vanish Said a person who does not understand why ShellShock was an issue and why it doesn't matter if a user who already has local access uses shellshock.

mugsie 11y ago
As someone who worked in a company that didn't always have access to the customer machine that ran our software, I can confirm. There might have been a magic page in the software that allowed raw SQL queries...
shangxiao 11y ago
> Vulnerabilities like ShellShock simply vanish I'm actually surprised at the naivety of whoever wrote this. I'm not an op at all and I understood enough about ShellShock to know that this isn't correct at all.

amalcon 11y ago

> Vulnerabilities like ShellShock simply vanish. What? No, they don't! ShellShock was about executing unintended code in a bash process where you control an environment variable but not stdin. If you control stdin, there's no need to sneak a command into an environment variable: you can just type it into the shell. Apart from this, the article makes no argument for removing SSH: it only argues for restricting access to privileged userids. That is to say, the standard best practice for decades. Root should be reserved for special situations, not used by default.

exarch 11y ago

This is what happens when dev teams hate ops teams. Troubleshooting a production bug without a terminal is a recipe for lengthy outages.

jacques_chester 11y ago
I worked on Cloud Foundry buildpacks for 7 months. I was working at the Pivotal Labs offices in NYC and I can tell you this for free: devs want SSH access too. When a box dies in staging, it's nice to learn why.
lamontcg 11y ago
Yes, exactly. This is an old-school ops team that learned about "devops" and "immutable infrastructure" and figured now they have all the necessary arguments to kick the devs off the servers once and for all. Not progress.

asymmetric 11y ago

This is not even a problem if you're using Ansible (and possibly others): you just say you want the latest version of the package installed, and ansible will to an `apt-get update` for you as a preliminary step, and install/update the package if needed. So basically, all the machines you're running ansible on will have the latest version (if that's what you want, of course)

stephen 11y ago

Interesting; so, boxfuse is one of these (..."micro" kernels? What's the name, of when you run in a hypervisor without a real OS, so no user isolation, no process isolation, etc.) OS/image-creation tools... I didn't realize they were production ready yet, but sounds pretty spiffy. What's interesting is that I think Boxfuse is spinning what most people would call a weakness ("if stuff breaks, I can't SSH into the shell and fix it (because, not only is that a 'bad idea', there just isn't a Linux OS there to SSH into anyway)" and calling it a strength. Nice spin. :-) If I tried boxfuse, I'd probably look at embedding the Apache Java-based SSHD server, not for app setup, but debugging ... assuming its terminal session would provide useful information in the Boxfuse "there is no OS" environment. (The boxfuse site uses the term "Secure Micro OS, Few MBs", so if I'm misinterpreting what their platform does, someone please correct me.)

kazinator 11y ago

"Sign up for your Boxfuse account." What? And disturb the tranquil immutability of boxfuse.com?

waspleg 11y ago
I admit I lol'd.
kaonashi 11y ago
It actually gets swapped with a new Boxfuse.com on write.

grhmc 11y ago

So I've considered this before, and I know it is coming from a good place. It seems great for the good times. At Clarify.io we use immutable infrastructure and we don't log in to boxes. Except when there are problems. And there are problems. Always. You don't want to have to fight your way in in the middle of a disaster. At Clarify.io we've considered having an on-login event fire when an admin uses SSH to log in to the box. This would schedule the server for termination within 24 hours. This gets you best of all worlds: - Gives tools to debug and restore service when there is a failure - Forces admins to not depend on SSH to bring a system up - Replaces servers when they are potentially drifting - Encourages a "cattle" mentality about the servers

aalbertson 11y ago
I really like this idea. Would be keen to see what your orchestration code looks like to handle that.
- grhmc 11y ago
  We already use JanitorMonkey. It would likely involve tagging the instance in AWS with a tainted tag, and adding code to JanitorMonkey to do the normal mark-and-sweep.
  - aalbertson 11y ago
    That's awesome! I don't know how I've missed JanitorMonkey amongst their tools, but that looks great. Definitely giving it a whirl.

HugoDaniel 11y ago

On *BSD kernels you can use securelevel to achieve proper immutability to some extent (like unchangeable pf rules, read-only raw disk devices, etc...). Disabling ssh only gives you the guarantee that ssh is going to be disabled, and that is quite different from immutability.

tgeek 11y ago

I find the better pattern here is to limit and discourage SSH and then monitor and log the hell out of it. There are numerous tools out there that can centralize any actions being taken on a host and sending it to a log that can be centralized. Outright removing all SSH puts you in a rough spot if things go south with some piece of software that your system monitoring/centralized logging don't cover 100%, and makes it way hard to do things like strace on a process.

grhmc 11y ago
Or if one of the problems of the server caused logging to not correctly start :)
- aalbertson 11y ago
  Which is certainly a valid case, but I would argue that if that IS the case, and it cannot autorecover, then that is a good candidate for the instance to self terminate.
- Sanddancer 11y ago
  If logging never started, then your service heartbeat script should have caught it and raised an alert. Logging being down is Kinda Important.
sneak 11y ago
Also, make it part of the process that each time ssh is used, logging or monitoring is set up to catch what it was used for, much the same way that a test is added when a bug is discovered.

mwcampbell 11y ago

A common response on this thread has been that if you take away SSH, you can't log into individual machines to debug. But you know the other situation where you can't do that? When your software is running on your users' machines rather than yours, i.e. mobile and desktop apps. I agree with the position of the OP, that it's better to make the software robust by design and through testing.

icebraining 11y ago
it's better to make the software robust by design and through testing. But that's a false dichotomy; having SSH in no way prevents you from designing and testing your software just as well.
- juliangregorian 11y ago
  I don't know, I think there's a legitimate argument to be made that some tools can become a crutch and a detriment. At one of my consults all the developers lean heavily on IDE debugging at the expense of developing well-defined contracts and interfaces. Nobody blinks twice at having a ton of threads mutating global variables because they have all these great debugging and tracing tools.
breischl 11y ago
If you're unable to debug due to the nature of your software then that sucks but it's life. And notice that lot of devs in that situation try to fix it with various remote debugging tools. The article seems to advocate putting yourself into that situation when you don't have to, which is silly.
- mwcampbell 11y ago
  The hard reality of no remote debugging in software deployed to end-user machines leads a wise developer to make potentially inconvenient trade-offs to compensate, e.g. reducing the chance of failure by minimizing dynamism. I believe, and I think the OP would agree, that applying the same discipline to server-side software leads to more robust applications, whereas if you know you have the crutch of remote debugging (via SSH or otherwise) when something goes wrong, you may well be more lax about preventing failures in the first place.

peterwwillis 11y ago

What is drift? Local edits, updates not applied, literally ntpd/ntpdate not updating the time. Your host is misconfigured; it's broken. How do you deal with systems breaking? By monitoring for them. It also helps to make them 'disposable', but this is not a replacement for monitoring. Good monitoring will tell you when time is off, when disks are about to fill up, when permissions are bad and when packages aren't up to date. Removing ssh does not give you monitoring. It does not prevent system state from changing. It really doesn't do anything but remove a very secure and simple method of communication and file transfer. Minimal/disposable images? That's fine. It won't make your servers or services immutable. One day your hardware's going to catch on fire and your first clue won't be the smoke billowing into the neighboring rack, it will be the errors in your logs.

deleted 11y ago

[deleted]

Sleaker 11y ago

Uhh... you now how we handle drift? We manage the update repository ourselves. New packages don't get released until we are ready to release them.... You don't need to limit ssh to help handle state drift. Article writer doesn't seem to understand the difference between user access, and software deployment.

njharman 11y ago

I ssh into machines for many reasons other than to change them. In fact most changes are done remotely with Ansible.

lamontcg 11y ago

This is just dumb as nails. Now I can't login to the production server in order to strace the process to figure out why production is misbehaving. And don't say you should always replicate the issue in preprod, sometimes you simply can't. Some issues only emerge at prod scale+load+latency that you don't have the financial resources or technical ability to be able to exactly replicate in a testbed. You are drunk on too much koolaid, go home.

juliangregorian 11y ago
That's great if strace et. al. can actually tell you why production is misbehaving. Of course, strace doesn't exist in vacuum, it changes the way your code executes. Everything is going to be slower, and race conditions and locks may not present themselves so readily. Then you also have the added stress of screwing around with a live server, what if you do something that messes it up worse? Sometimes it makes more sense just to kill that server and add a fresh one to the pool. Better yet if that process is automated. Good logging can actually be more valuable than mucking about in the live environment (or as I call it, panic mode).

siliconc0w 11y ago

I'm a fan of immutable infrastructure but there is still a need to get access to a container now and again to troubleshoot things. We don't run SSH inside the containers but it's easy enough to write some tooling to find a host running an instance and use docker exec to get a shell on it.

OriPekelman 11y ago

This is so the wrong question. Immutable infrastructure does not require nor warrant restricting the form of access you have to your services. Having everything build-oriented (basically building the builder on every deployment, than using it to build the service) is what prevents drift. Running on a read-only file system is what guarantees immutability and prevents drift. Not allowing SSH is plugging a single hole which guarantees nothing. Can the app-server write on its own disk? Can it modify configuration files? Maybe even code? As a developer I want to be able to write applications that are not specific to any one deployment method. I want to be able to change my mind to host this on different kind of infrastructure. These kind of limits force you to create a strong adherence between your application and its tooling and the deployment method. Of course, the Read/Write domain has to be explicit and limited, but this is good practice anyway. Running on a distributed storage grid based on something like CEPH can allow you to address this domain not only explicitly but also in a fault-tolerant and scalable way. When you cut off SSH you are cutting off half of the tooling we have and you make everything overly complicated. I want to be able to tail any log on any host. I want to be able to scp for inspection any file to my dev machine. At platform.sh (a PaaS running on a distributed grid of micro-containers) we run a build oriented immutable infrastructure so there is no, there can be no infrastructure drift. We proxy SSH so security stays the responsibility of the orchestration layer not any single host or service. This means we can filter by role any connection to any service, but developers can still just use the tools that work.