4 ms·
A better alternative is to kill entire servers every once in a while and reprovision them. That way, console access effectively becomes read-only, since any cha
by StavrosK 11y ago
A better alternative is to kill entire servers every once in a while and reprovision them. That way, console access effectively becomes read-only, since any changes will soon be lost, but you can still debug stuff.
- nailer 11y ago+1 netflix made Chaos Monkey specifically for that (not that anyone on HN couldn't code an equivalent, just that it's a technique used at scale).
- deleted 11y ago[deleted]
- atsaloli 11y agoWas the talk recorded?
- georgerobinson 11y agoSorry, it doesn't look like it! I did find the same presentation but from another event: https://www.youtube.com/watch?v=d6pU4C4PVoY https://www.youtube.com/watch?v=d6pU4C4PVoY However, I cannot recall whether this was a subject of the talk or whether I asked it in Q&A. I think it was the latter.
- nasalgoat 11y agoI hope that's a manual process, because I lose SSH sessions all the time and I'd hate for it to fire off while I was debugging.
- georgebarnett 11y agoAbsolutely correct. It's not clear from my reply however my intention is that servers should reset to a known config on restart. Edit; spelling
- ninkendo 11y agoFor that matter, you could just take away the "server" abstraction altogether and run isolated processes in production instead, each having an ephemeral filesystem in the form of a container. SSH access would be more like "heroku run bash" where you don't log into an existing instance, you log into a fresh ephemeral instance that just runs sshd and gets destroyed on logout. Debugging running instances is more interesting though, and I think is possible if each container has an SSH daemon with ephemeral keys, and a central authority they register with that you can hop through to get shell on any instance. Ops is a really fun job to be in when you stop thinking of servers as "things that people log into".