5 ms·
This kind of thing is foreign to me because I can't imagine writing an application, to run in the cloud (as I always do) where I care if a single instance has s
by ebikelaw 8y ago
This kind of thing is foreign to me because I can't imagine writing an application, to run in the cloud (as I always do) where I care if a single instance has some downtime. I always rely on something at a higher level proxying my traffic to the application, and when I need to push my application I just shoot each instance in the face and start another one elsewhere, usually on a different host:port. When I'm using the cheapest cloud capacity I can buy my application is constantly preempted by higher-paying tenants anyway, so I have to write my systems as if any instance will suddenly stop. Once you have that problem you can forget about graceful in-place restarts forever.
- twblalock 8y ago> Once you have that problem you can forget about graceful in-place restarts forever. Sure, unless you're the one running the load balancer or the service discovery system that enables other people to just kill their instances without worrying about graceful restarts.
- ebikelaw 8y agoSee my other comment on this thread. Even in that case you don't have to care.
- _wmd 8y agoThere are tons of applications where rolling upgrade doesn't necessarily work out.. anything that is even remotely latency sensitive or state-heavy, for example a game server or an order matching engine
- ebikelaw 8y agoEvery service that Google operates is released with rolling updates and is fully preemptible, including the most highly latency-sensitive services and all of the stateful storage services. Google is an existence proof for the applicability of rolling updates to services of all kinds.
- packetslave 8y agoNot every service
- uluyol 8y agoFor state heavy applications, you want graceful restart even if you can tolerate hard failures.
- kevin_nisbet 8y agoI would say this approach is for situation where the model you're using isn't always applicable. In the case of this article, it's discussing teleport, a drop in replacement SSH server. For many users, this is their administrative connection to a fleet of servers (teleport can operate as a transparent bastion, as well as on the end host). So in our case, it's a desirable feature to not interrupt existing shell sessions, while replacing the software serving those connections. This could mean the admin's own connection to the server doesn't fail mid-upgrade, or that hundreds of internal employee's don't groan because whatever they were doing that used a shell just got dropped and they need to restart. Disclaimer: I work with the author of the original article, and contributed to the article.
- otterley 8y agoI don't take any issue with treating this as a special case scenario, although I think the modern trend is to treat the need for long-lived SSH sessions for managing services as a design smell. What I do take issue with is the suggestion that handling such special cases is "table stakes." A little humility goes a long way in bolstering credibility.
- otterley 8y agoI agree. The characterization of live reload capability as "table stakes" for availability is nonsense. High-availability designs never require singleton services. Being able to replace vulnerable singletons that require process and TCP socket gymnastics with multiple instances having good fault isolation properties is the real "table stakes" for modern highly-available service architectures.
- old-gregg 8y ago> The characterization of live reload capability as "table stakes" for availability is nonsense. ... > replace vulnerable singletons You clearly didn't read (or failed to comprehend) the article. Not every protocol on the internet is stateless and no, Teleport is not a singleton.
- otterley 8y agoThe fact that I disagree with your characterization or opinions doesn't mean I didn't read or comprehend the article. Stateful protocols on the Internet are few as a proportion of overall traffic, and it was a wise choice for HTTP to be stateless. This improves its ability to be highly available. Teleport and SSH themselves are absolutely singletons in the most common use case: you want to SSH to the designated server, and that server alone. The service is "ssh on instance X." No other server is an acceptable substitute. That fact makes the service a singleton. Listen, don't get me wrong -- I'm very pleased that you put this article out as it's very insightful. The technical information in it speaks for itself; it didn't need to be burdened with controversial opinion.
- old-gregg 8y agoPerhaps the article was not explaining in detail what this daemon does. No, it's not just an SSH endpoint on instance X. For large deployments it is an SSH gateway (better implementation of jumphost) to your entire data center and it's running on multiple hosts behind an LB. Yet, every instance of it may be supporting hundreds of long-lived SSH connections originating from multiple teams, some of them can be deploying something critically important. So, yes, this socket acrobatics is absolutely table stakes and is required (and has been requested by corporate customers of T) for robust infrastructure upgrades. Old-timers like to refer to this as "carrier-grade" way of doing things. You may dislike SSH itself, but hey - that's what our current UNIX tooling depends on, so we have to make it work, and work well. Besides, even stateless HTTP sometimes needs reliable long-lived connections! Full disclaimer: I'm one of early Teleport contributors.
- Thaxll 8y agoYou know that some people work on servers that are stateful? Not everyone use REST type of transport where a failed node is ok because the lb will route traffic to another place. Your solution doesn't work for things like game servers.
- weberc2 8y agoPresumably you just drain the servers running the old version as games end and migrate traffic to the servers running the new version?
- viraptor 8y agoEven stateful services like game servers could be organised to handle losing a node. It's harder to do and you need some state replication in place. (versioned state if you want to do upgrades that way) It may not be cost effective to do so. But the possibility is there. I worked on a plan to have live migration of multiparty phone calls between data centres in realtime. The tech is available. But your planning outcome may be: it's cheaper to make single node more stable and deal with the occasional failure. "Stateful" only means you need to care about the state being moved. Things like Java hotswap, Erlang live code reload, and your custom solutions can handle that.
- otterley 8y agoDo you run game servers? They absolutely have failover capability.
- latchkey 8y agoMy usecase is that I have 1500+ raspberry pi class machines running my daemon process (golang agent). I want to be able to upgrade the binary, without downtime.
- dnautics 8y agoNext time consider using Elixir Nerves. This capability is built in to the deployment framework.
- latchkey 8y agoThanks for the suggestion. It seems like Nerves needs a whole firmware to support it. These are machines where the most I can do is copy a binary onto them (crypto mining hardware). golang was the simplest solution for cross compiling I could find at the time.
- coldseattle 8y agoI can update Erlang programs _while they're running_ without any downtime. We do this often at our company. See http://erlang.org/doc/design_principles/release_handling.html http://erlang.org/doc/design_principles/release_handling.htm... for the basics. With no shared state, and no global variables, and no side effects, you can update functions so the next time they're called they're something else.
- purplezooey 8y agoErlang is badass in this way