4 ms·

I can understand the technical argument against the stateless containerized model. It's not neat. But what is the business argument? That's where this perspecti

I can understand the technical argument against the stateless containerized model. It's not neat. But what is the business argument? That's where this perspective falls down for me. If you're building a database that manages a disk directly, I can see the argument for graceful shutdown and handover. In that case you really need the program to be colocated with the data and if you're down then the data is unavailable. I've built that once and it was hell to get right. But probably 90+% of programs I've ever worked on do not achieve any business advantage from graceful restarts (as opposed to drain and restart i.e. the stateless model).

bostonsre 6y ago
Kuberenetes is a leaky abstraction that by default allows the deployment of sloppy unreliable apps if you don't know what you're doing. If something is broken and it just restarts automatically and no one is paying attention, it will do that in perpetuity. Ideally, you would see those issues and iterate to make your app more stable, but it makes it easy to just forget about it. The additional layer of complexity is also harder to understand easily for most devs that haven't spent the time to learn kubernetes and even with devs that do spend the time it can still make understanding how everything works hard. Letting tech debt like unstable apps grow into mud balls can reduce a team's velocity in the long term and in some cases the short term as well. Kubernetes is great for easily sharing and executing recipes, but it can make it tough for people that want to become chefs that are able to use reason and intuition to build and understand complex systems.
- bweitzman 6y ago
  On the flip side, why bother spending time optimizing a process if nobody notices when it fails? Aren't there more important problems to be solving in at scenario?
  - knl 6y ago
    Maybe people do notice, but it’s not easy to report back? Or they notice the system runs slowly? Or they just accept the fact that all systems (extrapolating from several banking, public service, and publishing sites) are crap, and need to live with it?
  - bostonsre 6y ago
    Tech debt filled mud balls where feature after feature is clumped on top of one another without worrying what came before it and what will come after it are not great to maintain. Sure if you don't plan on maintaining or building on the component and you don't care if it restarts, then let it restart forever. But if it's a component that you plan on adding features to for a long period of time and stability issues are ignored, those types of issues can compound over time and make it more difficult to reason about and maintain.
    - bweitzman 6y ago
      I agree, things that change frequently need more attention to make sure you're not spending more time than you need to make changes. Acknowledging that I'm changing the goalposts a bit, I would consider a lack of understanding about the lifecycle of an application that causes development delays an abstract form of "noticing when it fails"
sargun 6y ago
When you have two systems that need to talk to each other, and one system expects the other system to be reliable, it’s difficult to completely toss out all graceful shutdown and handover. Usually this logic is handled at the service discovery or load balancing layer with the deployment bits, so people don’t notice. Kubernetes does a really good job at hiding this. When it hides this, it also hides a couple other things with software — like failing containers, or apps that require churn or fail. I mean this is literally the reason why things like monit were built in the older days — things broke if you didn’t restart them when they broke. In kubernetes, it also is easy to accidentally to change the system and make that behavior change (at a global level), that results in stuff like your app not being restarted as often, which results in elevated latency, and subsequent failures.
- paledot 6y ago
  No system is 100% reliable, though. Some systems are close enough that we grit our teeth and pretend they are and wave away the inevitable outages that have nothing to do with our app and everything to do with an unreliable dependency. In a way, you're better off with a 99% reliable dependency than 99.9999%. If you expect it to be down some of the time, you'll plan for it in your architecture and (in theory) gracefully handle the failure whether it's due to an insufficiency warmed container, a network issue, a bad deploy, or whatever else comes along.
mathgladiator 6y ago
I'd argue the biggest reason for a graceful restart would be reliability and latency.
- asdfasgasdgasdg 6y ago
  But if you have the app behind a load balancer that can discover new tasks, rolling restarts for updates do not negatively affect those parameters.
  - toast0 6y ago
    This works pretty well for request oriented protocols with short response times. It's not great for protocols with significant handshaking and potential for long connections. Things like chat services, shoutcast streaming, calling, imap, really anything realtime or notification driven would do better without forcing clients to reconnect. Especially since many will reconnect multiple times, as they may reconnect to servers which are running old code while the drain and restarts happen. If you need near instant cutover (which is pronably rare), in a drain and restart scenario, you basically need double the capacity, where hotload doesn't need anything extra.
    - geofft 6y ago
      So you pay a public cloud provider for 2x capacity for a couple of minutes while you do the rollout, and in exchange you get a much simpler design because the individual servers are stateless. I think that's better for the business.
      - mononcqc 6y ago
        Some services may require gigabytes of state to be downloaded to work with acceptable latency on local decisions, and that state is replicated in a way that is constantly updated. These could include ML models, large routing tables, or anything of the kind. Connections could be expected to be active for many minutes at a time (not everything is a web server serving short HTTP requests that are easy to resume), and so on. Changing the instance means having to re-transfer all of the data and re-establish all of that state, on all of the nodes. You could easily see draining of connections take 15-20 minutes, and booting back and scaling up to be taking 15-20 minutes as well, if you can do it for _all_ the instances at once (which may not be a guarantee, and you could need to stagger things to be more cost-effective). You start with each deploy taking easily over an hour. If you deploy 2-3 times a day and that your peak times line up with these, you can more than double your operating cost just to deploy, and that can take more than 4 figures to count. Some of the systems we maintained (not those we necessarily live deployed to, but still required rolling restarts) required over 5,000 instances and could not just be doubled in size without running into limits in specific regions for given instance types. If a blue/green deploy takes a couple minutes, you're probably not having a workload where this is worth thinking about that much.
ori_b 6y ago
> But what is the business argument? Complexity costs money.
- geofft 6y ago
  Not sure I follow. The solutions proposed in the article - designing your software so it can be live-reloaded, having people who understand the system deeply etc. - all seem more complex and more expensive than just stuffing everything in Kubernetes and letting pods restart.
  - mononcqc 6y ago
    1. You always need people that understand the system. Not everyone and not all of it, but you want that overall, people are able to understand and figure out what is going on. You similarly hope to always have people who understand the customers, the environment you operate in, and so on. If you can afford to do your thing without having people understanding how it works, you probably can afford to do your thing without a an actually good solution (just scrape by, it's alright) and can go on with whatever. I'd advance the theory that picking an off-the-shelf popular solution is going to be beneficial in that case because you externalize the costs of maintaining your expertise and knowledge to the rest of the ecosystem or industry, to other companies, and just never develop that fiber within your organization. It is, however, worthwhile to develop it for more things than just your tech stack. Everything having to do with on-boarding and dealing with legacy code is improved, along with broader dynamics if you try to be a learning organization that develops expertise in its people. 2. You always design your system in accordance to its deploy system, regardless of whether you realize it or not. If you ship binary artifacts and signed packages to customers who install them on their own devices, you will have a different development practice than if you do CI/CD with a single pipeline that always goes forwards. This will also be somewhat different if you work with open-source components that require paying attention to version schemes rather than just pushing a hash in a container image. If you use feature flags to help merging but also to control deployment, adopt A/B testing, and all these practices, you're intimately adapting your development approaches to deployment mechanisms that are available to you. It's not an extra cost, it's a cost you already pay today.

bostonsre 6y ago

Kuberenetes is a leaky abstraction that by default allows the deployment of sloppy unreliable apps if you don't know what you're doing. If something is broken and it just restarts automatically and no one is paying attention, it will do that in perpetuity. Ideally, you would see those issues and iterate to make your app more stable, but it makes it easy to just forget about it. The additional layer of complexity is also harder to understand easily for most devs that haven't spent the time to learn kubernetes and even with devs that do spend the time it can still make understanding how everything works hard. Letting tech debt like unstable apps grow into mud balls can reduce a team's velocity in the long term and in some cases the short term as well. Kubernetes is great for easily sharing and executing recipes, but it can make it tough for people that want to become chefs that are able to use reason and intuition to build and understand complex systems.

bweitzman 6y ago
On the flip side, why bother spending time optimizing a process if nobody notices when it fails? Aren't there more important problems to be solving in at scenario?
- knl 6y ago
  Maybe people do notice, but it’s not easy to report back? Or they notice the system runs slowly? Or they just accept the fact that all systems (extrapolating from several banking, public service, and publishing sites) are crap, and need to live with it?
- bostonsre 6y ago
  Tech debt filled mud balls where feature after feature is clumped on top of one another without worrying what came before it and what will come after it are not great to maintain. Sure if you don't plan on maintaining or building on the component and you don't care if it restarts, then let it restart forever. But if it's a component that you plan on adding features to for a long period of time and stability issues are ignored, those types of issues can compound over time and make it more difficult to reason about and maintain.
  - bweitzman 6y ago
    I agree, things that change frequently need more attention to make sure you're not spending more time than you need to make changes. Acknowledging that I'm changing the goalposts a bit, I would consider a lack of understanding about the lifecycle of an application that causes development delays an abstract form of "noticing when it fails"

sargun 6y ago

When you have two systems that need to talk to each other, and one system expects the other system to be reliable, it’s difficult to completely toss out all graceful shutdown and handover. Usually this logic is handled at the service discovery or load balancing layer with the deployment bits, so people don’t notice. Kubernetes does a really good job at hiding this. When it hides this, it also hides a couple other things with software — like failing containers, or apps that require churn or fail. I mean this is literally the reason why things like monit were built in the older days — things broke if you didn’t restart them when they broke. In kubernetes, it also is easy to accidentally to change the system and make that behavior change (at a global level), that results in stuff like your app not being restarted as often, which results in elevated latency, and subsequent failures.

paledot 6y ago
No system is 100% reliable, though. Some systems are close enough that we grit our teeth and pretend they are and wave away the inevitable outages that have nothing to do with our app and everything to do with an unreliable dependency. In a way, you're better off with a 99% reliable dependency than 99.9999%. If you expect it to be down some of the time, you'll plan for it in your architecture and (in theory) gracefully handle the failure whether it's due to an insufficiency warmed container, a network issue, a bad deploy, or whatever else comes along.

mathgladiator 6y ago

I'd argue the biggest reason for a graceful restart would be reliability and latency.

asdfasgasdgasdg 6y ago
But if you have the app behind a load balancer that can discover new tasks, rolling restarts for updates do not negatively affect those parameters.
- toast0 6y ago
  This works pretty well for request oriented protocols with short response times. It's not great for protocols with significant handshaking and potential for long connections. Things like chat services, shoutcast streaming, calling, imap, really anything realtime or notification driven would do better without forcing clients to reconnect. Especially since many will reconnect multiple times, as they may reconnect to servers which are running old code while the drain and restarts happen. If you need near instant cutover (which is pronably rare), in a drain and restart scenario, you basically need double the capacity, where hotload doesn't need anything extra.
  - geofft 6y ago
    So you pay a public cloud provider for 2x capacity for a couple of minutes while you do the rollout, and in exchange you get a much simpler design because the individual servers are stateless. I think that's better for the business.
    - mononcqc 6y ago
      Some services may require gigabytes of state to be downloaded to work with acceptable latency on local decisions, and that state is replicated in a way that is constantly updated. These could include ML models, large routing tables, or anything of the kind. Connections could be expected to be active for many minutes at a time (not everything is a web server serving short HTTP requests that are easy to resume), and so on. Changing the instance means having to re-transfer all of the data and re-establish all of that state, on all of the nodes. You could easily see draining of connections take 15-20 minutes, and booting back and scaling up to be taking 15-20 minutes as well, if you can do it for _all_ the instances at once (which may not be a guarantee, and you could need to stagger things to be more cost-effective). You start with each deploy taking easily over an hour. If you deploy 2-3 times a day and that your peak times line up with these, you can more than double your operating cost just to deploy, and that can take more than 4 figures to count. Some of the systems we maintained (not those we necessarily live deployed to, but still required rolling restarts) required over 5,000 instances and could not just be doubled in size without running into limits in specific regions for given instance types. If a blue/green deploy takes a couple minutes, you're probably not having a workload where this is worth thinking about that much.

ori_b 6y ago

> But what is the business argument? Complexity costs money.

geofft 6y ago
Not sure I follow. The solutions proposed in the article - designing your software so it can be live-reloaded, having people who understand the system deeply etc. - all seem more complex and more expensive than just stuffing everything in Kubernetes and letting pods restart.
- mononcqc 6y ago
  1. You always need people that understand the system. Not everyone and not all of it, but you want that overall, people are able to understand and figure out what is going on. You similarly hope to always have people who understand the customers, the environment you operate in, and so on. If you can afford to do your thing without having people understanding how it works, you probably can afford to do your thing without a an actually good solution (just scrape by, it's alright) and can go on with whatever. I'd advance the theory that picking an off-the-shelf popular solution is going to be beneficial in that case because you externalize the costs of maintaining your expertise and knowledge to the rest of the ecosystem or industry, to other companies, and just never develop that fiber within your organization. It is, however, worthwhile to develop it for more things than just your tech stack. Everything having to do with on-boarding and dealing with legacy code is improved, along with broader dynamics if you try to be a learning organization that develops expertise in its people. 2. You always design your system in accordance to its deploy system, regardless of whether you realize it or not. If you ship binary artifacts and signed packages to customers who install them on their own devices, you will have a different development practice than if you do CI/CD with a single pipeline that always goes forwards. This will also be somewhat different if you work with open-source components that require paying attention to version schemes rather than just pushing a hash in a container image. If you use feature flags to help merging but also to control deployment, adopt A/B testing, and all these practices, you're intimately adapting your development approaches to deployment mechanisms that are available to you. It's not an extra cost, it's a cost you already pay today.