3 ms·

> and automation of their own "just scp jarfile to server". And if the solution of any team fails and the right person is unavailable, you're suddenly bleeding

by ttymck 3y ago

> and automation of their own "just scp jarfile to server". And if the solution of any team fails and the right person is unavailable, you're suddenly bleeding money because no one knows how to get it back working. I think that's the heart of the issue. At a certain point, people other than the application developers bear responsibility for the operation of the application. The situation is not _fundamentally_ different: when it's 1-5 devs, everyone is responsible for the uptime. If one of those devs doesn't know how to deploy, you're still in the same boat as a larger team when something breaks. But the situation becomes _politically_ different: management decides there needs to be an additional layer of responsibility: the platform team. They are the backstop. And so, instead of the platform team agreeing to bear responsibility for N different bespoke systems, they prescribe some common API (kubernetes, etc). Of course they COULD prescribe "scp to a server and restart systemd unit" but then that's just not flexible enough. Some teams want different restart strategies, and it spirals out of control. At least kubernetes supports all that flexibility in a well-documented, battle-tested package. So, the parties to blame are two, with different degrees of culpability: 1. Management decides there needs to be more shared ownership (otherwise you bleed money, which is worse when you're bigger and make more money) 2. Platform team agrees to support kubernetes, because OF COURSE our developers need all the bells and whistles, and elastic beanstalk isn't good enough because what if we need* Feature X 3 years from now?

tetha 3y ago
> 2. Platform team agrees to support kubernetes, because OF COURSE our developers need all the bells and whistles, and elastic beanstalk isn't good enough because what if we need* Feature X 3 years from now? This is valid criticism. It's leading to an interesting rift or tension, at least at my current workplace: Infrastructure like this enables teams to be agile, but the infrastructure itself rapidly stops being agile and you need to have a 1-2 year planning horizon. A good platform enables you to deploy several times a day (if you want, you can always choose to go slower). But changing a core infrastructure piece if dozens of teams and dozens of dozens of service deployments depends on it... a year is suddenly a rather short time. In fact, I'd say you're not replacing such a core piece, you're rather adding a new deployment infrastructure and now you have 2 for a long time. As such, going for all the bells and whistles you might need 2-3 years down the road can be a very valid point - as long as you have evidence and reason to believe the complexity will be utilized. Otherwise, I'm totally a friend of having some Jenkins Job or Github action to rsync jar-files to from Artifactory to servers. Standardize the App-Server and the 1-2 java versions to have, have a proper systemd unit for it and have at it. We've run the company software like that for years. It just wouldn't work at the current scale.
avbanks 3y ago
> 2. Platform team agrees to support kubernetes, because OF COURSE our developers need all the bells and whistles, and elastic beanstalk isn't good enough because what if we need* Feature X 3 years from now? Whats so difficult about k8s? Kubernetes and especially the managed versions allow you to be a simple or complex as you want to be. What's the difficult part?
- ttymck 3y ago
  I don't think most would agree that k8s makes deploying a "simple" web service as "simple" as it could be, because it wasn't designed to. Pod spec needs to be wrapped in a deployment which needs to be fronted by a service which may or may not need an ingress. And for each of those entities, you need to know which of their dozens of configuration variables are relevant and what their values need to be. K8s is easy if you know how to use it, and use it correctly. It's extremely well-designed. It is not easy to learn how to use it correctly.
re-thc 3y ago
> Of course they COULD prescribe "scp to a server and restart systemd unit" but then that's just not flexible enough But what about if the server disappears or the disk failed and it won't start? It doesn't capture the entire process. I remember at some point we needed to deploy to a new region for a customer and yes it was "too difficult" because some of those manual steps were forgotten and lost. In became a x month long effort to work out all the small details.

by ttymck 3y ago

tetha 3y ago

> 2. Platform team agrees to support kubernetes, because OF COURSE our developers need all the bells and whistles, and elastic beanstalk isn't good enough because what if we need* Feature X 3 years from now? This is valid criticism. It's leading to an interesting rift or tension, at least at my current workplace: Infrastructure like this enables teams to be agile, but the infrastructure itself rapidly stops being agile and you need to have a 1-2 year planning horizon. A good platform enables you to deploy several times a day (if you want, you can always choose to go slower). But changing a core infrastructure piece if dozens of teams and dozens of dozens of service deployments depends on it... a year is suddenly a rather short time. In fact, I'd say you're not replacing such a core piece, you're rather adding a new deployment infrastructure and now you have 2 for a long time. As such, going for all the bells and whistles you might need 2-3 years down the road can be a very valid point - as long as you have evidence and reason to believe the complexity will be utilized. Otherwise, I'm totally a friend of having some Jenkins Job or Github action to rsync jar-files to from Artifactory to servers. Standardize the App-Server and the 1-2 java versions to have, have a proper systemd unit for it and have at it. We've run the company software like that for years. It just wouldn't work at the current scale.

avbanks 3y ago

> 2. Platform team agrees to support kubernetes, because OF COURSE our developers need all the bells and whistles, and elastic beanstalk isn't good enough because what if we need* Feature X 3 years from now? Whats so difficult about k8s? Kubernetes and especially the managed versions allow you to be a simple or complex as you want to be. What's the difficult part?

ttymck 3y ago
I don't think most would agree that k8s makes deploying a "simple" web service as "simple" as it could be, because it wasn't designed to. Pod spec needs to be wrapped in a deployment which needs to be fronted by a service which may or may not need an ingress. And for each of those entities, you need to know which of their dozens of configuration variables are relevant and what their values need to be. K8s is easy if you know how to use it, and use it correctly. It's extremely well-designed. It is not easy to learn how to use it correctly.

re-thc 3y ago

> Of course they COULD prescribe "scp to a server and restart systemd unit" but then that's just not flexible enough But what about if the server disappears or the disk failed and it won't start? It doesn't capture the entire process. I remember at some point we needed to deploy to a new region for a customer and yes it was "too difficult" because some of those manual steps were forgotten and lost. In became a x month long effort to work out all the small details.