5 ms·

Well I think it's neat. The bit I find most provoking is the "if you already have Kubernetes..." premise. I find myself having a hard time not wanting to shove

Well I think it's neat. The bit I find most provoking is the "if you already have Kubernetes..." premise. I find myself having a hard time not wanting to shove everything into the Kubernetes framework simply to avoid having to document what solutions I've chosen. `kubectl get all` gives me an overview of a project in a way that is impossible if every single project uses a different or bespoke management system. "simple/complex" is not the right paradigm. The real SRE controversy is "unique/standard". Yes, the standard approach is more complex. But it is better _in practice_ to have a single approach, rather than many individually-simpler, but in-aggregate-more-complex approaches. Kubernetes is never the perfect solution to an engineering problem, but it is almost always the most pragmatic solution to a business problem for a business with many such problems.

Spivak 2y ago
Is the mountain of k8s code on top of your cloud of choice not strictly more complex than the cloud services it's orchestrating? Like I think k8s is good software but to hear that you're not choosing it because it's good engineering is surprising. To me that's the only reason you choose it, because you want or need the control-loop based infrastructure management that you can't get on say AWS alone.
- marcinzm 2y ago
  There’s always a mountain of code on top of the cloud provider.
  - andrewstuart2 2y ago
    Furthermore, there's always a mountain of code aggregating heterogeneous systems into a uniform interface. Look at the linux kernel, or LLVM, or pretty much any other (successful) project that provides abstractions to many different backends.
  - Spivak 2y ago
    Yes but that code isn't typically running in production the way k8s is. Having a lot of terraform is just a fancy method of statically provisioning infrastructure. The code is gone once it applies. Bringing k8s into the mix is a whole other application layer that's providing services to your code. Things like sgs for pods, cni plugins, coredns, is code that wouldn't be there in a non k8s world. Your problems now include k8s networking as well as VPC networking.
    - marcinzm 2y ago
      > The code is gone once it applies Until you need to update a change or upgrade something or patch a security issue. The most dangerous code is code that doesn't run often. Like backup restores. Ticking time bombs unless you're really careful. > Bringing k8s into the mix is a whole other application layer that's providing services to your code. Things like sgs for pods, cni plugins, coredns, is code that wouldn't be there in a non k8s world. The same applies to everything running around your tech stack that you didn't write. That includes the language runtime, the compiler, the operating system, all OS libraries you use, and the libraries your app imports. The ship has sailed and K8S isn't much of an addition. I haven't hit a CNI bug in probably 5 years and that was only on Amazon's horrible CNI code. I've hit a ton of bugs in my software stack. And the libraries I depend on. And my deployment stack. And my cross-service networking stack at the application layer.
- zipmapfoldright 2y ago
  That's the same with.. the Linux kernel? I'd wager that most services people write have less complexity than the the kernel it runs on. We choose to do that not because we need its full feature set, but because it gives us useful abstractions.
  - cjk 2y ago
    I don’t know that that’s the right analogy. End users don’t have to write a mountain of configuration in order to effectively use the Linux kernel. If k8s were as complex as it was, but didn’t force that complexity onto the user, I’d agree.
    - cortesoft 2y ago
      You can deploy quite a bit to k8s with just a few lines of helm installs, and it will work with very little configuration.
    - Spivak 2y ago
      Exactly, I think I could have stated it better. Using k8s is strictly more complex from the perspective of the operator, and the mountain I'm referring to isn't actually from k8s itself but from all the 3rd party stuff you have to layer on to actually make it useful-- things like cluster-autoscaler, vpa, kube-prometheus, aws-load-balancer-controller, misc. operators, that fall on my side of the wall when it comes to fixing it when it's broken. I can understand that it doesn't come off that way but I actually like k8s and the surrounding ecosystem. But deploying k8s makes your infrastructure as complicated as k8s itself, which can be a massive win because it works in both directions.
      - skydhash 2y ago
        If it can’t be monolithic like *BSD, at least offer distributions like the Linux Sphere. Like a ready to go LAMP stack. Or a collection of services (Database, Storage, Logs,…)
        2y ago
akdor1154 2y ago
Yeah k8s is great. It gives you an infinite rope generator to let you hang yourself with ever increasing complexity, but with a bit of restraint you can orchestrate pretty much anything in a simple or at least standard way. I'd take a stack of yaml over a stack of bespoke aws managed service scripts any day.
- cqqxo4zV46cp 2y ago
  Not a day goes by where there isn’t some greybeard having a huge sulk about how nobody wants to use his collection of esoteric bash scripts that nobody else will ever understand, but HE does.
  - jauntywundrkind 2y ago
    Erulabs top comment nailed it so hard. There's been such a vocal force in comments for so long, of revulsion to "complexity", believing "simpler" is better. > Yes, the standard approach is more complex. But it is better _in practice_ to have a single approach, rather than many individually-simpler, but in-aggregate-more-complex approaches. There still remains such a strong reflexive "you don't need Kubernetes for that" response that shows up. But we really are seeing a lot more people who get the value of there being a standard way of doing things, of having common platform & expectations underneath. The cost of starting up may be significant, but it results in a more legible understandable result, and we can keep reusing that base of capabilities again and again and again. Its such an exciting change. Open source has been around for a long time, but running software has been the wild West, until very very recently. We now have this awesome, scalable, extensible way to manage containers, volumes, load balancers, databases, and whatever else have we. Love to see it. Folks aren't wrong about it having complexity, but so much of the complexity already existed, was just not clearly visible or well harnesses (different example is like how systemd makes it so easy to restrict services, give them temporary/unprivileged users). We are closer to a real shared understanding, to having so many particular decisions offloaded & so much more we can work together atop. Love it so much.
    - quectophoton 2y ago
paulddraper 2y ago
The bit I find the most provoking is calling k8 "standard." And not a collection of strung-together controllers.
politelemon 2y ago
> But it is better _in practice_ to have a single approach, rather than many individually-simpler, but in-aggregate-more-complex approaches. Very much depends on the point of view. It's great from an SRE point of view but not necessarily application/developers, who are being constrained to a platform's idiosyncracies and expressions of platform egos. The individually simpler solutions are only complex from a high level, ivory tower, or middle management perspective, not from the perspective of people who have to use and manage the application itself.
- ownagefool 2y ago
  I think just running a process and exposing a port is fine, but the second you get into running a bunch of services together, or caring about environments, the k8s abstraction is simpler. In the last 6 months my job has been to get production ready vault instances on azure. There's a bunch of complex, unreliable and not very fun APIs here. Much like AWS, there isn't really a StatefulSet ( PetSet ) abstraction. To you need to write a bunch of bespoke logic where you figure which IP addresses, names, IDs, and Disks you need to attach to a new VM. Whilst iterating, the azure APIs are eventually consistent that cause all sort of niggly problems. Resources that are created don't get added to TF when there's a failure. I create a new vault on a PR and it takes 20 minutes to deploy. The problem took a couple of months and I created a bunch of code. On k8s I can just deploy the helm chart in 2 minutes. The abstractions are cleaner, it's more reliable, and way more fun. At the end of the azure project, the team agreed to do a AKS poc, where we gave the task to the junior of the team, and the entire thing was done in a week. If you're not doing this type of work, maybe you don't need k8s. But if you're not doing ephemeral test environments, do you actually not see them as a positive, or is it an effort thing? Because it takes me no real effort.
thesnide 2y ago
If you only have a hammer, everything looks like a nail. Then, it mostly works, so management makes "economy of scales" and removes all screw drivers, since they need more skills anyway. Now, you are only using nails. Until someone else rediscovers screws and starts another hype cycle.
- TeMPOraL 2y ago
  Except in manufacturing, this actually works. Economies of scale are real and trump pretty much every other consideration, and they're what enable pretty much everything we have around us that isn't biological.

Spivak 2y ago

Is the mountain of k8s code on top of your cloud of choice not strictly more complex than the cloud services it's orchestrating? Like I think k8s is good software but to hear that you're not choosing it because it's good engineering is surprising. To me that's the only reason you choose it, because you want or need the control-loop based infrastructure management that you can't get on say AWS alone.

marcinzm 2y ago
There’s always a mountain of code on top of the cloud provider.
- andrewstuart2 2y ago
  Furthermore, there's always a mountain of code aggregating heterogeneous systems into a uniform interface. Look at the linux kernel, or LLVM, or pretty much any other (successful) project that provides abstractions to many different backends.
- Spivak 2y ago
  Yes but that code isn't typically running in production the way k8s is. Having a lot of terraform is just a fancy method of statically provisioning infrastructure. The code is gone once it applies. Bringing k8s into the mix is a whole other application layer that's providing services to your code. Things like sgs for pods, cni plugins, coredns, is code that wouldn't be there in a non k8s world. Your problems now include k8s networking as well as VPC networking.
  - marcinzm 2y ago
    > The code is gone once it applies Until you need to update a change or upgrade something or patch a security issue. The most dangerous code is code that doesn't run often. Like backup restores. Ticking time bombs unless you're really careful. > Bringing k8s into the mix is a whole other application layer that's providing services to your code. Things like sgs for pods, cni plugins, coredns, is code that wouldn't be there in a non k8s world. The same applies to everything running around your tech stack that you didn't write. That includes the language runtime, the compiler, the operating system, all OS libraries you use, and the libraries your app imports. The ship has sailed and K8S isn't much of an addition. I haven't hit a CNI bug in probably 5 years and that was only on Amazon's horrible CNI code. I've hit a ton of bugs in my software stack. And the libraries I depend on. And my deployment stack. And my cross-service networking stack at the application layer.
zipmapfoldright 2y ago
That's the same with.. the Linux kernel? I'd wager that most services people write have less complexity than the the kernel it runs on. We choose to do that not because we need its full feature set, but because it gives us useful abstractions.
- cjk 2y ago
  I don’t know that that’s the right analogy. End users don’t have to write a mountain of configuration in order to effectively use the Linux kernel. If k8s were as complex as it was, but didn’t force that complexity onto the user, I’d agree.
  - cortesoft 2y ago
    You can deploy quite a bit to k8s with just a few lines of helm installs, and it will work with very little configuration.
  - Spivak 2y ago
    Exactly, I think I could have stated it better. Using k8s is strictly more complex from the perspective of the operator, and the mountain I'm referring to isn't actually from k8s itself but from all the 3rd party stuff you have to layer on to actually make it useful-- things like cluster-autoscaler, vpa, kube-prometheus, aws-load-balancer-controller, misc. operators, that fall on my side of the wall when it comes to fixing it when it's broken. I can understand that it doesn't come off that way but I actually like k8s and the surrounding ecosystem. But deploying k8s makes your infrastructure as complicated as k8s itself, which can be a massive win because it works in both directions.
    - skydhash 2y ago
      If it can’t be monolithic like *BSD, at least offer distributions like the Linux Sphere. Like a ready to go LAMP stack. Or a collection of services (Database, Storage, Logs,…)
      - 2y ago

akdor1154 2y ago

Yeah k8s is great. It gives you an infinite rope generator to let you hang yourself with ever increasing complexity, but with a bit of restraint you can orchestrate pretty much anything in a simple or at least standard way. I'd take a stack of yaml over a stack of bespoke aws managed service scripts any day.

cqqxo4zV46cp 2y ago
Not a day goes by where there isn’t some greybeard having a huge sulk about how nobody wants to use his collection of esoteric bash scripts that nobody else will ever understand, but HE does.
- jauntywundrkind 2y ago
  Erulabs top comment nailed it so hard. There's been such a vocal force in comments for so long, of revulsion to "complexity", believing "simpler" is better. > Yes, the standard approach is more complex. But it is better _in practice_ to have a single approach, rather than many individually-simpler, but in-aggregate-more-complex approaches. There still remains such a strong reflexive "you don't need Kubernetes for that" response that shows up. But we really are seeing a lot more people who get the value of there being a standard way of doing things, of having common platform & expectations underneath. The cost of starting up may be significant, but it results in a more legible understandable result, and we can keep reusing that base of capabilities again and again and again. Its such an exciting change. Open source has been around for a long time, but running software has been the wild West, until very very recently. We now have this awesome, scalable, extensible way to manage containers, volumes, load balancers, databases, and whatever else have we. Love to see it. Folks aren't wrong about it having complexity, but so much of the complexity already existed, was just not clearly visible or well harnesses (different example is like how systemd makes it so easy to restrict services, give them temporary/unprivileged users). We are closer to a real shared understanding, to having so many particular decisions offloaded & so much more we can work together atop. Love it so much.
  - quectophoton 2y ago

paulddraper 2y ago

The bit I find the most provoking is calling k8 "standard." And not a collection of strung-together controllers.

politelemon 2y ago

> But it is better _in practice_ to have a single approach, rather than many individually-simpler, but in-aggregate-more-complex approaches. Very much depends on the point of view. It's great from an SRE point of view but not necessarily application/developers, who are being constrained to a platform's idiosyncracies and expressions of platform egos. The individually simpler solutions are only complex from a high level, ivory tower, or middle management perspective, not from the perspective of people who have to use and manage the application itself.

ownagefool 2y ago
I think just running a process and exposing a port is fine, but the second you get into running a bunch of services together, or caring about environments, the k8s abstraction is simpler. In the last 6 months my job has been to get production ready vault instances on azure. There's a bunch of complex, unreliable and not very fun APIs here. Much like AWS, there isn't really a StatefulSet ( PetSet ) abstraction. To you need to write a bunch of bespoke logic where you figure which IP addresses, names, IDs, and Disks you need to attach to a new VM. Whilst iterating, the azure APIs are eventually consistent that cause all sort of niggly problems. Resources that are created don't get added to TF when there's a failure. I create a new vault on a PR and it takes 20 minutes to deploy. The problem took a couple of months and I created a bunch of code. On k8s I can just deploy the helm chart in 2 minutes. The abstractions are cleaner, it's more reliable, and way more fun. At the end of the azure project, the team agreed to do a AKS poc, where we gave the task to the junior of the team, and the entire thing was done in a week. If you're not doing this type of work, maybe you don't need k8s. But if you're not doing ephemeral test environments, do you actually not see them as a positive, or is it an effort thing? Because it takes me no real effort.

thesnide 2y ago

If you only have a hammer, everything looks like a nail. Then, it mostly works, so management makes "economy of scales" and removes all screw drivers, since they need more skills anyway. Now, you are only using nails. Until someone else rediscovers screws and starts another hype cycle.

TeMPOraL 2y ago
Except in manufacturing, this actually works. Economies of scale are real and trump pretty much every other consideration, and they're what enable pretty much everything we have around us that isn't biological.