7 ms·

I don't get the hate for Kubernetes in this thread. TFA is from Figma. You can talk all day long about how early startups just don't need the kind of management

I don't get the hate for Kubernetes in this thread. TFA is from Figma. You can talk all day long about how early startups just don't need the kind of management benefits that Kubernetes offers, but the article isn't written by someone working for a startup, it's written by a company that nearly got sold to Adobe for $20 billion. Y'all really don't think a company like Figma stands to benefit from the flexibility that Kubernetes offers?

BobbyJo 2y ago
Kubernetes isn't even that complicated, and first party support from cloud providers often means you're doing something in K8s inleu of doing it in a cloud specific way (like ingress vs cloud specific load balancer setups). At a certain scale, K8s is the simple option. I think much of the hate on HN comes from the "ruby on rails is all you need" crowd.
- JohnMakin 2y ago
  > I think much of the hate on HN comes from the "ruby on rails is all you need" crowd. Maybe - people seem really gungho about serverless solutions here too
  - dexwiz 2y ago
    The hype for serverless cooled after that article about Prime Video dropping lambda. No one wants a product that a company won’t dogfood. I realize Amazon probably uses lambda elsewhere, but it was still a bad look.
    - LordKeren 2y ago
      I think it was much more about one specific use case of lambda that was a bad fit for the prime video team’s need and not a rejection of lambda/serverless. TBH, it kind of reflected more poorly on the team than lambda as a product
    - cmckn 2y ago
      > Amazon probably uses lambda elsewhere Yes, you could say that. :)
    - JohnMakin 2y ago
      not probably, their lambda service powers much of their control plane.
- eutropia 2y ago
  I guess the ones who quietly ship dozens of rails apps on k8s are too busy getting shit done to stop and share their boring opinions about pragmatically choosing the right tool for the job :)
  - BobbyJo 2y ago
    "But you can run your rails app on a single host with embedded SQLite, K8s is unnecessary."
    - ffsm8 2y ago
      And there is truth to that. Most deployments are at that level, and it absolutely is way more performant then the alternative. it just comes with several tradeoffs... But these tradeoffs are usually worth it for deployments with <10k concurrent users. Which Figma certainly isn't. Though you probably could still do it, but that's likely more trouble then it's worth (The 10k is just an arbitrary number I made up, there is no magic number which makes this approach unviable, it all depends on how the users interact with the platform/how often and where the data is inserted)
    - threeseed 2y ago
      Always said by people who haven't spent much time in the cloud. Because single hosts will always go down. Just a question of when.
      - BossingAround 2y ago
        I love k8s, but bringing back up a single app that crashed is a very different problem from "our k8s is down" - because if you think your k8s won't go down, you're in for a surprise. You can view a single k8s also as a single host, which will go down at some point (e.g. a botched upgrade, cloud network partition, or something similar). While much less frequent, also much more difficult to get out of. Of course, if you have a multi-cloud setup with automatic (and periodically tested!) app migration across clouds, well then... Perhaps that's the answer nowadays.. :)
- bamboozled 2y ago
  Agreed, we're a small team and we benefit greatly from managed k8s (EKS). I have to say the whole ecosystem just continues to improve as far as I can tell and the developer satisfaction is really high with it. Personally I think k8s is where it's at now. The innovation and open source contributions are immense. I'm glad we made the switch. I understand the frustrations of the past, but I think it was much harder to use 4+ years ago. Now, I don't see how anyone could mess it up so hard.
- MrDarcy 2y ago
  > Kubernetes isn't even that complicated I’ve been struggling to square this sentiment as well. I spend all day in AWS and k8s and k8s is at least an order of magnitude simpler than AWS. What are all the people who think operating k8s is too complicated operating on? Surely not AWS…
  - tbrownaw 2y ago
    The thing you already know tends to be less complicated than the thing you don't know.
  - rco8786 2y ago
    I think "k8s is complicated" and "AWS is even more complicated" can both be true. Doing anything in AWS is like pulling teeth.
  - brainzap 2y ago
    The sum is complex, specially with the custom operators.
- chrischen 2y ago
  There are also a lot of cog-in-the-machine engineers here that totally do not get the bigger picture or the vantage point from another department.
- deleted 2y ago
  [deleted]
- dorianmariefr 2y ago
  ruby on rails is all you need
logifail 2y ago
> it's written by a company that nearly got sold to Adobe for $20 billion (Apologies if this is a dumb question) but isn't Figma big enough to want to do any of their stuff on their own hardware yet? Why would they still be paying AWS rates? Or is it the case that a high-profile blog post about K8S and being provider-agnostic gets you sufficient discount on your AWS bill to still be value-for-money?
- jeffbee 2y ago
  There are a lot of ex-Dropbox people at Figma who might have learned firsthand that bringing your stuff on-prem under a theory of saving money is an intensely stupid idea.
  - logifail 2y ago
    > There are a lot of ex-Dropbox people at Figma who might have learned firsthand that bringing your stuff on-prem under a theory of saving money is an intensely stupid idea Well, that's one hypothesis. Another is that "Every maturing company with predictable products must be exploring ways to move workloads out of the cloud. AWS took your margin and isn't giving it back." ( https://news.ycombinator.com/item?id=35235775 https://news.ycombinator.com/item?id=35235775 )
- ozim 2y ago
  They are preparing for next blog post in a year - „how we cut costs by xx% by moving to our own servers”.
- hyperbolablabla 2y ago
  I work for a company making ~$9B in annual revenue and we use AWS for everything. I think a big aspect of that is just developer buy-in, as well as reliability guarantees, and being able to blame Amazon when things do go down
  - st3fan 2y ago
    Also, you don't have to worry about half of your stack? The shared responsibility model really works.
cwiggs 2y ago
k8s is complex, if you don't need the following you probably shouldn't use it: * Service discovery * Auto bin packing * Load Balancing * Automated rollouts and rollbacks * Horizonal scaling * Probably more I forgot about You also have secret and config management built in. If you use k8s you also have the added benefit of making it easier to move your workloads between clouds and bare metal. As long as you have a k8s cluster you can mostly move your app there. Problem is most companies I've worked at in the past 10 years needed multiple of the features above, and they decided to roll their own solution with Ansible/Chef, Terraform, ASGs, Packer, custom scripts, custom apps, etc. The solutions have always been worse than what k8s provides, and it's a bespoke tool that you can't hire for. For what k8s provides, it isn't complex, and it's all documented very well, AND it's extensible so you can build your own apps on top of it. I think there are more SWE on HN than Infra/Platform/Devops/buzzword engineers. As a result there are a lot of people who don't have a lot of experience managing infra and think that spinning up their docker container on a VM is the same as putting an app in k8s. That's my opinion on why k8s gets so much hate on HN.
- Osiris 2y ago
  Those all seem important to even moderately sized products.
  - worldsayshi 2y ago
    As long as your requirements are simple the config doesn't need to be complex either. Not much more than docker-compose. But once you start using k8s you probably tend to scope creep and find a lot of shiny things to add to your set up.
  - doctorpangloss 2y ago
    Some ways to tell if someone is a great developer are easy. JetBrains IDE? Ample storage space? Solving problems with the CLI? Consistently formatted code using the language's packaging ecosystem? No comments that look like this: # A verbose comment that starts capitalized, followed by a single line of code, cuing you that it was written by a ChatBot. Some ways to tell if someone is a great developer is hard. You can't tell if someone is a brilliant shipper of features, choosing exactly the right concerns to worry about at the moment, like doing more website authoring and less devops, with a grand plan for how to make everything cohere later; or, if the guy just doesn't know what the fuck he is doing. Kubernetes adoption is one of those, hard ones. It isn't a strong, bright signal like using PEP 8 and having a `pyproject.toml` with dependencies declared. So it may be obvious to you, "People adopt Kubernetes over ad-hoc decoupled solutions like Terraform because it has, in a Darwinian way, found the smallest set of easily surmountable concerns that should apply to most good applications." But most people just see, "Ahh! Why can't I just write the method bodies for Python function signatures someone else wrote for me, just like they did in CS50!!!"
tracerbulletx 2y ago
Even on a small project it's actually better imo than tying everything to a platform like netlify or vercel. I have this little notepad app that I deploy to a two node cluster in a github action and its an excellent workflow. The k8s to get everything deployed, provision tls and everything on commit is like 150 lines of mostly boilerplate yaml, I could pretty easily make it support branch previews or whatever too. https://github.com/SteveCastle/modelpad https://github.com/SteveCastle/modelpad
saaspirant 2y ago
Unrelated: What does _TFA_ mean here? Google and GPT didn't help (even with context)
- solatic 2y ago
  The Featured Article. (or, if you read it in a frustrated voice, The F**ing Article.)
  - Xixi 2y ago
    Related acronyms: RTFA (Read The F**ing Article) and RTFM (Read The F**ing Manual). The latter was a very common answer when struggling with Linux in the early 2000s...
osigurdson 2y ago
Kubernetes is the most amazing piece of software engineering that I have ever seen. Most of the hate is merely being directed at the learning curve.
- otabdeveloper4 2y ago
  No, k8s is shit. It's only useful for the degenerate "run lots of instances of webapp servers running slow interpreted languages" use case. Trying to do anything else in it is madness. And for the "webapp servers" use case they could have built something a thousand times simpler and more robust. Serving templated html ain't rocket science. (At least compared to e.g. running an OLAP database cluster.)
  - shakiXBT 2y ago
    Could you please bless us with another way to easily orchestrate thousands of containers in a cloud vendor agnostic fashion? Thanks! Oh, and just in case your first rebuttal is "having thousands of containers means you've already failed" - not everyone works in a mom n pop shop
    - otabdeveloper4 2y ago
      Read my post again. Just because k8s is the only game in town doesn't mean it is technically any good. As a technology it is a total shitshow. Luckily, the problem it solves ("orchestrating" slow webapp containers) is not a problem most professionals care about. Feature creep of k8s into domains it is utterly unsuitable for because devops wants a pay raise is a different issue.
      - shakiXBT 2y ago
        > Orchestrating containers is not a problem most professionals care about I truly wish you were right, but maybe it's good job security for us professionals!
      - osigurdson 2y ago
globular-toast 2y ago
I don't get the hate even if you are a small company. K8s has massively simplified our deployments. It used to be each app had it's own completely different deployment process. Could have been a shell script that SSHed to some VM. Who managed said VM? Did it do its own TLS termination? Fuck knows. Maybe they used Ansible. Great, but that's another tool to learn and do I really need to set up bare metal hosts from scratch for every service. No, so there's probably some other Ansible config somewhere that sets them up. And the secrets are stored where? Etc etc. People who say "you don't need k8s" never say what you do need. K8s gives us a uniform interface that works for everything. We just have a few YAML files for each app and it just works. We can just chuck new things on there and don't even have to think about networking. Just add a Service and it's magically available with a name to everything in the cluster. I know how to do this stuff from scratch and I do not want to be doing it every single time.
- shakiXBT 2y ago
  if you don't need High Availability you can even deploy to a single node k3s cluster. It's still miles better than having to setup systemd services, an Apache/NGINX proxy, etc. etc.
  - globular-toast 2y ago
    Yep, and you can get far with k3s "fake" load balancer (ServiceLB). Then when you need a more "real" cluster basically all the concepts are the same you just move to a new cluster.
roydivision 2y ago
One thing I learned when I started learning Kubernetes is that it is two disciplines that overlap, but are distinct none the less: - Platform build and management - App build and management Getting a stable K8s cluster up and running is quite different to building and running apps on it. Obviously there is overlap in the knowledge required, but there is a world of difference between using a cloud based cluster over your own home made one. We are a very small team and opted for cloud managed clusters, which really freed me up to concentrate on how to build and manage applications running on it.
smitelli 2y ago
I can only speak for myself, and some of the reasons why K8s has left a bad taste in my mouth: - It can be complex depending on the third-party controllers and operators in use. If you're not anticipating how they're going to make your resources behave differently than the documentation examples suggest they will, it can be exhausting to trace down what's making them act that way. - The cluster owners encounter forced software updates that seem to come at the most inopportune times. Yes, staying fresh and new is important, but we have other actual business goals we have to achieve at the same time and -- especially with the current cost-cutting climate -- care and feeding of K8s is never an organizational priority. - A bunch of the controllers we relied on felt like alpha grade toy software. We went into each control plane update (see previous point) expecting some damn thing to break and require more time investment to get the cluster simply working like it was before. - While we (cluster owners) begrudgingly updated, software teams that used the cluster absolutely did not. Countless support requests for broken deployments, which were all resolved by hand-holding the team through a Helm chart update that we advised them they'd need to do months earlier. - It's really not cheaper than e.g. ECS, again, in my experience. - Maybe this has/will change with time, but I really didn't see the "onboarding talent is easier because they already know it." They absolutely did not. If you're coming from a shop that used Istio/Argo and move to a Linkerd/Flux shop, congratulations, now there's a bunch to unlearn and relearn. - K8s is the first environment where I palpably felt like we as an industry reached a point where there were so many layers and layers on top of abstractions of abstractions that it became un-debuggable in practice. This is points #1-3 coming together to manifest as weird latency spikes, scaling runaways, and oncall runbooks that were tantamount to "turn it off and back on." Were some of these problems organizational? Almost certainly. But K8s had always been sold as this miracle technology that would relieve so many pain points that we would be better off than we had been. In my experience, it did not do that.
- ahoka 2y ago
  What would be the alternative?
  - smitelli 2y ago
    Truthfully, I don't know. But I suspect I'm not the only one who feels a kind of debilitating ennui about the way things have gone and how they continue to go.
deleted 2y ago
[deleted]

BobbyJo 2y ago

Kubernetes isn't even that complicated, and first party support from cloud providers often means you're doing something in K8s inleu of doing it in a cloud specific way (like ingress vs cloud specific load balancer setups). At a certain scale, K8s is the simple option. I think much of the hate on HN comes from the "ruby on rails is all you need" crowd.

JohnMakin 2y ago
> I think much of the hate on HN comes from the "ruby on rails is all you need" crowd. Maybe - people seem really gungho about serverless solutions here too
- dexwiz 2y ago
  The hype for serverless cooled after that article about Prime Video dropping lambda. No one wants a product that a company won’t dogfood. I realize Amazon probably uses lambda elsewhere, but it was still a bad look.
  - LordKeren 2y ago
    I think it was much more about one specific use case of lambda that was a bad fit for the prime video team’s need and not a rejection of lambda/serverless. TBH, it kind of reflected more poorly on the team than lambda as a product
  - cmckn 2y ago
    > Amazon probably uses lambda elsewhere Yes, you could say that. :)
  - JohnMakin 2y ago
    not probably, their lambda service powers much of their control plane.
eutropia 2y ago
I guess the ones who quietly ship dozens of rails apps on k8s are too busy getting shit done to stop and share their boring opinions about pragmatically choosing the right tool for the job :)
- BobbyJo 2y ago
  "But you can run your rails app on a single host with embedded SQLite, K8s is unnecessary."
  - ffsm8 2y ago
    And there is truth to that. Most deployments are at that level, and it absolutely is way more performant then the alternative. it just comes with several tradeoffs... But these tradeoffs are usually worth it for deployments with <10k concurrent users. Which Figma certainly isn't. Though you probably could still do it, but that's likely more trouble then it's worth (The 10k is just an arbitrary number I made up, there is no magic number which makes this approach unviable, it all depends on how the users interact with the platform/how often and where the data is inserted)
  - threeseed 2y ago
    Always said by people who haven't spent much time in the cloud. Because single hosts will always go down. Just a question of when.
    - BossingAround 2y ago
      I love k8s, but bringing back up a single app that crashed is a very different problem from "our k8s is down" - because if you think your k8s won't go down, you're in for a surprise. You can view a single k8s also as a single host, which will go down at some point (e.g. a botched upgrade, cloud network partition, or something similar). While much less frequent, also much more difficult to get out of. Of course, if you have a multi-cloud setup with automatic (and periodically tested!) app migration across clouds, well then... Perhaps that's the answer nowadays.. :)
bamboozled 2y ago
Agreed, we're a small team and we benefit greatly from managed k8s (EKS). I have to say the whole ecosystem just continues to improve as far as I can tell and the developer satisfaction is really high with it. Personally I think k8s is where it's at now. The innovation and open source contributions are immense. I'm glad we made the switch. I understand the frustrations of the past, but I think it was much harder to use 4+ years ago. Now, I don't see how anyone could mess it up so hard.
MrDarcy 2y ago
> Kubernetes isn't even that complicated I’ve been struggling to square this sentiment as well. I spend all day in AWS and k8s and k8s is at least an order of magnitude simpler than AWS. What are all the people who think operating k8s is too complicated operating on? Surely not AWS…
- tbrownaw 2y ago
  The thing you already know tends to be less complicated than the thing you don't know.
- rco8786 2y ago
  I think "k8s is complicated" and "AWS is even more complicated" can both be true. Doing anything in AWS is like pulling teeth.
- brainzap 2y ago
  The sum is complex, specially with the custom operators.
chrischen 2y ago
There are also a lot of cog-in-the-machine engineers here that totally do not get the bigger picture or the vantage point from another department.
deleted 2y ago
[deleted]
dorianmariefr 2y ago
ruby on rails is all you need

logifail 2y ago

> it's written by a company that nearly got sold to Adobe for $20 billion (Apologies if this is a dumb question) but isn't Figma big enough to want to do any of their stuff on their own hardware yet? Why would they still be paying AWS rates? Or is it the case that a high-profile blog post about K8S and being provider-agnostic gets you sufficient discount on your AWS bill to still be value-for-money?

jeffbee 2y ago
There are a lot of ex-Dropbox people at Figma who might have learned firsthand that bringing your stuff on-prem under a theory of saving money is an intensely stupid idea.
- logifail 2y ago
  > There are a lot of ex-Dropbox people at Figma who might have learned firsthand that bringing your stuff on-prem under a theory of saving money is an intensely stupid idea Well, that's one hypothesis. Another is that "Every maturing company with predictable products must be exploring ways to move workloads out of the cloud. AWS took your margin and isn't giving it back." ( https://news.ycombinator.com/item?id=35235775 https://news.ycombinator.com/item?id=35235775 )
ozim 2y ago
They are preparing for next blog post in a year - „how we cut costs by xx% by moving to our own servers”.
hyperbolablabla 2y ago
I work for a company making ~$9B in annual revenue and we use AWS for everything. I think a big aspect of that is just developer buy-in, as well as reliability guarantees, and being able to blame Amazon when things do go down
- st3fan 2y ago
  Also, you don't have to worry about half of your stack? The shared responsibility model really works.

cwiggs 2y ago

k8s is complex, if you don't need the following you probably shouldn't use it: * Service discovery * Auto bin packing * Load Balancing * Automated rollouts and rollbacks * Horizonal scaling * Probably more I forgot about You also have secret and config management built in. If you use k8s you also have the added benefit of making it easier to move your workloads between clouds and bare metal. As long as you have a k8s cluster you can mostly move your app there. Problem is most companies I've worked at in the past 10 years needed multiple of the features above, and they decided to roll their own solution with Ansible/Chef, Terraform, ASGs, Packer, custom scripts, custom apps, etc. The solutions have always been worse than what k8s provides, and it's a bespoke tool that you can't hire for. For what k8s provides, it isn't complex, and it's all documented very well, AND it's extensible so you can build your own apps on top of it. I think there are more SWE on HN than Infra/Platform/Devops/buzzword engineers. As a result there are a lot of people who don't have a lot of experience managing infra and think that spinning up their docker container on a VM is the same as putting an app in k8s. That's my opinion on why k8s gets so much hate on HN.

Osiris 2y ago
Those all seem important to even moderately sized products.
- worldsayshi 2y ago
  As long as your requirements are simple the config doesn't need to be complex either. Not much more than docker-compose. But once you start using k8s you probably tend to scope creep and find a lot of shiny things to add to your set up.
- doctorpangloss 2y ago
  Some ways to tell if someone is a great developer are easy. JetBrains IDE? Ample storage space? Solving problems with the CLI? Consistently formatted code using the language's packaging ecosystem? No comments that look like this: # A verbose comment that starts capitalized, followed by a single line of code, cuing you that it was written by a ChatBot. Some ways to tell if someone is a great developer is hard. You can't tell if someone is a brilliant shipper of features, choosing exactly the right concerns to worry about at the moment, like doing more website authoring and less devops, with a grand plan for how to make everything cohere later; or, if the guy just doesn't know what the fuck he is doing. Kubernetes adoption is one of those, hard ones. It isn't a strong, bright signal like using PEP 8 and having a `pyproject.toml` with dependencies declared. So it may be obvious to you, "People adopt Kubernetes over ad-hoc decoupled solutions like Terraform because it has, in a Darwinian way, found the smallest set of easily surmountable concerns that should apply to most good applications." But most people just see, "Ahh! Why can't I just write the method bodies for Python function signatures someone else wrote for me, just like they did in CS50!!!"

tracerbulletx 2y ago

Even on a small project it's actually better imo than tying everything to a platform like netlify or vercel. I have this little notepad app that I deploy to a two node cluster in a github action and its an excellent workflow. The k8s to get everything deployed, provision tls and everything on commit is like 150 lines of mostly boilerplate yaml, I could pretty easily make it support branch previews or whatever too. https://github.com/SteveCastle/modelpad https://github.com/SteveCastle/modelpad

saaspirant 2y ago

Unrelated: What does _TFA_ mean here? Google and GPT didn't help (even with context)

solatic 2y ago
The Featured Article. (or, if you read it in a frustrated voice, The F**ing Article.)
- Xixi 2y ago
  Related acronyms: RTFA (Read The F**ing Article) and RTFM (Read The F**ing Manual). The latter was a very common answer when struggling with Linux in the early 2000s...

osigurdson 2y ago

Kubernetes is the most amazing piece of software engineering that I have ever seen. Most of the hate is merely being directed at the learning curve.

otabdeveloper4 2y ago
No, k8s is shit. It's only useful for the degenerate "run lots of instances of webapp servers running slow interpreted languages" use case. Trying to do anything else in it is madness. And for the "webapp servers" use case they could have built something a thousand times simpler and more robust. Serving templated html ain't rocket science. (At least compared to e.g. running an OLAP database cluster.)
- shakiXBT 2y ago
  Could you please bless us with another way to easily orchestrate thousands of containers in a cloud vendor agnostic fashion? Thanks! Oh, and just in case your first rebuttal is "having thousands of containers means you've already failed" - not everyone works in a mom n pop shop
  - otabdeveloper4 2y ago
    Read my post again. Just because k8s is the only game in town doesn't mean it is technically any good. As a technology it is a total shitshow. Luckily, the problem it solves ("orchestrating" slow webapp containers) is not a problem most professionals care about. Feature creep of k8s into domains it is utterly unsuitable for because devops wants a pay raise is a different issue.
    - shakiXBT 2y ago
      > Orchestrating containers is not a problem most professionals care about I truly wish you were right, but maybe it's good job security for us professionals!
    - osigurdson 2y ago

globular-toast 2y ago

I don't get the hate even if you are a small company. K8s has massively simplified our deployments. It used to be each app had it's own completely different deployment process. Could have been a shell script that SSHed to some VM. Who managed said VM? Did it do its own TLS termination? Fuck knows. Maybe they used Ansible. Great, but that's another tool to learn and do I really need to set up bare metal hosts from scratch for every service. No, so there's probably some other Ansible config somewhere that sets them up. And the secrets are stored where? Etc etc. People who say "you don't need k8s" never say what you do need. K8s gives us a uniform interface that works for everything. We just have a few YAML files for each app and it just works. We can just chuck new things on there and don't even have to think about networking. Just add a Service and it's magically available with a name to everything in the cluster. I know how to do this stuff from scratch and I do not want to be doing it every single time.

shakiXBT 2y ago
if you don't need High Availability you can even deploy to a single node k3s cluster. It's still miles better than having to setup systemd services, an Apache/NGINX proxy, etc. etc.
- globular-toast 2y ago
  Yep, and you can get far with k3s "fake" load balancer (ServiceLB). Then when you need a more "real" cluster basically all the concepts are the same you just move to a new cluster.

roydivision 2y ago

One thing I learned when I started learning Kubernetes is that it is two disciplines that overlap, but are distinct none the less: - Platform build and management - App build and management Getting a stable K8s cluster up and running is quite different to building and running apps on it. Obviously there is overlap in the knowledge required, but there is a world of difference between using a cloud based cluster over your own home made one. We are a very small team and opted for cloud managed clusters, which really freed me up to concentrate on how to build and manage applications running on it.

smitelli 2y ago

I can only speak for myself, and some of the reasons why K8s has left a bad taste in my mouth: - It can be complex depending on the third-party controllers and operators in use. If you're not anticipating how they're going to make your resources behave differently than the documentation examples suggest they will, it can be exhausting to trace down what's making them act that way. - The cluster owners encounter forced software updates that seem to come at the most inopportune times. Yes, staying fresh and new is important, but we have other actual business goals we have to achieve at the same time and -- especially with the current cost-cutting climate -- care and feeding of K8s is never an organizational priority. - A bunch of the controllers we relied on felt like alpha grade toy software. We went into each control plane update (see previous point) expecting some damn thing to break and require more time investment to get the cluster simply working like it was before. - While we (cluster owners) begrudgingly updated, software teams that used the cluster absolutely did not. Countless support requests for broken deployments, which were all resolved by hand-holding the team through a Helm chart update that we advised them they'd need to do months earlier. - It's really not cheaper than e.g. ECS, again, in my experience. - Maybe this has/will change with time, but I really didn't see the "onboarding talent is easier because they already know it." They absolutely did not. If you're coming from a shop that used Istio/Argo and move to a Linkerd/Flux shop, congratulations, now there's a bunch to unlearn and relearn. - K8s is the first environment where I palpably felt like we as an industry reached a point where there were so many layers and layers on top of abstractions of abstractions that it became un-debuggable in practice. This is points #1-3 coming together to manifest as weird latency spikes, scaling runaways, and oncall runbooks that were tantamount to "turn it off and back on." Were some of these problems organizational? Almost certainly. But K8s had always been sold as this miracle technology that would relieve so many pain points that we would be better off than we had been. In my experience, it did not do that.

ahoka 2y ago
What would be the alternative?
- smitelli 2y ago
  Truthfully, I don't know. But I suspect I'm not the only one who feels a kind of debilitating ennui about the way things have gone and how they continue to go.

deleted 2y ago

[deleted]