8 ms·

I work on a 3-person DevOps team that just finished migrating ~20 services from GCE vms running docker-compose to GKE. It's taken us a little over a year. Part

I work on a 3-person DevOps team that just finished migrating ~20 services from GCE vms running docker-compose to GKE. It's taken us a little over a year. Partly because K8s has a steep learning curve, but also because safely transitioning services without disrupting product teams adds a lot of overhead. The investment is already yielding great returns. Developers are happy. Actual quote: "Kubernetes is the biggest quality-of-life improvement I've experienced in my career." The same is true for me as a DevOps engineer. I never have to write a script to orchestrate a rolling deployment ever again. I'll never have to touch Puppet or wrangle OS upgrades. I no longer need to use Terraform to scale out a service. Now that we've completed this transition, one of my goals for the next year is to get the team to a place where we spend 75% of our time on higher-level projects -- working on developer tools and Kubernetes extensions and setting site reliability standards, instead of the firefighting and operational work that has traditionally taken up so much of our time. Couldn't be happier about this change. ... That is until management decides we're overhead and fires us to free up budget for feature developers. ;)

z9e 6y ago
... Until they hire you back because entropy exists :P I'm on the DevOps side as well going through the same transition, k8s also allows insane customization, and I have some colleagues that are delaying our rollout unintentionally so they can play around with developing more tooling for deployments which is really frustrating. The k8s scene seems to be filled with constant scope creep and refactoring to get it just perfect before use. Either way, I agree the benefits far outweigh this annoyance that I've experienced. I'm so excited to work on developing tooling instead with my time. However, I don't think we're free entirely from managing servers the old way with Chef / Puppet / Ansible, unless you're purely hosted there's still the rule of thumb you shouldn't run services that hold state in k8s. But with persistent vol's I do see that changing, though I'm not sure if everyone agree's that's a good idea.
- vp8989 6y ago
  "I have some colleagues that are delaying our rollout unintentionally so they can play around with developing more tooling for deployments which is really frustrating." My impression is that the primary purpose of Kubernetes is to give SRE teams political air cover to rewrite a lot of their existing processes. Whether Kubernetes is actually required for that, or even net superior seems questionable. This unsexy work becomes justifiable because it's coupled to a mainstream accepted tech modernization. You see this same phenomenon with database migrations. Where what the team really needed to do is just rewrite an app to use the existing database properly. But no one is going to approve that work. So what happens is people convince themselves that the existing tech sucks and use that to rationalize doing the rewrite. The result ends up not always being net superior, because sure you did the rewrite but you are also eating the operational cost of integrating a new technology into the org.
  - Guest42 6y ago
    Yes, the number of proposed db switched I’ve seen is remarkably high. I once interviewed for a role as a database developer and was confused to find out they didn’t have the database that the role pertained to. One of the early questions in the interview was how quickly I could migrate production from ms sql to pg. Needless to say that was a gigantic red flag and I hope they found the right person for that job. I’ve also seen a switch from rdbms to Hadoop because a company had “millions” of rows. Luckily on this one I only had to rewrite a handful of queries.
    - eloff 6y ago
      > I’ve also seen a switch from rdbms to Hadoop because a company had “millions” of rows. Luckily on this one I only had to rewrite a handful of queries. Wat. That's gross. It probably costs them more per query now than the rdbms did.
      - Guest42 6y ago
        Didn’t even think about that part because I don’t know too much about Hadoop other than it seemed impractical.
    - yellow_lead 6y ago
      I guess millions sounded like a lot to a decision maker :)
    - jaywalk 6y ago
      I've got relatively modestly-specced SQL Servers handling tables with hundreds of millions and even billions of rows without breaking a sweat. Somebody either just really wanted a new toy to play with, or has no idea what indexes are.
      - Guest42 6y ago
        Exactly, I’ve seen Sql servers handle billions of rows with 2 thousand columns. I think also people that work too long at one company don’t realize how problems were solved elsewhere.
- anchochilis 6y ago
  Ha, are you me? I really pushed for us to follow the "change-as-little-as-possible and ship to prod quickly" route. Prod is where things get hard, and it's better to find out what's hard sooner rather than later. We are running a handful of stateful services in K8s (things like MongoDB for which GCP doesn't have a compelling and affordable managed offering). It's definitely more complex than transitioning a stateless service, but so far our experiences with StatefulSets and PersistentVolumes have been good. And this allows us to sunset Puppet/OS management completely. I should note that we _are_ being extremely careful about backups. We also run each stateful service in a dedicated node pool for isolation. Who knows, maybe a year from now we'll be shaking our heads and saying "that was a TERRIBLE idea" but for now, so far so good. We're running on GKE, so lots of things that would be hard in on-prem environments (ingress, networking, storage) are easy.
  - throwaway894345 6y ago
    > We're running on GKE, so lots of things that would be hard in on-prem environments (ingress, networking, storage) are easy. Agreed. The on-prem story is still really messy, but I think there's a lot of third-party work to build on-prem distributions that are cut and dry. Unfortunately, there are lots of them right now and it's not clear what the advantages and pitfalls are of each. Things will settle and this problem will be solved with time, but for now it's quite a pain point.
- AtlasBarfed 6y ago
  Management never hires people back, that's admitting failure. They hire other people with the supposedly same skillstack and then have them rebuild it from scratch.
vagrantJin 6y ago
> That is until management decides we're overhead and fires us to free up budget for feature developers. Snake eating it's own tail. But not only do I not understand K8s, I'm also an idiot.
vp8989 6y ago
"Kubernetes is the biggest quality-of-life improvement I've experienced in my career." Can you elaborate? What exactly about Kubernetes improved the developers life?
- yowlingcat 6y ago
  Can only speak for myself personally, but having gone through a similar moment, it was 50% docker QoL improvements and 50% k8s QoL improvements -- most significantly around prod/dev parity. Every engineer in my org really did gain a significant amount of confidence around being able to spin up and deploy new services (when necessary) without being nervous about something going wrong in prod that wasn't configured properly in a lower environment.
  - skinnyarms 6y ago
    I'm really happy with the local development story too. We use skaffold to pick and choose which resources we run locally, and then we use the same definition to generate our artifacts which keeps things in line. It reminds me of docker-compose file, except I can publish it too.
- anchochilis 6y ago
  1. Reliable rolling deployments. We didn't have these before. Yes, you can implement them without K8s (I have, at other companies) but to get the full set of features K8s provides, such as deploying N services in parallel, taking no more than X% of your capacity offline at a time, short-circuiting in the event the app is dead on arrival, connection draining with timeouts, you end up with a VERY complicated multi-threaded codebase. 2. Seamless horizontal scale-out. Want to scale up your app in a test environment from 3 replicas to 6 to do some performance testing? This used to be a DevOps ticket that would take a few days -- DevOps engineer tries running Terraform, but oh no, a CentOS package update seems to have broken our Puppet manifests and we have to fix that. Now the developer makes a PR to the GitOps repo where they adjust a single YAML setting. 3. GitOps/ArgoCD. ArgoCD is possibly my favorite piece of software of all time. It provides incredible visualizations for what's happening with your Kubernetes infrastructure. It really increases your confidence and trust in the system to be able watch a deployment rollout or scaling operation happen in real time. ArgoCD makes it spectacularly obvious when something has gone wrong -- you still sometimes need to go spelunking through the GCP console or use kubectl to inspect resources, but to a much lesser degree. I cannot emphasize enough how magical it is. These are all things that our initial implementation has delivered, we are also planning to try to leverage K8s for things like on-demand pull request preview environments, hosted developer environments, and canary deploys that are MUCH harder to implement in a world without K8s (trust me, I've done it).
Ianvdl 6y ago
As someone with limited experience with containers, how does K8s allow you to move away from things like Puppet for configuration management? Does it offer some substitute that alleviates the need for something like Puppet or Ansible?
- zdw 6y ago
  You still need something to provision the base OS and all the stuff under K8s (docker daemon, ntp, storage, networking, etc.) that it relies on, unless you go with a fully hosted solution. Ansible or Puppet still excel at that kind of work.
  - Spivak 6y ago
    And it looks like the parent went with a hosted solution which explains everything. Having to manage all the underlying services that k8s glues together is a huge PITA.
- specialp 6y ago
  Yes because whatever you used for app specific configuration like libraries and packages is now done in the Dockerfile and containerized. So the same thing run locally is run in the cloud. Then as far as the infrastructure for running code such as load balancers, service discovery, docker.. That is all given to you just by running K8s. So you are more concerned with shipping immutable containers to k8s than provisioning "machines". Then you can focus on containers which can be run, tested and built wherever without the fear of broken updates or one thing stepping on another. We found back in the days of ansible and chef that we had very low confidence in upgrading hosts live. So we would then do immutable hosts and blue green deploy them to production. But why think in the scope of hosts and VMs when really you have some application that needs to run somewhere. K8s IMO isn't the end all, I think eventually we will get to something that doesn't need containers at all and you run just processes. But it is a good step for now. Also once you have your stuff containerized it makes other non k8s stuff easy like AWS Lambda Edit: Also yes you can use those to set up generic k8s nodes but when we ran bare metal we used kubeadm to make coreOS immutable nodes. I don't think that is used anymore haven't checked but really the best way to set up k8s is to deploy really thin hosts that have nothing but Docker and k8s. VMware and others have solutions for this too where you don't have to mess with building hosts.
mushishi 6y ago
One great boon is to use Kotlin code to factor deployment descriptors into reusable parts that are somewhat statically type checked. I wasn't responsible for the migration from legacy system to Kubernetes, but that seemed to me a big win. Aside from being able to update the whole, and only parts of the system so trivially. For me the scariest part would be that it's not tech I would use (i.e. update the descriptors) regularly, so what if something goes wrong, how quick would I be able to identify the problem. I have no answer to that because I'm off the project.
pyrophane 6y ago
> I no longer need to use Terraform to scale out a service. Are you still using Terraform to specify your deployments? From experience, you need something there to manage all your yamls: deployments, configmaps, secrets. Especially if you have multiple environments.
neonological 6y ago
I’m tired of the complainers of the complainers. First a technology seems cool, then after engineers become experienced with the tool the warts rear itself and you get a vocal group of complainers. It doesn’t end there... after complaining for a really long time two things happen. First off so much time has passed that you get these domain experts (Devops people) whose entire job is to mess with kubernetes. Second the complainers have been complaining so long that people get tired of it. You now get people who are so tired of listening to people complain whose entire job entire job revolves around kubernetes that these people now start complaining about the complainers. It happened with JavaScript. Javascript was around for so long people started complaining about the complainers. In fact it’s been around so long that a whole generation of people who’ve never used any other language was literally born. These people started off as the complainers against the complainers but now they outnumber the complainers so you rarely see people talk shit about JavaScript anymore. Actually JavaScript has been around so long that the entire language has changed and part of the terribleness was fixed by making another language (typescript) compile into JavaScript. Which brings me back to kubernetes. Kubernetes is a bad tool with no alternative precisely because it requires a 3 man dedicated team a year to get things up and running. A good tool would be something like allows me to to get it up and running in a week just by reading some docs. Even better an hour. Could such a tool exist and replace Kubernetes? Yes. Does such a tool exist? No. I am a complainer and you are a complainer of complainers. What will likely happen some time from now is two possible things. Kube will be so integrated into the infrastructure ecosystem that wrappers will be written on top of kube just like how react and typescript have replaced JavaScript. If that doesn’t happen then a whole new tool will replace it. I’m sorry to say but the ideal we are shooting for here is a tool that will ultimately make Devops a general thing that all developers can deal with rather then an entire specialist team. Again no such tool exists yet but it certainly can exist, especially when the inventor of the tool has become a complainer. If the inventor of the tool becomes a complainer, that validates the complainers. And now the complainers of the complainers have nothing left to say.
- anchochilis 6y ago
  There already are a number of platforms that make deploying and operating apps very simple! Heroku, AppEngine, Lambda, so many others. Your 5-person startup does not need a DevOps engineer. Your 50-person org should start thinking about it. Your 500-person company probably needs multiple DevOps engineers -- having every dev team independently figure out how to handle things like deployments, reliability, and security is chaotic and wasteful. There are a lot of details that only start to matter at scale, and both Kubernetes and dedicated infrastructure teams are for this use case.
stickfigure 6y ago
> "Kubernetes is the biggest quality-of-life improvement I've experienced in my career." Just wait until the developers discover Google App Engine, Heroku, or DigitalOcean App Platform.
- sontek 6y ago
  Yeah, I think thats the point though. Kubernetes enables the orchestration and observability of a PaaS in a much more flexible way so that you can get all of that while still matching the requirements of your business. I think Heroku and DigitalOcean App Platform are still going to be popular for small setups (as will things like Amplify) but when you outgrow those (or realize you are paying too much for them) then Kubernetes is a reasonable option.
  - wdenniss 6y ago
    +1. One benefit I see of Kubernetes is that it can handle pretty much whatever you throw at it, so you can run everything in the one service. Need to run a bespoke database, yeah it can do that. Need to migrate an old service running in a VM that needs a disk, it can do that too.
- anchochilis 6y ago
  Oh man, I can't believe we didn't think of migrating our mature 20-service stack with complex hardware, security, and compliance requirements to App Engine. Dang it. I should've known to consult HN first!
lame88 6y ago
How much of this is specific to Kubernetes and how much is more accurately because you moved from managing your own VM infrastructure to using managed services? Cloud providers have (often long had) ways of saying "run this code" without managing a VM and without involving kubernetes, including with docker support. And they are often very easy to use though not without their faults. An example being Azure app service. I do agree that kubernetes is a pleasant experience from an application developer perspective with an existing cluster, but in my experience it was not without excessive pain and long hours by those administering the cluster. A year doesn't surprise me in your case, which brings to mind this question.
ascendantlogic 6y ago
That was my experience with Kubernetes when I set it up for a QA cluster a few years back. That learning curve is more like a sheer cliff face but once you get to the top the view is really nice. But getting there is a hell of a hike. It's easier now since every major cloud has a managed offering but my journey was pre-EKS so it was doubly as intense for me.
benlivengood 6y ago
> ... That is until management decides we're overhead and fires us to free up budget for feature developers. ;) It's a real risk, at least perceptually. Mitigate it by documenting the number of developers' hours you save with automation and better infrastructure. It's pretty hard to argue against a trend of increased developer productivity. Include your own hours; you're targeting saving 30 hours a week of your own time and that frees you up to improve other things even faster.
egh 6y ago
You spent more than 3 person-years migrating twenty services from one container orchestration system to another and you think it was worth it? Not sure that seems worth it to me...
- kevincox 6y ago
  If you have about 20 developers spending 10% of their time fighting with your current system it adds up fast. I mean 10% may be high but 20 may be low. It depends a lot on the exact situation if it was worth the time.
- anchochilis 6y ago
  Nope! We went from not having a container orchestration system to having one. If you're determined to be skeptical of the value-add from Kubernetes, my random post on HN ain't gonna convince you :) The developer experience improvement is obvious to people actually interacting with K8s. I do want to clarify that this isn't the only thing we accomplished in the past year. Just the thing we shipped that was our biggest priority and had (by far) the biggest impact.

z9e 6y ago

... Until they hire you back because entropy exists :P I'm on the DevOps side as well going through the same transition, k8s also allows insane customization, and I have some colleagues that are delaying our rollout unintentionally so they can play around with developing more tooling for deployments which is really frustrating. The k8s scene seems to be filled with constant scope creep and refactoring to get it just perfect before use. Either way, I agree the benefits far outweigh this annoyance that I've experienced. I'm so excited to work on developing tooling instead with my time. However, I don't think we're free entirely from managing servers the old way with Chef / Puppet / Ansible, unless you're purely hosted there's still the rule of thumb you shouldn't run services that hold state in k8s. But with persistent vol's I do see that changing, though I'm not sure if everyone agree's that's a good idea.

vp8989 6y ago
"I have some colleagues that are delaying our rollout unintentionally so they can play around with developing more tooling for deployments which is really frustrating." My impression is that the primary purpose of Kubernetes is to give SRE teams political air cover to rewrite a lot of their existing processes. Whether Kubernetes is actually required for that, or even net superior seems questionable. This unsexy work becomes justifiable because it's coupled to a mainstream accepted tech modernization. You see this same phenomenon with database migrations. Where what the team really needed to do is just rewrite an app to use the existing database properly. But no one is going to approve that work. So what happens is people convince themselves that the existing tech sucks and use that to rationalize doing the rewrite. The result ends up not always being net superior, because sure you did the rewrite but you are also eating the operational cost of integrating a new technology into the org.
- Guest42 6y ago
  Yes, the number of proposed db switched I’ve seen is remarkably high. I once interviewed for a role as a database developer and was confused to find out they didn’t have the database that the role pertained to. One of the early questions in the interview was how quickly I could migrate production from ms sql to pg. Needless to say that was a gigantic red flag and I hope they found the right person for that job. I’ve also seen a switch from rdbms to Hadoop because a company had “millions” of rows. Luckily on this one I only had to rewrite a handful of queries.
  - eloff 6y ago
    > I’ve also seen a switch from rdbms to Hadoop because a company had “millions” of rows. Luckily on this one I only had to rewrite a handful of queries. Wat. That's gross. It probably costs them more per query now than the rdbms did.
    - Guest42 6y ago
      Didn’t even think about that part because I don’t know too much about Hadoop other than it seemed impractical.
  - yellow_lead 6y ago
    I guess millions sounded like a lot to a decision maker :)
  - jaywalk 6y ago
    I've got relatively modestly-specced SQL Servers handling tables with hundreds of millions and even billions of rows without breaking a sweat. Somebody either just really wanted a new toy to play with, or has no idea what indexes are.
    - Guest42 6y ago
      Exactly, I’ve seen Sql servers handle billions of rows with 2 thousand columns. I think also people that work too long at one company don’t realize how problems were solved elsewhere.
anchochilis 6y ago
Ha, are you me? I really pushed for us to follow the "change-as-little-as-possible and ship to prod quickly" route. Prod is where things get hard, and it's better to find out what's hard sooner rather than later. We are running a handful of stateful services in K8s (things like MongoDB for which GCP doesn't have a compelling and affordable managed offering). It's definitely more complex than transitioning a stateless service, but so far our experiences with StatefulSets and PersistentVolumes have been good. And this allows us to sunset Puppet/OS management completely. I should note that we _are_ being extremely careful about backups. We also run each stateful service in a dedicated node pool for isolation. Who knows, maybe a year from now we'll be shaking our heads and saying "that was a TERRIBLE idea" but for now, so far so good. We're running on GKE, so lots of things that would be hard in on-prem environments (ingress, networking, storage) are easy.
- throwaway894345 6y ago
  > We're running on GKE, so lots of things that would be hard in on-prem environments (ingress, networking, storage) are easy. Agreed. The on-prem story is still really messy, but I think there's a lot of third-party work to build on-prem distributions that are cut and dry. Unfortunately, there are lots of them right now and it's not clear what the advantages and pitfalls are of each. Things will settle and this problem will be solved with time, but for now it's quite a pain point.
AtlasBarfed 6y ago
Management never hires people back, that's admitting failure. They hire other people with the supposedly same skillstack and then have them rebuild it from scratch.

vagrantJin 6y ago

> That is until management decides we're overhead and fires us to free up budget for feature developers. Snake eating it's own tail. But not only do I not understand K8s, I'm also an idiot.

vp8989 6y ago

"Kubernetes is the biggest quality-of-life improvement I've experienced in my career." Can you elaborate? What exactly about Kubernetes improved the developers life?

yowlingcat 6y ago
Can only speak for myself personally, but having gone through a similar moment, it was 50% docker QoL improvements and 50% k8s QoL improvements -- most significantly around prod/dev parity. Every engineer in my org really did gain a significant amount of confidence around being able to spin up and deploy new services (when necessary) without being nervous about something going wrong in prod that wasn't configured properly in a lower environment.
- skinnyarms 6y ago
  I'm really happy with the local development story too. We use skaffold to pick and choose which resources we run locally, and then we use the same definition to generate our artifacts which keeps things in line. It reminds me of docker-compose file, except I can publish it too.
anchochilis 6y ago
1. Reliable rolling deployments. We didn't have these before. Yes, you can implement them without K8s (I have, at other companies) but to get the full set of features K8s provides, such as deploying N services in parallel, taking no more than X% of your capacity offline at a time, short-circuiting in the event the app is dead on arrival, connection draining with timeouts, you end up with a VERY complicated multi-threaded codebase. 2. Seamless horizontal scale-out. Want to scale up your app in a test environment from 3 replicas to 6 to do some performance testing? This used to be a DevOps ticket that would take a few days -- DevOps engineer tries running Terraform, but oh no, a CentOS package update seems to have broken our Puppet manifests and we have to fix that. Now the developer makes a PR to the GitOps repo where they adjust a single YAML setting. 3. GitOps/ArgoCD. ArgoCD is possibly my favorite piece of software of all time. It provides incredible visualizations for what's happening with your Kubernetes infrastructure. It really increases your confidence and trust in the system to be able watch a deployment rollout or scaling operation happen in real time. ArgoCD makes it spectacularly obvious when something has gone wrong -- you still sometimes need to go spelunking through the GCP console or use kubectl to inspect resources, but to a much lesser degree. I cannot emphasize enough how magical it is. These are all things that our initial implementation has delivered, we are also planning to try to leverage K8s for things like on-demand pull request preview environments, hosted developer environments, and canary deploys that are MUCH harder to implement in a world without K8s (trust me, I've done it).

Ianvdl 6y ago

As someone with limited experience with containers, how does K8s allow you to move away from things like Puppet for configuration management? Does it offer some substitute that alleviates the need for something like Puppet or Ansible?

zdw 6y ago
You still need something to provision the base OS and all the stuff under K8s (docker daemon, ntp, storage, networking, etc.) that it relies on, unless you go with a fully hosted solution. Ansible or Puppet still excel at that kind of work.
- Spivak 6y ago
  And it looks like the parent went with a hosted solution which explains everything. Having to manage all the underlying services that k8s glues together is a huge PITA.
specialp 6y ago
Yes because whatever you used for app specific configuration like libraries and packages is now done in the Dockerfile and containerized. So the same thing run locally is run in the cloud. Then as far as the infrastructure for running code such as load balancers, service discovery, docker.. That is all given to you just by running K8s. So you are more concerned with shipping immutable containers to k8s than provisioning "machines". Then you can focus on containers which can be run, tested and built wherever without the fear of broken updates or one thing stepping on another. We found back in the days of ansible and chef that we had very low confidence in upgrading hosts live. So we would then do immutable hosts and blue green deploy them to production. But why think in the scope of hosts and VMs when really you have some application that needs to run somewhere. K8s IMO isn't the end all, I think eventually we will get to something that doesn't need containers at all and you run just processes. But it is a good step for now. Also once you have your stuff containerized it makes other non k8s stuff easy like AWS Lambda Edit: Also yes you can use those to set up generic k8s nodes but when we ran bare metal we used kubeadm to make coreOS immutable nodes. I don't think that is used anymore haven't checked but really the best way to set up k8s is to deploy really thin hosts that have nothing but Docker and k8s. VMware and others have solutions for this too where you don't have to mess with building hosts.

mushishi 6y ago

One great boon is to use Kotlin code to factor deployment descriptors into reusable parts that are somewhat statically type checked. I wasn't responsible for the migration from legacy system to Kubernetes, but that seemed to me a big win. Aside from being able to update the whole, and only parts of the system so trivially. For me the scariest part would be that it's not tech I would use (i.e. update the descriptors) regularly, so what if something goes wrong, how quick would I be able to identify the problem. I have no answer to that because I'm off the project.

pyrophane 6y ago

> I no longer need to use Terraform to scale out a service. Are you still using Terraform to specify your deployments? From experience, you need something there to manage all your yamls: deployments, configmaps, secrets. Especially if you have multiple environments.

neonological 6y ago

I’m tired of the complainers of the complainers. First a technology seems cool, then after engineers become experienced with the tool the warts rear itself and you get a vocal group of complainers. It doesn’t end there... after complaining for a really long time two things happen. First off so much time has passed that you get these domain experts (Devops people) whose entire job is to mess with kubernetes. Second the complainers have been complaining so long that people get tired of it. You now get people who are so tired of listening to people complain whose entire job entire job revolves around kubernetes that these people now start complaining about the complainers. It happened with JavaScript. Javascript was around for so long people started complaining about the complainers. In fact it’s been around so long that a whole generation of people who’ve never used any other language was literally born. These people started off as the complainers against the complainers but now they outnumber the complainers so you rarely see people talk shit about JavaScript anymore. Actually JavaScript has been around so long that the entire language has changed and part of the terribleness was fixed by making another language (typescript) compile into JavaScript. Which brings me back to kubernetes. Kubernetes is a bad tool with no alternative precisely because it requires a 3 man dedicated team a year to get things up and running. A good tool would be something like allows me to to get it up and running in a week just by reading some docs. Even better an hour. Could such a tool exist and replace Kubernetes? Yes. Does such a tool exist? No. I am a complainer and you are a complainer of complainers. What will likely happen some time from now is two possible things. Kube will be so integrated into the infrastructure ecosystem that wrappers will be written on top of kube just like how react and typescript have replaced JavaScript. If that doesn’t happen then a whole new tool will replace it. I’m sorry to say but the ideal we are shooting for here is a tool that will ultimately make Devops a general thing that all developers can deal with rather then an entire specialist team. Again no such tool exists yet but it certainly can exist, especially when the inventor of the tool has become a complainer. If the inventor of the tool becomes a complainer, that validates the complainers. And now the complainers of the complainers have nothing left to say.

anchochilis 6y ago
There already are a number of platforms that make deploying and operating apps very simple! Heroku, AppEngine, Lambda, so many others. Your 5-person startup does not need a DevOps engineer. Your 50-person org should start thinking about it. Your 500-person company probably needs multiple DevOps engineers -- having every dev team independently figure out how to handle things like deployments, reliability, and security is chaotic and wasteful. There are a lot of details that only start to matter at scale, and both Kubernetes and dedicated infrastructure teams are for this use case.

stickfigure 6y ago

> "Kubernetes is the biggest quality-of-life improvement I've experienced in my career." Just wait until the developers discover Google App Engine, Heroku, or DigitalOcean App Platform.

sontek 6y ago
Yeah, I think thats the point though. Kubernetes enables the orchestration and observability of a PaaS in a much more flexible way so that you can get all of that while still matching the requirements of your business. I think Heroku and DigitalOcean App Platform are still going to be popular for small setups (as will things like Amplify) but when you outgrow those (or realize you are paying too much for them) then Kubernetes is a reasonable option.
- wdenniss 6y ago
  +1. One benefit I see of Kubernetes is that it can handle pretty much whatever you throw at it, so you can run everything in the one service. Need to run a bespoke database, yeah it can do that. Need to migrate an old service running in a VM that needs a disk, it can do that too.
anchochilis 6y ago
Oh man, I can't believe we didn't think of migrating our mature 20-service stack with complex hardware, security, and compliance requirements to App Engine. Dang it. I should've known to consult HN first!

lame88 6y ago

How much of this is specific to Kubernetes and how much is more accurately because you moved from managing your own VM infrastructure to using managed services? Cloud providers have (often long had) ways of saying "run this code" without managing a VM and without involving kubernetes, including with docker support. And they are often very easy to use though not without their faults. An example being Azure app service. I do agree that kubernetes is a pleasant experience from an application developer perspective with an existing cluster, but in my experience it was not without excessive pain and long hours by those administering the cluster. A year doesn't surprise me in your case, which brings to mind this question.

ascendantlogic 6y ago

That was my experience with Kubernetes when I set it up for a QA cluster a few years back. That learning curve is more like a sheer cliff face but once you get to the top the view is really nice. But getting there is a hell of a hike. It's easier now since every major cloud has a managed offering but my journey was pre-EKS so it was doubly as intense for me.

benlivengood 6y ago

> ... That is until management decides we're overhead and fires us to free up budget for feature developers. ;) It's a real risk, at least perceptually. Mitigate it by documenting the number of developers' hours you save with automation and better infrastructure. It's pretty hard to argue against a trend of increased developer productivity. Include your own hours; you're targeting saving 30 hours a week of your own time and that frees you up to improve other things even faster.

egh 6y ago

You spent more than 3 person-years migrating twenty services from one container orchestration system to another and you think it was worth it? Not sure that seems worth it to me...

kevincox 6y ago
If you have about 20 developers spending 10% of their time fighting with your current system it adds up fast. I mean 10% may be high but 20 may be low. It depends a lot on the exact situation if it was worth the time.
anchochilis 6y ago
Nope! We went from not having a container orchestration system to having one. If you're determined to be skeptical of the value-add from Kubernetes, my random post on HN ain't gonna convince you :) The developer experience improvement is obvious to people actually interacting with K8s. I do want to clarify that this isn't the only thing we accomplished in the past year. Just the thing we shipped that was our biggest priority and had (by far) the biggest impact.