15 ms·

The premise of your question is invalid. Have you ever tried setting up a Kubernetes cluster and deploying apps in it? Kubernetes doesn't save work, it adds wor

The premise of your question is invalid. Have you ever tried setting up a Kubernetes cluster and deploying apps in it? Kubernetes doesn't save work, it adds work. In return, you get a lot of benefits, but it wasn't designed to reduce human work, nor was it designed to eliminate devops jobs. It was designed for scalability and availability more than anything. Most people using Kubernetes should be using something simpler, but that's a separate problem.

Thaxll 4y ago
> Have you ever tried setting up a Kubernetes cluster and deploying apps in it? Kubernetes doesn't save work This is wrong, deploying on Kubernetes is easy and quick for most apps, you have one docker image one deployment spec and that's it. https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#creating-a-deployment https://kubernetes.io/docs/concepts/workloads/controllers/de...
- caymanjim 4y ago
  And a service configuration, ingress, other networking, persistent volumes, a mechanism for updating deployed applications, management of the nodes (even with a managed service like EKS or other cloud solutions), logging, roles, security, etc. If you're a developer and all you have to worry about is one deployment spec, thank your devops team for making your life easier. Kubernetes is great for making the dev team's life easier, but someone did a lot of work to make it that easy for you.
  - spion 4y ago
    This is still easier with Kubernetes than with other tools. Installing e.g. fluentd and Kibana is again a configuration / chart which you apply to the cluster. For monitoring and visualization again you have an operator / chart that you can apply to the cluster. Yes there is a lot of complexity and learning involved but overall I was still pretty amazed at how quickly I was able to get a cluster up to speed with the available tools.
  - patrickserrano 4y ago
    Couldn't agree with this more. At my last company there was a fair bit of time put into making the deployment process for our microservices as simple as added a new YAML file to the deployment repo. That file pulled a custom chart, and as a dev you just needed to configure a few variables to get everything up an running. But if you were deploying something that couldn't use one of the pre-configured charts it was a bit more work, especially if you've never done it be hand before. Probably 98% of the devs were blissfully unaware of that complexity that the charts abstracted, and it let them focus on the services they were writing. I wasn't one of them, and always made sure to thank the devops team for simplifying the day to day deployments whenever I had to deal with writing a custom one.
    - iteratorx 4y ago
      You can now do something similar with Bunnyshell.com It Handles the devops jobs for dev teams. Full disclosure, I work for Bunnyshell.
      - macintux 4y ago
        In the absence of any description of what “something similar” or “handles the devops jobs” actually means, this comes across as spam, not informative.
        iteratorx 4y ago
        Sorry about that, I should have been more informative. Bunnyshell makes it easy to create and manage environments. (EaaS - environments as a service) You connect your k8s cluster(s) and git accounts/repos, it reads the docker-compose files and creates deployments on the cluster. You don’t need to know or write Kubernetes manifests, those are created for you. You also get auto updates and ephemeral/preview environments (when a PR is created against the branch of your env, Bunnyshell deploys a new env with the proposed changes). You are not restricted to creating resources only on the cluster, you can use Terraform for any resource that is external to the cluster ( like S3 buckets, RDS instances, anything Terraform can handle). Hope this helps,
  - sgarland 4y ago
    THIS. In my homelab, I spent roughly a day cutting services over from Docker-Compose to Kubernetes. That day included writing the Helm templates for everything, bootstrapping three bare metal nodes with a hypervisor (Proxmox), clustering said hypervisor for HA on its own isolated network, making images and then installing K3OS onto six VMs across the three nodes (3+3 control plane/worker), installing and configuring persistent distributed storage (Longhorn) with backups to a ZFS pool as an NFS target (Debian VM configured via Packer + Ansible), configuring MetalLB, and smoke testing everything. A day's work for one person to accomplish that isn't bad, IMO, but what that doesn't capture is the literal weeks I spent poring over documentation, trying things, running tests, learning what didn't work (Rook + Ceph is a nightmare), and so on. I went so far the day before the cutover as to recreate my homelab in Digital Ocean and run through the entire installation process. Having services that magically work is hard. Having a golden path so you can create a new one with a few clicks is even harder.
- ceejayoz 4y ago
  "Before you begin, make sure your Kubernetes cluster is up and running."
  - bayindirh 4y ago
    "And if it's not, please roll-back your systems and install it from scratch."
- blorenz 4y ago
  I'm expecting you are getting downvoted because of your assumptive and dismissive statement. The type of deployment you are suggesting is overkill to apply k8s as there is minimal to no orchestration.
  - WJW 4y ago
    Also because they are just outright missing the point of the post they were replying to. Yes, deploying thing on kubernetes is pretty simple. Deploying kubernetes itself is definitely not very simple.
    - Thaxll 4y ago
      People are confused, you should most likely not opperate k8s yourself if you don't know or does not have a team for it, nowdays managed k8s are easy to use and operate, with 3 clicks you get a cluster up and running with sane defaults.
      - blorenz 4y ago
        Not to sound trite but I feel like your continued pressing of this issue evokes the entire "Do you even lift, bro?" meme. There are PaaS that can get you up in 3 clicks, but now you are in bed with k8s which is critical for the infrastructure. I think it's quite necessary to definitely understand the inside-and-out of such a vital component for my infra.
- malwrar 4y ago
  Still need to maintain the cluster though, and boy does it require some fun maintenance. Kubernetes is more than just a distributed docker image scheduler, you also need to install and maintain basic dependencies like the pod networking implementation (a whole virtual IP space!!), DNS, loadbalancing, persistent volumes, monitoring, etc etc. Maybe your cloud provider sets it all up for you, but you're not going to escape having to debug and fix issues with that stuff, whether they be show stopping bugs or performance-impacting inconveniences like higher latency.
  - jmillikin 4y ago
    > the pod networking implementation (a whole virtual IP space!!) That part, at least, can be made simple: https://john-millikin.com/stateless-kubernetes-overlay-networks-with-ipv6 https://john-millikin.com/stateless-kubernetes-overlay-netwo... > DNS, loadbalancing, persistent volumes, monitoring, etc etc None of that is part of Kubernetes, and you'll need it (or not) regardless of how you choose to handle process scheduling. There's a sort of common idea that a "Kubernetes cluster" is an entire self-contained PaaS, and that (for example) monitoring in Kubernetes is somehow fundamentally different from what came before. It's easy to fall into the trap of creating an internal clone of Heroku, but Kubernetes itself doesn't require you to do so and it can be a lot faster to just run Nagios (etc).
    - rfoo 4y ago
      > None of that is part of Kubernetes Well, load balancing is part of Kubernetes (except that it does not provide a sane implementation), and tbh having to debug strange failures caused by seemingly-innocent Service configuration is my least-favorite part of Kubernetes. I agree with you on other points.
      - aliswe 4y ago
        DNS too
- ramraj07 4y ago
  Either you’re so good with k8s and are clueless about how rare that expertise is, or more likely, you’re so clueless of how complex and fickle k8s can be if configured wrong but you’re not even aware that you don’t know. I’ve seen actually decent engineers (maybe they’re not decent?) bring down prod because they accidentally Kubectl deploy’ed from their command line.
- bayindirh 4y ago
  You'd weep if you see an actual deployment yaml. What documentation provides is something minimal, and that's very long for something minimal. Also, self-healing can create interesting problems which are fun to trace, debug and understand.
globular-toast 4y ago
I mean, obviously it was designed to reduce human work. A tool that adds work wouldn't be very useful, would it?
- vsareto 4y ago
  I think the selling point here is getting k8s' benefits using other tools would result in more work than k8s
- zerkten 4y ago
  It reduces work, but imposes a high setup and operating cost. Once you get to the right scale then all the economics look correct. Large teams and organizations and absorb that cost more easily than many who are attempting to use k8s. Of course, being open source, anyone can try k8s, but the mileage is going to vary.
- hans1729 4y ago
  > A tool that adds work wouldn't be very useful, would it? Sure could, if it provides benefits that outweight the additional work. Insurance companies will happily pay a couple more IT-specialists if it reduces the amount of cases they have to cover by an arbitrary percentage. Tools very much can serve other purposes than reducing the friction of human attention. If you can containerize your infra and thereby mitigate threat vectors that may put your entire enterprise at risk if exploited, that's a good business call to make, even if the cost of your IT-department gets inflated by some margin.
- chkhd 4y ago
  The key question is what are you comparing to k8s in terms of complexity? Does it add work compared to setting up a VM with Docker and Watchtower? - for sure.. But does it add work compared to setting up something that gives you all the same benefits of k8s without using k8s? - imho definitely not.
  - Thaxll 4y ago
    > all the same benefits without using Kubernetes That's the catch, if running your app on a manually setup VM is equivalent to running your app on Kubernetes then you don't understand what Kuberentes is or provide.
stonemetal12 4y ago
It does save work, consider what you would have to do to provide those benefits without a tool. You would have to monitor usage and spin up new instances manually for scalability. You would have to manually update instances when updates came out. You would have to manually handle failover and stand by instances for availability. Manually configure load balancers. etc. Kubernetes is a great tool if you have scaling management problems.
- nullspace 4y ago
  That's an interesting second-order effect right? Before Kubernetes et al, it was so much work or so complex that few companies implemented those things. Kubernetes saves so much work, that now everyone is doing it and it feels like more work. "Things" here mean different things for different people. For me, it's secrets/certs, better rolling deploys and infra as code.
  - stonemetal12 4y ago
    Yeah, it is a premature optimization issue. Everyone is starting to implement scaling solutions before they have scaling problems.
planetafro 4y ago
I don't know my dude, all 3 major clouds offer "canned" k8s services that you can set up in a ridiculously short amount of time with Terraform and your CI platform of choice. I agree with some other comments in this thread about a general fervor in the Enterprise space to "modernize" needlessly. This conversation usually lands on the company copying what everyone else is doing or what Gartner tells them to do. Cue "DevOps". 100 percent agree with your comments on something simpler. I can't tell you how many times I've debated with our Analytics teams to just use Docker Compose/Swarm.
- Avalaxy 4y ago
  > all 3 major clouds offer "canned" k8s services that you can set up in a ridiculously short amount of time with Terraform and your CI platform of choice I don't agree. I spun up a Kubernetes cluster in Azure, which was indeed easy. But then I had to figure out how to write the correct deployment scripts to deploy my docker containers to it, and how to configure all the security stuff. After more than a week of trying to figure it out, I decided to ditch the whole solution and go for Azure Container Instances instead. It was too much for me to learn about all the concepts of Kubernetes, how you configure them, how to make it work for solutions that are not as simple as the example on the website, and how to navigate through the various different methods of deploying stuff. Maybe I'm just too dumb. But I wasn't going to invest a month of my time into doing something that should be simple enough for an average developer to accomplish.
  - sk8serboi 4y ago
    Similarly, we switched from self-managed k8s on EC2 to Fargate. Took about 2 months part-time with some consultants to get thing squared away. Once we deployed, we ran into all sorts of SRE-issues. Turns out AWS sets all those “sane limits” that our own folks never did. Still hunting ghosts from the rollout 6 months ago. Makes for good resume fodder, and makes me laugh at the prestigious titles and positions the folks who built this system went on to receive at big name firms. Guess it is someone else’s management problem now. :shrug:
  - throwaway894345 4y ago
boredumb 4y ago
The amount of benefits you receive is less than if you worked for the same benefits sans kubernetes. So it definitely saves a ton of work, but your latter point still stands that most of the time the benefits aren't always as... beneficial to some stages of development/scale so it's actually just adding work for the sake of it.
belter 4y ago
Anybody who watched "Kubernetes: The Documentary" knows the answer: https://youtu.be/BE77h7dmoQU https://youtu.be/BE77h7dmoQU Kubernetes only exists, because Google lost the Cloud Wars, and this was their Hail Mary pass.
- everfrustrated 4y ago
  And I might cynically offer it was "invented" to solve the problem of ex-googlers not having any useful immediately transferable skills as the Google internal tech stack had nothing in common with industry.
  - jvolkman 4y ago
    But the internal tech stack doesn't have much in common with Kubernetes, either.
  - jrockway 4y ago
    That's not really been my experience. The number of people who knew how to deploy software at Google was much smaller than the number of people writing software there. I was certainly the only person on my team who ever dealt with the nitty gritty of running stuff in production. That seems about like the industry standard; I'd say the average software engineer in industry, inside or outside Google, considers their work done when their code is merged. At my first job outside of Google we used something called Convox, which was very similar to running things in production at Google. You triggered a package build from your workstation (!) and then adjusted production to pick that up. Very similar to mpm packages and GCL files. (The difference is that Google had some machinery to say "while the build might have been triggered from a workstation, all this code has actually been checked in and reviewed". Convox did not have that part, so yeah, you could just edit some files and push them to production without checking them in, which wasn't great. But when you are a 4 person development team, not the end of the world by any means.)
  - lacker 4y ago
    In 2005 Google search got into a sticky spot, where they were completely unable to deploy new versions of the search engine from the master branch of the code for a while, because there were performance regressions that nobody could find. Deployment and related things like "deploy a copy of the production configuration to these spare 2000 machines" were manual slogs. I was the noob on the team doing such important yet unfulfilling tasks as "backport this Python performance testing script to Python 1.x because our only working test environment doesn't have the same version of Python as production does". This was before borg aka kubernetes, and let me tell you, a whole bunch of stuff was dysfunctional and broken. All this is not to say that Kubernetes is the right solution to your problem, or to almost anyone's problem. It was just an improvement to Google's infrastructure, for the sort of problems that Google had. For some people it makes sense... for you it might not.
picozeta 4y ago
That's actually my point. If Kubernetes does indeed provide the best solution to provide scalability and availability, one can argue that this would result in a decreased demand for dev ops engineers, as they "would just have to use Kubernetes" and it would decrease their workload. In reality this does not seem to be the case, that's why I asked.
- n_e 4y ago
  > If Kubernetes does indeed provide the best solution to provide scalability and availability, one can argue that this would result in a decreased demand for dev ops engineers, as they "would just have to use Kubernetes". I'd say it would result in either: - the same scalability and availability with fewer DevOps engineers - better scalability and availability with a similar number or more DevOps engineers In my experience, it's almost always the second case that happens. For example, a service would be moved from a few (virtualized or physical) servers that can only be scaled manually, to a k8s cluster with either autoscaling or at least scaling by changing a configuration file.
  - throwaway894345 4y ago
    Right. Most companies aren't content to settle for doing the same thing they were doing but with fewer engineers when they realize they could be using those other engineers to automate even more things (via Kubernetes operators).
nine_k 4y ago
It's a good point: tools made by giants for giants, such as Kubernetes, Bazel, etc, may not make sense for a smaller operation. But what would you suggest in lieu of Kubernetes? What would save work for a shop which is not yet a giant but has already overgrown the capabilities of 2-3 manually managed boxes / instances? I can think of several options. Management by Ansible / Saltstack / Chef can easily become a rabbit hole comparable to maintaining K8s's YAML. In your experience, does Nomad save SRE work at smaller scales? Does Terraform? CloudFormation?
- nprateem 4y ago
  A cloud service that can run containers directly, e.g. Amazon ACS, Google Cloud Run, etc.
  - throwaway894345 4y ago
    > e.g. Amazon ACS It's "Amazon ECS", and it works pretty well for standard fare CRUD web services, but more complex use cases quickly end up pulling in more and more AWS services (for example, if you need to run a cron job, you can use EventBridge to trigger an ECS task or just do it all with Lambda). This isn't dramatically worse--it's mostly just different. Kubernetes APIs are more consistent than in AWS, and Kubernetes tends to be more flexible and less limited than AWS. It's also much easier to extend Kubernetes or find a pre-existing extension than in AWS. But mostly it's not going to make or break your project one way or the other. If you're just running CRUD web services, this is fine, but if you need to spin up a background job or do more complex orchestration then it can quickly become advantageous to
- wasmitnetzen 4y ago
  We run Nomad with about 7 to 9 client instances in prod. Quite happy with it.
- neurostimulant 4y ago
  If you only limit yourself to a subset of kubernetes's features, it's actually really great for small operations. I run a small cluster for my side projects consisting of 3 nodes. 1 node is dedicated to run database containers and control plane node (it's also a worker node but strictly for database statefulsets only), and the other 2 as worker nodes, but one of them also run an NFS server that mounted as persistent volumes so every worker nodes can have access to it. Self-healing? Internet-scale? I have on interest for them on my small cluster. I just want my apps to not go down while I'm updating them without writing complicated blue-green deployment scripts, ability to move around pods to other nodes when one got overloaded, and ability to add or remove nodes when needed without starting again from scratch. I basically treat it like docker-compose on steroid. So far it works really well.
glintik 4y ago
> Kubernetes doesn't save work, it adds work. It saves, but for certain heavy scenarios like High Availability, autoscaling etc. I hope, one day we can see Kubernetes simplified, that will fit 95% of SOHO users.
namaria 4y ago
This is exactly what the "automation destroys jobs" argument gets wrong. When aggregate output becomes cheaper, aggregate demand increases.
spyspy 4y ago
> Most people using Kubernetes should be using something simpler, but that's a separate problem. This has been the biggest drain on my career. Everyone wants to be an "engineer" ready to handle every problem like its the next facebook. Like bruh, this service is going to get like 100 req/hour max and only when the sun is up - just **ing throw it on cloud run. We can tell the only thing you want to build is your resume.
tyingq 4y ago
My favorite part of this is that now K8S by itself isn't enough. People are laying "service mesh" and other things on top of it. And, yeah, I get the sales pitch for it, but at some point, it's too much.
bborud 4y ago
It could have been designed better to be easier to set up and use. Especially for ad-hoc or more casual use. If it is easy to use casually, people will use it more and they will learn how to use it faster. It is also easier to see why you would invest in learning to create more complex configurations. Pretty much every piece of software I've written in the past decade that tends to have configs in production can also work with little or no config. Including clustered software that just exploits things that are widely available like Zeroconf to get a decent low effort cluster up and running. No, you probably won't be able to (or want to) use those features in production, but that's beside the point. The point is to lower the thresholds. And then keep aiming at lowering them wherever you can because asking other people to care about your software is seriously uncool. Other people will never care as much as you do. It is normal for programmers to become defensive about software. Be it their own or software they really like. But it is far more productive to assume that when users think something is awkward, perhaps it is because it is awkward. And perhaps it could have been done better. Nobody actually gives a crap what someone thinks Kubernetes was built for -- and what kind of rubbish experience they think is deeply justified by the goals. It is either needlessly awkward to use or it isn't. And guess what: most people think it is awkward. And I seriously doubt it needs to be this awkward.
- jljljl 4y ago
  Most companies I've seen end up building some kind of framework around Kubernetes to make the developer experience tolerable. The threshold for getting started and getting a basic deployment up and running is way too high. Of course, there's a huge cost to building your own framework as well... And it's easy to get wrong. I started https://www.jetpack.io/ https://www.jetpack.io/ recently to try and build a better solution on top of Kubernetes. If you're interested in checking it out and giving us early feedback, you can signup on the website, or email us at `demo at jetpack.io`
  - bborud 4y ago
    Thanks, I will have a look!
- devonbleak 4y ago
50yearsold 4y ago
Just use AWS ECS Fargate - probably much easier to manage/setup than your own k8s cluster or even ECS k8s route. Azure/GCP may provide something similar. I think any Google product is always needlessly complex that only solves their own use case and probably many other companies need not blindly adopt those unless absolutely necessary.
varispeed 4y ago
It also by design locks you in one of the cloud providers in a quite deceptive way. There is no singular interface for networking, so you cannot really port your infrastructure from cloud to bare metal for instance. Bare metal also was heavily underdeveloped, so setting it up was fragile and fiddly and quite limited.
dsr_ 4y ago
The thread here full of people incorrecting each other about how to use Kubernetes seems pretty solid evidence for your thesis "it adds work".
aidenn0 4y ago
What is this "something simpler"? I have to manage like half a dozen docker images. K8s seems like a massive overkill, but managing by hand is rather error-prone.
engineerDave 4y ago
Agreed. Kube makes the scaling out easier not the bootstrapping. An important difference.
KaiserPro 4y ago
> It was designed for scalability and availability more than anything. yet its really not great at the first, and I'm strongly suspicious about the latter claim. Its very simple to make boneheaded decisions about networking that makes thing fragile. In terms of scale, you have a limit of 5k nodes. given how fucking chatty it is, it costs a lot to run loads of nodes.

Thaxll 4y ago

> Have you ever tried setting up a Kubernetes cluster and deploying apps in it? Kubernetes doesn't save work This is wrong, deploying on Kubernetes is easy and quick for most apps, you have one docker image one deployment spec and that's it. https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#creating-a-deployment https://kubernetes.io/docs/concepts/workloads/controllers/de...

caymanjim 4y ago
And a service configuration, ingress, other networking, persistent volumes, a mechanism for updating deployed applications, management of the nodes (even with a managed service like EKS or other cloud solutions), logging, roles, security, etc. If you're a developer and all you have to worry about is one deployment spec, thank your devops team for making your life easier. Kubernetes is great for making the dev team's life easier, but someone did a lot of work to make it that easy for you.
- spion 4y ago
  This is still easier with Kubernetes than with other tools. Installing e.g. fluentd and Kibana is again a configuration / chart which you apply to the cluster. For monitoring and visualization again you have an operator / chart that you can apply to the cluster. Yes there is a lot of complexity and learning involved but overall I was still pretty amazed at how quickly I was able to get a cluster up to speed with the available tools.
- patrickserrano 4y ago
  Couldn't agree with this more. At my last company there was a fair bit of time put into making the deployment process for our microservices as simple as added a new YAML file to the deployment repo. That file pulled a custom chart, and as a dev you just needed to configure a few variables to get everything up an running. But if you were deploying something that couldn't use one of the pre-configured charts it was a bit more work, especially if you've never done it be hand before. Probably 98% of the devs were blissfully unaware of that complexity that the charts abstracted, and it let them focus on the services they were writing. I wasn't one of them, and always made sure to thank the devops team for simplifying the day to day deployments whenever I had to deal with writing a custom one.
  - iteratorx 4y ago
    You can now do something similar with Bunnyshell.com It Handles the devops jobs for dev teams. Full disclosure, I work for Bunnyshell.
    - macintux 4y ago
      In the absence of any description of what “something similar” or “handles the devops jobs” actually means, this comes across as spam, not informative.
      - iteratorx 4y ago
        Sorry about that, I should have been more informative. Bunnyshell makes it easy to create and manage environments. (EaaS - environments as a service) You connect your k8s cluster(s) and git accounts/repos, it reads the docker-compose files and creates deployments on the cluster. You don’t need to know or write Kubernetes manifests, those are created for you. You also get auto updates and ephemeral/preview environments (when a PR is created against the branch of your env, Bunnyshell deploys a new env with the proposed changes). You are not restricted to creating resources only on the cluster, you can use Terraform for any resource that is external to the cluster ( like S3 buckets, RDS instances, anything Terraform can handle). Hope this helps,
- sgarland 4y ago
  THIS. In my homelab, I spent roughly a day cutting services over from Docker-Compose to Kubernetes. That day included writing the Helm templates for everything, bootstrapping three bare metal nodes with a hypervisor (Proxmox), clustering said hypervisor for HA on its own isolated network, making images and then installing K3OS onto six VMs across the three nodes (3+3 control plane/worker), installing and configuring persistent distributed storage (Longhorn) with backups to a ZFS pool as an NFS target (Debian VM configured via Packer + Ansible), configuring MetalLB, and smoke testing everything. A day's work for one person to accomplish that isn't bad, IMO, but what that doesn't capture is the literal weeks I spent poring over documentation, trying things, running tests, learning what didn't work (Rook + Ceph is a nightmare), and so on. I went so far the day before the cutover as to recreate my homelab in Digital Ocean and run through the entire installation process. Having services that magically work is hard. Having a golden path so you can create a new one with a few clicks is even harder.
ceejayoz 4y ago
"Before you begin, make sure your Kubernetes cluster is up and running."
- bayindirh 4y ago
  "And if it's not, please roll-back your systems and install it from scratch."
blorenz 4y ago
I'm expecting you are getting downvoted because of your assumptive and dismissive statement. The type of deployment you are suggesting is overkill to apply k8s as there is minimal to no orchestration.
- WJW 4y ago
  Also because they are just outright missing the point of the post they were replying to. Yes, deploying thing on kubernetes is pretty simple. Deploying kubernetes itself is definitely not very simple.
  - Thaxll 4y ago
    People are confused, you should most likely not opperate k8s yourself if you don't know or does not have a team for it, nowdays managed k8s are easy to use and operate, with 3 clicks you get a cluster up and running with sane defaults.
    - blorenz 4y ago
      Not to sound trite but I feel like your continued pressing of this issue evokes the entire "Do you even lift, bro?" meme. There are PaaS that can get you up in 3 clicks, but now you are in bed with k8s which is critical for the infrastructure. I think it's quite necessary to definitely understand the inside-and-out of such a vital component for my infra.
malwrar 4y ago
Still need to maintain the cluster though, and boy does it require some fun maintenance. Kubernetes is more than just a distributed docker image scheduler, you also need to install and maintain basic dependencies like the pod networking implementation (a whole virtual IP space!!), DNS, loadbalancing, persistent volumes, monitoring, etc etc. Maybe your cloud provider sets it all up for you, but you're not going to escape having to debug and fix issues with that stuff, whether they be show stopping bugs or performance-impacting inconveniences like higher latency.
- jmillikin 4y ago
  > the pod networking implementation (a whole virtual IP space!!) That part, at least, can be made simple: https://john-millikin.com/stateless-kubernetes-overlay-networks-with-ipv6 https://john-millikin.com/stateless-kubernetes-overlay-netwo... > DNS, loadbalancing, persistent volumes, monitoring, etc etc None of that is part of Kubernetes, and you'll need it (or not) regardless of how you choose to handle process scheduling. There's a sort of common idea that a "Kubernetes cluster" is an entire self-contained PaaS, and that (for example) monitoring in Kubernetes is somehow fundamentally different from what came before. It's easy to fall into the trap of creating an internal clone of Heroku, but Kubernetes itself doesn't require you to do so and it can be a lot faster to just run Nagios (etc).
  - rfoo 4y ago
    > None of that is part of Kubernetes Well, load balancing is part of Kubernetes (except that it does not provide a sane implementation), and tbh having to debug strange failures caused by seemingly-innocent Service configuration is my least-favorite part of Kubernetes. I agree with you on other points.
    - aliswe 4y ago
      DNS too
ramraj07 4y ago
Either you’re so good with k8s and are clueless about how rare that expertise is, or more likely, you’re so clueless of how complex and fickle k8s can be if configured wrong but you’re not even aware that you don’t know. I’ve seen actually decent engineers (maybe they’re not decent?) bring down prod because they accidentally Kubectl deploy’ed from their command line.
bayindirh 4y ago
You'd weep if you see an actual deployment yaml. What documentation provides is something minimal, and that's very long for something minimal. Also, self-healing can create interesting problems which are fun to trace, debug and understand.

globular-toast 4y ago

I mean, obviously it was designed to reduce human work. A tool that adds work wouldn't be very useful, would it?

vsareto 4y ago
I think the selling point here is getting k8s' benefits using other tools would result in more work than k8s
zerkten 4y ago
It reduces work, but imposes a high setup and operating cost. Once you get to the right scale then all the economics look correct. Large teams and organizations and absorb that cost more easily than many who are attempting to use k8s. Of course, being open source, anyone can try k8s, but the mileage is going to vary.
hans1729 4y ago
> A tool that adds work wouldn't be very useful, would it? Sure could, if it provides benefits that outweight the additional work. Insurance companies will happily pay a couple more IT-specialists if it reduces the amount of cases they have to cover by an arbitrary percentage. Tools very much can serve other purposes than reducing the friction of human attention. If you can containerize your infra and thereby mitigate threat vectors that may put your entire enterprise at risk if exploited, that's a good business call to make, even if the cost of your IT-department gets inflated by some margin.
chkhd 4y ago
The key question is what are you comparing to k8s in terms of complexity? Does it add work compared to setting up a VM with Docker and Watchtower? - for sure.. But does it add work compared to setting up something that gives you all the same benefits of k8s without using k8s? - imho definitely not.
- Thaxll 4y ago
  > all the same benefits without using Kubernetes That's the catch, if running your app on a manually setup VM is equivalent to running your app on Kubernetes then you don't understand what Kuberentes is or provide.

stonemetal12 4y ago

It does save work, consider what you would have to do to provide those benefits without a tool. You would have to monitor usage and spin up new instances manually for scalability. You would have to manually update instances when updates came out. You would have to manually handle failover and stand by instances for availability. Manually configure load balancers. etc. Kubernetes is a great tool if you have scaling management problems.

nullspace 4y ago
That's an interesting second-order effect right? Before Kubernetes et al, it was so much work or so complex that few companies implemented those things. Kubernetes saves so much work, that now everyone is doing it and it feels like more work. "Things" here mean different things for different people. For me, it's secrets/certs, better rolling deploys and infra as code.
- stonemetal12 4y ago
  Yeah, it is a premature optimization issue. Everyone is starting to implement scaling solutions before they have scaling problems.

planetafro 4y ago

I don't know my dude, all 3 major clouds offer "canned" k8s services that you can set up in a ridiculously short amount of time with Terraform and your CI platform of choice. I agree with some other comments in this thread about a general fervor in the Enterprise space to "modernize" needlessly. This conversation usually lands on the company copying what everyone else is doing or what Gartner tells them to do. Cue "DevOps". 100 percent agree with your comments on something simpler. I can't tell you how many times I've debated with our Analytics teams to just use Docker Compose/Swarm.

Avalaxy 4y ago
> all 3 major clouds offer "canned" k8s services that you can set up in a ridiculously short amount of time with Terraform and your CI platform of choice I don't agree. I spun up a Kubernetes cluster in Azure, which was indeed easy. But then I had to figure out how to write the correct deployment scripts to deploy my docker containers to it, and how to configure all the security stuff. After more than a week of trying to figure it out, I decided to ditch the whole solution and go for Azure Container Instances instead. It was too much for me to learn about all the concepts of Kubernetes, how you configure them, how to make it work for solutions that are not as simple as the example on the website, and how to navigate through the various different methods of deploying stuff. Maybe I'm just too dumb. But I wasn't going to invest a month of my time into doing something that should be simple enough for an average developer to accomplish.
- sk8serboi 4y ago
  Similarly, we switched from self-managed k8s on EC2 to Fargate. Took about 2 months part-time with some consultants to get thing squared away. Once we deployed, we ran into all sorts of SRE-issues. Turns out AWS sets all those “sane limits” that our own folks never did. Still hunting ghosts from the rollout 6 months ago. Makes for good resume fodder, and makes me laugh at the prestigious titles and positions the folks who built this system went on to receive at big name firms. Guess it is someone else’s management problem now. :shrug:
- throwaway894345 4y ago

boredumb 4y ago

The amount of benefits you receive is less than if you worked for the same benefits sans kubernetes. So it definitely saves a ton of work, but your latter point still stands that most of the time the benefits aren't always as... beneficial to some stages of development/scale so it's actually just adding work for the sake of it.

belter 4y ago

Anybody who watched "Kubernetes: The Documentary" knows the answer: https://youtu.be/BE77h7dmoQU https://youtu.be/BE77h7dmoQU Kubernetes only exists, because Google lost the Cloud Wars, and this was their Hail Mary pass.

everfrustrated 4y ago
And I might cynically offer it was "invented" to solve the problem of ex-googlers not having any useful immediately transferable skills as the Google internal tech stack had nothing in common with industry.
- jvolkman 4y ago
  But the internal tech stack doesn't have much in common with Kubernetes, either.
- jrockway 4y ago
  That's not really been my experience. The number of people who knew how to deploy software at Google was much smaller than the number of people writing software there. I was certainly the only person on my team who ever dealt with the nitty gritty of running stuff in production. That seems about like the industry standard; I'd say the average software engineer in industry, inside or outside Google, considers their work done when their code is merged. At my first job outside of Google we used something called Convox, which was very similar to running things in production at Google. You triggered a package build from your workstation (!) and then adjusted production to pick that up. Very similar to mpm packages and GCL files. (The difference is that Google had some machinery to say "while the build might have been triggered from a workstation, all this code has actually been checked in and reviewed". Convox did not have that part, so yeah, you could just edit some files and push them to production without checking them in, which wasn't great. But when you are a 4 person development team, not the end of the world by any means.)
- lacker 4y ago
  In 2005 Google search got into a sticky spot, where they were completely unable to deploy new versions of the search engine from the master branch of the code for a while, because there were performance regressions that nobody could find. Deployment and related things like "deploy a copy of the production configuration to these spare 2000 machines" were manual slogs. I was the noob on the team doing such important yet unfulfilling tasks as "backport this Python performance testing script to Python 1.x because our only working test environment doesn't have the same version of Python as production does". This was before borg aka kubernetes, and let me tell you, a whole bunch of stuff was dysfunctional and broken. All this is not to say that Kubernetes is the right solution to your problem, or to almost anyone's problem. It was just an improvement to Google's infrastructure, for the sort of problems that Google had. For some people it makes sense... for you it might not.

picozeta 4y ago

That's actually my point. If Kubernetes does indeed provide the best solution to provide scalability and availability, one can argue that this would result in a decreased demand for dev ops engineers, as they "would just have to use Kubernetes" and it would decrease their workload. In reality this does not seem to be the case, that's why I asked.

n_e 4y ago
> If Kubernetes does indeed provide the best solution to provide scalability and availability, one can argue that this would result in a decreased demand for dev ops engineers, as they "would just have to use Kubernetes". I'd say it would result in either: - the same scalability and availability with fewer DevOps engineers - better scalability and availability with a similar number or more DevOps engineers In my experience, it's almost always the second case that happens. For example, a service would be moved from a few (virtualized or physical) servers that can only be scaled manually, to a k8s cluster with either autoscaling or at least scaling by changing a configuration file.
- throwaway894345 4y ago
  Right. Most companies aren't content to settle for doing the same thing they were doing but with fewer engineers when they realize they could be using those other engineers to automate even more things (via Kubernetes operators).

nine_k 4y ago

It's a good point: tools made by giants for giants, such as Kubernetes, Bazel, etc, may not make sense for a smaller operation. But what would you suggest in lieu of Kubernetes? What would save work for a shop which is not yet a giant but has already overgrown the capabilities of 2-3 manually managed boxes / instances? I can think of several options. Management by Ansible / Saltstack / Chef can easily become a rabbit hole comparable to maintaining K8s's YAML. In your experience, does Nomad save SRE work at smaller scales? Does Terraform? CloudFormation?

nprateem 4y ago
A cloud service that can run containers directly, e.g. Amazon ACS, Google Cloud Run, etc.
- throwaway894345 4y ago
  > e.g. Amazon ACS It's "Amazon ECS", and it works pretty well for standard fare CRUD web services, but more complex use cases quickly end up pulling in more and more AWS services (for example, if you need to run a cron job, you can use EventBridge to trigger an ECS task or just do it all with Lambda). This isn't dramatically worse--it's mostly just different. Kubernetes APIs are more consistent than in AWS, and Kubernetes tends to be more flexible and less limited than AWS. It's also much easier to extend Kubernetes or find a pre-existing extension than in AWS. But mostly it's not going to make or break your project one way or the other. If you're just running CRUD web services, this is fine, but if you need to spin up a background job or do more complex orchestration then it can quickly become advantageous to
wasmitnetzen 4y ago
We run Nomad with about 7 to 9 client instances in prod. Quite happy with it.
neurostimulant 4y ago
If you only limit yourself to a subset of kubernetes's features, it's actually really great for small operations. I run a small cluster for my side projects consisting of 3 nodes. 1 node is dedicated to run database containers and control plane node (it's also a worker node but strictly for database statefulsets only), and the other 2 as worker nodes, but one of them also run an NFS server that mounted as persistent volumes so every worker nodes can have access to it. Self-healing? Internet-scale? I have on interest for them on my small cluster. I just want my apps to not go down while I'm updating them without writing complicated blue-green deployment scripts, ability to move around pods to other nodes when one got overloaded, and ability to add or remove nodes when needed without starting again from scratch. I basically treat it like docker-compose on steroid. So far it works really well.

glintik 4y ago

> Kubernetes doesn't save work, it adds work. It saves, but for certain heavy scenarios like High Availability, autoscaling etc. I hope, one day we can see Kubernetes simplified, that will fit 95% of SOHO users.

namaria 4y ago

This is exactly what the "automation destroys jobs" argument gets wrong. When aggregate output becomes cheaper, aggregate demand increases.

spyspy 4y ago

> Most people using Kubernetes should be using something simpler, but that's a separate problem. This has been the biggest drain on my career. Everyone wants to be an "engineer" ready to handle every problem like its the next facebook. Like bruh, this service is going to get like 100 req/hour max and only when the sun is up - just **ing throw it on cloud run. We can tell the only thing you want to build is your resume.

tyingq 4y ago

My favorite part of this is that now K8S by itself isn't enough. People are laying "service mesh" and other things on top of it. And, yeah, I get the sales pitch for it, but at some point, it's too much.

bborud 4y ago

It could have been designed better to be easier to set up and use. Especially for ad-hoc or more casual use. If it is easy to use casually, people will use it more and they will learn how to use it faster. It is also easier to see why you would invest in learning to create more complex configurations. Pretty much every piece of software I've written in the past decade that tends to have configs in production can also work with little or no config. Including clustered software that just exploits things that are widely available like Zeroconf to get a decent low effort cluster up and running. No, you probably won't be able to (or want to) use those features in production, but that's beside the point. The point is to lower the thresholds. And then keep aiming at lowering them wherever you can because asking other people to care about your software is seriously uncool. Other people will never care as much as you do. It is normal for programmers to become defensive about software. Be it their own or software they really like. But it is far more productive to assume that when users think something is awkward, perhaps it is because it is awkward. And perhaps it could have been done better. Nobody actually gives a crap what someone thinks Kubernetes was built for -- and what kind of rubbish experience they think is deeply justified by the goals. It is either needlessly awkward to use or it isn't. And guess what: most people think it is awkward. And I seriously doubt it needs to be this awkward.

jljljl 4y ago
Most companies I've seen end up building some kind of framework around Kubernetes to make the developer experience tolerable. The threshold for getting started and getting a basic deployment up and running is way too high. Of course, there's a huge cost to building your own framework as well... And it's easy to get wrong. I started https://www.jetpack.io/ https://www.jetpack.io/ recently to try and build a better solution on top of Kubernetes. If you're interested in checking it out and giving us early feedback, you can signup on the website, or email us at `demo at jetpack.io`
- bborud 4y ago
  Thanks, I will have a look!
devonbleak 4y ago

50yearsold 4y ago

Just use AWS ECS Fargate - probably much easier to manage/setup than your own k8s cluster or even ECS k8s route. Azure/GCP may provide something similar. I think any Google product is always needlessly complex that only solves their own use case and probably many other companies need not blindly adopt those unless absolutely necessary.

varispeed 4y ago

It also by design locks you in one of the cloud providers in a quite deceptive way. There is no singular interface for networking, so you cannot really port your infrastructure from cloud to bare metal for instance. Bare metal also was heavily underdeveloped, so setting it up was fragile and fiddly and quite limited.

dsr_ 4y ago

The thread here full of people incorrecting each other about how to use Kubernetes seems pretty solid evidence for your thesis "it adds work".

aidenn0 4y ago

What is this "something simpler"? I have to manage like half a dozen docker images. K8s seems like a massive overkill, but managing by hand is rather error-prone.

engineerDave 4y ago

Agreed. Kube makes the scaling out easier not the bootstrapping. An important difference.

KaiserPro 4y ago

> It was designed for scalability and availability more than anything. yet its really not great at the first, and I'm strongly suspicious about the latter claim. Its very simple to make boneheaded decisions about networking that makes thing fragile. In terms of scale, you have a limit of 5k nodes. given how fucking chatty it is, it costs a lot to run loads of nodes.