16 ms·

Falling for Kubernetes

164 points by icyfox 4y ago

anotherhue 4y ago
Vanilla k8s is pretty good. But once the 8 trillion vendors have you 'curl | helm' ing you end up with a knot of a system. Keep it simple, use GitOps (ArgoCD is great), let k8s do what it's good at, managing workloads, not as a delivery mechanism for a vendor. As an aside, the existence of the '{{ | indent 4 }}' function in helm should disqualify it from any serious use. Render, don't template.
- p_l 4y ago
  I judiciously delete any and all helm I can find, and fight text-based templating where possible :/ Especially since k8s doesn't even use "text" for the format, it's just happens that JSON is the serialization, use datastructures and serialize them dammit :/
- bushbaba 4y ago
  I'm fairly convinced that helm in production is an anti-pattern. With you instead having all K8s manifests checked into your Git repository with CI/CD to handle any changes. Helm just has too much auto-magic and remove's one of k8s best features, the git-blame / git-diff for Infrastructure.
  - blamarvt 4y ago
    I don't understand why this isn't the prevailing philosophy. I'm over here Terraforming helm deployments and having no idea what is actually happening under the covers.
    - cortesoft 4y ago
      We have our CI run a helm —-dry-run —-debug before the deploy step, so we can see the exact kubernetes resources that are going to be applied. You can even save that as an artifact so you can diff it later.
      - a012 4y ago
        There's Helm plugin (https://github.com/databus23/helm-diff https://github.com/databus23/helm-diff) that show diff results for you, for example helm diff upgrade <release> <repo> --namespace <namespace>
  - the_duke 4y ago
    Helm still saves you an incredible amount of work for setting up all the third party services. If your team is big enough you can just write your own configs, but that takes a lot of time and often quite a bit of knowledge about the relevant service. Rollbacks and resource cleanup is not to be underestimated either if you are not getting that from other tools like Argo. Note: You can still use the Git-centric approach by generating configuration via `helm template`.
    - 0x457 4y ago
      I'm convinced that the only reason Helm saves you time with 3rd party services is because 3rd party services only provide Helm chart or don't provide means of deployment to k8s at all. I've been using kustomize to deploy some things with ArgoCD and it's so much easier. Now, I'm trying to never use helm for 3rd party. However, for your internal things, Helm is hard to replace. It's easy to start a chart that is capable of deploying most of your internal services, but maintaining it is a nightmare. Actually, using helm, as in `helm` the binary, directly sounds bonkers to me and I wouldn't wish this upon anyone.
  - FridgeSeal 4y ago
    I use Helm only for installing “supporting applications” -elastic searchoperator, jupyterHub, etc. Our normal deployments are standard K8s configs. These apps use Helm because a lot of the settings are complicated, Co dependent, etc. Absolutely would not write helm charts from scratch for normal deployments, and if I got these apps in a better format than helm, I’d probably drop it immediately.
    - bottled_poe 4y ago
      It’s the fact that they are complicated which warrants manual configuration. If the org can’t write the configuration, how will they support it when something goes wrong? It’s a problem waiting to happen.
    - FridgeSeal 4y ago
      I can't edit this anymore - but to anyone reading, I ought to specify that I do _not_ use Helm directly, only via Flux, our CD tool. It ablates a lot of the issues of dealing with helm charts.
- halfmatthalfcat 4y ago
  I don't understand this comment. How else are you going to deploy pieces of k8s infra into k8s if not with Helm and Helm Charts? Sure, you can use Argo to deploy and sync Helm charts into k8s but...you're still going to be using Helm (if not indirectly via Argo) and you will inevitably need to template things that need to be dynamically configured at render-time.
  - justinsaccount 4y ago
    See https://xeiaso.net/blog/dhall-kubernetes-2020-01-25 https://xeiaso.net/blog/dhall-kubernetes-2020-01-25
    - halfmatthalfcat 4y ago
      So instead of Go Templates, I'm going to use Dhall? Why? I'd be losing interop with the entire Helm ecosystem too, so there go dependencies and such to maintain my IaC. That blog post doesn't alleviate any issues one might have using the traditional Go Templates + Helm.
  - qbasic_forever 4y ago
    Just kubectl apply the manifests. You can even use kubectl -k for the Kustomize configuration engine that can more or less replace most of what helm does today.
    - halfmatthalfcat 4y ago
      So what, I'm going to have a big Makefile or something with a bunch of kubectl applies? For each environment too? What if one of my dependencies (cert-manager for example) doesn't support directly applying via kubectl but have to be rendered with Helm? How do I manage versions of these dependencies too? For better or for worse Helm is the defacto standard for deploying into k8s. Kustomizing toy apps or simple things may work but I have yet to see a large production stack use anything but Helm.
      - 4y ago
- n3t 4y ago
  > As an aside, the existence of the '{{ | indent 4 }}' function in helm should disqualify it from any serious use. Render, don't template. This. My first thought when I saw the indentation hack was "it can't be serious, production-ready software". My take on this is as follows. If you have a simple use case, write your K8s manifests directly. If you have a complex use case, Helm is often more pain than it's worth. Use alternatives, for example Jsonnet[0] with kubecfg[1]. Or emit manifests from your language of choice. Just don't use Helm. [0]: https://jsonnet.org/ https://jsonnet.org/ [1]: https://github.com/kubecfg/kubecfg https://github.com/kubecfg/kubecfg
  - 015a 4y ago
    For a complex use case: cdk8s is cool https://cdk8s.io/ https://cdk8s.io/
  - vlunkr 4y ago
    It is shocking that such a clearly bad design choice has stuck, and that Helm has become so popular in spite of it. I've had my eye on jsonnet, I'll have to try that next time I do something in k8s.
    - runevault 4y ago
      We are using helm at work but I've never touched it so I cannot comment on it being bad. However cargo cult habits have made quite a few technologies over the years take off when they never should have just because someone with either the right sort of intelligence and/or charisma made people believe it was good, and then the ripple effect made it take off.
      - nfhshy68 4y ago
        I had someone on here assume we were a small dev shop because we don't use microservices the other day. I don't think they were expecting 2k devs to be what I responded with. Apparently monoliths a scale a lot bigger than you'd think.
    - 0xbadcafebee 4y ago
- topspin 4y ago
  > not as a delivery mechanism for a vendor Amen. I got turned off from k8s following a tutorial that used Helm. I ran it and watched mysteries and line noise scroll past and walked away for a year. I thought "no, I will never put this squirming mass of tapeworms in front of anyone." Then I took up with k3s and got underway.
- jrockway 4y ago
  > As an aside, the existence of the '{{ | indent 4 }}' function in helm should disqualify it from any serious use. Render, don't template. Yeah, I think helm will be the death of Kubernetes. Some other workload management tool will come out, it will have a computer-science-inspired templating system (think Lisp macros, not C #defines) that is also easy to use, and the increased maintainability will be a breath of fresh air to people in helm hell and they'll switch over, even if the workload management fundamentals are worse. It is a shame that ksonnet was abandoned. jsonnet is a very pleasant programming language for problems of similar scope. I think that people have to see some adoption or they give up; so Helm stays motivated to continue existing despite the fact that the design is fundamentally flawed, while alternatives give up when nobody uses them. If you're looking for a lesson in "worse is better", look no farther from helm. Easy things are hard, but everything is possible. That turns out to be important. I also liked kustomize a lot, but it definitely sacrificed features for security and a clean overall design posture. I don't really know what it's missing, but it must be missing something. (I use kustomize for everything, but it seems rare in the industry. And I don't use it for complicated things like "make a namespace and app deployment for every PR"; I think to manage things like that it's missing features people want. I say just run your development branch on your workstation and commit it when it works. Running the app you write shouldn't be rocket science, and so a computer program that configures it shouldn't be necessary. The industry definitely disagrees with me there, though.) One of the biggest problems I have with helm is that it's not easy to make local modifications to charts. That needs to be a first class feature, and something everyone knows how to use. As a vendor who ships a helm chart, I feel like almost all of my work these days is hearing from users "our k8s security team requires that every manifest set field X", and then I have to go add that for them and release it. If kustomize were the default, they'd just add that themselves instead of asking. But hey, opportunity to provide good customer service, I guess. (Luckily, most of the requests are compatible with each other. At one point we ran as root because that's the default, a customer required running as a user, and now everyone can run a locked down version; nobody NEEDED to run as root. So these requests generally improve the product, but it's a little tedious to make them ask.)
  - moondev 4y ago
    kustomize supports using a helm chart as a resource https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/helmcharts/ https://kubectl.docs.kubernetes.io/references/kustomize/kust...
- whalesalad 4y ago
  I really think this is why most people dislike and misunderstand the value of kube. If you raw dog it, it’s pretty amazing what you can build. It’s not very hard to roll your own Heroku (tailored to your workflows and workloads) with kube, if you shy away from all the noise of helm and other vendors as you say.
umvi 4y ago
> This deployment needed to serve docker images on boot-up, which instance templates do support. However they don't support them via API, only in the web console or the CLI. Not exactly true. I got around this limitation by sending a startup script[1] in API metadata which basically just does invokes `docker run ...` and it works just fine. This allows spinning up/down container-based VMs via API only which is nice. [1] https://cloud.google.com/compute/docs/instances/startup-scripts/linux https://cloud.google.com/compute/docs/instances/startup-scri...
- dilyevsky 4y ago
  Yeah you can do the same thing via any image that supports cloud-init or ignition (eg fedora coreos) and have the same exact setup deployed in almost any cloud
throwaway787544 4y ago
> Bare metal is always my first choice both for past projects and startups. Is this a new Tech Hipster thing? Like writing a letter with a typewriter rather than an ink pen or computer/printer? "You don't understand, man; a virtual machine is, like, not authentic, man."
- EddySchauHai 4y ago
  Bare metal can still have containers, I assumed it just meant on their own hardware?
  - BoorishBears 4y ago
    Nope, you can follow the link where they describe the stack and they're very proud of the exclusion of containerization
    - EddySchauHai 4y ago
      Ah! I'm all for owning hardware but docker is the best tech from the last couple years IMO for development.
    - icyfox 4y ago
      OP here. To clarify, the blog doesn't use containers just because it didn't really call for it. Ended up creating a simpler CI pipeline to just push updated static files directly. For everything else that I work on, I'm 100% in on containers. Just those containers running mainly on bare metal and newly on k8.
- pm90 4y ago
  There's benefits to using bare metal, but generally only for situations where you really need them. e.g. if you want to run hardware accelerated apps, run your own virtualization/VMM (like firecracker) etc. So yes, for the most part it is a hipster thing.
  - heavenlyblue 4y ago
umvi 4y ago
My problem with Kubernetes is my problem with with front-end web frameworks - they introduce too much complexity to the point of being esoteric for simple systems. If you have a simple website built on boring technologies like HTML, CSS, and vanilla JS, then nearly anyone can read, understand, and make changes to it, even backend developers. If you instead wrote it in React/Webpack/etc. then suddenly only frontend experts can understand and contribute and debug. Same with k8s. If you make a cloud backend using boring technologies like plain old programs calling APIs then nearly anyone can read, understand, and make changes to it. But if you instead make it use a big pile of configuration files with a million switches then suddenly only k8s experts can understand and contribute and debug. I'm not saying don't use ReactJS or Kubernetes, I'm just saying make sure that the benefits you get from switching to it outweigh the new complexity and therefore expertise required to understand and debug it.
- initplus 4y ago
  K8s provides a cloud-agnostic solution for (mostly stateless) configuration management. You have to record which ports your applications listen on, which subdomains are used for what, how to access that storage bucket with the data files in it etc. somewhere. The moment you have more than a couple of individual services managing this all by hand becomes a nightmare. You need a really strong culture of good practice to make sure that the exact production config actually ends up in source control. You can be a k8s expert, or an expert in your own home-spun configuration management/git-ops workflow. But not having any workflow here just isn't an option in my opinion. You too easily end up with spaghetti infrastructure that everybody is too scared to touch. With k8s I can take my repo of yaml and change cloud providers in an afternoon.
  - XorNot 4y ago
    The problem is kubernetes doesn't include any notions of obvious things I'd like to do like parameterizing my deployment - even though all the providers want to sell me kubernetes clusters as a service.
    - FridgeSeal 4y ago
tillmannhgl 4y ago
This sounded like an article from 2017.
- yjftsjthsd-h 4y ago
  What's changed since?
topspin 4y ago
"Azure is the only major provider that still has free control panels" Oracle Cloud Infrastructure does as well. Perhaps it does not yet qualify as major... It's major to Oracle, that's for sure.
kelnos 4y ago
> Keep it managed since that legitimately becomes a reliability headache. This is the thing that I think will always give me pause. If I have to pay a third party to manage my cluster orchestration backplane, that seems like a pretty big piece of overhead. Sure, I can do it myself, but then I have to deal with said reliability headache. It seems ironic that a cluster management framework -- that touts its ability to reliably keep your applications running -- has its own reliability issues.
- yunwal 4y ago
  I don't disagree with you, but if you can reliably trade your dataplane outages for control plane outages, that's still usually a good tradeoff.
  - kelnos 4y ago
    That's a really good point. Certainly you don't want either to go down, but outages are more tolerable if customer requests are still getting serviced regardless.
- lmarcos 4y ago
  Exactly. We are building these incredible open source tools... but they grow so complex that we need to pay others in order to use them effectively? What would you say if you had to pay Google if you want to use Golang in an effective way (because the language has become so complex that it's difficult to handle it on your own?). Crazy. I wanted to take a look at how to use K8s on my own cluster, and damn it, to install the whole thing is not that straightforward. So, now to keep my sanity I need to pay a cloud provider to use k8s! I guess that's the trick: build some open source monster that's very hard to install/maintain/use but has cool features. People will love it and they'll pay for it.
  - vbezhenar 4y ago
    I think people are overestimating the difficulty of setting up k8s. It's not that hard. Grab Debian, install containerd.io, set one sysctl, load one kernel module and you're all set. Install few more packages, run kubeadm init and that's all. The only thing that I've found truly hard is autoscaling VMs. Managed clouds got it easy, I was not able to make it so far. But it doesn't seem that hard either, I just need to write one particularly careful bash script, I just don't have time for that yet. Google does have some secret sauce. For example there's horizontal scaling which spins up more pods as load grows. There's vertical scaling that adjusts resources for individual pods. Those are generally incompatible with each other and people adjust resources manually. GKE autopilot has multi-dimensional scaling which they didn't open source, it does both. May be I wasn't hit with something nasty yet. But so far I think that those complains about managing k8s are somewhat strange. Yes, if you manage thousands of VMs, it might get another full-time job, but that's a scale. I manage few VMs and that's not an issue.
master_crab 4y ago
The fact that there is a think piece everyday either extolling or cursing Kubernetes is a key indicator this bit of tech has some serious teething issues.
azurezyq 4y ago
We are currently building a Database-as-a-service platform (tidbcloud.com) using Kubernetes. I have to say it is a love-and-hate story. On the bright side, k8s is almost the only option of an abstraction layer on top of different Clouds, for a complex system with tens of components. Database is more than masters and workers, there are so many components you need to take care of. For example, we may need monitoring agents, certificate managers, health checkers, admin proxies, etc. Without k8s, you have to be the owner of a kindergarten. On the other side, k8s IS complicated. It's like an unexplored garden. People just enter it and try to use whatever they see, and cause all kinds of problems. What we met are: * Try to apply the operator pattern to everything, debugging is really painful. Learning curve is steep. * Small services still cost a lot. VPA is not mature enough and many tiny services may be just better off on lambda. * k8s is not really designed for small per-tenant clusters. Managing a fleet of clusters is no easy job, but it is something SaaS companies have to deal with.
- nijave 4y ago
  Probably the next closest is just plain VMs (and potentially backplane/management layer running on k8s or whatever) But yeah... Even then each cloud has quirks with Kubernetes and there's still quite a few resources just to stand up a cluster. Kubernetes can partially solve the initial provisioning but you generally need the cluster running with nodes before you can use something like CAPI or Crossplane (so you still need Terraform or Pulumi or scripts or whatever) Having worked with a similar system, shared tenancy with tenant per namespace is just as bad but in a different way (if you use the classic operator pattern with 1 operator per cluster, you potentially have a massive blast radius). Then there's security...
  - azurezyq 4y ago
    1 operator per cluster is not ideal since most clusters are "stable" and don't need much care. Having plenty of them should be a headache. The operator crash on our side does sound scary. But as a DBaaS system, as long as the blast radius doesn't touch the data plane, it is manageable.
- Beltalowda 4y ago
fuziontech 4y ago
For anyone managing a k8s cluster and are fatigued with memorizing and reciting kubectl commands should definitely take a look at k9s[0]. It provides a curses like interface for managing k8s which makes it really easy to operate and dive into issues when debugging. Move from grabbing logs for a pod to being at a terminal on the container and then back out to looking at or editing the yaml for the resource definition in only a few key presses. [0] https://k9scli.io/ https://k9scli.io/
- _carbyau_ 4y ago
  I had to look up k9s because I wondered what you meant by "curses like interface" - it couldn't be where my mind went:"f*ck_u_k8s -doWhatIwant -notWhatISaid" And upon lookup I was transported back to my youth of ascii interfaces.
  - kortilla 4y ago
    https://en.wikipedia.org/wiki/Ncurses https://en.wikipedia.org/wiki/Ncurses
- yuppie_scum 4y ago
  Can’t vouch for k9s enough, it’s great and I think it helped me to gain a much better understanding of the resource/api system.
  - bbeausej 4y ago
    k9s is by far the most productive tool in my toolshed for managing & operating k8s clusters.
- timothylaurent 4y ago
  It's really better than any GUI I've used. I like it better than Weave Scope, Lens, and Rancher.
- quickthrower2 4y ago
  I like Lens, as more of a GUIs fan, and very occasional k8s-er. It has saved me a lot of time.
draw_down 4y ago
stn_za 4y ago
Amazon EKS + Fargate. No bare metal to manage. Control plane complexity is abrastacted away. Fargate namespace + profiles, no worker node configuration. EKS cost will be $90/m Thereafter you only pay for what cpu/mem limits you assign to your deployments/pods. Otherwise, why bare metal? For your basic needs, bare metal, self managed control planes, etc are definitely over complicating things.
- deleted 4y ago
  [deleted]
- denvrede 4y ago
  Using Fargate for long / permanently running workloads in EKS is only an option when the costs are none of your concern.
  - stn_za 4y ago
    As opposed to? Fargate is cheap, you can have permanently running workloads and still pay less than non-fargate options.
- easton 4y ago
  If you're abstracting away most of the complexity of k8s, why not just go use ECS and spend nothing on the cluster? You will probably have to do some rewriting of your deployment scripts when you move off of EKS anyway (just like when you move off of ECS), so you might as well use ECS and save the $90/m (and it's generally easier to use).
deleted 4y ago
[deleted]
stickfigure 4y ago
...or deploy your code on Google App Engine, Heroku, Elastic Beanstalk, Digital Ocean App Platform, Fly.io (etc etc) and spend all your time implementing user-facing features instead of maintaining infrastructure. Yeah, I get it, compared to maintaining bare metal, k8s is amazing. But you're still wasting your time working on plumbing.
- latchkey 4y ago
  Google Cloud Functions for the win. Very reasonable pricing model too. Doing 55 requests per second, 24/7, and it is about $100 a month, including the managed Cloud SQL postgres instance.
  - sonium 4y ago
    That's what I thought as well, but now I do have some long-running jobs that exceed GCF's 60min limit. So I'm stuck with docker on Compute Engine, where GCP treats you like a 2nd class citizen as the OP found out.
    - fvdessen 4y ago
      you need to split those jobs into smaller ones that read their parameters from a queue. Then it will fit in serverless and also be more reliable
      - simiones 4y ago
        And much harder to maintain and understand...
        fvdessen 4y ago
        We're doing that with cloud functions, pubsub and pulumi, the infra code to set that up is trivial, and it is actually a lot easier to maintain since it's fully serverless & you get retries and parallelism 'for free'. With cronjobs on vms the job itself might be a bit easier to code, but everything around it is a lot harder. (What happens if your 5h job crashes in the middle, who restarts it ? How do you manage locks to prevent concurrent execution ? How do you prevent that job from overloading the system ? etc ...) just to clarify our setup: - 1 pubsub 'job' queue - 1 cloud function triggered by a scheduled event populates the job queue - 1 idempotent cloud function to handle a job, triggered by events on the queue.
daemond 4y ago
From the Google trend[0] and the historical data of Kubernetes repo on GitHub[1], K8s has crossed the chasm to become the dominant choice for infrastructure. Whether it is the result of several companies working together, or from developers' choice. I think K8s will remain mainstream until there are big changes in the infrastructure world. [0] https://trends.google.com/trends/explore?date=2014-06-01%202022-08-10&q=Kubernetes https://trends.google.com/trends/explore?date=2014-06-01%202... [1] https://ossinsight.io/analyze/kubernetes/kubernetes https://ossinsight.io/analyze/kubernetes/kubernetes
mihaigalos 4y ago
For very bare minimum playing around, I use k3s on Linode (1 shared CPU, 1 GB RAM, 25 GB disk for a master and another one for worker node) at 10$ / month.
lbriner 4y ago
Lots of people complain about Kubernetes complexity but I have found it is as complex as you make it. If you are running some simple workloads then once you have the pipeline setup, there is almost no maintenance required. When people complain and then start talking about super complex configuration, bespoke networking functionality and helm charts that have "too many options" then surely that just means you don't have the skills to use the system to that degree? I could say that .Net is too complicated because it has MSIL and library binding sequences involving versions and public keys and the fact you can not always link e.g. netfx with netstandard but these are either just things you need to learn, or things that you can't use until you do learn them. It's like someone complaining that a Ferrari is too complicated because you can't work out how to change the cylinder head when most people will just drive it.
- jiggawatts 4y ago
  This is a bit like people being apologetic for PHP. Sure, technically, it is possible to write good PHP code. It doesn't have to turn into a spaghetti. I have several issues with Kubernetes that superficially look like I'm just avoiding the complexity, but I've dealt with systems that are much more complex with ease. 1. In most orgs and environments, its a cloud-on-a-cloud. A few years ago I used to joke with people that the virtualisation in the cloud is 7 layers deep and no human can understand or troubleshoot it any longer. Adding something like Docker adds 1-2 layers. Kubernetes doubles the layers. Everything you do with the underlying cloud is duplicated in Kubernetes, but incompatible. E.g.: Azure has: Kubernetes has: Resource Groups Namespaces Tags Labels Disks PVs & Container Images VMs Nodes (various) Pods Load balancers Ingress NSGs & FWs (various) Policy Policies Key Vault etcd ARM Templates Helm charts JSON APIs gRPC APIs Azure AD Pluggable auth Azure Metrics Prometheus Log Analytics (various) PowerShell cli tool These interact in weird and wonderful ways. Azure NATs all traffic, and then Kubernetes NATs it again by default. There's "security" at every layer, but just to be irritating, all Kubernetes traffic comes from unpredictable IPs in a single Subnet, making firewalling a nightmare. You're running cloud VMs already, but Windows containers run in nested VMs by default on Kubernetes. Kubernetes has its own internal DNS service for crying out loud! 2. Trying to do everything is to be less than optimal for everyone. There are four distinct ways of managing a cluster, and they're not compatible. You can run imperative commands, upload Helm charts, sync the cluster with an entire folder of stuff, or use a plugin like Flux to do GitOps. But if different people in a large team mix these styles, then this causes a giant mess. (To be fair, this is an issue with all of the major public cloud providers also.) 3. Google-isms everywhere. Every aspect of Kubernetes uses their internal shorthand. I'm not an ex-Googler. Nobody at any of my customers is. Nobody around here "speaks this dialect", because we're 12,000 kilometres from Silicon Valley. I'm sure this is not deliberate, but there are definite "cliques" with distinct cultures in the IT world. As FAANG employees flush with cash jump ship to start projects and startups like Kubernetes, they take their culture with them. In my opinion, mixing these together at random into a larger enterprise architecture is generally a mistake. 4. Kubernetes is not much better than bare VMs for developers, especially when compared with something like Azure App Service. The latter will "hold your hand for you" and has dozens of slick diagnostic tools integrated with it. On Kubernetes, if you want to do something as simple as capture crash dumps when there's a memory leak detected, you have to set this up yourself. 5. Microservices and RPC-oriented by default. Sure: you're not forced to implement this pattern, but it's a very steep slippery slope with arrows pointing downhill. In my experience, this is unnecessary complexity 99.99% of the time. Just last week I had to talk a developer out of adding Kubernetes to a trivial web application. Notice that I said "a" developer? Yes, a solo developer was keen on adopting this fad "just because". He was seriously planning on splitting individual REST endpoints out into containerised microservices. He's the third solo developer I've had to talk out of adopting Kubernetes this year. 6. Startup culture. Kubernetes shipped too early in my opinion. Really basic things are still being worked out, and it is already littered with deprecation warnings in the documentation. It's the type of product that should have been fleshed out a bit better at one or two large early adopter customers, and only then released to the general public. But its authors had a lot of competition (Docker Swarm, etc...) so they felt a lot of pressure to ship an MVP and iterate fast. That's fine I suppose, for them, but as an end-user I have to deal with a lot of churn and breakage. A case-in-point is that the configuration file formats are so inadequate that they have spawned a little ecosystem of config-file-generator-generators. I don't even know how deep those layers go these days. (Again, to be fair, Azure now has Bicep -> ARM as a standard transpilation step.) 7. Weak security by default because containers aren't security boundaries as far as Linus Torvalds or Microsoft Security Response Center are concerned. Everyone I've ever talked to about Kubernetes in the wild assumes the opposite, that it's magically more secure than hypervisors. I get the purpose of Kubernetes in the same way that I get the purpose of something like Haskell, coq, or hand-rolled cryptography. They all have their legitimate uses, and can be useful in surprising ways. Should they be the norm for typical development teams? I'm not convinced. Maybe one day Kubernetes v3 will be mature, stable, and useful for a wider audience. Especially once the underlying cloud is removed and there are smaller providers offering "pure Kubernetes clouds" where node pools are bare metal and there's no NAT and there isn't an external load balancer in front of the Kubernetes load balancer to make things extra spicy when diagnosing performance issues late at night across multiple incompatible metrics collector systems...
gizzlon 4y ago
Seems like everyone is forgetting about PaaS, and I don't understand why .. For many use-cases it's going to be much simpler and cheaper than a manged k8. There's no lock-in with Cloud Run than GKE. (actual lock-in comes with proprietary databases and the like.) edit: Missed the GPU part, might make the OP's project the exception to the rule People also forget about auto-scaling groups of VM's such as Managed Instance Groups in GCP: https://cloud.google.com/compute/docs/instance-groups/ https://cloud.google.com/compute/docs/instance-groups/
lsofzz 4y ago
This is one of the pet peeves of HN submitters and readers :) Sure, here is my two cents FWIW - Kubernetes is complex for some set of folks but not for others. So, the answer is - depends; On a lot of external factors outside of just technical capabilities of team. Kubernetes solves many non-trivial challenges but its not a silver bullet. I could certainly learn from Mercedes platform/infra team's "story from the trenches" (they reportedly run 900+ k8s clusters on production:)
woopwoop24 4y ago
i still believe most of the companies are better off with just deploying containers on vm's with autoscaling groups with a LB in front of it and some kind of terraform + ansible | CI-pipeline deployment to manage it. all the complexity you have to buy with dealing with k8s. you have to update it frequently, update all the charts, fix all the breaking things in a developing ecosystem, and still have to deal with the hard parts of dynamic certs, storage and network. and you would still need to dedicate "normal" vms to host databases and your monitoring and your storage because if your cluster goes down, so is your management layer. i have been with k8s since 1.3 and it is so disproportionate that i will not touch it for most of the times

164 points by icyfox 4y ago

Vanilla k8s is pretty good. But once the 8 trillion vendors have you 'curl | helm' ing you end up with a knot of a system. Keep it simple, use GitOps (ArgoCD is great), let k8s do what it's good at, managing workloads, not as a delivery mechanism for a vendor. As an aside, the existence of the '{{ | indent 4 }}' function in helm should disqualify it from any serious use. Render, don't template.

p_l 4y ago
I judiciously delete any and all helm I can find, and fight text-based templating where possible :/ Especially since k8s doesn't even use "text" for the format, it's just happens that JSON is the serialization, use datastructures and serialize them dammit :/
bushbaba 4y ago
I'm fairly convinced that helm in production is an anti-pattern. With you instead having all K8s manifests checked into your Git repository with CI/CD to handle any changes. Helm just has too much auto-magic and remove's one of k8s best features, the git-blame / git-diff for Infrastructure.
- blamarvt 4y ago
  I don't understand why this isn't the prevailing philosophy. I'm over here Terraforming helm deployments and having no idea what is actually happening under the covers.
  - cortesoft 4y ago
    We have our CI run a helm —-dry-run —-debug before the deploy step, so we can see the exact kubernetes resources that are going to be applied. You can even save that as an artifact so you can diff it later.
    - a012 4y ago
      There's Helm plugin (https://github.com/databus23/helm-diff https://github.com/databus23/helm-diff) that show diff results for you, for example helm diff upgrade <release> <repo> --namespace <namespace>
- the_duke 4y ago
  Helm still saves you an incredible amount of work for setting up all the third party services. If your team is big enough you can just write your own configs, but that takes a lot of time and often quite a bit of knowledge about the relevant service. Rollbacks and resource cleanup is not to be underestimated either if you are not getting that from other tools like Argo. Note: You can still use the Git-centric approach by generating configuration via `helm template`.
  - 0x457 4y ago
    I'm convinced that the only reason Helm saves you time with 3rd party services is because 3rd party services only provide Helm chart or don't provide means of deployment to k8s at all. I've been using kustomize to deploy some things with ArgoCD and it's so much easier. Now, I'm trying to never use helm for 3rd party. However, for your internal things, Helm is hard to replace. It's easy to start a chart that is capable of deploying most of your internal services, but maintaining it is a nightmare. Actually, using helm, as in `helm` the binary, directly sounds bonkers to me and I wouldn't wish this upon anyone.
- FridgeSeal 4y ago
  I use Helm only for installing “supporting applications” -elastic searchoperator, jupyterHub, etc. Our normal deployments are standard K8s configs. These apps use Helm because a lot of the settings are complicated, Co dependent, etc. Absolutely would not write helm charts from scratch for normal deployments, and if I got these apps in a better format than helm, I’d probably drop it immediately.
  - bottled_poe 4y ago
    It’s the fact that they are complicated which warrants manual configuration. If the org can’t write the configuration, how will they support it when something goes wrong? It’s a problem waiting to happen.
  - FridgeSeal 4y ago
    I can't edit this anymore - but to anyone reading, I ought to specify that I do _not_ use Helm directly, only via Flux, our CD tool. It ablates a lot of the issues of dealing with helm charts.
halfmatthalfcat 4y ago
I don't understand this comment. How else are you going to deploy pieces of k8s infra into k8s if not with Helm and Helm Charts? Sure, you can use Argo to deploy and sync Helm charts into k8s but...you're still going to be using Helm (if not indirectly via Argo) and you will inevitably need to template things that need to be dynamically configured at render-time.
- justinsaccount 4y ago
  See https://xeiaso.net/blog/dhall-kubernetes-2020-01-25 https://xeiaso.net/blog/dhall-kubernetes-2020-01-25
  - halfmatthalfcat 4y ago
    So instead of Go Templates, I'm going to use Dhall? Why? I'd be losing interop with the entire Helm ecosystem too, so there go dependencies and such to maintain my IaC. That blog post doesn't alleviate any issues one might have using the traditional Go Templates + Helm.
- qbasic_forever 4y ago
  Just kubectl apply the manifests. You can even use kubectl -k for the Kustomize configuration engine that can more or less replace most of what helm does today.
  - halfmatthalfcat 4y ago
    So what, I'm going to have a big Makefile or something with a bunch of kubectl applies? For each environment too? What if one of my dependencies (cert-manager for example) doesn't support directly applying via kubectl but have to be rendered with Helm? How do I manage versions of these dependencies too? For better or for worse Helm is the defacto standard for deploying into k8s. Kustomizing toy apps or simple things may work but I have yet to see a large production stack use anything but Helm.
    - 4y ago
n3t 4y ago
> As an aside, the existence of the '{{ | indent 4 }}' function in helm should disqualify it from any serious use. Render, don't template. This. My first thought when I saw the indentation hack was "it can't be serious, production-ready software". My take on this is as follows. If you have a simple use case, write your K8s manifests directly. If you have a complex use case, Helm is often more pain than it's worth. Use alternatives, for example Jsonnet[0] with kubecfg[1]. Or emit manifests from your language of choice. Just don't use Helm. [0]: https://jsonnet.org/ https://jsonnet.org/ [1]: https://github.com/kubecfg/kubecfg https://github.com/kubecfg/kubecfg
- 015a 4y ago
  For a complex use case: cdk8s is cool https://cdk8s.io/ https://cdk8s.io/
- vlunkr 4y ago
  It is shocking that such a clearly bad design choice has stuck, and that Helm has become so popular in spite of it. I've had my eye on jsonnet, I'll have to try that next time I do something in k8s.
  - runevault 4y ago
    We are using helm at work but I've never touched it so I cannot comment on it being bad. However cargo cult habits have made quite a few technologies over the years take off when they never should have just because someone with either the right sort of intelligence and/or charisma made people believe it was good, and then the ripple effect made it take off.
    - nfhshy68 4y ago
      I had someone on here assume we were a small dev shop because we don't use microservices the other day. I don't think they were expecting 2k devs to be what I responded with. Apparently monoliths a scale a lot bigger than you'd think.
  - 0xbadcafebee 4y ago
topspin 4y ago
> not as a delivery mechanism for a vendor Amen. I got turned off from k8s following a tutorial that used Helm. I ran it and watched mysteries and line noise scroll past and walked away for a year. I thought "no, I will never put this squirming mass of tapeworms in front of anyone." Then I took up with k3s and got underway.
jrockway 4y ago
> As an aside, the existence of the '{{ | indent 4 }}' function in helm should disqualify it from any serious use. Render, don't template. Yeah, I think helm will be the death of Kubernetes. Some other workload management tool will come out, it will have a computer-science-inspired templating system (think Lisp macros, not C #defines) that is also easy to use, and the increased maintainability will be a breath of fresh air to people in helm hell and they'll switch over, even if the workload management fundamentals are worse. It is a shame that ksonnet was abandoned. jsonnet is a very pleasant programming language for problems of similar scope. I think that people have to see some adoption or they give up; so Helm stays motivated to continue existing despite the fact that the design is fundamentally flawed, while alternatives give up when nobody uses them. If you're looking for a lesson in "worse is better", look no farther from helm. Easy things are hard, but everything is possible. That turns out to be important. I also liked kustomize a lot, but it definitely sacrificed features for security and a clean overall design posture. I don't really know what it's missing, but it must be missing something. (I use kustomize for everything, but it seems rare in the industry. And I don't use it for complicated things like "make a namespace and app deployment for every PR"; I think to manage things like that it's missing features people want. I say just run your development branch on your workstation and commit it when it works. Running the app you write shouldn't be rocket science, and so a computer program that configures it shouldn't be necessary. The industry definitely disagrees with me there, though.) One of the biggest problems I have with helm is that it's not easy to make local modifications to charts. That needs to be a first class feature, and something everyone knows how to use. As a vendor who ships a helm chart, I feel like almost all of my work these days is hearing from users "our k8s security team requires that every manifest set field X", and then I have to go add that for them and release it. If kustomize were the default, they'd just add that themselves instead of asking. But hey, opportunity to provide good customer service, I guess. (Luckily, most of the requests are compatible with each other. At one point we ran as root because that's the default, a customer required running as a user, and now everyone can run a locked down version; nobody NEEDED to run as root. So these requests generally improve the product, but it's a little tedious to make them ask.)
- moondev 4y ago
  kustomize supports using a helm chart as a resource https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/helmcharts/ https://kubectl.docs.kubernetes.io/references/kustomize/kust...
whalesalad 4y ago
I really think this is why most people dislike and misunderstand the value of kube. If you raw dog it, it’s pretty amazing what you can build. It’s not very hard to roll your own Heroku (tailored to your workflows and workloads) with kube, if you shy away from all the noise of helm and other vendors as you say.

umvi 4y ago

> This deployment needed to serve docker images on boot-up, which instance templates do support. However they don't support them via API, only in the web console or the CLI. Not exactly true. I got around this limitation by sending a startup script[1] in API metadata which basically just does invokes `docker run ...` and it works just fine. This allows spinning up/down container-based VMs via API only which is nice. [1] https://cloud.google.com/compute/docs/instances/startup-scripts/linux https://cloud.google.com/compute/docs/instances/startup-scri...

dilyevsky 4y ago
Yeah you can do the same thing via any image that supports cloud-init or ignition (eg fedora coreos) and have the same exact setup deployed in almost any cloud

throwaway787544 4y ago

> Bare metal is always my first choice both for past projects and startups. Is this a new Tech Hipster thing? Like writing a letter with a typewriter rather than an ink pen or computer/printer? "You don't understand, man; a virtual machine is, like, not authentic, man."

EddySchauHai 4y ago
Bare metal can still have containers, I assumed it just meant on their own hardware?
- BoorishBears 4y ago
  Nope, you can follow the link where they describe the stack and they're very proud of the exclusion of containerization
  - EddySchauHai 4y ago
    Ah! I'm all for owning hardware but docker is the best tech from the last couple years IMO for development.
  - icyfox 4y ago
    OP here. To clarify, the blog doesn't use containers just because it didn't really call for it. Ended up creating a simpler CI pipeline to just push updated static files directly. For everything else that I work on, I'm 100% in on containers. Just those containers running mainly on bare metal and newly on k8.
pm90 4y ago
There's benefits to using bare metal, but generally only for situations where you really need them. e.g. if you want to run hardware accelerated apps, run your own virtualization/VMM (like firecracker) etc. So yes, for the most part it is a hipster thing.
- heavenlyblue 4y ago

umvi 4y ago

My problem with Kubernetes is my problem with with front-end web frameworks - they introduce too much complexity to the point of being esoteric for simple systems. If you have a simple website built on boring technologies like HTML, CSS, and vanilla JS, then nearly anyone can read, understand, and make changes to it, even backend developers. If you instead wrote it in React/Webpack/etc. then suddenly only frontend experts can understand and contribute and debug. Same with k8s. If you make a cloud backend using boring technologies like plain old programs calling APIs then nearly anyone can read, understand, and make changes to it. But if you instead make it use a big pile of configuration files with a million switches then suddenly only k8s experts can understand and contribute and debug. I'm not saying don't use ReactJS or Kubernetes, I'm just saying make sure that the benefits you get from switching to it outweigh the new complexity and therefore expertise required to understand and debug it.

initplus 4y ago
K8s provides a cloud-agnostic solution for (mostly stateless) configuration management. You have to record which ports your applications listen on, which subdomains are used for what, how to access that storage bucket with the data files in it etc. somewhere. The moment you have more than a couple of individual services managing this all by hand becomes a nightmare. You need a really strong culture of good practice to make sure that the exact production config actually ends up in source control. You can be a k8s expert, or an expert in your own home-spun configuration management/git-ops workflow. But not having any workflow here just isn't an option in my opinion. You too easily end up with spaghetti infrastructure that everybody is too scared to touch. With k8s I can take my repo of yaml and change cloud providers in an afternoon.
- XorNot 4y ago
  The problem is kubernetes doesn't include any notions of obvious things I'd like to do like parameterizing my deployment - even though all the providers want to sell me kubernetes clusters as a service.
  - FridgeSeal 4y ago

tillmannhgl 4y ago

This sounded like an article from 2017.

yjftsjthsd-h 4y ago
What's changed since?

topspin 4y ago

"Azure is the only major provider that still has free control panels" Oracle Cloud Infrastructure does as well. Perhaps it does not yet qualify as major... It's major to Oracle, that's for sure.

kelnos 4y ago

> Keep it managed since that legitimately becomes a reliability headache. This is the thing that I think will always give me pause. If I have to pay a third party to manage my cluster orchestration backplane, that seems like a pretty big piece of overhead. Sure, I can do it myself, but then I have to deal with said reliability headache. It seems ironic that a cluster management framework -- that touts its ability to reliably keep your applications running -- has its own reliability issues.

yunwal 4y ago
I don't disagree with you, but if you can reliably trade your dataplane outages for control plane outages, that's still usually a good tradeoff.
- kelnos 4y ago
  That's a really good point. Certainly you don't want either to go down, but outages are more tolerable if customer requests are still getting serviced regardless.
lmarcos 4y ago
Exactly. We are building these incredible open source tools... but they grow so complex that we need to pay others in order to use them effectively? What would you say if you had to pay Google if you want to use Golang in an effective way (because the language has become so complex that it's difficult to handle it on your own?). Crazy. I wanted to take a look at how to use K8s on my own cluster, and damn it, to install the whole thing is not that straightforward. So, now to keep my sanity I need to pay a cloud provider to use k8s! I guess that's the trick: build some open source monster that's very hard to install/maintain/use but has cool features. People will love it and they'll pay for it.
- vbezhenar 4y ago
  I think people are overestimating the difficulty of setting up k8s. It's not that hard. Grab Debian, install containerd.io, set one sysctl, load one kernel module and you're all set. Install few more packages, run kubeadm init and that's all. The only thing that I've found truly hard is autoscaling VMs. Managed clouds got it easy, I was not able to make it so far. But it doesn't seem that hard either, I just need to write one particularly careful bash script, I just don't have time for that yet. Google does have some secret sauce. For example there's horizontal scaling which spins up more pods as load grows. There's vertical scaling that adjusts resources for individual pods. Those are generally incompatible with each other and people adjust resources manually. GKE autopilot has multi-dimensional scaling which they didn't open source, it does both. May be I wasn't hit with something nasty yet. But so far I think that those complains about managing k8s are somewhat strange. Yes, if you manage thousands of VMs, it might get another full-time job, but that's a scale. I manage few VMs and that's not an issue.

master_crab 4y ago

The fact that there is a think piece everyday either extolling or cursing Kubernetes is a key indicator this bit of tech has some serious teething issues.

azurezyq 4y ago

We are currently building a Database-as-a-service platform (tidbcloud.com) using Kubernetes. I have to say it is a love-and-hate story. On the bright side, k8s is almost the only option of an abstraction layer on top of different Clouds, for a complex system with tens of components. Database is more than masters and workers, there are so many components you need to take care of. For example, we may need monitoring agents, certificate managers, health checkers, admin proxies, etc. Without k8s, you have to be the owner of a kindergarten. On the other side, k8s IS complicated. It's like an unexplored garden. People just enter it and try to use whatever they see, and cause all kinds of problems. What we met are: * Try to apply the operator pattern to everything, debugging is really painful. Learning curve is steep. * Small services still cost a lot. VPA is not mature enough and many tiny services may be just better off on lambda. * k8s is not really designed for small per-tenant clusters. Managing a fleet of clusters is no easy job, but it is something SaaS companies have to deal with.

nijave 4y ago
Probably the next closest is just plain VMs (and potentially backplane/management layer running on k8s or whatever) But yeah... Even then each cloud has quirks with Kubernetes and there's still quite a few resources just to stand up a cluster. Kubernetes can partially solve the initial provisioning but you generally need the cluster running with nodes before you can use something like CAPI or Crossplane (so you still need Terraform or Pulumi or scripts or whatever) Having worked with a similar system, shared tenancy with tenant per namespace is just as bad but in a different way (if you use the classic operator pattern with 1 operator per cluster, you potentially have a massive blast radius). Then there's security...
- azurezyq 4y ago
  1 operator per cluster is not ideal since most clusters are "stable" and don't need much care. Having plenty of them should be a headache. The operator crash on our side does sound scary. But as a DBaaS system, as long as the blast radius doesn't touch the data plane, it is manageable.
Beltalowda 4y ago

fuziontech 4y ago

For anyone managing a k8s cluster and are fatigued with memorizing and reciting kubectl commands should definitely take a look at k9s[0]. It provides a curses like interface for managing k8s which makes it really easy to operate and dive into issues when debugging. Move from grabbing logs for a pod to being at a terminal on the container and then back out to looking at or editing the yaml for the resource definition in only a few key presses. [0] https://k9scli.io/ https://k9scli.io/

_carbyau_ 4y ago
I had to look up k9s because I wondered what you meant by "curses like interface" - it couldn't be where my mind went:"f*ck_u_k8s -doWhatIwant -notWhatISaid" And upon lookup I was transported back to my youth of ascii interfaces.
- kortilla 4y ago
  https://en.wikipedia.org/wiki/Ncurses https://en.wikipedia.org/wiki/Ncurses
yuppie_scum 4y ago
Can’t vouch for k9s enough, it’s great and I think it helped me to gain a much better understanding of the resource/api system.
- bbeausej 4y ago
  k9s is by far the most productive tool in my toolshed for managing & operating k8s clusters.
timothylaurent 4y ago
It's really better than any GUI I've used. I like it better than Weave Scope, Lens, and Rancher.
quickthrower2 4y ago
I like Lens, as more of a GUIs fan, and very occasional k8s-er. It has saved me a lot of time.

draw_down 4y ago

stn_za 4y ago

Amazon EKS + Fargate. No bare metal to manage. Control plane complexity is abrastacted away. Fargate namespace + profiles, no worker node configuration. EKS cost will be $90/m Thereafter you only pay for what cpu/mem limits you assign to your deployments/pods. Otherwise, why bare metal? For your basic needs, bare metal, self managed control planes, etc are definitely over complicating things.

deleted 4y ago
[deleted]
denvrede 4y ago
Using Fargate for long / permanently running workloads in EKS is only an option when the costs are none of your concern.
- stn_za 4y ago
  As opposed to? Fargate is cheap, you can have permanently running workloads and still pay less than non-fargate options.
easton 4y ago
If you're abstracting away most of the complexity of k8s, why not just go use ECS and spend nothing on the cluster? You will probably have to do some rewriting of your deployment scripts when you move off of EKS anyway (just like when you move off of ECS), so you might as well use ECS and save the $90/m (and it's generally easier to use).

deleted 4y ago

[deleted]

stickfigure 4y ago

...or deploy your code on Google App Engine, Heroku, Elastic Beanstalk, Digital Ocean App Platform, Fly.io (etc etc) and spend all your time implementing user-facing features instead of maintaining infrastructure. Yeah, I get it, compared to maintaining bare metal, k8s is amazing. But you're still wasting your time working on plumbing.

latchkey 4y ago
Google Cloud Functions for the win. Very reasonable pricing model too. Doing 55 requests per second, 24/7, and it is about $100 a month, including the managed Cloud SQL postgres instance.
- sonium 4y ago
  That's what I thought as well, but now I do have some long-running jobs that exceed GCF's 60min limit. So I'm stuck with docker on Compute Engine, where GCP treats you like a 2nd class citizen as the OP found out.
  - fvdessen 4y ago
    you need to split those jobs into smaller ones that read their parameters from a queue. Then it will fit in serverless and also be more reliable
    - simiones 4y ago
      And much harder to maintain and understand...
      - fvdessen 4y ago
        We're doing that with cloud functions, pubsub and pulumi, the infra code to set that up is trivial, and it is actually a lot easier to maintain since it's fully serverless & you get retries and parallelism 'for free'. With cronjobs on vms the job itself might be a bit easier to code, but everything around it is a lot harder. (What happens if your 5h job crashes in the middle, who restarts it ? How do you manage locks to prevent concurrent execution ? How do you prevent that job from overloading the system ? etc ...) just to clarify our setup: - 1 pubsub 'job' queue - 1 cloud function triggered by a scheduled event populates the job queue - 1 idempotent cloud function to handle a job, triggered by events on the queue.

daemond 4y ago

From the Google trend[0] and the historical data of Kubernetes repo on GitHub[1], K8s has crossed the chasm to become the dominant choice for infrastructure. Whether it is the result of several companies working together, or from developers' choice. I think K8s will remain mainstream until there are big changes in the infrastructure world. [0] https://trends.google.com/trends/explore?date=2014-06-01%202022-08-10&q=Kubernetes https://trends.google.com/trends/explore?date=2014-06-01%202... [1] https://ossinsight.io/analyze/kubernetes/kubernetes https://ossinsight.io/analyze/kubernetes/kubernetes

mihaigalos 4y ago

For very bare minimum playing around, I use k3s on Linode (1 shared CPU, 1 GB RAM, 25 GB disk for a master and another one for worker node) at 10$ / month.

lbriner 4y ago

Lots of people complain about Kubernetes complexity but I have found it is as complex as you make it. If you are running some simple workloads then once you have the pipeline setup, there is almost no maintenance required. When people complain and then start talking about super complex configuration, bespoke networking functionality and helm charts that have "too many options" then surely that just means you don't have the skills to use the system to that degree? I could say that .Net is too complicated because it has MSIL and library binding sequences involving versions and public keys and the fact you can not always link e.g. netfx with netstandard but these are either just things you need to learn, or things that you can't use until you do learn them. It's like someone complaining that a Ferrari is too complicated because you can't work out how to change the cylinder head when most people will just drive it.

jiggawatts 4y ago
This is a bit like people being apologetic for PHP. Sure, technically, it is possible to write good PHP code. It doesn't have to turn into a spaghetti. I have several issues with Kubernetes that superficially look like I'm just avoiding the complexity, but I've dealt with systems that are much more complex with ease. 1. In most orgs and environments, its a cloud-on-a-cloud. A few years ago I used to joke with people that the virtualisation in the cloud is 7 layers deep and no human can understand or troubleshoot it any longer. Adding something like Docker adds 1-2 layers. Kubernetes doubles the layers. Everything you do with the underlying cloud is duplicated in Kubernetes, but incompatible. E.g.: Azure has: Kubernetes has: Resource Groups Namespaces Tags Labels Disks PVs & Container Images VMs Nodes (various) Pods Load balancers Ingress NSGs & FWs (various) Policy Policies Key Vault etcd ARM Templates Helm charts JSON APIs gRPC APIs Azure AD Pluggable auth Azure Metrics Prometheus Log Analytics (various) PowerShell cli tool These interact in weird and wonderful ways. Azure NATs all traffic, and then Kubernetes NATs it again by default. There's "security" at every layer, but just to be irritating, all Kubernetes traffic comes from unpredictable IPs in a single Subnet, making firewalling a nightmare. You're running cloud VMs already, but Windows containers run in nested VMs by default on Kubernetes. Kubernetes has its own internal DNS service for crying out loud! 2. Trying to do everything is to be less than optimal for everyone. There are four distinct ways of managing a cluster, and they're not compatible. You can run imperative commands, upload Helm charts, sync the cluster with an entire folder of stuff, or use a plugin like Flux to do GitOps. But if different people in a large team mix these styles, then this causes a giant mess. (To be fair, this is an issue with all of the major public cloud providers also.) 3. Google-isms everywhere. Every aspect of Kubernetes uses their internal shorthand. I'm not an ex-Googler. Nobody at any of my customers is. Nobody around here "speaks this dialect", because we're 12,000 kilometres from Silicon Valley. I'm sure this is not deliberate, but there are definite "cliques" with distinct cultures in the IT world. As FAANG employees flush with cash jump ship to start projects and startups like Kubernetes, they take their culture with them. In my opinion, mixing these together at random into a larger enterprise architecture is generally a mistake. 4. Kubernetes is not much better than bare VMs for developers, especially when compared with something like Azure App Service. The latter will "hold your hand for you" and has dozens of slick diagnostic tools integrated with it. On Kubernetes, if you want to do something as simple as capture crash dumps when there's a memory leak detected, you have to set this up yourself. 5. Microservices and RPC-oriented by default. Sure: you're not forced to implement this pattern, but it's a very steep slippery slope with arrows pointing downhill. In my experience, this is unnecessary complexity 99.99% of the time. Just last week I had to talk a developer out of adding Kubernetes to a trivial web application. Notice that I said "a" developer? Yes, a solo developer was keen on adopting this fad "just because". He was seriously planning on splitting individual REST endpoints out into containerised microservices. He's the third solo developer I've had to talk out of adopting Kubernetes this year. 6. Startup culture. Kubernetes shipped too early in my opinion. Really basic things are still being worked out, and it is already littered with deprecation warnings in the documentation. It's the type of product that should have been fleshed out a bit better at one or two large early adopter customers, and only then released to the general public. But its authors had a lot of competition (Docker Swarm, etc...) so they felt a lot of pressure to ship an MVP and iterate fast. That's fine I suppose, for them, but as an end-user I have to deal with a lot of churn and breakage. A case-in-point is that the configuration file formats are so inadequate that they have spawned a little ecosystem of config-file-generator-generators. I don't even know how deep those layers go these days. (Again, to be fair, Azure now has Bicep -> ARM as a standard transpilation step.) 7. Weak security by default because containers aren't security boundaries as far as Linus Torvalds or Microsoft Security Response Center are concerned. Everyone I've ever talked to about Kubernetes in the wild assumes the opposite, that it's magically more secure than hypervisors. I get the purpose of Kubernetes in the same way that I get the purpose of something like Haskell, coq, or hand-rolled cryptography. They all have their legitimate uses, and can be useful in surprising ways. Should they be the norm for typical development teams? I'm not convinced. Maybe one day Kubernetes v3 will be mature, stable, and useful for a wider audience. Especially once the underlying cloud is removed and there are smaller providers offering "pure Kubernetes clouds" where node pools are bare metal and there's no NAT and there isn't an external load balancer in front of the Kubernetes load balancer to make things extra spicy when diagnosing performance issues late at night across multiple incompatible metrics collector systems...

gizzlon 4y ago

Seems like everyone is forgetting about PaaS, and I don't understand why .. For many use-cases it's going to be much simpler and cheaper than a manged k8. There's no lock-in with Cloud Run than GKE. (actual lock-in comes with proprietary databases and the like.) edit: Missed the GPU part, might make the OP's project the exception to the rule People also forget about auto-scaling groups of VM's such as Managed Instance Groups in GCP: https://cloud.google.com/compute/docs/instance-groups/ https://cloud.google.com/compute/docs/instance-groups/

lsofzz 4y ago

This is one of the pet peeves of HN submitters and readers :) Sure, here is my two cents FWIW - Kubernetes is complex for some set of folks but not for others. So, the answer is - depends; On a lot of external factors outside of just technical capabilities of team. Kubernetes solves many non-trivial challenges but its not a silver bullet. I could certainly learn from Mercedes platform/infra team's "story from the trenches" (they reportedly run 900+ k8s clusters on production:)

woopwoop24 4y ago

i still believe most of the companies are better off with just deploying containers on vm's with autoscaling groups with a LB in front of it and some kind of terraform + ansible | CI-pipeline deployment to manage it. all the complexity you have to buy with dealing with k8s. you have to update it frequently, update all the charts, fix all the breaking things in a developing ecosystem, and still have to deal with the hard parts of dynamic certs, storage and network. and you would still need to dedicate "normal" vms to host databases and your monitoring and your storage because if your cluster goes down, so is your management layer. i have been with k8s since 1.3 and it is so disproportionate that i will not touch it for most of the times