14 ms·

What would a Kubernetes 2.0 look like

• recursivedoubts 1y ago
  please make it look like old heroku for us normies
• mootoday 1y ago
  Why containers when you can have Wasm components on wasmCloud :-)?! https://wasmcloud.com/ https://wasmcloud.com/
• zdw 1y ago
  Related to this, a 2020 take on the topic from the MetalLB dev: https://blog.dave.tf/post/new-kubernetes/ https://blog.dave.tf/post/new-kubernetes/
  • deleted 1y ago
    [deleted]
  • jauntywundrkind 1y ago
    152 comments on A Better Kubernetes, from the Ground Up, https://news.ycombinator.com/item?id=25243159 https://news.ycombinator.com/item?id=25243159
• zug_zug 1y ago
  Meh, imo this is wrong. What Kubernetes is missing most is a 10 year track record of simplicity/stability. What it needs most to thrive is a better reputation of being hard to foot-gun yourself with. It's just not a compelling business case to say "Look at what you can do with kubernetes, and you only need a full-time team of 3 engineers dedicated to this technology at tho cost of a million a year to get bin-packing to the tune of $40k." For the most part Kubernetes is becoming the common-tongue, despite all the chaotic plugins and customizations that interact with each other in a combinatoric explosion of complexity/risk/overhead. A 2.0 would be what I'd propose if I was trying to kill kuberenetes.
  • candiddevmike 1y ago
    Kubernetes is what happens when you need to support everyone's wants and desires within the core platform. The abstraction facade breaks and ends up exposing all of the underlying pieces because someone needs feature X. So much of Kubernetes' complexity is YAGNI (for most users). Kubernetes 2.0 should be a boring pod scheduler with some RBAC around it. Let folks swap out the abstractions if they need it instead of having everything so tightly coupled within the core platform.
    • echelon 1y ago
      Kubernetes doesn't need a flipping package manager or charts. It needs to do one single job well: workload scheduling. Kubernetes clusters shouldn't be bespoke and weird with behaviors that change based on what flavor of plugins you added. That is antithetical to the principal of the workloads you're trying to manage. You should be able to headshot the whole thing with ease. Service discovery is just one of many things that should be a different layer.
      • KaiserPro 1y ago
        > Service discovery is just one of many things that should be a different layer. hard agree. Its like jenkins, good idea, but its not portable.
        • 12_throw_away 1y ago
          > Its like jenkins Having regretfully operated both k8s and Jenkins, I fully agree with this, they have some deep DNA in common.
    • sitkack 1y ago
      Kubernetes is when you want to sell complexity because complexity makes money and and naturally gets you vendor lockin even while being ostensibly vendor neutral. Never interrupt the customer while they are foot gunning themselves. Swiss Army Buggy Whips for Everyone!
• Bluecobra 1y ago
  [flagged]
  • neepi 1y ago
    Surely it’s Kubernetes 2.0: the search for more complexity by trying to get rid of it.
    • mulmen 1y ago
      Bluecobra made a Spaceballs reference.
  • esafak 1y ago
    Where's the schwartz when you need it?
  • deleted 1y ago
    [deleted]
• dijit 1y ago
  Honestly; make some blessed standards easier to use and maintain. Right now running K8S on anything other than cloud providers and toys (k3s/minikube) is disaster waiting to happen unless you're a really seasoned infrastructure engineer. Storage/state is decidedly not a solved problem, debugging performance issues in your longhorn/ceph deployment is just pain. Also, I don't think we should be removing YAML, we should instead get better at using it as an ILR (intermediate language representation) and generating the YAML that we want instead of trying to do some weird in-place generation (Argo/Helm templating) - Kubernetes sacrificed a lot of simplicity to be eventually consistent with manifests, and our response was to ensure we use manifests as little as possible, which feels incredibly bizzare. Also, the design of k8s networking feels like it fits ipv6 really well, but it seems like nobody has noticed somehow.
  • lawn 1y ago
    k3s isn't a toy though.
    • dijit 1y ago
      * Uses Flannel bi-lateral NAT for SDN * Uses local-only storage provider by default for PVC * Requires entire cluster to be managed by k3s, meaning no freebsd/macos/windows node support * Master TLS/SSL Certs not rotated (and not talked about). k3s is very much a toy - a nice toy though, very fun to play with.
      • bigstrat2003 1y ago
        None of those things make it a toy. It is in fact a useful tool when you want to run kubernetes in a small environment.
        • dijit 1y ago
          Nothing wrong with toys, they’re meant to be played with. If you deployed this to production though, the problem is not that it’s a toy: its you not understanding what technical trade offs they are making to give you an easy environment. I can tell you’re defensive though, might be good to learn what I mean instead of trying to ram a round peg into a square hole.
• jcastro 1y ago
  For the confusion around verified publishing, this is something the CNCF encourages artifact authors and their projects to set up. Here are the instructions for verifying your artifact: https://artifacthub.io/docs/topics/repositories/ https://artifacthub.io/docs/topics/repositories/ You can do the same with just about any K8s related artifact. We always encourage projects to go through the process but sometimes they need help understanding that it exists in the first place. Artifacthub is itself an incubating project in the CNCF, ideas around making this easier for everyone are always welcome, thanks! (Disclaimer: CNCF Staff)
  • calcifer 1y ago
    > We always encourage projects to go through the process but sometimes they need help understanding that it exists in the first place. Including ingress-nginx? Per OP, it's not marked as verified. If even the official components don't bother, it's hard to recommend it to third parties.
    • aritashion 1y ago
      As far as I'm aware, ingress-nginx is just the defacto ingress, but not actually an official/ reference implementation?
      • mdaniel 1y ago
        I think that's about 20% of the complaints in this thread: there isn't an official/reference implementation of just any of the components, because it matters a great deal what functionality you already have lying around versus needing to bring with you
• johngossman 1y ago
  Not a very ambitious wishlist for a 2.0 release. Everyone I talk to complains about the complexity of k8s in production, so I think the big question is could you do a 2.0 with sufficient backward compatibility that it could be adopted incrementally and make it simpler. Back compat almost always mean complexity increases as the new system does its new things and all the old ones.
  • herval 1y ago
    The question is always what part of that complexity can be eliminated. Every “k8s abstraction” I’ve seen to date either only works for a very small subset of stuff (eg the heroku-like wrappers) or eventually develops a full blown dsl that’s as complex as k8s (and now you have to learn that job-specific dsl)
    • mdaniel 1y ago
      Relevant: Show HN: Canine – A Heroku alternative built on Kubernetes - https://news.ycombinator.com/item?id=44292103 https://news.ycombinator.com/item?id=44292103 - June, 2025 (125 comments)
      • herval 1y ago
        yep, that's the latest of a long lineage of such projects (one of which I worked on myself). Ohers include kubero, dokku, porter, kr0, etc. There was a moment back in 2019 where every big tech company was trying to roll out their own K8s DSL (I know of Twitter, Airbnb, WeWork, etc). For me, the only thing that really changed was LLMs - chatgpt is exceptional at understanding and generating valid k8s configs (much more accurately than it can do coding). It's still complex, but it feels I have a second brain to look at it now
        • programd 1y ago
          Maybe that should be the future of K8s 2.0. Instead of changing the core of the beast tweak it minimally to get rid of whatever limitations are annoying and instead allocate resources to put a hefty AI in front of it so that human toil is reduced. At some point you won't need a fully dedicated ops team. I think a lot of people in this discussion are oblivious to where this is heading.
• mrweasel 1y ago
  What I would add is "sane defaults", as in unless you pick something different, you get a good enough load balancer/network/persistent storage/whatever. I'd agree that YAML isn't a good choice, but neither is HCL. Ever tried reading Terraform, yeah, that's bad too. Inherently we need a better way to configure Kubernetes clusters and changing out the language only does so much. IPv6, YES, absolutely. Everything Docker, container and Kubernetes should have been IPv6 only internal from the start. Want IPv4? That should be handle by a special case ingress controller.
  • zdw 1y ago
    Sane defaults is in conflict with "turning you into a customer of cloud provider managed services". The longer I look at k8s, the more I see it "batteries not included" around storage, networking, etc, with the result being that the batteries come with a bill attached from AWS, GCP, etc. K8s is less of an open source project, and more as a way encourage dependency on these extremely lucrative gap filler services from the cloud providers.
    • JeffMcCune 1y ago
      Except you can easily install calico, istio, and ceph on used hardware in your garage and get an experience nearly identical to every hyper scaler using entirely free open source software.
      • zdw 1y ago
        Having worked on on-prem K8s deployments, yes, you can do this. But getting it to production grade is very different than a garage-quality proof of concept.
        • mdaniel 1y ago
          I think OP's point was: but how much of that production grade woe is the fault of Kubernetes versus, sure, turns out booting up an PaaS from scratch is hard as nails. I think that k8s pluggable design also blurs that boundary in most people's heads. I can't think of the last time the control plane shit itself, versus everyone and their cousin has a CLBO story for the component controllers installed on top of k8s
• tayo42 1y ago
  > where k8s is basically the only etcd customer left. Is that true. No one is really using it? I think one thing k8s would need is some obvious answer for stateful systems(at scale, not mysql at a startup). I think there are some ways to do it? Where I work there is basically everything on k8s, then all the databases on their own crazy special systems to support they insist its impossible and costs to much. I work in the worst of all worlds now supporting this. re: comments about k8s should just schedule pods. mesos with aurora or marathon was basically that. If people wanted that those would have done better. The biggest users of mesos switched to k8s
  • haiku2077 1y ago
    I had to go deep down the etcd rabbit hole several years ago. The problems I ran into: 1. etcd did an fsync on every write and required all nodes to complete a write to report a write as successful. This was not configurable and far higher a guarantee than most use cases actually need - most Kubernetes users are fine with snapshot + restore an older version of the data. But it really severely impacts performance. 2. At the time, etcd had a hard limit of 8GB. Not sure if this is still there. 3. Vanilla etcd was overly cautious about what to do if a majority of nodes went down. I ended up writing a wrapper program to automatically recover from this in most cases, which worked well in practice. In conclusion there was no situation where I saw etcd used that I wouldn't have preferred a highly available SQL DB. Indeed, k3s got it right using sqlite for small deployments.
    • nh2 1y ago
      For (1), I definitely want my production HA databases to fsync every write. Of course configurability is good (e.g. for automated fasts tests you don't need it), but safe is a good default here, and if somebody sets up a Kubernetes cluster, they can and should afford enterprise SSDs where fsync of small data is fast and reliable (e.g. 1000 fsyncs/second).
      • haiku2077 1y ago
        > I definitely want my production HA databases to fsync every write. I didn't! Our business DR plan only called for us to restore to an older version with short downtime, so fsync on every write on every node was a reduction in performance for no actual business purpose or benefit. IIRC we modified our database to run off ramdisk and snapshot every few minutes which ran way better and had no impact on our production recovery strategy. > if somebody sets up a Kubernetes cluster, they can and should afford enterprise SSDs where fsync of small data is fast and reliable At the time one of the problems I ran into was that public cloud regions in southeast asia had significantly worse SSDs that couldn't keep up. This was on one of the big three cloud providers. 1000 fsyncs/second is a tiny fraction of the real world production load we required. An API that only accepts 1000 writes a second is very slow! Also, plenty of people run k8s clusters on commodity hardware. I ran one on an old gaming PC with a budget SSD for a while in my basement. Great use case for k3s.
• rwmj 1y ago
  Make there be one, sane way to install it, and make that method work if you just want to try it on a single node or single VM running on a laptop.
  • mdaniel 1y ago
    My day job makes this request of my team right now, and yet when trying to apply this logic to a container and cloud-native control plane, there are a lot more devils hiding in those details. Use MetalLB for everything, even if NLBs are available? Use Ceph for storage even if EBS is available? Definitely don't use Ceph on someone's 8GB laptop. I can keep listing "yes, but" items that make doing such a thing impossible to troubleshoot because there's not one consumer So, to circle back to your original point: rke2 (Apache 2) is a fantastic, airgap-friendly, intelligence community approved distribution, and pairs fantastic with rancher desktop (also Apache 2). It's not the kubernetes part of that story which is hard, it's the "yes, but" part of the lego build - https://github.com/rancher/rke2/tree/v1.33.1%2Brke2r1#quick-start https://github.com/rancher/rke2/tree/v1.33.1%2Brke2r1#quick-... - https://github.com/rancher-sandbox/rancher-desktop/releases https://github.com/rancher-sandbox/rancher-desktop/releases
• fatbird 1y ago
  How many places are running k8s without OpenShift to wrap it and manage a lot of the complexity?
  • jitl 1y ago
    I’ve never used OpenShift nor do I know anyone irl who uses it. Sample from SF where most people I know are on AWS or GCP.
    • coredog64 1y ago
      You can always go for the double whammy and run ROSA: RedHat OpenShift on AWS
    • p_l 1y ago
      OpenShift is somewhat popular as turnkey solution for on-premise kubernetes if you can stomach Red Hat pricing.
  • raincom 1y ago
    Openshift, if IBM and Redhat want to milk the license and support contracts. There are other vendors that sell k8s: Rancher, for instance. SuSe bought Rancher.
• Melatonic 1y ago
  MicroVM's
• geoctl 1y ago
  I would say k8s 2.0 needs: 1. gRPC/proto3-based APIs to make controlling k8s clusters easier using any programming language not just practically Golang as is the case currently and this can even make dealing with k8s controllers easier and more manageable, even though it admittedly might actually complicates things at the API server-side when it comes CRDs. 2. PostgreSQL or pluggable storage backend by default instead of etcd. 3. Clear identity-based, L7-aware ABAC-based access control interface that can be implemented by CNIs for example. 4. Applying userns by default 5. Easier pluggable per-pod CRI system where microVMs and container-based runtimes can easily co-exist based on the workload type.
  • jitl 1y ago
    All the APIs, including CRDs, already have a well described public & introspectable OpenAPI schema you can use to generate clients. I use the TypeScript client generated and maintained by Kubernetes organization. I don’t see what advantage adding a binary serialization wire format has. I think gRPC makes sense when there’s some savings to be had with latency, multiplexing, streams etc but control plane things like Kubernetes don’t seem to me like it’s necessary.
    • geoctl 1y ago
      I haven't used CRDs myself for a few years now (probably since 2021), but I still remember developing CRDs was an ugly and hairy experience to say the least, partly due to the flaws of Golang itself (e.g. no traits like in Rust, no macros, no enums, etc...). With protobuf you can easily compile your definitions to any language with clear enum, oneof implementations, you can use the standard protobuf libraries to do deepCopy, merge, etc... for you and you can also add basic validations in the protobuf definitions and so on. gRPC/protobuf will basically allow you to develop k8s controllers very easily in any language.
      • mdaniel 1y ago
        CRDs are not tied to golang in any way whatsoever; <https://www.redhat.com/en/blog/writing-custom-controller-python https://www.redhat.com/en/blog/writing-custom-controller-pyt...> and <https://metacontroller.github.io/metacontroller/guide/create.html https://metacontroller.github.io/metacontroller/guide/create...> are two concrete counter-examples, with the latter being the most "microservices" extreme. You can almost certainly implement them in bash if you're trying to make the front page of HN
• pm90 1y ago
  Hard disagree with replacing yaml with HCL. Developers find HCL very confusing. It can be hard to read. Does it support imports now? Errors can be confusing to debug. Why not use protobuf, or similar interface definition languages? Then let users specify the config in whatever language they are comfortable with.
  • geoctl 1y ago
    You can very easily build and serialize/deserialize HCL, JSON, YAML or whatever you can come up with outside Kubernetes from the client-side itself (e.g. kubectl). This has actually nothing to do with Kubernetes itself at all.
    • Kwpolska 1y ago
      There aren’t that many HCL serialization/deserialization tools. Especially if you aren’t using Go.
      • demosthanos 1y ago
        Given that we're apparently discussing an entire k8s 2.0 based on HCL that hardly seems like a barrier. You'd have needed to write the HCL tooling to get the 2.0 working anyway.
  • dilyevsky 1y ago
    Maybe you know this but Kubernetes interface definitions are already protobufs (except for crds)
    • cmckn 1y ago
      Sort of. The hand-written go types are the source of truth and the proto definitions are generated from there, solely for the purpose of generating protobuf serializers for the hand-written go types. The proto definition is used more as an intermediate representation than an “API spec”. Still useful, but the ecosystem remains centered on the go types and their associated machinery.
• darkwater 1y ago
  I totally dig the HCL request. To be honest I'm still mad at Github that initially used HCL for Github Actions and then ditched it for YAML when they went stable.
  • lillecarl 1y ago
    I detest HCL, the module system is pathetic. It's not composable at all and you keep doing gymnastics to make sure everything is known at plan time (like using lists where you should use dictionaries) and other anti-patterns. I use Terranix to make config.tf.json which means I have the NixOS module system that's composable enough to build a Linux distro at my fingertips to compose a great Terraform "state"/project/whatever. It's great to be able to run some Python to fetch some data, dump it in JSON, read it with Terranix, generate config.tf.json and then apply :)
    • jitl 1y ago
      What’s the list vs dictionary issue in Terraform? I use a lot of dictionaries (maps in tf speak), terraform things like for_each expect a map and throw if handed a list.
      • lillecarl 1y ago
        Internally a lot of modules cast dictionaries to lists of the same length because the keys of the dict might not be known at plan time or something. The "Terraform AWS VPC module does this internally for many things. I couldn't tell you exactly, but modules always end up either not exposing enough or exposing too much. If I were to write my module with Terranix I can easily replace any value in any resource from the module I'm importing using "resource.type.name.parameter = lib.mkForce "overridenValue";" without having to expose that parameter in the module "API". The nice thing is that it generates "Terraform"(config.tf.json) so the supremely awesome state engine and all API domain knowledge bound in providers work just the same and I don't have to reach for something as involved as Pulumi. You can even mix Terranix with normal HCL since config.tf.json is valid in the same project as HCL. A great way to get started is to generate your provider config and other things where you'd reach to Terragrunt/friends. Then you can start making options that makes resources at your own pace. The terraform LSP sadly doesn't read config.tf.json yet so you'll get warnings regarding undeclared locals and such but for me it's worth it, I generally write tf/tfnix with the provider docs open and the language (Nix and HCL) are easy enough to write without full LSP. https://terranix.org/ https://terranix.org/ says it better than me, but by doing it with Nix you get programatical access to the biggest package library in the world to use at your discretion (Build scripts to fetch values from weird places, run impure scripts with null_resource or it's replacements) and an expressive functional programming language where you can do recursion and stuff, you can use derivations to run any language to transform strings with ANY tool. It's like Terraform "unleashed" :) Forget "dynamic" blocks, bad module APIs and hacks (While still being able to use existing modules too if you feel the urge).
• mdaniel 1y ago
  > Allow etcd swap-out From your lips to God's ears. And, as they correctly pointed out, this work is already done, so I just do not understand the holdup. Folks can continue using etcd if it's their favorite, but mandating it is weird. And I can already hear the butwhataboutism yet there is already a CNCF certification process and a whole subproject just for testing Kubernetes itself, so do they believe in the tests or not? > The Go templates are tricky to debug, often containing complex logic that results in really confusing error scenarios. The error messages you get from those scenarios are often gibberish And they left off that it is crazypants to use a textual templating language for a whitespace sensitive, structured file format. But, just like the rest of the complaints, it's not like we don't already have replacements, but the network effect is very real and very hard to overcome That barrier of "we have nicer things, but inertia is real" applies to so many domains, it just so happens that helm impacts a much larger audience
• jonenst 1y ago
  What about kustomize and kpt ? I'm using them (instead of helm) but but: * kpt is still not 1.0 * both kustomize and kpt require complex setups to programatically generate configs (even for simple things like replicas = replicasx2)
• a8zjd-p24 1y ago
  [dead]
• draw_down 1y ago
  [dead]
• jitl 1y ago
  I feel like I’m already living in the Kubernetes 2.0 world because I manage my clusters & its applications with Terraform. - I get HCL, types, resource dependencies, data structure manipulation for free - I use a single `tf apply` to create the cluster, its underlying compute nodes, related cloud stuff like S3 buckets, etc; as well as all the stuff running on the cluster - We use terraform modules for re-use and de-duplication, including integration with non-K8s infrastructure. For example, we have a module that sets up a Cloudflare ZeroTrust tunnel to a K8s service, so with 5 lines of code I can get a unique public HTTPS endpoint protected by SSO for whatever running in K8s. The module creates a Deployment running cloudflared as well as configures the tunnel in the Cloudflare API. - Many infrastructure providers ship signed well documented Terraform modules, and Terraform does reasonable dependency management for the modules & providers themselves with lockfiles. - I can compose Helm charts just fine via the Helm terraform provider if necessary. Many times I see Helm charts that are just “create namespace, create foo-operator deployment, create custom resource from chart values” (like Datadog). For these I opt to just install the operator & manage the CRD from terraform directly or via a thin Helm pass through chat that just echos whatever HCL/YAML I put in from Terraform values. Terraform’s main weakness is orchestrating the apply process itself, similar to k8s with YAML or whatever else. We use Spacelift for this.
  • ofrzeta 1y ago
    In a way it's redundant to have the state twice: once in Kubernetes itself and once in the Terraform state. This can lead to problems when resources are modified through mutating webhooks or similar. Then you need to mark your properties as "computed fields" or something like that. So I am not a fan of managing applications through TF. Managing clusters might be fine, though.
    • theyinwhy 1y ago
      Exactly our experience hence we moved to plain K8s objects 6 years ago and never looked back.
    • jitl 1y ago
      That's fair, and I think why more people don't go the Terraform route. Our setup is pretty simple which helps. We treat entire clusters more like namespaces, where a cluster will run a single main application and its support services as a form of "application level availability zone". We still get bin packing for all that stuff with each cluster, but maybe not MAXIMUM BIN PACKING that we'd get if we ran all the applications in a single big cluster, and there's some EKS cost paying for more clusters. We do sometimes have the mutating webhook stuff, for example when running 3rd party JVM stuff we tell the Datadog operator to inject JMX agents into applications using a mutating webhook. For those kinds of things we manage the application using the Helm provider pass-through I mentioned, so what Terraform stores in its state, diffs, and manages on change is the input manifests passed through Helm; for those resources it never inspects the Kubernetes API objects directly - it will just trigger a new helm release if the inputs change on the Terraform side or delete the helm release if removed. This is not a beautiful solution but it works well in practice with minimal fuss when we hit those Kubernetes provider annoyances.
• moomin 1y ago
  Let me add one more: give controllers/operators a defined execution order. Don’t let changes flow both ways. Provide better ways for building things that don’t step on everyone else’s toes. Make whatever replaces helm actually maintain stuff rather than just splatting it out.
  • clvx 1y ago
    This is a hard no for me. This is the whole thing about reconciliation loop. You can just push something to the api/etcd and eventually it will become ready when all the dependencies exist. Now, rejecting manifests because crd’s don’t exist yet is a different discussion. I’m down to have a cache of manifests to be deployed waiting for crd but if the crd isn't deployed, then a garbage collection alike tool removes them from cache. This is what fluxcd and argocd already do in a way but I would like to Have it natively.
    • moomin 1y ago
      I don't think what I'm proposing would prevent that? It would mean you needed things to be DAGs, but it's cyclic dependencies that mess you up anyway.
• dzonga 1y ago
  I thought this would be written along the lines of an lllm going through your code - spinning up a railway file. then say have tf for few of the manual dependencies etc that can't be easily inferred. & get automatic scaling out of the box etc. a more simplified flow rather than wrangling yaml or hcl in short imaging if k8s was a 2-3 max 5 line docker compose like file
• singularity2001 1y ago
  More like wasm?
  • mdaniel 1y ago
    As far as I know one can do that right now, since wasmedge (Apache 2) exposes a CRI interface https://wasmedge.org/docs/develop/deploy/oci-runtime/crun#prerequisites https://wasmedge.org/docs/develop/deploy/oci-runtime/crun#pr... (et al)
• nunez 1y ago
  I _still_ think Kubernetes is insanely complex, despite all that it does. It seems less complex these days because it's so pervasive, but complex it remains. I'd like to see more emphasis on UX for v2 for the most common operations, like deploying an app and exposing it, then doing things like changing service accounts or images without having to drop into kubectl edit. Given that LLMs are it right now, this probably won't happen, but no harm in dreaming, right?
  • Pet_Ant 1y ago
    Kubernetes itself contains so many layers of abstraction. There are pods, which is the core new idea, and it's great. But now there are deployments, and rep sets, and namespaces... and it makes me wish we could just use Docker Swarm. Even Terraform seems to live on just a single-layer and was relatively straight-forward to learn. Yes, I am in the middle of learning K8s so I know exactly how steep the curve is.
    • nunez 1y ago
      The abstractions are fine and make a lot of sense the deeper you get into k8s. CRs for example. CRs are an amazing design concept. The fact that everything is a CR, with some exceptions (like everything in the v1 API) and their controllers are always Pods running somewhere is incredible. Debugging cluster irrops is so much easier once you internalize this. However, this exemplifies my desire for better UX. It would be great if "kubectl logs" accepted an CR kind, which would have the API server would automatically find its associated controller and work with the controller-manager to pump out its logs. Shit, I should make that PR, actually. Even better would be a (much improved) built-in UI that UIops people can use to do something like this. This will become extremely important once the ex-VMware VIadmins start descending onto Kubernetes clusters, which will absolutely happen given that k8s is probably the best vCenter alternative that exists (as non-obvious as that seems right now, though I work at Red Hat and am focused on OpenShift, so take that with two grains of salt)
    • jakewins 1y ago
      The core idea isn’t pods. The core idea is reconciliation loops: you have some desired state - a picture of how you’d like a resource to look or be - and little controller loops that indefinitely compare that to the world, and update the world. Much of the complexity then comes from the enormous amount of resource types - including all the custom ones. But the basic idea is really pretty small. I find terraform much more confusing - there’s a spec, and the real world.. and then an opaque blob of something I don’t understand that terraform sticks in S3 or your file system and then.. presumably something similar to a one-shot reconciler that wires that all together each time you plan and apply?
• mountainriver 1y ago
  We have started working on a sort of Kubernetes 2.0 with https://github.com/agentsea/nebulous https://github.com/agentsea/nebulous -- still pre-alpha Things we are aiming to improve: * Globally distributed * Lightweight, can easily run as a single binary on your laptop while still scaling to thousands of nodes in the cloud. * Tailnet as the default network stack * Bittorrent as the default storage stack * Multi-tenant from the ground up * Live migration as a first class citizen Most of these needs were born out of building modern machine learning products, and the subsequent GPU scarcity. With ML taking over the world though this may be the norm soon.
  • hhh 1y ago
    Wow… Cool stuff, the live migration is very interesting. We do autoscaling across clusters across clouds right now based on pricing, but actual live migration is a different beast
  • znpy 1y ago
    > * Globally distributed Non-requirement? > * Tailnet as the default network stack That would probably be the first thing I look to rip out if I ever was to use that. Kubernetes assuming the underlying host only has a single NIC has been a plague for the industry, setting it back ~20 years and penalizing everyone that's not running on the cloud. Thank god there are multiple CNI implementation. Only recently with Multus (https://www.redhat.com/en/blog/demystifying-multus https://www.redhat.com/en/blog/demystifying-multus) some sense seem to be coming back into that part of the infrastructure. > * Multi-tenant from the ground up How would this be any different from kubernetes? > * Bittorrent as the default storage stack Might be interesting, unless you also mean seeding public container images. Egress traffic is crazy expensive.
    • nine_k 1y ago
      > Non-requirement > the first thing I look to rip out This only shows how varied the requirements are across the industry. One size does not fit all, hence multiple materially different solutions spring up. This is only good.
• Dedime 1y ago
  From someone who was recently tasked with "add service mesh" - make service mesh obsolete. I don't want to install a service mesh. mTLS or some other from of encryption between pods should just happen automatically. I don't want some janky ass sidecar being injected into my pod definition ala linkerd, and now I've got people complaining that cilium's god mode is too permissive. Just have something built-in, please.
  • mdaniel 1y ago
    For my curiosity, what threat model is mTLS and encryption between pods driving down? Do you run untrusted workloads in your cluster and you're afraid they're going to exfil your ... I dunno, SQL login to the in-cluster Postgres? As someone who has the same experience you described with janky sidecars blowing up normal workloads, I'm violently anti service-mesh. But, cert expiry and subjectAltName management is already hard enough, and you would want that to happen for every pod? To say nothing of the TLS handshake for every connection?
  • ahmedtd 1y ago
    Various pieces support pieces for pod to pod mTLS are slowly being brought into the main Kubernetes project. Take a look at https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/4317-pod-certificates https://github.com/kubernetes/enhancements/tree/master/keps/..., which is hopefully landing as alpha in Kubernetes 1.34. It lets you run a controller that issues certificates, and the certificates get automatically plumbed down into pod filesystems, and refresh is handled automatically. Together with ClusterTrustBundles (KEP 3257), these are all the pieces that are needed for someone to put together a controller that distributes certificates and trust anchors to every pod in the cluster.
• benced 1y ago
  I found Kubernetes insanely intuitive coming from the frontend world. I was used to writing code that took in data and made the UI react to that - now I write code that the control panel uses reconciles resources with config.
• znpy 1y ago
  I'd like to add my points of view: 1. Helm: make it official, ditch the text templating. The helm workflow is okay, but templating text is cumbersome and error-prone. What we should be doing instead is patching objects. I don't know how, but I should be setting fields, not making sure my values contain text that are correctly indented (how many spaces? 8? 12? 16?) 2. Can we get a rootless kubernetes already, as a first-class citizen? This opens a whole world of possibilities. I'd love to have a physical machine at home where I'm dedicating only an unprivileged user to it. It would have limitations, but I'd be okay with it. Maybe some setuid-binaries could be used to handle some limited privileged things.
• d4mi3n 1y ago
  I agree with the author that YAML as a configuration format leaves room for error, but please, for the love of whatever god or ideals you hold dear, do not adopt HCL as the configuration language of choice for k8s. While I agree type safety in HCL beats that of YAML (a low bar), it still leaves a LOT to be desired. If you're going to go through the trouble of considering a different configuration language anyway, let's do ourselves a favor and consider things like CUE[1] or Starlark[2] that offer either better type safety or much richer methods of composition. 1. https://cuelang.org/docs/introduction/#philosophy-and-principles https://cuelang.org/docs/introduction/#philosophy-and-princi... 2. https://github.com/bazelbuild/starlark?tab=readme-ov-file#design-principles https://github.com/bazelbuild/starlark?tab=readme-ov-file#de...
  • mdaniel 1y ago
    I repeatedly see this "yaml isn't typesafe" claim but have no idea where it's coming from since all the Kubernetes APIs are OpenAPI, and thus JSON Schema, and since YAML is a superset of JSON it is necessarily typesafe Every JSON Schema aware tool in the universe will instantly know this PodSpec is wrong: kind: 123 metadata: [ {you: wish} ] I think what is very likely happening is that folks are -- rightfully! -- angry about using a text templating language to try and produce structured files. If they picked jinja2 they'd have the same problem -- it does not consider any textual output as "invalid" so jinja2 thinks this is a-ok jinja2.Template("kind: {{ youbet }}").render(youbet=True) I am aware that helm does *YAML* sanity checking, so one cannot just emit whatever crazypants yaml they wish, but it does not then go one step further to say "uh, your json schema is fubar friend"
• fragmede 1y ago
  Instead of yaml, json, or HCL, how about starlark? It's a stripped down Python, used in production by bazel, so it's already got the go libraries.
  • fjasdfwa 1y ago
    kube-apiserver uses a JSON REST API. You can use whatever serializes to JSON. YAML is the most common and already works directly with kubectl. I personally use TypeScript since it has unions and structural typing with native JSON support but really anything can work.
    • mdaniel 1y ago
      Fun fact, while digging into the sibling comment's complaint about the OpenAPI spec, I learned that it actually advertises multiple content-types: application/json application/json;stream=watch application/vnd.kubernetes.protobuf application/vnd.kubernetes.protobuf;stream=watch application/yaml which I presume all get coerced into protobuf before being actually interpreted
  • mdaniel 1y ago
    As the sibling comment points out, I think that would be a perfectly fine helm replacement, but I would never ever want to feed starlark into k8s apis directly
    • dilyevsky 1y ago
      Not directly, first render it to protos. Couple gigs ago I made this - https://github.com/cruise-automation/isopod https://github.com/cruise-automation/isopod
• NathanFlurry 1y ago
  The #1 problem with Kubernetes is it's not something that "Just Works." There's a very small subset of engineers who can stand up services on Kubernetes without having it fall over in production – not to mention actually running & maintaining a Kubernetes cluster on your own VMs. In response, there's been a wave of "serverless" startups because the idea of running anything yourself has become understood as (a) a time sink, (b) incredibly error prone, and (c) very likely to fail in production. I think a Kubernetes 2.0 should consider what it would look like to have a deployment platform that engineers can easily adopt and feel confident running themselves – while still maintaining itself as a small-ish core orchestrator with strong primitives. I've been spending a lot of time building Rivet to itch my own itch of an orchestrator & deployment platform that I can self-host and scale trivially: https://github.com/rivet-gg/rivet https://github.com/rivet-gg/rivet We currently advertise as the "open-source serverless platform," but I often think of the problem as "what does Kubernetes 2.0 look like." People are already adopting it to push the limits into things that Kubernetes would traditionally be good at. We've found the biggest strong point is that you're able to build roughly the equivalent of a Kubernetes controller trivially. This unlocks features more complex workload orchestration (game servers, per-tenant deploys), multitenancy (vibe coding per-tenant backends, LLM code interpreters), metered billing per-tenant, more powerful operators, etc.
  • stuff4ben 1y ago
    I really dislike this take and I see it all the time. Also I'm old and I'm jaded, so it is what it is... Someone decides X technology is too heavy-weight and wants to just run things simply on their laptop because "I don't need all that cruft". They spend time and resources inventing technology Y to suit their needs. Technology Y gets popular and people add to it so it can scale, because no one runs shit in production off their laptops. Someone else comes along and says, "damn, technology Y is too heavyweight, I don't need all this cruft..." "There are neither beginnings nor endings to the Wheel of Time. But it was a beginning.”
    • adrianmsmith 1y ago
      It’s also possible for things to just be too complex. Just because something’s complex doesn’t necessarily mean it has to be that complex.
• nikisweeting 1y ago
  It should natively support running docker-compose.yml configs, essentially treating them like swarm configurations and "automagically" deploying them with sane defaults for storage and network. Right now the gap between compose and full-blown k8s is too big.
  • mdaniel 1y ago
    So, what I'm hearing is that it should tie itself to a commercial company, who now have a private equity master to answer to, versus an open source technology run by a foundation Besides, easily half of this thread is whining about helm for which docker-compose has no answer whatsoever. There is no $(docker compose run oci://example.com/awesome --version 1.2.3 --set-string root-user=admin)
    • arccy 1y ago
      docker-compose -f $(curl example.com/v1.2.3/awesome-compose.yaml) helm is an anti pattern, nobody should touch it with a 10 foot pole
    • nikisweeting 1y ago
      What? I didn't say any of that, OCI and docker-compose are both open formats, do it with podman for all I care.
  • ChocolateGod 1y ago
    > Right now the gap between compose and full-blown k8s is too big. It's Hashicorp so you have to be weary, but Nomad fills this niche https://developer.hashicorp.com/nomad https://developer.hashicorp.com/nomad
    • nikisweeting 1y ago
      I really want to love nomad but I've never been able to get it working well for my workloads. I'm not a big fan of HCL, I really just want docker-compose++.
  • 1y ago
• woile 1y ago
  What bothers me: - it requires too much RAM to run in small machines (1GB RAM). I want to start small but not have to worry about scalability. docker swarm was nice in this regard. - use KCL lang or CUE lang to manage templates
• otterley 1y ago
  First, K8S doesn't force anyone to use YAML. It might be idiomatic, but it's certainly not required. `kubectl apply` has supported JSON since the beginning, IIRC. The endpoints themselves speak JSON and grpc. And you can produce JSON or YAML from whatever language you prefer. Jsonnet is quite nice, for example. Second, I'm curious as to why dependencies are a thing in Helm charts and why dependency ordering is being advocated, as though we're still living in a world of dependency ordering and service-start blocking on Linux or Windows. One of the primary idioms in Kubernetes is looping: if the dependency's not available, your app is supposed to treat that is a recoverable error and try again until the dependency becomes available. Or, crash, in which case, the ReplicaSet controller will restart the app for you. You can't have dependency conflicts in charts if you don't have dependencies (cue "think about it" meme here), and you install each chart separately. Helm does let you install multiple versions of a chart if you must, but woe be unto those who do that in a single namespace. If an app truly depends on another app, one option is to include the dependency in the same Helm chart! Helm charts have always allowed you to have multiple application and service resources.
  • wutwutwat 1y ago
    try to use a crd before its definition is installed. hard failure. no graceful retries. sometimes dependency order is forced
    • otterley 1y ago
      Surely there’s a way to fix this problem other than introduce dependency ordering.
      • wutwutwat 1y ago
        one thing depends on another. how else would you solve for ordered dependencies other than ordering dependencies?
  • Arrowmaster 1y ago
    You say supposed to. That's great when building your own software stack in house but how much software is available that can run on kubenetes but was created before it existed. But somebody figured out it could run in docker and then later someone realized it's not that hard to make it run in kubenetes because it already runs in docker. You can make an opinionated platform that does things how you think is the best way to do them, and people will do it how they want anyway with bad results. Or you can add the features to make it work multiple ways and let people choose how to use it.
• cyberax 1y ago
  I would love: 1. Instead of recreating the "gooey internal network" anti-pattern with CNI, provide strong zero-trust authentication for service-to-service calls. 2. Integrate with public networks. With IPv6, there's no _need_ for an overlay network. 3. Interoperability between several K8s clusters. I want to run a local k3s controller on my machine to develop a service, but this service still needs to call a production endpoint for a dependent service.
  • mdaniel 1y ago
    To the best of my knowledge, nothing is stopping you from doing any of those things right now. Including, ironically, authentication for pod-to-pod calls, since that's how Service Accounts work today. That even crosses the Kubernetes API boundary thanks to IRSA and, if one were really advanced, any OIDC compliant provider that would trust the OIDC issuer in Kubernetes. The eks-anywhere distribution even shows how to pull off this stunt from your workstation via publishing the JWKS to S3 or some other publicly resolvable https endpoint I am not aware of any reason why you couldn't connect directly to any Pod, which necessarily includes the kube-apiserver's Pod, from your workstation except for your own company's networking policies
    • cyberax 1y ago
      Service accounts are _close_ to what I want, but not quite. They are not quite seamless enough for service-to-service calls. > I am not aware of any reason why you couldn't connect directly to any Pod, which necessarily includes the kube-apiserver's Pod, from your workstation except for your own company's networking policies I don't think I can create a pod with EKS that doesn't have any private network?
      • clvx 1y ago
        You may want to take a look to SPIRE. I think that's what you are referring to.
      • p_l 1y ago
        That's an EKS problem, not a k8s problem. K8s will happily let you run a pod with host network, and even the original "kubenet" network implementation allowed directly calling pods if your network wasn't more damaged than a brain after multiple strokes combined with headshot tank cannon (aka most enterprise networks in my experience)
• solatic 1y ago
  I don't get the etcd hate. You can run single-node etcd in simple setups. You can't easily replace it because so much of the Kubernetes API is a thin wrapper around etcd APIs like watch that are quite essential to writing controllers and don't map cleanly to most other databases, certainly not sqlite or frictionless hosted databases like DynamoDB. What actually makes Kubernetes hard to set up by yourself are a) CNIs, in particular if you both intend to avoid cloud-provider specific CNIs, support all networking (and security!) features, and still have high performance; b) all the cluster PKI with all the certificates for all the different components, which Kubernetes made an absolute requirement because, well, prpduction-grade security. So if you think you're going to make an "easier" Kubernetes, I mean, you're avoiding all the lessons learned and why we got here in the first place. CNI is hardly the naive approach to the problem. Complaining about YAML and Helm are dumb. Kubernetes doesn't force you to use either. The API server anyway expects JSON at the end. Use whatever you like.
  • mdaniel 1y ago
    > I don't get the etcd hate. I'm going out on a limb to say you've only ever used hosted Kubernetes, then. A sibling comment mentioned their need for vanity tooling to babysit etcd and my experience was similar. If you are running single node etcd, that would also explain why you don't get it: you've been very, very, very, very lucky never to have that one node fail, and you've never had to resolve the very real problem of ending up with just two etcd nodes running
    • solatic 1y ago
      No, I ran three-node etcd clusters for years that Kops set up. Kops deployed an etcd-operator to babysit them and take care of backups and the like. It was set-and-forget. We had controlled disruptions all the time as part of Kubernetes control plane updates, no issues. And you know... etcd supports five-node clusters, precisely to help support people who are paranoid about extended single node failure.
• 0xbadcafebee 1y ago
  > Ditch YAML for HCL Hard pass. One of the big downsides to a DSL is it's linguistic rather than programmatic. It depends on a human to learn a language and figuring out how to apply it correctly. I have written a metric shit-ton of terraform in HCL. Yet even I struggle to contort my brain into the shape it needs to think of how the fuck I can get Terraform to do what I want with its limiting logic and data structures. I have become almost completely reliant on saved snippet examples, Stackoverflow, and now ChatGPT, just to figure out how to deploy the right resources with DRY configuration in a multi-dimensional datastructure. YAML isn't a configuration format (it's a data encoding format) but it does a decent job at not being a DSL, which makes things way easier. Rather than learn a language, you simply fill out a data structure with attributes. Any human can easily follow documentation to do that without learning a language, and any program can generate or parse it easily. (Now, the specific configuration schema of K8s does suck balls, but that's not YAML's fault) > I still remember not believing what I was seeing the first time I saw the Norway Problem It's not a "Norway Problem". It's a PEBKAC problem. The "problem" is literally that the user did not read the YAML spec, so they did not know what they were doing, then did the wrong thing, and blamed YAML. It's wandering into the forest at night, tripping over a stump, and then blaming the stump. Read the docs. YAML is not crazy, it's a pretty simple data format. > Helm is a perfect example of a temporary hack that has grown to be a permanent dependency Nobody's permanently dependent on Helm. Plenty of huge-ass companies don't use it at all. This is where you proved you really don't know what you're talking about. (besides the fact that helm is a joy to use compared to straight YAML or HCL)
  • deleted 1y ago
    [deleted]
• rcarmo 1y ago
  One word: Simpler.
  • TacticalCoder 1y ago
    [dead]
• aranw 1y ago
  YAML and Helm are my two biggest pain points with k8s and I would love to see them replaced with something else. CUE for YAML would be really nice. As for replacing Helm, I'm not too sure really. Perhaps with YAML being replaced by CUE maybe something more powerful and easy to understand could evolve from using CUE?
• fideloper 1y ago
  "Low maintenance", welp. I suppose that's true in one sense - in that I'm using EKS heavily, and don't maintain cluster health myself (other than all the creative ways I find to fuck up a node). And perhaps in another sense: It'll try its hardest to run some containers so matter how many times I make it OOMkill itself. Buttttttttt Kubernetes is almost pure maintenance in reality. Don't get me wrong, it's amazing to just submit some yaml and get my software out into the world. But the trade off is pure maintenance. The workflows to setup a cluster, decide which chicken-egg trade-off you want to get ArgoCD running, register other clusters if you're doing a hub-and-spoke model ... is just, like, one single act in the circus. Then there's installing all the operators of choice from https://landscape.cncf.io/ https://landscape.cncf.io/. I mean that page is a meme, but how many of us run k8s clusters without at least 30 pods running "ancillary" tooling? (Is "ancillary" the right word? It's stuff we need, but it's not our primary workloads). A repeat circus is spending hours figuring out just the right values.yaml (or, more likely, hours templating it, since we're ArgoCD'ing it all, right?) > As an side, I once spent HORUS figuring out to (incorrectly) pass boolean values around from a Secrets Manager Secret, to a k8s secret - via External Secrets, another operator! - to an ArgoCD ApplicationSet definition, to another values.yaml file. And then you have to operationalize updating your clusters - and all the operators you installed/painstakingly configured. Given the pace of releases, this is literally, pure maintenance that is always present. Finally, if you're autoscaling (Karpenter in our case), there's a whole other act in the circus (wait, am I still using that analogy?) of replacing your nodes "often" without downtime, which gets fun in a myriad of interesting ways (running apps with state is fun in kubernetes!) So anyway, there's my rant. Low fucking maintenance!
  • ljm 1y ago
    I’ve been running k3s on hetzner for over 2 years now with 100% uptime. In fact, it was so low maintenance that I lost my SSH key for the master node and I had to reprovision the entire cluster. Took about 90 mins including the time spent updating my docs. If it was critical I could have got that down to 15 mins tops. 20€/mo for a k8s cluster using k3s, exclusively on ARM, 3 nodes 1 master, some storage, and a load balancer with automatic dns on cloudflare.
    • 1y ago
• hosh 1y ago
  While we're speculating: I disagree that YAML is so bad. I don't particularly like HCL. The tooling I use don't care though -- as long as I can stil specify things in JSON, then I can generate (not template) what I need. It would be more difficult to generate HCL. I'm not a fan of Helm, but it is the de facto package manager. The main reason I don't like Helm has more to do with its templating system. Templated YAML is very limiting, when compared to using a full-fledged language platform to generate a datastructure that can be converted to JSON. There are some interesting things you can do with that. (cdk8s is like this, but it is not a good example of what you can do with a generator). On the other hand, if HCL allows us to use modules, scoping, and composition, then maybe it is not so bad after all.
  • hamishwhc 1y ago
    > cdk8s is like this, but it is not a good example of what you can do with a generator As a die-hard fan of cdk8s (and the AWS CDK), I am curious to hear about this more. What do you feel is missing or could be done better?
    • hosh 1y ago
      I used Typescript cdk8s, or tried to. Manipulating the objects were unwieldy. I wrote https://github.com/matsuri-rb/matsuri https://github.com/matsuri-rb/matsuri ... I have not really promoted it. I tried cdk8s because the team I was working with used Typescript and not Ruby, and I thought cdk8s would have worked well, since it generates manifests instead of templates it. Matsuri takes advantage of language features in Ruby not found in Typescript (and probably not Python) that allows for being able to compose things together. Instead of trying to model objects, it is based around constructing a hash that is then converted to JSON. It uses fine-grained method overloading to allow for (1) mixins, and (2) configuration from default values. The result is that with very little ceremony, I can get something to construct the manifest I needed. There were a lot of extra ceremony and boiler plate needed to do anything in the Typescript cdk8s. While I can use class inheritance with Matsuri, over the years, I had moved away from it because it was not as robust as mixins (compositions). It was quite the shock to try to work with Typescript cdk8s and the limitations of that approach. The main reason I had not promoted Matsuri is because this tool is really made for people who know Ruby well ... but that might have been a career mistake not to try. Instead of having 10 years to slowly get enough support behind it, people want something better supported such as cdk8s or Helmfiles.
• mikeocool 1y ago
  How about release 2.0 and then don’t release 2.1 for a LONG time. I get that in the early days such a fast paced release/EOL schedule made sense. But now something that operates at such a low level shouldn’t require non-security upgrades every 3 months and have breaking API changes at least once a year.
• akdor1154 1y ago
  I think the yaml / HCL and package system overlap.. I wouldnt so much go HCL as something like JSONnet, Pkl, Dhall, or even (inspiration not recommendation) Nix - we need something that allows a schema for powering an LSP, with enough expressitivity to void the need for Helm's templating monstrosity, and ideally with the ability for users to override things that library/package authors haven't provided explicit hooks for. Does that exist yet? Probably not, but the above languages are starting to approach it.
  • pas 1y ago
    k8s could provide a strongly-typed DSL, like cdk8s+ it's okay to be declarative (the foundation layer), but people usually think in terms of operations (deploy this, install that, upgrade this, assign some vhost, filter that, etc.) and even for the declarative stuff, a builder pattern works well, it can be super convenient, sufficient terse, and typed, and easily composed (and assembled via normál languages, not templates) ... well. anyway. maybe by the time k8s2 rolls around Go will have at least normal error handling.
• ra7 1y ago
  The desired package management system they describe sounds a lot like Carvel's kapp-controller (https://carvel.dev/kapp-controller/ https://carvel.dev/kapp-controller/). The Carvel ecosystem, which includes its own YAML templating tool called 'ytt', isn't the most user friendly in my experience and can feel a bit over-engineered. But it does get the idea of Kubernetes-native package management using CRDs mostly right.
• 1oooqooq 1y ago
  systemd, but distributed. and with config files redone from scratch (and ideally not in yaml)
  • mdaniel 1y ago
    I'll bite: what is the systemd primitive for secret injection? for declaring which EBS volume it needs mounted where? (I don't mean Mount= I mean a conceptual MountVolumeByTag[Name=my-pgdata-0]=/var/lib/postgres/data ) Oh, that's another great question: what is the systemd primitive for making 3 copies of a service if you find that one copy isn't getting it done? Maybe all of those "implementation details" are what you meant by "config files redone" and then perhaps by the follow-on "simple matter of programming" to teach C how to have new behaviors that implement those redone config files
    • 1oooqooq 1y ago
      those are available in machine files.
    • arccy 1y ago
      secret injection: systemd-creds the other parts don't exist because that's really only useful for distributed systems, and systems is not that right now
  • 15155 1y ago
    How is a systemd service file functionally different from a Pod spec?
    • 1oooqooq 1y ago
      exactly. now add the missing distributed part.
• bigcat12345678 1y ago
  》Ditch YAML for HCL I maintained borgcfg 2015-2019 The biggest lesson k8s drew from borg is to replace bcl (borgcfg config language) with yaml (by Brian Grant) Then this article suggests to reverse Yep, knowledge not experienced is just fantasy
  • Too 1y ago
    Interested to hear more about this. Care to elaborate more? Maybe there are already articles written about this?
  • simoncion 1y ago
    I've spent many, many, many years working with YAML at $DAYJOB. YAML is just a really bad choice for configuration files that are longer than a handful of lines. The fact that YAML encourages you to never put double-quotes around strings means that (based on my experience at $DAYJOB) one gets turbofucked by YAML mangling input by converting something that was intended to be a string into another datatype at least once a year. On top of that, the whitespace-sensitive mode makes long documents extremely easy to get lost in, and hard to figure out how to correctly edit. On top of that, the fact that the default mode of operation for every YAML parser I'm aware of emits YAML in the whitespace-sensitive mode means that approximately zero of the YAML you will ever encounter is written in the (more sane) whitespace-insensitive mode. [0] It may be that bcl is even worse than YAML. I don't know, as I've never worked with bcl. YAML might be manna from heaven in comparison... but that doesn't mean that it's actually good. [0] And adding to the mess is the fact that there are certain constructions (like '|') that you can't use in the whitespace-insensitive mode... so some config files simply can't be straightforwardly expressed in the other mode. "Good" job there, YAML standard authors.
  • jitl 1y ago
    Just because borgcfg is painful (?) and YAML is a better option for the public, doesn't mean we can't continue and improve on YAML with more limited expression syntax. Terraform/HCL is often annoying because it doesn't give users tools like easy function declarations for re-use. You get one re-use abstraction: create a module, which is heavy-weight - needs a new directory, "provider version" declarations, a lot of boilerplate per input and output. In return this means the amount of creative shenanigans one needs to understand is typically pretty low.
• anonfordays 1y ago
  So odd seeing all the HCL hate here. It's dead simple to read, much more so than YAML. It grew out of Mitchell's hatred for YAML. If your "developers" are having problems with HCL, it's likely a skills issue.
  • Too 1y ago
    Yeah. I can't get the amount of hate seen here either. It's super natural to read and has very few concepts to learn. Just resources, lists, sets, dictionaries, variables, ternary operator, function calls and templates. Paired with a LSP and syntax highlighting it gets even easier to see the boundaries between code and variables. The two different ways to write dicts are a bit strange and foreach has some gotchas but honestly not a big deal. Especially not when put in comparison to YAML which has nothing even close and even more invisible footguns. Seeing comments like "it does not have if-statements". Duh, it's not imperative, for many good reasons. Not understanding that yeah it's clearly a skills issue.
    • mystifyingpoi 1y ago
      > Duh, it's not imperative, for many good reasons You can be declarative and still have imperative constructs hidden behind pure functions let's say. That's how Ansible does it - it is super useful to define a custom filter that's called like "{{ myvariable | my_filter }}" but underneath there is a bunch of imperative Python datastructure wrangling (without visible side effects, of course - just in memory stuff), with arbitrary code. Doing the same in HCL is I believe impossible in general case.
• moondev 1y ago
  I was all ready to complain about HCL because of the horrible ergonomics of multiline strings, which would be a deal breaker as a default config format. I just looked though and it seems they now support it in a much cleaner fashion. https://developer.hashicorp.com/terraform/language/expressions/strings#heredoc-strings https://developer.hashicorp.com/terraform/language/expressio... This actually makes me want to give HCL another chance
  • mdaniel 1y ago
    Be careful about linking to terraform docs for HCL, as it can be very easy to blur terraform's implementation of the behaviors from HCL itself https://github.com/hashicorp/hcl/blob/v2.23.0/hclsyntax/spec.md#template-expressions https://github.com/hashicorp/hcl/blob/v2.23.0/hclsyntax/spec... I believe is the actual language specification of heredocs
• lukaslalinsky 1y ago
  What I'd like to see the most is API stability. At small scale, it's extremely hard to catch up with Kubernetes releases and the whole ecosystem is paced around those. It's just not sustainable running Kubernetes unless you have someone constantly upgrading everything (or you pay someone/something to do it for you). By the years, we should have a good idea for a wide range of APIs that are useful and stick with those.
• fragmede 1y ago
  Kubernetes 2.0 would just be kubernetes with batteries included
• brikym 1y ago
  The bit about Helm templating resonated with me. Stringly typed indentation hell.
• jerry1979 1y ago
  Not sure where buildkit is at these days, but k8s should have reproducible builds.
• jeffrallen 1y ago
  Simpler? A guy can dream anyway.
• Too 1y ago
  > "Kubernetes isn't opinionated enough" yes please, and then later... > "Allow etcd swap-out" I don't have any strong opinions about etcd, but man... can we please just have one solution, neatly packaged and easy to deploy. When your documentation is just a list of abstract interfaces, conditions or "please refer to your distribution", no wonder that nobody wants to maintain a cluster on-prem.
• jillesvangurp 1y ago
  > What would a Kubernetes 2.0 look like A lot simpler hopefully. It never really took off but docker swarm had a nice simplicity to it. Right idea, but Docker Inc. mismanaged it. Unfortunately, Kubernetes evolved into a bit of a monster. Designed to be super complicated. Full of pitfalls. In need of vast amounts of documentation, training, certification, etc. Layers and layers of complexity, hopelessly overengineered. I.e. lots of expensive hand holding. My mode with technology is that if the likes of Red Hat, IBM, etc. get really excited: run away. Because they are seeing lots of dollars for exactly the kind of stuff I don't want in my life. Leaving Kubernetes 2.0 to the people that did 1.0 is probably just going to lead to more of the same. The people behind this require this to be a combination of convoluted and hard to use. That's how they make money. If it was easy, they'd be out of business.
• brunoborges 1y ago
  I'm not surprised, but somewhat disappointed that the author did not mention Java EE Application Servers. IMO one of the disadvantages of that solution compared to Kuberentes was that they were Java-specific and not polyglot. But everything else about installation, management, scalling, upgrading, etc, was quite well done especially for BEA/Oracle WebLogic.
• donperignon 1y ago
  Probably like docker swarm
• ExoticPearTree 1y ago
  I wish for k8s 2.0 to be less verbose when it comes to deploying anything on it. I want to be able to say in two or five lines of YAML: - run this as 3 pods with a max of 5 - map port 80 to this load balancer - use this environment variables I don't really care if it's YAML or HCL. Moving from YAML to HCL it's going to be an endless issue of "I forgot to close a curly bracket somewhere" versus "I missed an ident somewhere".
  • edude03 1y ago
    knative might be what you're looking for
  • kachapopopow 1y ago
    Helm does that, kinda. I use the bitnami base chart for my own deployments and it works pretty well.
• liampulles 1y ago
  The whole reason that Kubernetes is driven off YAML (or JSON) is that the YAML is a source of truth of what the user intention is. Piping HCL, which has dynamically determined values, directly to the k8s API would make it harder to figure out what the desired state was at apply time when you are troubleshooting issues later. The easy solution here is to generate the YAML from the HCL (or from helm, or whatever other abstraction you choose) and to commit and apply the YAML. More broadly, I think Kubernetes has a bit of a marketing problem. There is a core 20% of the k8s API which is really good and then a remaining 80% of niche stuff which only big orgs with really complex deployments need to worry about. You likely don't need (and should not use) that cloud native database that works off CRDs. But if you acknowledge this aspect of its API and just use the 20%, then you will be happy.
• darqis 1y ago
  Sorry, but k8s is not low maintenance. You have to keep the host systems updated and you have to keep the containers updated. And then you have paradigm changes like ingress falloff and emergence of gateway API. k8s is very time intensive, that's why I am not using it. It adds complexity and overhead. That might be acceptable for large organizations, but not for the single dev or small company.
• smetj 1y ago
  Hard no to HCL! Yaml is simple. A ton of tools can parse and process it. I understand the author's gripes (long files, indentation, type confusions) but then I would even prefer JSON as an alternative. Just use better tooling that helps you address your problems/gripes. Yaml is just fine.
• smetj 1y ago
  Kubernetes has all the capabilities to address the needs of pretty much every architecture/design out there. You only need one architecture/design for your particular use case. Nonetheless, you will have to cary all that weight with you, although you will never use it.
• vbezhenar 1y ago
  I have some experience with Kubernetes, both managed and self-maintained. So here are my wishes: 1. Deliver Kubernetes as a complete immutable OS image. Something like Talos. It should auto-update itself. 2. Take opinionated approach. Do not allow for multiple implementations of everything, especially as basic as networking. There should be hooks for integration with underlying cloud platform, of course. 3. The system must work reliably out of the box. For example, kubeadm clusters are not set up properly, when it takes to memory limits. You could easily make node unresponsive by eating memory in your pod. 4. Implement built-in monitoring. Built-in centralized logs. Built-in UI. Right now, kubeadm cluster if not usable. You need to spend a lot of time, intalling prometheus, loki, grafana, configure dashboards, configure every piece of software. Those are very different softwares from different vendors. It's a mess. It requires a lot of processing power and RAM to work. It should not be like that. 5. Implement user management, with usernames and passwords. Right now you need to set up keycloak, configure oauth authentication, complex realm configuration. It's a mess. It requires a lot of RAM to work. It should not be like that. 6. Remove certificates, keys. Cluster should just work, no need to refresh anything. Join node and it stays there. So basically I want something like Linux. Which just works. I don't need to set up Prometheus to look at my 15-min load average or CPU consumption. I don't need to set up Loki to look at logs, I have journald which is good enough for most tasks. I don't need to install CNI to connect to network. I don't need to install Keycloak to create user. It won't stop working, because some internal certificate has expired. I also want lower resources consumption. Right now Kubernetes is very hungry. I need to dedicate like 2 GiB RAM to master node, probably more. I don't even want to know about master nodes. Basic Linux system eats like 50 MiB RAM. I can dedicate another 50 MiB to Kubernetes, rest is for me, please. Right now it feels that Kubernetes was created to create more jobs. It's a very necessary system, but it could be so much better.
• nuker 1y ago
  No one yet mentioned AWS ECS and Fargate?
  • mdaniel 1y ago
    Because I can't run AWS ECS nor Fargate on my laptop Also, unless something has radically changed recently, ECS == Amazon Managed Docker (one can see a lot of docker-compose inspired constructs: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-definition-template.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/... ) which, as this thread is continuously debating, reasonable people can differ on whether that's a limitation of a simplification
• bionhoward 1y ago
  Would rust be a lot better than HCL to replace yaml? Just learning Kubernetes and want to say I hope Kubernetes 2.0 uses a “real” programming language with a decent type system. A big benefit could be for the infrastructure language to match the developer language. However, knowing software, reinventing something like Kubernetes is a bottomless pit type of task, best off just dealing with it and focusing on the Real Work (TM), right?
• dhorthy 1y ago
  Overall I like this but confused about this part “Yaml doesn’t enforce types but HCL does” Is the same schema-based validation that is 1) possible client-side with HCL and 2) enforced server-side by k8s not also trivial to enforce client side in an ide?
• AtlasBarfed 1y ago
  In my opinion we should start at the command line. If you want to run a program on a computer, the most basic way is to open a command line and invoke the program. And that executes it on one computer number CPUs TBD. But with modern networking primitives and foundations, why can I not open a command line and have a concise way of orchestrating and execution of a program across multiple machines? I have tried several times to do this writing utility code for Cassandra. I got in my opinion very temptingly close to being able to do this. Likewise with docker, vagrant, and yes, kubernetes, with their CLI interfaces for running commands on containers, the CLI fundamentals are also there. Others taking a shot at this are salt, stack, ansible and those types of things, but they really seem to be concerned mostly about Enterprise contracts and at the core of pure CLI execution. Security is really a pain in the ass when it comes to things like this. Your CLI prompt has a certain security assurance with it. You've already logged in. That's a side note. One of the frustrations I started running to as I was doing this is the Enterprise obsession with getting a manual login / totp code still access anything. Holy hell do I have to jump through hopes in order to automate things across multiple machines when they have totp barriers for accessing them. The original kubernetes kind of handwaved a lot of this away by forcing the removal, jump boxes, a flat network plane, etc.
• ziggure 1y ago
  What I've learned comments here is that k8s is essentially already perfect, YAML is actually awesome, and that any criticism of k8s just proves the ignorance of the critic. I guess this means k8s 2.0 should look exactly like k8s 1.0.
• pjmlp 1y ago
  Hopefully written in a more developer friendly language like Rust. Kubernetes is one of the few reasons I need to care about Go.
  • mdaniel 1y ago
    I know I'm running the risk, but for my curiosity what makes Rust more developer friendly in your mind? I have seen my fair share of golang panics in the Kubernetes ecosystem, but I can't recall very many memory corruption issues that would warrant porting off of golang
    • pjmlp 1y ago
      Not having a culture anti-language design, that praises lack of features, only to get them several years later, partly broken, as they have to accommodate into what exists, instead of designing them from the beginning. In Go circles even a language like Java is considered PhD skill level programming languages, it is sooo hard. Pascal and C like enumerations require a kludge with iota and const, dependencies was an whole show between community and Google, generics took their sweet time and the implementation has a few gotchas , makes direct references to git repos for packages and requires tricks with HTML meta headers to redirect requests, if != err boilerplate,... Had it not been for Docker and Kubernetes adoption success, it might have not taken off.