5 ms·
How I think about Kubernetes
- NewJazz 10mo agoLove the HN title mod here lol
- stavros 10mo agoHN removes a "how" if the post starts with it, sometimes making it hilarious.
- dkdcio 10mo agoI’ve seen this a few times now, what’s the context/background on why this is done?
- NewJazz 10mo agoI think maybe they check for "How to [...]" and other variations? "How I broke TLS 1.3" -> "I broke TLS 1.3"
- dkdcio 10mo agoyeah but why?
- JCattheATM 10mo agoOver-engineering to solve a problem that doesn't exist, thereby making one.
- dkdcio 10mo agosure…what’s the motivation? is this seriously that difficult of a question to answer? this “solution” was put in place for a reason, what’s the reason? does nobody actually know?
- NewJazz 10mo agoSomeone was annoyed by certain cliche titles I guess. They also remove numbers. So if you post "1850 Bloody Island Massacre" it removes the number. But it is meant to remove eg. "13 reasons why you should impeac[...]" B/c apparently the number of reasons why is too much of a cliche or bother? Idk. Email dang haha
- zem 10mo agoyeah, really wish they would fix that one!
- frisovv 10mo agoTbh the missing how is probably why I followed the link. And I appreciated the post, so net positive outcome here.
- DonHopkins 10mo agoOtherwise you might confuse it with the HN mod tomhow. Same reason they remove "dang" if the post starts with it, like the discussion about "Dang! Who ate the middle out of the daddy longlegs". https://ifunny.co/picture/dang-who-ate-the-middle-out-of-the-daddy-longlegs-OeLw20CcA https://ifunny.co/picture/dang-who-ate-the-middle-out-of-the...
- websiteapi 10mo agoI always wonder if things can be simpler. When you think of a really simple DB you think of SQLite. What's the really simple K8s? Even doing a single node deployment these days seems complicate with Prometheus, Grafana, etc. etc. docker/podman compose up with quadlets and all of this stuff just seems so eh. I really like the idea of something like Firebase, but it never seems to work out or just move the complexity to the vendor, which is fine, but I like knowing I can roll my own.
- eyeris 10mo agoBig question is which feature subset you want to replicate. Kubernetes means everything to everyone. At its core, I think it’s being able to read/write distributed state (which doesn’t need to be etcd) and being able for all the components (especially container hosts) to follow said state. But the ecosystem has expanded significantly beyond that.
- jauntywundrkind 10mo agoIMO this is what keeps people from building systems that might challenge kubernetes. Everyone wants to say Kuberentes is too complex, so we built something that does much less. I respect that! But I think it usually fails to grok what Kubernetes is and why it's such an interesting and vital rallying point, that's so thoroughly captured our systems-making. Let's look at the premise: > That’s why I like to think of Kubernetes as a runtime for declarative infrastructure with a type system. You can go build a simple way to deploy containers or ship apps: but you are missing what I think allows Kubernetes to be such a big tent, thats a core useful platform for so many. Kubernetes works the same for all types, for everything you want to manage. It's the same desired state management + autonomic systems patterns, whatever you are doing. An extensible platform with a very simple common core. There are other takes and other tries, but managing desired state for any kind of type is a huge win that allows many people to find their own uses for kube, that is absolutely the cornerstone to it's popularity. If you do want less, the one project I'd point to that is kubernetes without the kubernetes complexity is KCP. It's just the control plane. It doesn't do anything at all. This to me is much simpler. It's not finding a narrowly defined use case to focus on, it's distilling out the general system into it's simplest parts. Rebuilding a good simple bespoke app container launching platform around KCP would be doable, and maintain the overarching principles that make Kube actually interesting. I seriously think there is something deeply rotten with our striving for simplicity. I know we've all been burned, and there's so often we want to throw up our hands, and I get it. But the way out is through. I'd rather dance the dance & try to scout for better further futures, than reject & try to walk back.
- zsoltkacsandi 10mo ago> Thinking of Kubernetes as a runtime for declarative infrastructure instead of a mere orchestrator results in very practical approaches to operate your cluster. Unpopular opinion, but the source of most of the problems I've seen with infrastructures using Kubernetes came from exactly this kind of approach. Problems usually come when we use tools to solve things that they weren't made for. That is why - in my opinion - it is super important to treat a container orchestrator a container orchestrator.
- szundi 10mo agoIt would have helped if you tell us why you don’t like this approach.
- zsoltkacsandi 10mo agoIt's right there: > the source of most of the problems I've seen with infrastructures using Kubernetes came from exactly this kind of approach But some more concrete stories: Once, while I was on call, I got paged because a Kubernetes node was running out of disk space. The root cause was the logging pipeline. Normally, debugging a "no space left on device" issue in a logging pipeline is fairly straightforward, if the tools are used as intended. This time, they weren't. The entire pipeline was managed by a custom-built logging operator, designed to let teams describe logging pipelines declaratively. The problem? The resource definitions alone were around 20,000 lines of YAML. In the middle of the night, I had to reverse-engineer how the operator translated that declarative configuration into an actual pipeline. It took three days and multiple SREs to fully understand and fix the issue. Without such a declarative magic it takes usually 1 hour to solve such an issue. Another example: external-dns. It's commonly used to manage DNS declaratively in Kubernetes. We had multiple clusters using Route 53 in the same AWS account. Route 53 has a global API request limit per account. When two or more clusters tried to reconcile DNS records at the same time, one would hit the quota. The others would partially fail, drift out of sync, and trigger retries - creating one of the messiest cross-cluster race conditions I've ever dealt with. And I have plenty more stories like these.
- 10mo ago
- btown 10mo agoOne approach if "dang it, someone/I needed to use kubectl during the outage, how do we get gitops/poor-mans-gitops back in place to match reality" is, either agentically-looping or artisanally-looping, to try simple gitops configurations (or diffs to current gitops configurations) until a dry-run diff with your live configuration results in no changes. For instance, with Helm, I've had success using Helmfile's diffs (which in turn use https://github.com/databus23/helm-diff https://github.com/databus23/helm-diff) to do this. There's more of a spectrum between these than you think, in a way that can be agile for small teams without dedicated investment in gitops. Even with the messes that can occur, I'd take it over the Heroku CLI any day.
- blackjack_ 10mo agoYes, there is a term for a system that handles a declarative state of infrastructure and does reconciliation versus current state; a control plane. We have been talking about control planes in devops/ SRE for a number of years now! Welcome to the conversation.
- anymouse123456 10mo agoThe allure of declarative approaches to complex problem solving has finally been worn down to nothing for me and Kubernetes was the last straw, nearly 10 years ago. The mental gymnastics required to express oneself in yaml, rather than, say, literally anything else, invariably generates a horror show of extremely verbose boilerplate, duplication, bloat, delays and pain. If you're not Google, please for the love of god, please consider just launching a monolith and database on a Linux box (or two) in the corner and see how beautifully simple life can be. They'll hum along quietly serving many thousands of actual customers and likely cost less to purchase than a single month (or at worst, quarter) of today's cloud-based muggings. When you pay, you'll pay for bandwidth and that's real value that also happens to make your work environment more efficient.
- themgt 10mo agoIf you're not Google, please for the love of god, please consider just launching a monolith and database on a Linux box (or two) in the corner and see how beautifully simple life can be. You can literally get a Linux box (or two) in the corner and run: curl -sfL https://get.k3s.io | sh - cat <<EOF | kubectl apply -f - ...(json/yaml here) EOF How am I installing a monolith and a database on this Linux box without Kubernetes? Be specific, just show the commands for me to run. Kubernetes that will work for ~anything. HNers spend more tokens complaining about the complexity than it takes to setup. The mental gymnastics required to express oneself in yaml, rather than, say, literally anything else Like, brainfuck? Like bash? Like Terraform HCL puppet chef ansible pile-o-scripts? The effort required to output your desired infrastructure's definition as JSON shouldn't really be that gargantuan. You express yourself in anything else but it can't be dumped to JSON?
- hbogert 10mo agoI'm saying this as Kubernetes certified service provider: Just because you can install it with 1 command doesn't mean it's not complex, it's just made easier, not simpler.
- anymouse123456 10mo ago
- paddw 10mo ago> Thinking of Kubernetes as a runtime for declarative infrastructure instead of a mere orchestrator results in very practical approaches to operate your cluster. This is a pretty good definition. I think part of the challenge is the evolution of K8s over time sometimes makes it feel less like a coherent runtime and more like a pile of glue amalgamated from several different components all stuck together. That and you will have to be aware of how those abstractions stick together with the abstractions from your cloud provider, etc...
- NewJazz 10mo agoSometimes the interfaces are clear and consistent (loadbalancers, for example) and the line between, e.g., an AWS NLB and your ingress gateway, is clear-cut. Other times there is a significant degree of portability pains and sparse feature matrices (CSI, IME).
- tbrownaw 10mo agoIt's an application server for multi-part containerized applications, like Tomcat is an application server for applications that can be turned into .war files.
- liampulles 10mo agoOn the use of GitOps for k8s, I think it makes sense for application workloads, and less sense for raw infrastructure definitions (unless you are running at such a scale that your infrastructure is often scaled like an application). For my infrastructure definition repo, I will apply it in my terminal with kubectl, watch, and then merge the PR/commit to master. I often need to do this progressively just to roll back if I see resource consumption or other issues, it would be quite dangerous to let the CI pipeline apply everything and then for me to try and change declarations whilst the control plane API is totally starved for resources. Also (and maybe this is me not doing "proper devops", I don't care), I will often want to tinker a bit with the declaration, trying a bunch of little changes, and then commiting once all is satisfactory. That "dev loop" is less productive if I have to wait for a CI pipeline for every step.
- thiagoeh 10mo agoThis more direct interaction by using kubectl happens on a development cluster? Or is it on the same that runs production workloads?
- liampulles 10mo agoBoth, though I will get it working in dev first (and yes, I've tended to separate clusters by environment).
- EdwardDiego 10mo agoWhen you say infrastructure in this context, are you referring to the actual K8s cluster infra? Or is it what I tend to call "intra-cluster infra" - DBs / Prometheus / Kafka etc. Infra that support apps?
- liampulles 10mo agoSort of both - I tend to use a managed Kubernetes cluster (like EKS), so the infra repo has YAML for operators to deal with logging, monitoring, storage, secret management, etc. Anything that is not an application workload.
- oogali 10mo agoI sometimes joke that Kubernetes is a mass experiment in teaching people how to write Go via YAML. The giant nested YAML you come across is the input (pre-deserialization)/output (post-serialization) for the declared types: https://github.com/kubernetes/api/blob/master/core/v1/types.go https://github.com/kubernetes/api/blob/master/core/v1/types.... Fortunately, or unfortunately, I am the only person that finds humor in this.
- hbogert 10mo agoWriting go in yaml and forgetting everything else we learned software engineering. Proper ide's, being able to make abstractions, not copy pasting, structured templating and thus not string based templating, should I go on?
- postalrat 10mo agoI think many people have the wrong idea what a container is (or i do) and make it sound more exotic than it is. Sure they have some level of isolation but for someone learning this stuff its better to think of them as just a process, like all the other processes running on your computer. And kubernetes as a system that runs and networks processes running on multiple computers.
- NewJazz 10mo agoWell, a group of processes... With some separation between them. Like you can't listen on localhost and call from another pod. But with processes you can do that. Same thing with UDS.
- wlonkly 10mo agoThe author repeats: > that runtime continuously works to make the infrastructure match your intent. The flipside of that is that the infrastructure, at any given time, might not match your intent, or might be continuously working to try to match your intent, which means the state of the infrastructure often does not match the state of its configuration, which is hell during an incident. It makes me wonder if declarative, converging systems are actually what we want, or if they're what we ended up with, or if all the alternatives are just worse.