10 ms·

Accidental complexity, essential complexity, and Kubernetes

137 points by paulgb 4y ago

ordiel 4y ago
Having worked in two of the largest cloud providers there's only one thing I can say which is that wherever I've seen kubernetees been used it has mostly served to make understanding of the application 10 time more complicated ensuring you spend 10 times longer to fix deployment issues Once you finally get it working is "ok" at least until you have to reconfigure it... Not like you couldn't have launch a few containers on "restart always" mode... and no, this is not me not understanding autoscaling and the like, IMHO I think that goes to the realm of accidental complexity since the use cases for that are more a corner case than a necesity for 95% of the scenarios where I've seen it being used
bob1029 4y ago
Also see closely related paper: http://curtclifton.net/papers/MoseleyMarks06a.pdf http://curtclifton.net/papers/MoseleyMarks06a.pdf This one inspired our most recent system architecture.
- candiddevmike 4y ago
  Thank you for sharing this, much more insightful than the article
mati365 4y ago
Kubernetes and its complexity again..
tsimionescu 4y ago
The article is pretty lite, though I agree with most of the points. However, I think that, especially in the context context of Kubernetes, this part is completely wrong: > Containers were a great advance, but much of their complexity falls into the “accidental” category. There’s no fundamental reason that I should need anything except my compiler to produce a universally deployable unit of code. Containers are not used in Kubernetes or other similar orchestrators because of their support for bundling dependencies - that is a bonus at best. Instead, they are used because they are a standard way of using cgroups to tightly control what pieces of the system a process has access to, so that multiple processes running on the same system can't accidentally affect each other, and so that a process can't accidentally depend on the system it is running on (including things like open ports). These are key properties for a system that seeks to efficiently distribute workloads on a number of computers without having to understand the specifics of what workload it's running. They are also used because container registries are a ready made secure Linux distribution-agnostic way of retrieving software and referring to it with a unique name. quay.io/python:3.7 will work on Ubuntu, SuSE, RedHat or any other base system, unlike relying on apt/yum/etc.
- zamalek 4y ago
  I think they mean different things to different people. From the problems I have faced with customer-controlled OS installations, the biggest thing that they offer is configuration isolation (or rather independence). I have seen some truly crazy shit done by customer's administrators and even crazier shit done by management software that they install. Taking away that autonomy is huge.
  - tsimionescu 4y ago
    Absolutely, but the context of the article was specifically Kubernetes.
    - paulgb 4y ago
      In the context of Kubernetes, what do containers have to do with cgroups? I get that container runtimes typically provide a nicer interface for setting them up, but Kubernetes doesn’t use those interfaces anyway. If it was just about cgroups, they could have just run a regular process in cgroups.
      - samus 4y ago
        Containers are a construct created by employing cgroups + namespaces together. Namespaces provide isolations in terms of filesystem, network, users, processes, and probably more.
        paulgb 4y ago
        Yes, but those are features of Linux that containers use; containers don’t have exclusive use of them. My original point was: > Containers were a great advance, but much of their complexity falls into the “accidental” category. There’s no fundamental reason that I should need anything except my compiler to produce a universally deployable unit of code. I stand by that. We might just be arguing semantics about whether that unit of code would be called a “container” or not. I’m leaving open the possibility that it would be (say) a WebAssembly module or static binary.
deleted 4y ago
[deleted]
threeseed 4y ago
> The fact that you need a special distribution like minikube, Kind, k3s, microk8s, or k0s to run on a single-node instance. (The fact that I named five such distributions is a canary of its own.) They serve completely different purposes though. Some are Docker-based designed to be lightweight for testing e.g Kind, others are designed to be scalable to full clusters e.g k3s. And I think having the choice is a good thing as it proves that Kubernetes is vendor-agnostic.
- ClumsyPilot 4y ago
  > The fact that I named five such distributions is a canary of its own. Seriously? Linux still can't even agree on tooling to display UI to the user, and its been like 30 years. Or how many package managers and their supposed replacements are there?
- moondev 4y ago
  It's also not even accurate. You can absolutely run a single node cluster with kubeadm - aka upstream Kubernetes
- spoiler 4y ago
  > The fact that I named five such distributions is a canary of its own That can be said for programming language implementations. Or programming languages as a concept, too. Or configuration formats. Or even arbitrary products or libraries that overlap and handle the same domain. Another comment mentioned Linux distributions. My points is: what is your point? The whole industry is like this (for better or worse). Picking something you don't like and slapping "it has more than one implementation" is hardly an argumentative against it (or any of the other implementations).
orf 4y ago
> One way to think of Kubernetes is as a distributed framework for control loops. Broadly speaking, control loops create a declarative layer on top of an imperative system. Finally: a post about Kubernetes that actually understands what it fundamentally is. > Kubernetes abstracts away the decision of which computer a pod runs on, but reality has a way of poking through. For example, if you want multiple pods to access the same persistent storage volume, whether or not they are running on the same node suddenly becomes your concern again. I'm not sure I agree that this itself constitutes a leaky abstraction. Kubernetes still abstracts away the decision of which computer a pod runs on, the persistent storage volume is just another declarative constraint on where a pod can be scheduled. It's no different from specifying you need "40" CPU cores, and thus only being able to be scheduled on nodes with >40 cores. > There’s no fundamental reason that I should need anything except my compiler to produce a universally deployable unit of code. I agree - but you can do that with 'containers'. You just create an empty "FROM scratch" container then copy your binary in. The end result is that your OCI image is just a single .tar.gz file containing your binary alongside a JSON document specifying things like the entrypoint and any labels. Just like a shipping container, this plugs into anything that ships things shaped like that. You get a bunch of nice stuff for free with this (tagged + content addressable remote storage with garbage collection! inter-image layer caching!), even if you're slinging about WASM binaries I'd still package them as OCI images. > The use of YAML as the primary interface, which is notoriously full of foot-guns. There's a lot more to say here, and it's more of a legitimate criticism of Kubernetes than most of the "hurr dur k8s complex" critisisms you commonly see. The ecosystem has kind of centered around Helm as a way of templating resources, but it's... horrible. It's all so horrible. A huge step up from ktml or other rubbish from the past, but go's template language isn't fun. But I'm not sure how it could have gone any differently. JSON is as standard and simple as it gets and is simple to auto-generate, but it's not user friendly. So either the kubernetes getting started guide starts with "first go auto-generate your JSON using some tool we don't distribute", or you provide a more user-friendly way for people to onboard. YAML is a good middle ground here between giving users the ability to auto-generate resources from other systems (i.e spitting out JSON and shovelling it into a k8s API) and something user-friendly for people to interact with and view using kubectl/k9s/whatever.
- dinosaurdynasty 4y ago
deleted 4y ago
[deleted]
hbrn 4y ago
The more I think about it, the more I realize that generic declarative style, despite sounding very promising, might not be a good fit for modern deployment. Mainly due to technology fragmentation. There's no one true way to deploy an abstract app, each tech stack is fairly unique. Two apps can look exactly the same from the desired state perspective, but have two extremely different deployment processes. They might even have exactly the same tech stack, but different reliability requirements. Somehow you need to be able to accommodate for those differences in your declarative framework. So you'll pay the abstraction costs (complexity). But you will only reap the benefits if you're constantly switching the components of your architecture. And that typically doesn't happen: by the time you get to a point where you need a deployment framework, your architecture fairly rigid. Maybe k8s makes a lot of sense if you're Google. But 99.99% of companies are not Google.
- spoiler 4y ago
  I'm not sure what you're trying to say. You list a number of problems that are addressed relatively well within Kubernetes whilst staying somewhat ergonomic; or at least more so than Chef, puppet, ansible, etc. I'm not sure if you had the misfortune of working with any of the others, but it's extremely frustrating having to untangle them, especially after the 50th variation of it, because it's always done subtly different either because of scripts or something else. Kubernetes and Kustomize remove all these issues. You learn the language (or schema, I guess) once, and your never have to re-learn idiosyncrasies, dialects, variable nomenclatures, and others used between different projects. Also, here's little room for creativity with templating languages that break tooling (I'm aware of Helm, but I also dislike Helm personally). I also have a feeling this is what people dislike about Kubernetes. It's boring, but it's meant to be.
jrockway 4y ago
I agree very much about the accidental complexity of containers. Ignoring the runtime concerns (cgroups, namespaces, networking, etc.), the main problem seems to have been "we can't figure out how to get Python code to production", and the solution was "just ship a disk image of some developer's workstation to production". To do this, they created a new file format ("OCI image format", though not called that at the time), a new protocol for exchanging files between computers ("OCI distribution spec"), a new way to run executables on Linux ("OCI runtime spec"), and various proprietary versions of Make, with a caching mechanism that isn't aware of the actual dependencies that go into your final product. The result is a staggering level of complexity, all to work around the fact that nobody even bothered trying to add a build system to Python. Like the author, I tend to write software in a language that produces a single statically linked binary, so I don't need any of this distribution stuff. I don't need layers, I don't need a Make-alike, I don't need a build cache, but I still have to go out of my way to wrap the generated binary and push it to a super special server. Imagine a world where we just skipped all of this, and your k8s manifest looks like: containers: - name: foo-app image: - architecture: amd64 os: linux binary: https://github.com/whatever/download/foo-app/foo-app-1.0.0-linux-amd64 checksum: sha256@1234567890abcdef - architecture: arm64 os: linux binary: https://github.com/whatever/download/foo-app/foo-app-1.0.0-linux-arm64 checksum: sha256@abdcef1234567890 Meanwhile, if you don't want to orchestrate your deployment and just want to run your app: wget https://github.com/whatever/download/foo-app/foo-app-1.0.0-linux-amd64 chmod a+x foo-app-1.0.0-linux-amd64 ./foo-app-1.0.0-linux-amd64 I dunno. I feel like, as an industry, we spent billions of dollars, founded brand new companies, created a new job title ("Devops", don't get me started), all so that we could avoid making Python output a single binary. I'm not sure that random chaos did the right thing, and you're right to be bitter about it.
- jen20 4y ago
  This is effectively what Nomad permits - scheduling things that are not Docker containers.
- BwackNinja 4y ago
  You /can/ package python applications into a single binary -- just a zip file prepended with a python shebang. You can do much the same with the base python libraries. No one does it, I presume, because the community is against vendoring dependencies that way, the same way they are against static binaries. https://bwackninja.github.io/2020/11/25/packaging-python.html https://bwackninja.github.io/2020/11/25/packaging-python.htm...
theteapot 4y ago
> Through Brooks’ accidental-vs-essential lens, a lot of discussion around when to use Kubernetes boils down to the idea that essential complexity can become accidental complexity when you use the wrong tool for the job. The number of states my microwave can enter is essential complexity if I want to heat food, but accidental complexity if I just want a timer. With that in mind ... I'm twisting my mind trying to grasp this interpretation in Brooks' complexity paradigm. I'm sure Brooks would be interested to learn there exists so called wrong-tools that can reduce essential complexity to accidental :). I think Brooks would put it that the essential complexity is the same and irreducible, but the accidental complexity is increased when the wrong tool is used.
bvrmn 4y ago
> The number of states my microwave can enter is essential complexity Two dials to set power and clock-timer is enough. Microwaves with digital panel and led display are a perfect example of accidental complexity.
- SoftTalker 4y ago
  Consumers don't stop and think about what they really need, or what the simplest solution to their problem is. They equate "more features" to "better." I think it was Joel Spolsky who said (paraphrasing) "nothing we did with our software ever created more new revenue than adding features."
liftm 4y ago
> When you run a container on Kubernetes, you don’t specify which computer they run on; you (hopefully) don’t care. And then you want to deploy something rack aware and you have to deal with `requiredDuringSchedulingIgnoredDuringExecution` in your yaml. Even greater fun ensues if you need some kind of special-purpose hardware.
- jiggawatts 4y ago
  Also if you have both Windows 2019 and 2022. Or if you have licensing based on "hardware cores" or "servers".
abraxas 4y ago
A Kubernetes based architecture exposes to the developer and the operator a lot of things that superior hardware can abstract almost completely. Things like Mainframes can offer performance and uptime that easily beats pretty large commodity hardware clusters yet we seem to be pushing in this weird direction as an industry while ignoring why the mainframes came to be.
- jiggawatts 4y ago
  Think of Kubernetes as a "software defined mainframe". It's an alternative implementation of many of the same concepts. You don't have to go to IBM and purchase a monolithic box with POWER processors in it.
  - abraxas 4y ago
    But the thing is that mainframes abstract a lot of the fault tolerance in hardware thus absolving the developer of a lot of the responsibility and the cognitive overhead. A mainframe in software pitch sounds a bit like that 90s RAM expansion software. Could it possibly work? To a limited degree, perhaps. Would I want it instead of an extra RAM stick? Hell no!
    - jiggawatts 4y ago
      Not software emulated, but software defined. For example, you can get a pair of enormous 100 Tbps core switch/routers for your data centre from Cisco. Upgrading these is terrifying, they cost an incredible amount of money, and if one fails you lose 50% capacity. Conversely, if you run them active-passive, you waste 50% of your capital investment. Alternatively, you could use a cluster of 1000 VMs on commodity hardware to run your traffic management, with the rules defined purely in software. This can be an order of magnitude cheaper, scale to infinity, and allow smooth upgrades and change rollouts. The former is what every large enterprise does, the latter is what the public cloud hosting providers do. You can't argue that it's some sort of "fake substitute" when the clouds operate production networks at a scale that far outstrips anything you'd find in any private data centre! Similarly, many large enterprises use and prefer monolithic mainframes, but in the cloud the preferred option is something like Kubernetes. For example, with K8s on a typical cloud, it's possible to use "spot" instances to temporarily scale out workloads at night for batch processing to use tens of thousands of CPU cores. No on-prem mainframe can compete with that on any metric. Not absolute throughput, not price, not price/performance.
_gabe_ 4y ago
> With that in mind, I want to talk about the sources of complexity in Kubernetes. I don't care about the complexities in Kubernetes, I care about the complexities it (cl)aims to be solving. I reread the article a second time to see if I missed something, but the whole article just seems to be talking about the complexities within kubernetes and justifying those. I still don't understand why every application must be able to scale to infinity and maintain 99.999% reliability. Not everyone is Google and I feel like we'd be a lot better if if the not-Google companies just recognized that instead of layering on accidental complexity in the form of infinite scaling for your service that gets all of 5,000 requests a month. It's funny, this article mentions Fred Brook's "No Silver Bullet" and then proceeds to tout a solution developed for Google scale problems as the silver bullet to whatever it is Kubernetes is actually solving. Like what does it even mean to be using kubernetes if you only ever have 1 or 2 instances (pods?) running and deploy infrequently (which I suspect is the norm rather than the exception)?
- shadycuz 4y ago
  I have been using Kubernetes for my side projects and personal home services for almost 5 years now. I only ever have 1 pod per service and I deploy pretty infrequently. For me its because how much easier Kubernetes makes my life. I'm using digital ocean's managed kubernetes offering. This means I don't have to worry about the cluster. The OS and updates and all that are managed for me. Same thing with cluster scaling. I just keep adding or removing services and they make sure the cluster is right sized. Basically I just dont ever have to worry about infra. The next nice thing is the community support. Do I want a load balancer/reverse proxy with auto certificate renewal? It's a simple install command away from running. How about a HA database? MFA? Block storage? It's like magic. The third thing is that most services are going to have a pretty similar deployment/testing strategy. So when I want to stand up a new one. I can copy 2 or 3 files from another repo, do a find and replace on the App name and port number and deploy a new service with just one command. Again, its like magic. While I dont use this feature, I like the fact that I know I can move my setup to anywhere I want. Move my cluster into my house? on another cloud provider? This would be really easy to lift and shift.
  - 4y ago
KronisLV 4y ago
A person by the name of Venkat Subramaniam actually did a pretty nice presentation about accidental and domain (essential) complexity a few years ago: [video] https://www.youtube.com/watch?v=4MEKu2TcEHM https://www.youtube.com/watch?v=4MEKu2TcEHM It's a bit humorous and long winded, but personally I rather enjoyed it and it struck a chord with me, especially the bits about how even things like configuration can be a source of frustration. In regards to Kubernetes, of course, there's also lots of nuance and cases where it is good, as well as those where something simpler (like Nomad/Swarm/Compose) might be a better fit. Either way, personally I think that containers are pretty good overall, unless you want to ship Vagrant images or full VM images.

137 points by paulgb 4y ago

Having worked in two of the largest cloud providers there's only one thing I can say which is that wherever I've seen kubernetees been used it has mostly served to make understanding of the application 10 time more complicated ensuring you spend 10 times longer to fix deployment issues Once you finally get it working is "ok" at least until you have to reconfigure it... Not like you couldn't have launch a few containers on "restart always" mode... and no, this is not me not understanding autoscaling and the like, IMHO I think that goes to the realm of accidental complexity since the use cases for that are more a corner case than a necesity for 95% of the scenarios where I've seen it being used

bob1029 4y ago

Also see closely related paper: http://curtclifton.net/papers/MoseleyMarks06a.pdf http://curtclifton.net/papers/MoseleyMarks06a.pdf This one inspired our most recent system architecture.

candiddevmike 4y ago
Thank you for sharing this, much more insightful than the article

mati365 4y ago

Kubernetes and its complexity again..

tsimionescu 4y ago

The article is pretty lite, though I agree with most of the points. However, I think that, especially in the context context of Kubernetes, this part is completely wrong: > Containers were a great advance, but much of their complexity falls into the “accidental” category. There’s no fundamental reason that I should need anything except my compiler to produce a universally deployable unit of code. Containers are not used in Kubernetes or other similar orchestrators because of their support for bundling dependencies - that is a bonus at best. Instead, they are used because they are a standard way of using cgroups to tightly control what pieces of the system a process has access to, so that multiple processes running on the same system can't accidentally affect each other, and so that a process can't accidentally depend on the system it is running on (including things like open ports). These are key properties for a system that seeks to efficiently distribute workloads on a number of computers without having to understand the specifics of what workload it's running. They are also used because container registries are a ready made secure Linux distribution-agnostic way of retrieving software and referring to it with a unique name. quay.io/python:3.7 will work on Ubuntu, SuSE, RedHat or any other base system, unlike relying on apt/yum/etc.

zamalek 4y ago
I think they mean different things to different people. From the problems I have faced with customer-controlled OS installations, the biggest thing that they offer is configuration isolation (or rather independence). I have seen some truly crazy shit done by customer's administrators and even crazier shit done by management software that they install. Taking away that autonomy is huge.
- tsimionescu 4y ago
  Absolutely, but the context of the article was specifically Kubernetes.
  - paulgb 4y ago
    In the context of Kubernetes, what do containers have to do with cgroups? I get that container runtimes typically provide a nicer interface for setting them up, but Kubernetes doesn’t use those interfaces anyway. If it was just about cgroups, they could have just run a regular process in cgroups.
    - samus 4y ago
      Containers are a construct created by employing cgroups + namespaces together. Namespaces provide isolations in terms of filesystem, network, users, processes, and probably more.
      - paulgb 4y ago
        Yes, but those are features of Linux that containers use; containers don’t have exclusive use of them. My original point was: > Containers were a great advance, but much of their complexity falls into the “accidental” category. There’s no fundamental reason that I should need anything except my compiler to produce a universally deployable unit of code. I stand by that. We might just be arguing semantics about whether that unit of code would be called a “container” or not. I’m leaving open the possibility that it would be (say) a WebAssembly module or static binary.

deleted 4y ago

[deleted]

threeseed 4y ago

> The fact that you need a special distribution like minikube, Kind, k3s, microk8s, or k0s to run on a single-node instance. (The fact that I named five such distributions is a canary of its own.) They serve completely different purposes though. Some are Docker-based designed to be lightweight for testing e.g Kind, others are designed to be scalable to full clusters e.g k3s. And I think having the choice is a good thing as it proves that Kubernetes is vendor-agnostic.

ClumsyPilot 4y ago
> The fact that I named five such distributions is a canary of its own. Seriously? Linux still can't even agree on tooling to display UI to the user, and its been like 30 years. Or how many package managers and their supposed replacements are there?
moondev 4y ago
It's also not even accurate. You can absolutely run a single node cluster with kubeadm - aka upstream Kubernetes
spoiler 4y ago
> The fact that I named five such distributions is a canary of its own That can be said for programming language implementations. Or programming languages as a concept, too. Or configuration formats. Or even arbitrary products or libraries that overlap and handle the same domain. Another comment mentioned Linux distributions. My points is: what is your point? The whole industry is like this (for better or worse). Picking something you don't like and slapping "it has more than one implementation" is hardly an argumentative against it (or any of the other implementations).

orf 4y ago

> One way to think of Kubernetes is as a distributed framework for control loops. Broadly speaking, control loops create a declarative layer on top of an imperative system. Finally: a post about Kubernetes that actually understands what it fundamentally is. > Kubernetes abstracts away the decision of which computer a pod runs on, but reality has a way of poking through. For example, if you want multiple pods to access the same persistent storage volume, whether or not they are running on the same node suddenly becomes your concern again. I'm not sure I agree that this itself constitutes a leaky abstraction. Kubernetes still abstracts away the decision of which computer a pod runs on, the persistent storage volume is just another declarative constraint on where a pod can be scheduled. It's no different from specifying you need "40" CPU cores, and thus only being able to be scheduled on nodes with >40 cores. > There’s no fundamental reason that I should need anything except my compiler to produce a universally deployable unit of code. I agree - but you can do that with 'containers'. You just create an empty "FROM scratch" container then copy your binary in. The end result is that your OCI image is just a single .tar.gz file containing your binary alongside a JSON document specifying things like the entrypoint and any labels. Just like a shipping container, this plugs into anything that ships things shaped like that. You get a bunch of nice stuff for free with this (tagged + content addressable remote storage with garbage collection! inter-image layer caching!), even if you're slinging about WASM binaries I'd still package them as OCI images. > The use of YAML as the primary interface, which is notoriously full of foot-guns. There's a lot more to say here, and it's more of a legitimate criticism of Kubernetes than most of the "hurr dur k8s complex" critisisms you commonly see. The ecosystem has kind of centered around Helm as a way of templating resources, but it's... horrible. It's all so horrible. A huge step up from ktml or other rubbish from the past, but go's template language isn't fun. But I'm not sure how it could have gone any differently. JSON is as standard and simple as it gets and is simple to auto-generate, but it's not user friendly. So either the kubernetes getting started guide starts with "first go auto-generate your JSON using some tool we don't distribute", or you provide a more user-friendly way for people to onboard. YAML is a good middle ground here between giving users the ability to auto-generate resources from other systems (i.e spitting out JSON and shovelling it into a k8s API) and something user-friendly for people to interact with and view using kubectl/k9s/whatever.

dinosaurdynasty 4y ago

deleted 4y ago

[deleted]

hbrn 4y ago

The more I think about it, the more I realize that generic declarative style, despite sounding very promising, might not be a good fit for modern deployment. Mainly due to technology fragmentation. There's no one true way to deploy an abstract app, each tech stack is fairly unique. Two apps can look exactly the same from the desired state perspective, but have two extremely different deployment processes. They might even have exactly the same tech stack, but different reliability requirements. Somehow you need to be able to accommodate for those differences in your declarative framework. So you'll pay the abstraction costs (complexity). But you will only reap the benefits if you're constantly switching the components of your architecture. And that typically doesn't happen: by the time you get to a point where you need a deployment framework, your architecture fairly rigid. Maybe k8s makes a lot of sense if you're Google. But 99.99% of companies are not Google.

spoiler 4y ago
I'm not sure what you're trying to say. You list a number of problems that are addressed relatively well within Kubernetes whilst staying somewhat ergonomic; or at least more so than Chef, puppet, ansible, etc. I'm not sure if you had the misfortune of working with any of the others, but it's extremely frustrating having to untangle them, especially after the 50th variation of it, because it's always done subtly different either because of scripts or something else. Kubernetes and Kustomize remove all these issues. You learn the language (or schema, I guess) once, and your never have to re-learn idiosyncrasies, dialects, variable nomenclatures, and others used between different projects. Also, here's little room for creativity with templating languages that break tooling (I'm aware of Helm, but I also dislike Helm personally). I also have a feeling this is what people dislike about Kubernetes. It's boring, but it's meant to be.

jrockway 4y ago

I agree very much about the accidental complexity of containers. Ignoring the runtime concerns (cgroups, namespaces, networking, etc.), the main problem seems to have been "we can't figure out how to get Python code to production", and the solution was "just ship a disk image of some developer's workstation to production". To do this, they created a new file format ("OCI image format", though not called that at the time), a new protocol for exchanging files between computers ("OCI distribution spec"), a new way to run executables on Linux ("OCI runtime spec"), and various proprietary versions of Make, with a caching mechanism that isn't aware of the actual dependencies that go into your final product. The result is a staggering level of complexity, all to work around the fact that nobody even bothered trying to add a build system to Python. Like the author, I tend to write software in a language that produces a single statically linked binary, so I don't need any of this distribution stuff. I don't need layers, I don't need a Make-alike, I don't need a build cache, but I still have to go out of my way to wrap the generated binary and push it to a super special server. Imagine a world where we just skipped all of this, and your k8s manifest looks like: containers: - name: foo-app image: - architecture: amd64 os: linux binary: https://github.com/whatever/download/foo-app/foo-app-1.0.0-linux-amd64 checksum: sha256@1234567890abcdef - architecture: arm64 os: linux binary: https://github.com/whatever/download/foo-app/foo-app-1.0.0-linux-arm64 checksum: sha256@abdcef1234567890 Meanwhile, if you don't want to orchestrate your deployment and just want to run your app: wget https://github.com/whatever/download/foo-app/foo-app-1.0.0-linux-amd64 chmod a+x foo-app-1.0.0-linux-amd64 ./foo-app-1.0.0-linux-amd64 I dunno. I feel like, as an industry, we spent billions of dollars, founded brand new companies, created a new job title ("Devops", don't get me started), all so that we could avoid making Python output a single binary. I'm not sure that random chaos did the right thing, and you're right to be bitter about it.

jen20 4y ago
This is effectively what Nomad permits - scheduling things that are not Docker containers.
BwackNinja 4y ago
You /can/ package python applications into a single binary -- just a zip file prepended with a python shebang. You can do much the same with the base python libraries. No one does it, I presume, because the community is against vendoring dependencies that way, the same way they are against static binaries. https://bwackninja.github.io/2020/11/25/packaging-python.html https://bwackninja.github.io/2020/11/25/packaging-python.htm...

theteapot 4y ago

> Through Brooks’ accidental-vs-essential lens, a lot of discussion around when to use Kubernetes boils down to the idea that essential complexity can become accidental complexity when you use the wrong tool for the job. The number of states my microwave can enter is essential complexity if I want to heat food, but accidental complexity if I just want a timer. With that in mind ... I'm twisting my mind trying to grasp this interpretation in Brooks' complexity paradigm. I'm sure Brooks would be interested to learn there exists so called wrong-tools that can reduce essential complexity to accidental :). I think Brooks would put it that the essential complexity is the same and irreducible, but the accidental complexity is increased when the wrong tool is used.

bvrmn 4y ago

> The number of states my microwave can enter is essential complexity Two dials to set power and clock-timer is enough. Microwaves with digital panel and led display are a perfect example of accidental complexity.

SoftTalker 4y ago
Consumers don't stop and think about what they really need, or what the simplest solution to their problem is. They equate "more features" to "better." I think it was Joel Spolsky who said (paraphrasing) "nothing we did with our software ever created more new revenue than adding features."

liftm 4y ago

> When you run a container on Kubernetes, you don’t specify which computer they run on; you (hopefully) don’t care. And then you want to deploy something rack aware and you have to deal with `requiredDuringSchedulingIgnoredDuringExecution` in your yaml. Even greater fun ensues if you need some kind of special-purpose hardware.

jiggawatts 4y ago
Also if you have both Windows 2019 and 2022. Or if you have licensing based on "hardware cores" or "servers".

abraxas 4y ago

A Kubernetes based architecture exposes to the developer and the operator a lot of things that superior hardware can abstract almost completely. Things like Mainframes can offer performance and uptime that easily beats pretty large commodity hardware clusters yet we seem to be pushing in this weird direction as an industry while ignoring why the mainframes came to be.

jiggawatts 4y ago
Think of Kubernetes as a "software defined mainframe". It's an alternative implementation of many of the same concepts. You don't have to go to IBM and purchase a monolithic box with POWER processors in it.
- abraxas 4y ago
  But the thing is that mainframes abstract a lot of the fault tolerance in hardware thus absolving the developer of a lot of the responsibility and the cognitive overhead. A mainframe in software pitch sounds a bit like that 90s RAM expansion software. Could it possibly work? To a limited degree, perhaps. Would I want it instead of an extra RAM stick? Hell no!
  - jiggawatts 4y ago
    Not software emulated, but software defined. For example, you can get a pair of enormous 100 Tbps core switch/routers for your data centre from Cisco. Upgrading these is terrifying, they cost an incredible amount of money, and if one fails you lose 50% capacity. Conversely, if you run them active-passive, you waste 50% of your capital investment. Alternatively, you could use a cluster of 1000 VMs on commodity hardware to run your traffic management, with the rules defined purely in software. This can be an order of magnitude cheaper, scale to infinity, and allow smooth upgrades and change rollouts. The former is what every large enterprise does, the latter is what the public cloud hosting providers do. You can't argue that it's some sort of "fake substitute" when the clouds operate production networks at a scale that far outstrips anything you'd find in any private data centre! Similarly, many large enterprises use and prefer monolithic mainframes, but in the cloud the preferred option is something like Kubernetes. For example, with K8s on a typical cloud, it's possible to use "spot" instances to temporarily scale out workloads at night for batch processing to use tens of thousands of CPU cores. No on-prem mainframe can compete with that on any metric. Not absolute throughput, not price, not price/performance.

_gabe_ 4y ago

> With that in mind, I want to talk about the sources of complexity in Kubernetes. I don't care about the complexities in Kubernetes, I care about the complexities it (cl)aims to be solving. I reread the article a second time to see if I missed something, but the whole article just seems to be talking about the complexities within kubernetes and justifying those. I still don't understand why every application must be able to scale to infinity and maintain 99.999% reliability. Not everyone is Google and I feel like we'd be a lot better if if the not-Google companies just recognized that instead of layering on accidental complexity in the form of infinite scaling for your service that gets all of 5,000 requests a month. It's funny, this article mentions Fred Brook's "No Silver Bullet" and then proceeds to tout a solution developed for Google scale problems as the silver bullet to whatever it is Kubernetes is actually solving. Like what does it even mean to be using kubernetes if you only ever have 1 or 2 instances (pods?) running and deploy infrequently (which I suspect is the norm rather than the exception)?

shadycuz 4y ago
I have been using Kubernetes for my side projects and personal home services for almost 5 years now. I only ever have 1 pod per service and I deploy pretty infrequently. For me its because how much easier Kubernetes makes my life. I'm using digital ocean's managed kubernetes offering. This means I don't have to worry about the cluster. The OS and updates and all that are managed for me. Same thing with cluster scaling. I just keep adding or removing services and they make sure the cluster is right sized. Basically I just dont ever have to worry about infra. The next nice thing is the community support. Do I want a load balancer/reverse proxy with auto certificate renewal? It's a simple install command away from running. How about a HA database? MFA? Block storage? It's like magic. The third thing is that most services are going to have a pretty similar deployment/testing strategy. So when I want to stand up a new one. I can copy 2 or 3 files from another repo, do a find and replace on the App name and port number and deploy a new service with just one command. Again, its like magic. While I dont use this feature, I like the fact that I know I can move my setup to anywhere I want. Move my cluster into my house? on another cloud provider? This would be really easy to lift and shift.
- 4y ago

KronisLV 4y ago

A person by the name of Venkat Subramaniam actually did a pretty nice presentation about accidental and domain (essential) complexity a few years ago: [video] https://www.youtube.com/watch?v=4MEKu2TcEHM https://www.youtube.com/watch?v=4MEKu2TcEHM It's a bit humorous and long winded, but personally I rather enjoyed it and it struck a chord with me, especially the bits about how even things like configuration can be a source of frustration. In regards to Kubernetes, of course, there's also lots of nuance and cases where it is good, as well as those where something simpler (like Nomad/Swarm/Compose) might be a better fit. Either way, personally I think that containers are pretty good overall, unless you want to ship Vagrant images or full VM images.