3 ms·
This is literally my job and I kind of hate how unnecessarily complex it all is. Not in the sense of "distributed systems are hard", but that we don't actually
by 0xbadcafebee 6y ago
This is literally my job and I kind of hate how unnecessarily complex it all is. Not in the sense of "distributed systems are hard", but that we don't actually need the onion to have 20 layers, we really only need 2 or 3.
The reason we need an orchestration system for our orchestration system, is the orchestration system is a snowflake. We need an orchestration system because the container system is a snowflake. We need a container system because containers are a snowflake. None of it is really standard or easy to implement, because none of it was designed in the spirit of Unix. It was all just some random companies who threw some crap together so they could start making money selling ads or hosting services. (I don't mean cgroups, I mean the tools that use them, although cgroups are kind of warty in themselves)
All most people need to do is run a couple processes on a couple hosts, send packets between services (and networks), and have something start/stop/restart those processes. You can have all of that with, say, systemd (ugh), or three different standard interfaces that any program can use with standard libraries/ABIs. Notice I didn't say "with 5 different daemons running on 3 hosts that need complex configuration, constant maintenance, and a migration effort every 3 months".
The Open Container Initiative seems close to getting the first thing done. If we get the other two standardized, we can lose a whole bunch of the onion layers. Consequently a lot of people will need to find a new way to make money, because there'll be not nearly as much need to pay people to deal with the onion.
- baq 6y agoThe way you described it one has to wonder what would the tech landscape look like had Plan 9 received adoption instead of Linux, so you didn’t have to invent all those layers of complexity to manage services on multiple hosts. Just mount the network interface, cpu, ram and start multiple processes...
- yjftsjthsd-h 6y agoNah, we'd still find ways to make it complex. I mean, yes, I think Plan 9 would've been a nicer base to work from and maybe reduced it a bit, but people are excellent at making things more complex, sometimes for legitimate reasons. Off the top of my head, I don't think Plan 9 has a good "orchestration" story itself (ex. "run 17 instances of this process wherever there's enough free resources in our 100-server cluster"), and while the "Plan 9 k8s" of this alternate universe would be simpler than the k8s we have in reality, it would still exist and add one or more layers of abstraction, and then people would stick management layers in front of that, and...
- ColanR 6y ago> I don't think Plan 9 has a good "orchestration" story itself I'm not super familiar with plan 9, but given what I do know, I'm pretty sure that could be done with an rc script. I'm pretty sure that a) you can fork processes after starting them on remote systems, and b) you can query remote systems to figure out which ones have free resources. Given that, it's just a shell / rc script away. Then you just need another script to do the management. ;)
- nitrogen 6y agoAs another example of something orchestration-like ("run 17 processes where possible on this 100 node cluster"), there were also OpenMOSIX and Beowulf clusters on Linux, which if I remember right would load balance work across available systems, in the same way that the kernel's scheduler balances work across CPU cores. https://en.wikipedia.org/wiki/OpenMosix https://en.wikipedia.org/wiki/OpenMosix
- peterwwillis 6y agoYes. Mosix was the kind of black magic that makes you think software today is still several generations behind the times (like Plan9 before it). Even its distributed network filesystem is superior in ease of use to all others. But the devil's in the details, and distributing threads and shared memory is just really difficult, and the patchsets never made it into vanilla. So there goes yet another amazing design down the drain. It reminds me of how electric cars are a 190 year old technology whose actual peak popularity was 120 years ago. Yet here we are, with manufacturers predicting they'll finally be popular again, in 10 to 15 years.
- pjmlp 6y agoHere is the secret, one also doesn't need to follow IT fashion with every fad that pops up. Still delivering on VMs, might be out of fashion, but it works, everyone gets it, and I get back home on time.
- menage 6y agoThe funny thing is that one of the complaints many people have about the cgroups API (text-based control files in a virtual filesystem, which came originally from the cpusets system) is very much in style of Plan 9.
- sjy 6y ago> All most people need to do is run a couple processes on a couple hosts … You can have all of that with, say, systemd (ugh) Why ugh? I do have all of that with systemd, and it’s great. Rather than running each service in a separate filesystem and requiring them to communicate over a virtual network, I use ProtectSystem and ProtectHome to keep services as isolated as they need to be. Rather than creating a network bridge and doing NAT inside my server, if services require network isolation, I use NetworkNamespacePath to assign them to namespaces that contain different VLAN interfaces, which are configured in my router just like physical interfaces. Rather than building a huge container image for every application, I use my OS’s package manager to install apps and manage their dependencies. The skills required to do this can be applied to any systemd-based Linux distribution, and are needed to properly understand and troubleshoot systemd-based Linux distributions running in containers anyway. I’d need some new tools (eg. Ansible) to scale this setup past one or two hosts, but I’m baffled by the people setting up complex Docker/Kubernetes systems just to self-host a few web apps.
- ColanR 6y agoI'd render a guess that the 'ugh' has to do with the general assessment that systemd falls prey to the same flaws that GP identified in the onion layers of orchestration and containerization systems: it's unnecessarily complicated.
- peterwwillis 6y agoSystemd is basically another Kubernetes. The Unix way does not require a patriarchal set of daemons and a proprietary framework to manage the interoperability of everything else in a way that's incompatible with anything else. That said, it's still several layers less than K8s and its dependencies are today.