11 ms·
The thing we are missing still is the distributed OS. Kubernetes only exists because of the missing abstractions in Linux to be able to do computation, discover
by throwaway787544 5y ago
The thing we are missing still is the distributed OS. Kubernetes only exists because of the missing abstractions in Linux to be able to do computation, discovery, message passing/IO, instrumentation over multiple nodes. If you could do ps -A and see all processes on all nodes, or run a program and have it automatically execute on a random node, or if (grumble grumble) Systemd unit files would schedule a minimum of X processes on N nodes, most of the K8s ecosystem would become redundant. A lot of other components like unified AuthZ for linux already exist, as well as networking (WireGuard anyone?).
- ohYi55 5y ago
- pjmlp 5y agoKubernetes only exists because people wanted to do Application Servers in any language, and now they are rediscovering them trying to sell us on Kubernetes + WebAssembly, the irony.
- zozbot234 5y agoThe abstractions are there in Linux, largely imported from plan 9. And work is ongoing to support further abstractions, such as easy checkpoint/restore of whole containers. Kubernetes is a very new framework intended to support large-scale orchestration and deployment in a mostly automated way, driven by 'declarative' configuration; at some point, these features will be rewritten in a way that's easier to understand and perhaps extend further.
- MisterTea 5y ago> The abstractions are there in Linux, largely imported from plan 9. Which abstractions are those?
- zozbot234 5y ago> to be able to do computation, discovery, message passing/IO, instrumentation over multiple nodes. Kernel namespaces are the building blocks for this, because an app that accesses all kernel-managed resources via separate namespaces is insulated from the specifics of any single node, and can thus be transparently migrated elsewhere. It enables the kind of location independence that OP is arguing for here.
- glorfindel66 5y agoThat’s not at all what Linux namespaces permit. It’s a side effect of using them that could be leveraged using something like CRIU, sure, but it’s not what they’re for and they’re not a building block for anything mentioned in the portion of their comment you quoted. Namespaces simply make the kernel lie when asked about sockets and users and such. It’s intended for isolation on a single server. They’re next to useless in distributed work, particularly the kind being discussed here (Plan 9ish). You actually want the opposite: to accomplish that, you want the kernel to lie even harder and make things up in the context of those interfaces, rather than hide things. Namespaces don’t really get you there in their current form.
- zozbot234 5y ago> That’s not at all what Linux namespaces permit. Isolating processes from the specifics of the system they're running on is a key feature of the namespace-based model; it seems weird to call it a "side effect only". We should keep in mind that CRIU itself is still a fairly new feature that's only entered mainline recently, and the kernel already has plenty of ways to "make up" more virtual resources that are effectively controlled by userspace. While it may be true that these things are largely ad hoc for now, it's not clear that this will be an obstacle in the future,
- gnufx 5y agoI can talk about namespaces in HPC distributed systems, and they don't look anything like Plan 9 to me. They make life harder in various respects, and even dangerous with Linux features that don't take them into account (like at least one of the "zero-copy" add-on modules used by MPI shared memory implementations).
- NavinF 5y agoEh I can’t see Linux getting a built-in distributed kv store (etcd) any time soon. Same goes for distributed filesystems. All you have out of the box is nfs which gives you the worst of both worlds: Every nfs server is a SPOF yet these servers don’t take advantage of their position to guarantee even basic consistency (atomic appends) that you get for free everywhere else. And besides how would you even implement all those features you listed without recreating k8s? A distributed “ps -A” that just runs “for s in servers; ssh user@$s ps; done” and sorts the output would be trivial, but anything more complex (e.g. keeping at least 5 instances of an app running as machines die) requires distributed and consistent state.
- zozbot234 5y ago> requires distributed and consistent state Distributed yes, but not necessarily consistent. You can use CRDTs to manage "partial, flexible" consistency requirements. This might mean, e.g. sometimes having more than 5 instances running, but should come with increased flexibility overall.
- throwaway787544 5y agoFwiw those features existed in Mosix (a Linux SSI patch) 2 decades ago... I feel like we could probably do it again In terms of CAP, yeah it might not have been technically as reliable. But there's different levels of reliability for different applications; we could implement a lot of it in userland and tailor as needed
- NavinF 5y agoI call BS. I can’t find any details about how mosix handled storage, but what I did find suggests nfs semantics. That’s totally unusable which is probably why the project died decades ago. (And apparently you had to recompile every app because they changed the syscall ABI to add a node ID to every inode or something? Guess they were speedrunning obsolescence) > we could implement a lot of it in userland Yeah that’s k8s, etcd, ceph, and the distributed database of the week.
- 5y ago
- wwalexander 5y agoPlan 9 was designed in this way, but never took off. Rob Pike: > This is 2012 and we're still stitching together little microcomputers with HTTPS and ssh and calling it revolutionary. I sorely miss the unified system view of the world we had at Bell Labs, and the way things are going that seems unlikely to come back any time soon.
- nautilus12 5y agoGlad to see Plan 9 getting some love in the comments even if it didn't make it into the article.
- jlpom 5y agoThis describe more a Single System Image [0] to me (WPD includes Plan 9 as one but considering it does not does not supports process migration I find it moot). LinuxPMI [1] seems to be a good idea but they seems to be based on Linux 2.6, so you would have to heavily patch newer kernel. The only thing that seems to support process migration with current software / still active are CRIU [2] (which doesn't support graphical/wayland programs) and DragonflyBSD [3] (in their own words very basic). [0]: https://en.wikipedia.org/wiki/Single_system_image https://en.wikipedia.org/wiki/Single_system_image [1]: http://linuxpmi.org http://linuxpmi.org [2]: criu.org [3]: https://man.dragonflybsd.org/?command=sys_checkpoint§ion=2 https://man.dragonflybsd.org/?command=sys_checkpoint§ion...
- zozbot234 5y agoGraphical programs could be checkpointed and restored as long as they don't directly connect to the hardware. (Because the checkpoint/restore system has no idea how to grab the hardware's relevant state or replicate it on restore.) This means running those apps in a hardware-independent way (e.g. using a separate Wayland instance that connects to the system one), but aside from that it ought to be usable.
- jlpom 5y agoFor CRIU it is not supported: https://criu.org/Integration#Wayland.2FWeston https://criu.org/Integration#Wayland.2FWeston, also in my experience it doesn't work. Are you talking about an other software?
- deleted 5y ago[deleted]
- ff317 5y agoThere were older attempts at this stuff, in the 90s with "Beowulf" clusters that had cross-machine process management and whatnot. It's a lot harder than it seems to make this approach make sense in the real world, as the abstraction hides important operational details. The explicit container + orchestration abstraction is probably closer to the ideal than trying to stretch linux/systemd/cgroups across the network "seamlessly". It's clearer what's going on and what the operational trade-offs are.
- mnd999 5y agoImagine a Beowulf cluster of hot grits in soviet Russia with CowboyNeal.
- gnufx 5y ago> in the 90s with "Beowulf" clusters In case of any confusion, that sort of thing wasn't a generic Beowulf feature, but it sounds like Bproc. I don't know if it's still used. (The Sourceforge version is ancient.) https://updates.penguincomputing.com/clusterware/6/docs/clusterware-docs/prog-guide/bproc.html https://updates.penguincomputing.com/clusterware/6/docs/clus... https://sourceforge.net/projects/bproc/ https://sourceforge.net/projects/bproc/ Containers actually only make it harder to "orchestrate" your distributed processes in an HPC system.
- oceanplexian 5y agoAm I the one who doesn’t want this? The entire point of UNIX philosophy (Which seems to be something they aren’t teaching in software development these days) is to do one thing and do it well. We don’t need Linux operating operating as a big declarative distributed system with a distributed scheduling systems and a million half-baked APIs to interact with it, the way K8s works. If you want that you should build something to your specific requirements, not shove more things into the kernel.
- goodpoint 5y agoLinux/UNIX does not have to turn into a mess like k8s to be natively distributed. Plan9 was doing it with a tiny codebase in comparison.
- Karrot_Kream 5y agoThe UNIX philosophy made more sense as an abstraction for a computer when computers were simpler. Computers nowadays (well at least since 2006-ish) have multiple cores executing simultaneously with complicated amounts of background logic, interrupt-driven logic, shared caches, etc. The UNIX philosophy doesn't map to this reality at all. Right now there's no set of abstractions except machine code that exposes the machine's distributed systems' in a coherent abstraction. Nothing is stopping someone else from writing a UNIX abstraction atop this though.
- generalizations 5y agoThe idea of doing one thing, and doing it well, isn't dependent on the simplicity of the underlying system (I imagine that PDP-11 systems seemed impressively complicated in their time, too). The UNIX philosophy is a paradigm for managing complexity. To me, that seems more relevant with modern computers, not less. > “A program is generally exponentially complicated by the number of notions that it invents for itself. To reduce this complication to a minimum, you have to make the number of notions zero or one, which are two numbers that can be raised to any power without disturbing this concept. Since you cannot achieve much with zero notions, it is my belief that you should base systems on a single notion.” - Ken Thompson
- 5y ago
- evandrofisico 5y agoActually, at some point in the 2.4 kernel it was possibile to do that, with single image systems, such as openmosix, that handled process discovery, computation and much more, but underneath the simple user interface it was complex, kinda insecure and so, was never abandoned and never ported to newer versions.
- icedchai 5y agoI think OpenVMS did this... in the 80's.
- Karrot_Kream 5y agoI think you're looking at the wrong abstraction level. You're thinking on a node (computer) basis. Even on a single computer, many of the things that happen are distributed. DMA controllers, input interrupts, kernel-forced context switches, there's a lot going on there but we still pretend that our computers are just executing sequential code. I agree with the OP and think it's high time we treat the computer as the distributed system it is. Fuschia and GenodeOS are both making developments in this direction.
- f0e4c2f7 5y agoI very much agree with this and while Kubernetes is better than a poke in the eye, I look forward to the day when there is a true distributed OS available in the way you describe. It's possible Kubernetes could even grow into that somehow.
- als0 5y agoI remember the Barrelfish OS was trying to tackle this problem head on https://barrelfish.org/ https://barrelfish.org/
- gnufx 5y agoDistributed computation with message passing (and RDMA) is the essence of HPC systems. SGI systems supported multi-node Linux single system images up to ~1024 cores a fair few years ago, but they depend on a coherent interconnect (NUMAlink, originally from the MPIS-based systems under Irix). However, you don't ignore the distributed nature of even single HPC nodes unless you want to risk perhaps an order of magnitude performance loss. SMP these days doesn't stand for Symmetric Multi-Processing.
- zozbot234 5y agoDistributed shared memory is feasible in theory even via being provided in-software by the OS. You're right that this would not change the physical reality of message passing, but it would allow a single multi-processor application code to operate seamlessly using either shared memory on a single node, or distributed memory on a large cluster.
- gnufx 5y agoI talk about the practice in HPC, not theory, and this stuff is literally standard (remote memory of various types and the same thing running the same, modulo performance and resources, on a 32-core node as on one core each of 32 nodes). However, you still need to consider network non-uniformity at levels from at least NUMA nodes up, at least if you want performance in general.
- uvdn7 5y agoAbstract a fleet of machines as single super computer sounds nice. But how about partial failures? It's something that a real stateful distributed system would have to deal with all the time but a single host machine almost never deals with (do you worry about a single cacheline failure when writing a program?).
- marcosdumay 5y agoThere is a huge amount of research about distributed OSes (really, they were very fashionable at the 90's and early 00's). Plenty of people worked on this problem, and it's basically solved (as in, we don't have any optimal solution, but it won't be a problem on a real system).
- NavinF 5y agoIt’s “basically solved” in the sense that everyone gave up on distributed OSes and used k8s instead.
- zozbot234 5y agoK8s is doing distributed OS's on easy mode, supporting basically ephemeral 'webscale' workloads for pure horizontal scaling. Even then it introduces legendary amounts of non-essential complexity in pursuit of this goal. It gets used because "Worse is better" is a thing, not because anyone thinks it's an unusually effective way to address these problems.
- pjmlp 5y agoI see K8S as Application Servers for everyone, with containers replacing EARs, it certainly gets some WebLogic/WebSphere vibes when looking at those yml files and how we used to setup an Application Server cluster.
- marcosdumay 5y agoK8s just pushes the problem into the application layer, where every developer must solve it independently. It's not like it solves anything.
- cultofmetatron 5y agoI built my startup in elixir and the erlang VM provides all of these. its kind of amazing. Things we've been able to have out of the box * a metrics dashaboard that gives you a ps -A for all our nodes * intermachine pubsub without having to setup a third party message queue * auto failover for all our microservices * spinning up a microservice takes about as much effort as adding a controller in rails * cronjobs where one node can trigger a job on another node in the network. Hell, we have crons scheduled where we don't even know which machine it'll run on. it just gets done
- sethammons 5y agoI was thinking elixir/erlang too. I've only been using it for a couple of months but I've quickly come to the conclusion that their claims of robustness are not in line with what I want/need. For example, the pubsub lacks persistence and if the node that carries that info dies, you lose that data. There is no built in consensus that tries to maintain state in the face of failure, so you bring in Oban. I've yet to experience the advantages of elixir. Sure, I can hot reload a module, but I spend probably an hour a day waiting for things to compile. I prefer k8s and Go but figure that may be because I'm still new to the ecosystem.
- cultofmetatron 5y agoso far we've had little need for persistence in pubsub, for the few places we do, we have used oban the same way you have. It would be easy to pull in a library like Yggdrasil and abstract it away. For the most part, we just haven't needed it enough to justify setting up rabbitmq or kafka. k8 is indeed useful but the benefit of elixir here is that I can setup the supervision tree in pure elixir. By keeping things simple, we've been able to focus on pushing out features instead of worrying about infrastructure.