5 ms·

A lot of people miss the fact that Plan 9 was a real distributed operating system. It's not just UNIX with a couple features ("ooh everything is a file" "ooh UT

A lot of people miss the fact that Plan 9 was a real distributed operating system. It's not just UNIX with a couple features ("ooh everything is a file" "ooh UTF8"). You can effortlessly execute any program across multiple hosts on a network. You can use any resource of any host on the network, including files, processes, graphics, networks, disks. It all just works like magic, using a single message-oriented protocol. If Linux worked like this today, you would not need Kubernetes. You could run systemd on a single host and it would just schedule services across all the hosts on the network. Namespaces would already be independent and distributed, so containers would only be used for their esoteric features (network address translation, chroot environment, image layers). Configurations, files, secrets, etc would just be regular files that any process in any container on any host could access using regular permissions models. About 50 layers of abstraction BS would disappear. I think because nobody's actually seen how it can work, they can't imagine it being this simple.

bombela 6y ago
Single point of failure comes to mind though.
- PuercoPop 6y ago
  Why? There isn't a centralized message queue. Its just everything uses the same abstraction and is presented as filesystem or a file.
  - hashkb 6y ago
    > systemd on a single host How would a HA Plan 9 deployment handle disappearance of the "leader"?
    - viraptor 6y ago
      The way I understand it, Plan9 provides the basics, so you can delegate to external nodes over 9P. But it doesn't provide an orchestration solution, so the answer would be "HA Plan 9 deployment" is not a thing - you'd need an extra layer. Unless I missed something while researching P9 years ago.
      - hashkb 6y ago
        So the argument then goes something like: A HA k8s- (or Heroku)- esque platform is more easily built, understood, and operated with Plan 9 because it comes out of the box with many of Docker (and Swarm's?) features. Is that right?
        throwaway823882 6y ago
        The main argument is it comes with primitives that makes most of the orchestration systems' features redundant, because all the components of regular applications can just interoperate over a single unified messaging protocol and API. To consume some custom service, you just read/write a file. To expose a custom service, you just create a fileserver. To do wacky networking, you create a union of files. No more masses of abstractions and glue to tie pieces together because they all speak the same language. In terms of HA, the implementation will be up to your preferred architecture (master/slave, master/master, p2p). Plan9 doesn't schedule services itself, so systemd would need to be modified to adapt one of those architectures. But it wouldn't be a whole lot of work, and it could reuse Plan9's abstractions for most of it. Etcd's key/value store becomes just a filesystem, and Paxos/Raft could be implemented either as a network driver or a userland app, which combined with a union fs, means you just manipulate a single directory of files. You don't even need to futz around with TLS, and again: filesystem permissions. It's not a 1/1 replacement for K8s, but I bet about 80% of the codebase would go away, and most (if not all) of the abstractions.
- stjohnswarts 6y ago
  Surely they designed in failovers?
  - dmytrish 6y ago
    No, they did not. Plan 9 was designed long before any high availability/CAP theorem/distributed databases lessons were learned: it embodies the Unix mindset of reliably available nodes talking over tcp. Failover/distributed consensus/orchestration/load balancing/network split tolerance/replication could be built on top of 9P servers, but all of these concepts are pretty alien to Plan 9 itself.
Shared404 6y ago
> I think because nobody's actually seen how it can work, they can't imagine it being this simple. What's a good way to try it? Cluster of raspberry pi's, or just any given home-lab setup?
- viraptor 6y ago
  https://elinux.org/Plan_9_on_Raspberry_Pi https://elinux.org/Plan_9_on_Raspberry_Pi
mjevans 6y ago
Part of that magic is the trust in the network computer. This could work very well for a corporate setting with thin clients working with a distributed cluster of network services where everything is owned by the corporation. I'm not so sure that this model of trust works with the way computers have evolved since then. Comparing the issues E.G. X11 has vs modern workarounds for direct user IO for games I also wonder how the security model and composition of file layers could negatively impact the experience. Taking the ideas of Plan 9 as inspiration, the more realtime elements could be filtered in kernel and message passed to other processes via a single centralized security model. That might also include exposing shared memory via a memory mapped file, or possibly via a higher level message passing abstraction.
- zozbot234 6y ago
  > That might also include exposing shared memory via a memory mapped file, or possibly via a higher level message passing abstraction. Distributed shared memory needs quite a bit more than the simple "read" and "write" primitives that something like 9P provides. You basically need to replicate a low-level coherency protocol in software. Of course, expect it to be quite slow.
  - deleted 6y ago
    [deleted]
- rakoo 6y ago
  > Part of that magic is the trust in the network computer All network connections are authenticated. Thanks to the way everything in Plan 9 is implemented using it in a network will be closer to how a VPN works: your "view" of the network will be of only trusted computers, and going to the outside world will go through a machine that can act as a firewall.
jonjacky 6y ago
Yes, and it's helpful to remember why it's a distributed system. Plan 9 was created to support people working together in groups at the project or department level. The Plan 9 creators - the original Unix guys - liked the idea of a time-shared computer, where there is just one system to administer and everyone can easily access all the files and other resources. Then it became feasible to use many computers instead of just one, but they wanted to use them with not much more administrative effort than a single time-shared computer, and no additional barriers to sharing files etc. So in the original Plan 9 installations the computers used as terminals were stateless - you could walk up to any terminal and log in to your own customized environment that mounted just the files systems and other resources you wanted. Also, they made use of specialized computers - the ones with nice displays were terminals, there were compute servers with powerful CPUs and file servers with big disks. Some computers were quite specialized, like the ones with WORM drives that supported the Venti versioned file system, that provided seamless automatic backups and even a sort of version control. Now Plan 9 lives on, used (as far as I know) mostly by lone individuals. So now the Plan 9 terminal, file server, and compute server usually all run on the same computer. It works, but it's not the original vision. I think one of the reasons Linux but not Plan 9 took off, besides licensing, is that this vision of a project-scale distributed system fell out of style. Many of the people who adopted Linux in the 1990s wanted a largely self-contained computer they could run themselves, they didn't want a terminal to connect to a distributed system. The original Plan 9 stateless terminals don't really fit in a world where everyone is carrying around their own laptop. So now we have a world with a lot of mostly self-contained individual computers, that use cloud services far away run by huge corporations. The intermediate scale organized around projects and small groups isn't explicitly supported by the computer systems themselves. Plan 9 can live on in this world, but it's not the world it was originally designed for.

bombela 6y ago

Single point of failure comes to mind though.

PuercoPop 6y ago
Why? There isn't a centralized message queue. Its just everything uses the same abstraction and is presented as filesystem or a file.
- hashkb 6y ago
  > systemd on a single host How would a HA Plan 9 deployment handle disappearance of the "leader"?
  - viraptor 6y ago
    The way I understand it, Plan9 provides the basics, so you can delegate to external nodes over 9P. But it doesn't provide an orchestration solution, so the answer would be "HA Plan 9 deployment" is not a thing - you'd need an extra layer. Unless I missed something while researching P9 years ago.
    - hashkb 6y ago
      So the argument then goes something like: A HA k8s- (or Heroku)- esque platform is more easily built, understood, and operated with Plan 9 because it comes out of the box with many of Docker (and Swarm's?) features. Is that right?
      - throwaway823882 6y ago
        The main argument is it comes with primitives that makes most of the orchestration systems' features redundant, because all the components of regular applications can just interoperate over a single unified messaging protocol and API. To consume some custom service, you just read/write a file. To expose a custom service, you just create a fileserver. To do wacky networking, you create a union of files. No more masses of abstractions and glue to tie pieces together because they all speak the same language. In terms of HA, the implementation will be up to your preferred architecture (master/slave, master/master, p2p). Plan9 doesn't schedule services itself, so systemd would need to be modified to adapt one of those architectures. But it wouldn't be a whole lot of work, and it could reuse Plan9's abstractions for most of it. Etcd's key/value store becomes just a filesystem, and Paxos/Raft could be implemented either as a network driver or a userland app, which combined with a union fs, means you just manipulate a single directory of files. You don't even need to futz around with TLS, and again: filesystem permissions. It's not a 1/1 replacement for K8s, but I bet about 80% of the codebase would go away, and most (if not all) of the abstractions.
stjohnswarts 6y ago
Surely they designed in failovers?
- dmytrish 6y ago
  No, they did not. Plan 9 was designed long before any high availability/CAP theorem/distributed databases lessons were learned: it embodies the Unix mindset of reliably available nodes talking over tcp. Failover/distributed consensus/orchestration/load balancing/network split tolerance/replication could be built on top of 9P servers, but all of these concepts are pretty alien to Plan 9 itself.

Shared404 6y ago

> I think because nobody's actually seen how it can work, they can't imagine it being this simple. What's a good way to try it? Cluster of raspberry pi's, or just any given home-lab setup?

viraptor 6y ago
https://elinux.org/Plan_9_on_Raspberry_Pi https://elinux.org/Plan_9_on_Raspberry_Pi

mjevans 6y ago

Part of that magic is the trust in the network computer. This could work very well for a corporate setting with thin clients working with a distributed cluster of network services where everything is owned by the corporation. I'm not so sure that this model of trust works with the way computers have evolved since then. Comparing the issues E.G. X11 has vs modern workarounds for direct user IO for games I also wonder how the security model and composition of file layers could negatively impact the experience. Taking the ideas of Plan 9 as inspiration, the more realtime elements could be filtered in kernel and message passed to other processes via a single centralized security model. That might also include exposing shared memory via a memory mapped file, or possibly via a higher level message passing abstraction.

zozbot234 6y ago
> That might also include exposing shared memory via a memory mapped file, or possibly via a higher level message passing abstraction. Distributed shared memory needs quite a bit more than the simple "read" and "write" primitives that something like 9P provides. You basically need to replicate a low-level coherency protocol in software. Of course, expect it to be quite slow.
- deleted 6y ago
  [deleted]
rakoo 6y ago
> Part of that magic is the trust in the network computer All network connections are authenticated. Thanks to the way everything in Plan 9 is implemented using it in a network will be closer to how a VPN works: your "view" of the network will be of only trusted computers, and going to the outside world will go through a machine that can act as a firewall.

jonjacky 6y ago

Yes, and it's helpful to remember why it's a distributed system. Plan 9 was created to support people working together in groups at the project or department level. The Plan 9 creators - the original Unix guys - liked the idea of a time-shared computer, where there is just one system to administer and everyone can easily access all the files and other resources. Then it became feasible to use many computers instead of just one, but they wanted to use them with not much more administrative effort than a single time-shared computer, and no additional barriers to sharing files etc. So in the original Plan 9 installations the computers used as terminals were stateless - you could walk up to any terminal and log in to your own customized environment that mounted just the files systems and other resources you wanted. Also, they made use of specialized computers - the ones with nice displays were terminals, there were compute servers with powerful CPUs and file servers with big disks. Some computers were quite specialized, like the ones with WORM drives that supported the Venti versioned file system, that provided seamless automatic backups and even a sort of version control. Now Plan 9 lives on, used (as far as I know) mostly by lone individuals. So now the Plan 9 terminal, file server, and compute server usually all run on the same computer. It works, but it's not the original vision. I think one of the reasons Linux but not Plan 9 took off, besides licensing, is that this vision of a project-scale distributed system fell out of style. Many of the people who adopted Linux in the 1990s wanted a largely self-contained computer they could run themselves, they didn't want a terminal to connect to a distributed system. The original Plan 9 stateless terminals don't really fit in a world where everyone is carrying around their own laptop. So now we have a world with a lot of mostly self-contained individual computers, that use cloud services far away run by huge corporations. The intermediate scale organized around projects and small groups isn't explicitly supported by the computer systems themselves. Plan 9 can live on in this world, but it's not the world it was originally designed for.