4 ms·

Plan 9 was designed in this way, but never took off. Rob Pike: > This is 2012 and we're still stitching together little microcomputers with HTTPS and ssh and

Plan 9 was designed in this way, but never took off. Rob Pike: > This is 2012 and we're still stitching together little microcomputers with HTTPS and ssh and calling it revolutionary. I sorely miss the unified system view of the world we had at Bell Labs, and the way things are going that seems unlikely to come back any time soon.

nautilus12 5y ago
Glad to see Plan 9 getting some love in the comments even if it didn't make it into the article.
jlpom 5y ago
This describe more a Single System Image [0] to me (WPD includes Plan 9 as one but considering it does not does not supports process migration I find it moot). LinuxPMI [1] seems to be a good idea but they seems to be based on Linux 2.6, so you would have to heavily patch newer kernel. The only thing that seems to support process migration with current software / still active are CRIU [2] (which doesn't support graphical/wayland programs) and DragonflyBSD [3] (in their own words very basic). [0]: https://en.wikipedia.org/wiki/Single_system_image https://en.wikipedia.org/wiki/Single_system_image [1]: http://linuxpmi.org http://linuxpmi.org [2]: criu.org [3]: https://man.dragonflybsd.org/?command=sys_checkpoint&section=2 https://man.dragonflybsd.org/?command=sys_checkpoint&section...
- zozbot234 5y ago
  Graphical programs could be checkpointed and restored as long as they don't directly connect to the hardware. (Because the checkpoint/restore system has no idea how to grab the hardware's relevant state or replicate it on restore.) This means running those apps in a hardware-independent way (e.g. using a separate Wayland instance that connects to the system one), but aside from that it ought to be usable.
  - jlpom 5y ago
    For CRIU it is not supported: https://criu.org/Integration#Wayland.2FWeston https://criu.org/Integration#Wayland.2FWeston, also in my experience it doesn't work. Are you talking about an other software?
    - zozbot234 5y ago
      It has been done "virtually" by going through e.g. VNC https://criu.org/VNC https://criu.org/VNC . Alternately, CRIU apps could be required to use virt-* devices, which CRIU might checkpoint and restore similar to VM's.
- stormbrew 5y ago
  I don't really see any reason to consider process migration a required feature of either a distributed os or a single system image. Even on a single computer this isn't always practical or desireable (ie. you can't 'migrate' a program running on your gpu to your cpu, and you can't trivially migrate a thread from one process to another either). Not all units of computation are interchangeable, and a system that recognizes this and doesn't try to shoehorn everything down to the lowest common denominator actually gains some expressive power over a uniform system (else we would not have threads).
- gnufx 5y ago
  For what it's worth, the HPC-standard way of checkpointing/migrating distributed execution (in userspace, unlike CRIU) is https://dmtcp.sourceforge.io/ https://dmtcp.sourceforge.io/ It supports X via VNC -- I've never tried -- but I guess you could use xpra.
- MisterTea 5y ago
  > his describe more a Single System Image [0] to me No, Plan 9 is not a SSI OS. The idea is all resources are exposed via a single unified file oriented protocol: 9p. All devices are files which means all communication happens over fd's meaning you look at your computer like a patch bay of resources, all communicated with via read() and write(). e.g.: [physical disk]<-->[kernel: sd(3)]-----< /dev/sdE0/ [audio card] <---->[kernel: audio(3)]--< /dev/audio [keyboard]-------->[kernel: kbd(3)]----< /dev/kbd Looking above it looks like Unix but with MAJOR differences. First off the disk is a directory containing partitions which are just files who's size is the partitions size. You can read or write those files as you please. Since the kernel only cares about exposing hardware as files, the file system on a partition needs to be translated to 9p. We do this with a program that is a file server which interprets e.g. a fat32 fs and serves it via 9p (dossrv(4)). Your disk based file system is just a user-space program. And since files are the interface you can bind over them to replace them with a different service like mixfs(4). /dev/audio is like the old linux oss where only one program could open a sound card at a time. To remedy this on plan 9 you run mixfs which opens /dev/audio and then binds itself over /dev replacing /dev/audio in that namespace with a multiplexed /dev/audio from mixfs. Now you start your window manager and the children programs will see mixfs's /dev/audio instead of the kernel /dev/audio. Your programs can now play audio simultaneously without changing ANYTHING. Now compare that simplicity to the trash fire linux audio has been and continues to be with yet another audio subsystem. Keyboard keymaps are a filter program sitting between /dev/kbd and your program. All it does is read in key codes and maps key presses according to a key map which is just a file with key->mapping lines. Again, keyboards are files so a user space file server can be a keyboard such as a GUI keyboard that binds itself over /dev/kbd. Now all those files can be exported or imported to other machines, regardless of CPU architecture. Unix is an OS built on top of a single machine. Plan 9 is a Unix built on top of a network. It's the closest I can get to computing nirvana where all my resources are available from any machine with simple commands that are part of the base OS which is tiny compared to the rest.
  - emteycz 5y ago
    Best explanation of Plan 9 I've ever seen
  - nyanpasu64 5y ago
gnufx 5y ago
If you yearn for Plan 9 -- I'm not sure I do -- Minnich's current incarnation of the inspiration seems to be https://github.com/u-root/cpu https://github.com/u-root/cpu
monocasa 5y ago
Are there any good walkthroughs of what a good, distributed plan 9 setup looks like from either a development or a administration perspective? Particularly an emphasis on many distributed compute nodes (or cpu servers in plan 9 parlance).
NavinF 5y ago
Meh. Every time a 9p server dies, every client dies. Plan9 is not comparable to k8s.
jasonwatkinspdx 5y ago
I think Rob is right to call out the problem, but is being a bit rose colored about Plan 9. Plan 9 was definitely ahead of its time, but it's also a far cry from the sort of distributed OS we need today. "Everything is a remote posix file" ends up being a really bad abstraction for distributed computing. What people are doing today with warehouse scale clusters indeed has a ton of layers of crap in there, and I think it's obvious to yern for sweeping that away. But there's no chance you could do that with P9 as it was designed.
- wahern 5y ago
  "Everything is a file" originally referred to read and write as universal object interfaces. It's similar to Smalltalk's send/receive as an idealized model for object-based programming. Hierarchical filesystem namespaces for object enumeration and acquisition is tangential, though it often works well because most namespaces (DNS, etc) tend to be hierarchical. (POSIX filesystem semantics doesn't really figure into Plan 9 except, perhaps, incidentally.) Filesystem namespacing isn't quite as abstract, though (open, readdir, etc, are much more concrete interfaces), making impedance mismatch more likely. The abstraction is sound. We ended up with TCP and HTTP instead of IL and 9P (and at scale, URLs instead of file descriptors), because of trust issues, but that's not surprising. Ultimately the interface of read/write sits squarely in the middle of all of them, and most others. To build a distributed system with different primitives at the core, for example, send/receive, requires creating significantly stronger constraints on usage and implementation environments. People do that all the time, but in practice they do so by building atop the file interface model. That's what makes the "everything is a file" model so powerful--it's an interoperability sweet spot; an axis around which you can expect most large-scale architectures to revolve around at their core, even if the read/write abstraction isn't visible at the point users (e.g. application developers) interact with the architecture.
  - jasonwatkinspdx 5y ago
    A hierarchical namespace is fine, but the open/read/write/sync/close protocol on byte based files is definitely inadequate. The constraints on usage you decry are in fact fundamental constraints of distributed computing that are at odds with the filesystem abstraction. And this is exactly what I was getting at in talking about rose colored glasses with P9. It in no way is a replacement for something like Colossus or Spanner.

nautilus12 5y ago

Glad to see Plan 9 getting some love in the comments even if it didn't make it into the article.

jlpom 5y ago

This describe more a Single System Image [0] to me (WPD includes Plan 9 as one but considering it does not does not supports process migration I find it moot). LinuxPMI [1] seems to be a good idea but they seems to be based on Linux 2.6, so you would have to heavily patch newer kernel. The only thing that seems to support process migration with current software / still active are CRIU [2] (which doesn't support graphical/wayland programs) and DragonflyBSD [3] (in their own words very basic). [0]: https://en.wikipedia.org/wiki/Single_system_image https://en.wikipedia.org/wiki/Single_system_image [1]: http://linuxpmi.org http://linuxpmi.org [2]: criu.org [3]: https://man.dragonflybsd.org/?command=sys_checkpoint&section=2 https://man.dragonflybsd.org/?command=sys_checkpoint&section...

zozbot234 5y ago
Graphical programs could be checkpointed and restored as long as they don't directly connect to the hardware. (Because the checkpoint/restore system has no idea how to grab the hardware's relevant state or replicate it on restore.) This means running those apps in a hardware-independent way (e.g. using a separate Wayland instance that connects to the system one), but aside from that it ought to be usable.
- jlpom 5y ago
  For CRIU it is not supported: https://criu.org/Integration#Wayland.2FWeston https://criu.org/Integration#Wayland.2FWeston, also in my experience it doesn't work. Are you talking about an other software?
  - zozbot234 5y ago
    It has been done "virtually" by going through e.g. VNC https://criu.org/VNC https://criu.org/VNC . Alternately, CRIU apps could be required to use virt-* devices, which CRIU might checkpoint and restore similar to VM's.
stormbrew 5y ago
I don't really see any reason to consider process migration a required feature of either a distributed os or a single system image. Even on a single computer this isn't always practical or desireable (ie. you can't 'migrate' a program running on your gpu to your cpu, and you can't trivially migrate a thread from one process to another either). Not all units of computation are interchangeable, and a system that recognizes this and doesn't try to shoehorn everything down to the lowest common denominator actually gains some expressive power over a uniform system (else we would not have threads).
gnufx 5y ago
For what it's worth, the HPC-standard way of checkpointing/migrating distributed execution (in userspace, unlike CRIU) is https://dmtcp.sourceforge.io/ https://dmtcp.sourceforge.io/ It supports X via VNC -- I've never tried -- but I guess you could use xpra.
MisterTea 5y ago
> his describe more a Single System Image [0] to me No, Plan 9 is not a SSI OS. The idea is all resources are exposed via a single unified file oriented protocol: 9p. All devices are files which means all communication happens over fd's meaning you look at your computer like a patch bay of resources, all communicated with via read() and write(). e.g.: [physical disk]<-->[kernel: sd(3)]-----< /dev/sdE0/ [audio card] <---->[kernel: audio(3)]--< /dev/audio [keyboard]-------->[kernel: kbd(3)]----< /dev/kbd Looking above it looks like Unix but with MAJOR differences. First off the disk is a directory containing partitions which are just files who's size is the partitions size. You can read or write those files as you please. Since the kernel only cares about exposing hardware as files, the file system on a partition needs to be translated to 9p. We do this with a program that is a file server which interprets e.g. a fat32 fs and serves it via 9p (dossrv(4)). Your disk based file system is just a user-space program. And since files are the interface you can bind over them to replace them with a different service like mixfs(4). /dev/audio is like the old linux oss where only one program could open a sound card at a time. To remedy this on plan 9 you run mixfs which opens /dev/audio and then binds itself over /dev replacing /dev/audio in that namespace with a multiplexed /dev/audio from mixfs. Now you start your window manager and the children programs will see mixfs's /dev/audio instead of the kernel /dev/audio. Your programs can now play audio simultaneously without changing ANYTHING. Now compare that simplicity to the trash fire linux audio has been and continues to be with yet another audio subsystem. Keyboard keymaps are a filter program sitting between /dev/kbd and your program. All it does is read in key codes and maps key presses according to a key map which is just a file with key->mapping lines. Again, keyboards are files so a user space file server can be a keyboard such as a GUI keyboard that binds itself over /dev/kbd. Now all those files can be exported or imported to other machines, regardless of CPU architecture. Unix is an OS built on top of a single machine. Plan 9 is a Unix built on top of a network. It's the closest I can get to computing nirvana where all my resources are available from any machine with simple commands that are part of the base OS which is tiny compared to the rest.
- emteycz 5y ago
  Best explanation of Plan 9 I've ever seen
- nyanpasu64 5y ago

gnufx 5y ago

If you yearn for Plan 9 -- I'm not sure I do -- Minnich's current incarnation of the inspiration seems to be https://github.com/u-root/cpu https://github.com/u-root/cpu

monocasa 5y ago

Are there any good walkthroughs of what a good, distributed plan 9 setup looks like from either a development or a administration perspective? Particularly an emphasis on many distributed compute nodes (or cpu servers in plan 9 parlance).

NavinF 5y ago

Meh. Every time a 9p server dies, every client dies. Plan9 is not comparable to k8s.

jasonwatkinspdx 5y ago

I think Rob is right to call out the problem, but is being a bit rose colored about Plan 9. Plan 9 was definitely ahead of its time, but it's also a far cry from the sort of distributed OS we need today. "Everything is a remote posix file" ends up being a really bad abstraction for distributed computing. What people are doing today with warehouse scale clusters indeed has a ton of layers of crap in there, and I think it's obvious to yern for sweeping that away. But there's no chance you could do that with P9 as it was designed.

wahern 5y ago
"Everything is a file" originally referred to read and write as universal object interfaces. It's similar to Smalltalk's send/receive as an idealized model for object-based programming. Hierarchical filesystem namespaces for object enumeration and acquisition is tangential, though it often works well because most namespaces (DNS, etc) tend to be hierarchical. (POSIX filesystem semantics doesn't really figure into Plan 9 except, perhaps, incidentally.) Filesystem namespacing isn't quite as abstract, though (open, readdir, etc, are much more concrete interfaces), making impedance mismatch more likely. The abstraction is sound. We ended up with TCP and HTTP instead of IL and 9P (and at scale, URLs instead of file descriptors), because of trust issues, but that's not surprising. Ultimately the interface of read/write sits squarely in the middle of all of them, and most others. To build a distributed system with different primitives at the core, for example, send/receive, requires creating significantly stronger constraints on usage and implementation environments. People do that all the time, but in practice they do so by building atop the file interface model. That's what makes the "everything is a file" model so powerful--it's an interoperability sweet spot; an axis around which you can expect most large-scale architectures to revolve around at their core, even if the read/write abstraction isn't visible at the point users (e.g. application developers) interact with the architecture.
- jasonwatkinspdx 5y ago
  A hierarchical namespace is fine, but the open/read/write/sync/close protocol on byte based files is definitely inadequate. The constraints on usage you decry are in fact fundamental constraints of distributed computing that are at odds with the filesystem abstraction. And this is exactly what I was getting at in talking about rose colored glasses with P9. It in no way is a replacement for something like Colossus or Spanner.