7 ms·

A Pi-Powered Plan 9 Cluster

157 points by katzeilla 6y ago

rcarmo 6y ago
This is great. I’ve had a single machine running for a long time, and a lot of the knowledge is still around, but buried in ancient pages. Having someone write about it and do a little hardware project is sure to spark interest.
m463 6y ago
The best kind of project. Well done - case, os, keyboard - but for no reason at all.
- whalesalad 6y ago
  until it ends up in a landfill because an hour later the author was bored lol
pandatigox 6y ago
Does anyone have a link to that monitor? It looks uber cute!
- zhte415 6y ago
  Looks a lot like this but with a different stand https://www.beetronics.com/12-inch-monitor-white https://www.beetronics.com/12-inch-monitor-white (i.e. probably not that :) but similar).
  - bullen 6y ago
    I can recommend the 15" 4:3 which has a ok quality display for non-moving graphics (the brand is Sharp): https://www.beetronics.be/ecran-15-pouces-en-metal-4-3 https://www.beetronics.be/ecran-15-pouces-en-metal-4-3 But beware, these are cheap Chinese monitors available from alibaba company jwell (cheaper but with bad display) so be prepared for a bad experience with the remote. On the upside, these monitors have Composite (for use with older computers) and can be powered by 12v (span 9-33v, not tested though)!
msk-lywenn 6y ago
It would have been nice to know how the software side of the clustering was set up. How easy/painful was it to let the different pies know they were part of a single cluster, how easy/painful was it to have different programs seemlessly use resources from different pies, etc.
- arghwhat 6y ago
  On plan9, all your services are file based. The difference between a cluster and a standalone setup is just whether you mount local instances of a service, or remote instances. For a cluster, you'll usually have a cwfs/hjfs (persistent storage) server, a faktotum (auth) server, and one or more CPU servers, and possibly a terminal machine. You can then freely use or connect to CPU servers. As everything is shared, and the GUI is also file-based, everything will feel local. https://9p.io/wiki/plan9/Expanding_your_Grid/index.html https://9p.io/wiki/plan9/Expanding_your_Grid/index.html
  - pjmlp 6y ago
    > everything will feel local. Not in what concerns graphics rendering performance.
    - MisterTea 6y ago
      Depends on the latency which is the achilles heel of 9p and plan 9 in general. With 9p, for every T message there's an R message that is waited on before the next T is sent. So if there's 100ms of latency you have to wait 100ms between EVERY 9p message. Pain and suffering. The labs people never focused on performance but actually building a clean comprehensive system with the small team they had. It was a pragmatic approach for sure and compromises were made. Though when using local ethernet, heavy graphics isn't that bad. I can play doom over local ethernet using drawterm to a cpu server which is actually running the doom processes. Of course blowing up the game screen makes it suffer as doom doesn't use dev draw. Another thing to realize is that plan 9 graphics are done via dev draw which is a sort of 2d graphics engine that you load text and bitmaps into then issue commands to draw and move them around on screen. So a local dev draw on the terminal only receives draw commands for the loaded resources so there is some caching and heavy lifting done at the terminal end. It's not like vnc or rdp where you are viewing a remote frame buffer.
aap_ 6y ago
What did this have to do with plan 9? This post is mostly about building a cute case and keyboard for 4 raspis. No clustering and almost no plan 9 involved at all.
- TabTwo 6y ago
  I thought the same. Just booting the first rpi and it ends where the real fun would start.
flobosg 6y ago
Devine Lu Linvega has been running Plan 9 on a Raspberry Pi for a while: https://wiki.xxiivv.com/site/plan9.html https://wiki.xxiivv.com/site/plan9.html
- katzeilla 6y ago
  Interesting! Those screenshots are pretty.
- hestefisk 6y ago
  Read about his setup here: https://wiki.xxiivv.com/site/usagi.html https://wiki.xxiivv.com/site/usagi.html So cool.
varispeed 6y ago
While this looks cool it is far from useful. I took time to build such cluster with the Pies and shortly after it landed in the box of things I may need in the future. It's just too slow to do anything useful. Watching stuff compile or even boot was like watching paint dry. It is much better to get a bunch of used NUC if you are space conscious or just get VMWare Workstation or similar tool to simulate number of machines on your desktop.
- psychoslave 6y ago
  Come on, it’s just plain beautiful. Sure it might not be very useful as a modern desktop, but just looking at the result, it looks like a peace of art to me. The person probably enjoyed doing it, most likely didn’t harm anyone in the process, and possibly learnt some things on the road. Not everything need to be useful beyond the joy of doing it.
- ClumsyPilot 6y ago
  I second this - the PI's are great for hacking and hardware, but obsession with PI for compute does not make sence. Any used PC/NUC/Intel Atom board is much, much faster and some cost around the same.
  - OskarS 6y ago
    I think the utility is more with experimenting with distributed computing technologies in a relatively affordable way, with hardware that is familiar and simple to set up.
    - magicalhippo 6y ago
      I made a small SAN, just for learning. Four pi4's serving two USB-SATA disks each via iSCSI to my NAS, which mounts them as a ZFS pool. The pi4's are connected to a switch with a 10G link to the NAS. Again, mainly just for fun and experience.
  - bullen 6y ago
    They are much faster because they draw more energy. The Raspberry 4 at 1 Gflops/watt is a better option if you want to power the thing with lead acid batteries as backup power for more than 24 hours. Also Raspberry 4 can be cooled passively with a smaller heatsink so the total size and weight is less.
ordu 6y ago
> The project provided an excuse to make use of a Vortex Core 47-key keyboard, which together with a mini white HDMI monitor, provided a particularly compact and suitably futuristic feeling setup. As for me this keyboard doesn't seem futuristic at all. It reminds me of the first computer I experienced in my life.[1] I mean, it is lovely keyboard, I really like it. Just "futuristic" is a strange way to describe it. It would be even more nice, if some keys of bright colors were added. Like in MK-61[2]. Though photo in wikipedia doesn't capture it (or maybe my memory is wrong), but those blue key was of a so beautiful shade, I really loved it in my childhood and was eager to learn how to use MK-61 just for a joy of using this very key. Yellow and red keys were also likeable but not so much, moreover the function of red key was trivial and I lost interest to it very quickly. But the blue key was beautiful and mysterious... My first love. Maybe I became a programmer due to color of this key. [1] https://en.wikipedia.org/wiki/Electronika_BK https://en.wikipedia.org/wiki/Electronika_BK [2] https://en.wikipedia.org/wiki/Elektronika_MK-61 https://en.wikipedia.org/wiki/Elektronika_MK-61
- dexen 6y ago
  >futuristic Presumably through Vortex'[0] similarity to the Symbolics PN 365407[1][2] keyboard, with its very distinctive design that manages to capture at once both futurism and timelessness. Disregard the 2020's clunky cherry switch keyboards, acquire the OG Best Keyboard. -- [0] https://mechanicalkeyboards.com/shop/images/products/large_VTG47BEG_5.jpg https://mechanicalkeyboards.com/shop/images/products/large_V... [1] http://xahlee.info/kbd/symbolics_keyboard_pn_365407.html http://xahlee.info/kbd/symbolics_keyboard_pn_365407.html [2] http://coder.zoomquiet.top/data/20130716102310/index.html http://coder.zoomquiet.top/data/20130716102310/index.html
  - rbanffy 6y ago
    I believe you are right on spot with that - the minimalism, of the keyboard, case and the Plan 9 UI evoke some sort of timelessness, the same way a Connection Machine still looks futuristic today.
- zepto 6y ago
  Maybe ‘retro-futuristic’ is a better term.
iuguy 6y ago
Flagged because a post about a "Pi-Powered Plan 9 Cluster" should at least pay lip service to running Plan 9 or setting up a cluster.
MisterTea 6y ago
Of course those sd card image links are broken. I recommend using the 9front fork which is actively developed with a nearly daily active mailing list and irc channel. http://9front.org/releases/ http://9front.org/releases/ (please use the torrent links if possible)
amelius 6y ago
Here's an old video of a cluster with 3000 RPis: https://www.youtube.com/watch?v=78H-4KqVvrg https://www.youtube.com/watch?v=78H-4KqVvrg
charlieo88 6y ago
Something that always bothers me about these Pi clusters is the Ethernet cable lengths. Isn't there a minimum length?
- dsr_ 6y ago
  10cm for 1000BASE-CX, nothing prior specified a minimum length.
  - rwmj 6y ago
    That's really interesting - do you know why there is a minimum? Maximum lengths are easy to understand because of voltage drops, interference and impedance over long distances. And of course in the old CSMA/CD days multiple senders couldn't be separated by too large a distance.
    - MisterTea 6y ago
      signal reflection most likely. https://en.wikipedia.org/wiki/Signal_reflection https://en.wikipedia.org/wiki/Signal_reflection
moth-fuzz 6y ago
I discovered this article a while ago when I was toying around with using plan 9 as it was designed, as a fully transparent distributed operating system. After reading this article I was a little shocked because the author didn't do any configuration or setup for the other machines, so I thought, wow! Plan 9 was so advanced that all you need are multiple machines present on the same network and it will automatically 'clusterize' them. So I went out to make my own cluster and realized that no, sadly, the author simply did not activate nor configure the other machines at all. I still love plan 9 and still use it on the occasion, but I'd love more detail on how to actually set up a plan9 cluster and really 'get into' the power of distributed computing.
asim 6y ago
We tried hosting Micro (micro.mu) on Pi last year as a test to create some sort of global distributed network for services. Micro is written in Go and as Plan 9 focuses everything as a file, micro focuses on everything as a service. This post is really fascinating. In some ways the popularity of it makes me think self hosted Pi for cloud services might be viable. Even to the point of designing a packaged solution for devs. For reference https://micro.mu/blog/2019/12/05/building-a-microservices-network.html https://micro.mu/blog/2019/12/05/building-a-microservices-ne...
Naac 6y ago
I think I missed something. Is there any setup required in software for the actual cluster? It looks like there are 4 Pis all running a Pi Plan 9 image, and they're wired together via ethernet. But where is the software setup to link them all together? Or does this just happen automatically on Plan 9?

157 points by katzeilla 6y ago

This is great. I’ve had a single machine running for a long time, and a lot of the knowledge is still around, but buried in ancient pages. Having someone write about it and do a little hardware project is sure to spark interest.

m463 6y ago

The best kind of project. Well done - case, os, keyboard - but for no reason at all.

whalesalad 6y ago
until it ends up in a landfill because an hour later the author was bored lol

pandatigox 6y ago

Does anyone have a link to that monitor? It looks uber cute!

zhte415 6y ago
Looks a lot like this but with a different stand https://www.beetronics.com/12-inch-monitor-white https://www.beetronics.com/12-inch-monitor-white (i.e. probably not that :) but similar).
- bullen 6y ago
  I can recommend the 15" 4:3 which has a ok quality display for non-moving graphics (the brand is Sharp): https://www.beetronics.be/ecran-15-pouces-en-metal-4-3 https://www.beetronics.be/ecran-15-pouces-en-metal-4-3 But beware, these are cheap Chinese monitors available from alibaba company jwell (cheaper but with bad display) so be prepared for a bad experience with the remote. On the upside, these monitors have Composite (for use with older computers) and can be powered by 12v (span 9-33v, not tested though)!

msk-lywenn 6y ago

It would have been nice to know how the software side of the clustering was set up. How easy/painful was it to let the different pies know they were part of a single cluster, how easy/painful was it to have different programs seemlessly use resources from different pies, etc.

arghwhat 6y ago
On plan9, all your services are file based. The difference between a cluster and a standalone setup is just whether you mount local instances of a service, or remote instances. For a cluster, you'll usually have a cwfs/hjfs (persistent storage) server, a faktotum (auth) server, and one or more CPU servers, and possibly a terminal machine. You can then freely use or connect to CPU servers. As everything is shared, and the GUI is also file-based, everything will feel local. https://9p.io/wiki/plan9/Expanding_your_Grid/index.html https://9p.io/wiki/plan9/Expanding_your_Grid/index.html
- pjmlp 6y ago
  > everything will feel local. Not in what concerns graphics rendering performance.
  - MisterTea 6y ago
    Depends on the latency which is the achilles heel of 9p and plan 9 in general. With 9p, for every T message there's an R message that is waited on before the next T is sent. So if there's 100ms of latency you have to wait 100ms between EVERY 9p message. Pain and suffering. The labs people never focused on performance but actually building a clean comprehensive system with the small team they had. It was a pragmatic approach for sure and compromises were made. Though when using local ethernet, heavy graphics isn't that bad. I can play doom over local ethernet using drawterm to a cpu server which is actually running the doom processes. Of course blowing up the game screen makes it suffer as doom doesn't use dev draw. Another thing to realize is that plan 9 graphics are done via dev draw which is a sort of 2d graphics engine that you load text and bitmaps into then issue commands to draw and move them around on screen. So a local dev draw on the terminal only receives draw commands for the loaded resources so there is some caching and heavy lifting done at the terminal end. It's not like vnc or rdp where you are viewing a remote frame buffer.

aap_ 6y ago

What did this have to do with plan 9? This post is mostly about building a cute case and keyboard for 4 raspis. No clustering and almost no plan 9 involved at all.

TabTwo 6y ago
I thought the same. Just booting the first rpi and it ends where the real fun would start.

flobosg 6y ago

Devine Lu Linvega has been running Plan 9 on a Raspberry Pi for a while: https://wiki.xxiivv.com/site/plan9.html https://wiki.xxiivv.com/site/plan9.html

katzeilla 6y ago
Interesting! Those screenshots are pretty.
hestefisk 6y ago
Read about his setup here: https://wiki.xxiivv.com/site/usagi.html https://wiki.xxiivv.com/site/usagi.html So cool.

varispeed 6y ago

While this looks cool it is far from useful. I took time to build such cluster with the Pies and shortly after it landed in the box of things I may need in the future. It's just too slow to do anything useful. Watching stuff compile or even boot was like watching paint dry. It is much better to get a bunch of used NUC if you are space conscious or just get VMWare Workstation or similar tool to simulate number of machines on your desktop.

psychoslave 6y ago
Come on, it’s just plain beautiful. Sure it might not be very useful as a modern desktop, but just looking at the result, it looks like a peace of art to me. The person probably enjoyed doing it, most likely didn’t harm anyone in the process, and possibly learnt some things on the road. Not everything need to be useful beyond the joy of doing it.
ClumsyPilot 6y ago
I second this - the PI's are great for hacking and hardware, but obsession with PI for compute does not make sence. Any used PC/NUC/Intel Atom board is much, much faster and some cost around the same.
- OskarS 6y ago
  I think the utility is more with experimenting with distributed computing technologies in a relatively affordable way, with hardware that is familiar and simple to set up.
  - magicalhippo 6y ago
    I made a small SAN, just for learning. Four pi4's serving two USB-SATA disks each via iSCSI to my NAS, which mounts them as a ZFS pool. The pi4's are connected to a switch with a 10G link to the NAS. Again, mainly just for fun and experience.
- bullen 6y ago
  They are much faster because they draw more energy. The Raspberry 4 at 1 Gflops/watt is a better option if you want to power the thing with lead acid batteries as backup power for more than 24 hours. Also Raspberry 4 can be cooled passively with a smaller heatsink so the total size and weight is less.

ordu 6y ago

> The project provided an excuse to make use of a Vortex Core 47-key keyboard, which together with a mini white HDMI monitor, provided a particularly compact and suitably futuristic feeling setup. As for me this keyboard doesn't seem futuristic at all. It reminds me of the first computer I experienced in my life.[1] I mean, it is lovely keyboard, I really like it. Just "futuristic" is a strange way to describe it. It would be even more nice, if some keys of bright colors were added. Like in MK-61[2]. Though photo in wikipedia doesn't capture it (or maybe my memory is wrong), but those blue key was of a so beautiful shade, I really loved it in my childhood and was eager to learn how to use MK-61 just for a joy of using this very key. Yellow and red keys were also likeable but not so much, moreover the function of red key was trivial and I lost interest to it very quickly. But the blue key was beautiful and mysterious... My first love. Maybe I became a programmer due to color of this key. [1] https://en.wikipedia.org/wiki/Electronika_BK https://en.wikipedia.org/wiki/Electronika_BK [2] https://en.wikipedia.org/wiki/Elektronika_MK-61 https://en.wikipedia.org/wiki/Elektronika_MK-61

dexen 6y ago
>futuristic Presumably through Vortex'[0] similarity to the Symbolics PN 365407[1][2] keyboard, with its very distinctive design that manages to capture at once both futurism and timelessness. Disregard the 2020's clunky cherry switch keyboards, acquire the OG Best Keyboard. -- [0] https://mechanicalkeyboards.com/shop/images/products/large_VTG47BEG_5.jpg https://mechanicalkeyboards.com/shop/images/products/large_V... [1] http://xahlee.info/kbd/symbolics_keyboard_pn_365407.html http://xahlee.info/kbd/symbolics_keyboard_pn_365407.html [2] http://coder.zoomquiet.top/data/20130716102310/index.html http://coder.zoomquiet.top/data/20130716102310/index.html
- rbanffy 6y ago
  I believe you are right on spot with that - the minimalism, of the keyboard, case and the Plan 9 UI evoke some sort of timelessness, the same way a Connection Machine still looks futuristic today.
zepto 6y ago
Maybe ‘retro-futuristic’ is a better term.

iuguy 6y ago

Flagged because a post about a "Pi-Powered Plan 9 Cluster" should at least pay lip service to running Plan 9 or setting up a cluster.

MisterTea 6y ago

Of course those sd card image links are broken. I recommend using the 9front fork which is actively developed with a nearly daily active mailing list and irc channel. http://9front.org/releases/ http://9front.org/releases/ (please use the torrent links if possible)

amelius 6y ago

Here's an old video of a cluster with 3000 RPis: https://www.youtube.com/watch?v=78H-4KqVvrg https://www.youtube.com/watch?v=78H-4KqVvrg

charlieo88 6y ago

Something that always bothers me about these Pi clusters is the Ethernet cable lengths. Isn't there a minimum length?

dsr_ 6y ago
10cm for 1000BASE-CX, nothing prior specified a minimum length.
- rwmj 6y ago
  That's really interesting - do you know why there is a minimum? Maximum lengths are easy to understand because of voltage drops, interference and impedance over long distances. And of course in the old CSMA/CD days multiple senders couldn't be separated by too large a distance.
  - MisterTea 6y ago
    signal reflection most likely. https://en.wikipedia.org/wiki/Signal_reflection https://en.wikipedia.org/wiki/Signal_reflection

moth-fuzz 6y ago

I discovered this article a while ago when I was toying around with using plan 9 as it was designed, as a fully transparent distributed operating system. After reading this article I was a little shocked because the author didn't do any configuration or setup for the other machines, so I thought, wow! Plan 9 was so advanced that all you need are multiple machines present on the same network and it will automatically 'clusterize' them. So I went out to make my own cluster and realized that no, sadly, the author simply did not activate nor configure the other machines at all. I still love plan 9 and still use it on the occasion, but I'd love more detail on how to actually set up a plan9 cluster and really 'get into' the power of distributed computing.

asim 6y ago

We tried hosting Micro (micro.mu) on Pi last year as a test to create some sort of global distributed network for services. Micro is written in Go and as Plan 9 focuses everything as a file, micro focuses on everything as a service. This post is really fascinating. In some ways the popularity of it makes me think self hosted Pi for cloud services might be viable. Even to the point of designing a packaged solution for devs. For reference https://micro.mu/blog/2019/12/05/building-a-microservices-network.html https://micro.mu/blog/2019/12/05/building-a-microservices-ne...

Naac 6y ago

I think I missed something. Is there any setup required in software for the actual cluster? It looks like there are 4 Pis all running a Pi Plan 9 image, and they're wired together via ethernet. But where is the software setup to link them all together? Or does this just happen automatically on Plan 9?