5 ms·

What a coincidence, I just finished setting up my 64GB RAM / 2 TB storage Raspberry Pi 4 PoE cluster tonight (only 1 power cable!): https://www.dropbox.com/s/um

What a coincidence, I just finished setting up my 64GB RAM / 2 TB storage Raspberry Pi 4 PoE cluster tonight (only 1 power cable!): https://www.dropbox.com/s/um5mjt8bucfkrbk/picluster.jpg?dl=0 https://www.dropbox.com/s/um5mjt8bucfkrbk/picluster.jpg?dl=0 I was really impressed that I got line speed Ethernet between the Pis, 112MB/s doing a wget of a 1G file from one to another. Things have improved a lot since the original Raspberry Pi from 2012. I need to figure out how to get the 8th Pi working though, since I'm using one of the PoE ports to power a Mikrotik that I'm using as a wireless bridge onto my existing broadcast domain. Technically I don't need the bridge, since each Pi can connect to my wireless AP over 5GHz 802.11ac just fine, but it feels nicer to funnel upstream data through one point. More importantly, it reduces WiFi collisions since there wouldn't be 8 devices broadcasting simultaneously and interfereing with each other (especially at those close distances) if they all needed to download something from the Internet at the same time... like a Docker image for a Kubernetes deployment :)

youreanidi0t 6y ago
You spent at least $720 on the Pis not including case, network and console cables, storage, PSU and PDU harness, switch, (missing) thermal management and a management system. Congrats on not saving any money (or power usage) over buying a Ryzen 5 desktop, which would absolutely murder the Pi4 cluster performance-wise. Better memory bandwidth, AES-NI...for about the same wattage. Seriously, go look at the benchmarks. For compression, a Ryzen 3600 is TWENTY times faster than a Pi4, just to give one example. On top of performance you wouldn't have to deal with hardware consoles, slow disk IO, the network config would be virtual and have a fraction of the latency with much greater bandwidth...and 8x8GB of ram is far less flexible than 1x64GB of ram. You could put the whole thing into sleep/suspend mode when you aren't using it, too. Pi4s are cool but I'm so sick of people using Pi's because they're trendy and will get them attention, instead of them actually making sense for the application.
- danmur 6y ago
  Burn :P One small thing they're nice for is if you want to play with a cluster (e.g. nomad, kubernetes, whatever) on a budget. Of course it's not going to be great bang for buck just for performance but if you're trying out some distributed computing it's nice to be working to real limitations, network and RAM and particular.
- hddherman 6y ago
  I'm sure that if their goal was a fast server with a modern CPU, then they would have gone for it. I see their project as a fun hobby. It's very fun to put a lot of tiny low-power computers together in a cluster and learn as you go. Sure, you should take into account the limitations, but you should not feel discouraged if all you want to do is to have fun, learn and build something that is quite unique.
- panpanna 6y ago
  Most pi clusters are created for learning devops theologies. Speaking of that, happy to see LXD on the front page. Its not as hyped as Docker but I really like it.
- herpderperator 6y ago
  As others have correctly pointed out, this is a fun, ongoing project on server clusters built using real hardware with real limitations. My laptop from 2018 has 64GB RAM... and would it even matter to say that I already have several high-performance machines? I am not sure what your point is. I did not buy Raspberry Pis to save money, I bought them to have a bunch of small physical servers to play around with. I could have easily started 8 VMs bridged to the network in the same way with virt-manager/QEMU and got 8 much more powerful Linux instances for free if that's what I wanted.
  - fapjacks 6y ago
    Like looking at a jeweler's workshop and asking "Why all these shitty little hammers? Don't you know how well a sledge works?"
- oarsinsync 6y ago
  In terms of raw compute, you're absolutely right. In terms of understanding distributed computing and the failure modes that can occur, the GP's approach is definitely better. If you build distributed systems only testing them through virtualisation, you'll run into all sorts of issues that come from adding real world constraints that are otherwise masked through the virtualised abstraction.
- jacquesm 6y ago
  And I'm sick of people pissing on other people's nice projects out of envy or malice. Why don't you simply fuck off? Flagged.
- barumi 6y ago
  This post just adds noise to an otherwise interesting discussion while missing the whole point of learning how to set up a cluster. Even the tone itself is non-constructive, all wrong and off-putting. I'm sure you can understand that someone who bolts on computers over WiFi and praises a 100MB/s connection is not aiming for performance. This is not what HN is all about.
hiq 6y ago
Have you tried to experiment with 802.11s and mesh networks to remove some of these Ethernet cables by any chance? I'm looking into some "advanced" router features these days, and I was surprised that there is a standard for this, although it seems badly supported. That would seem like a good option for a more flexible (in terms of networking) cluster if the nodes are that close to each other that wireless networking shortcomings aren't much of an issue. Of course that's only the theory...
- herpderperator 6y ago
  I couldn't tell by your question if you were referring to the Pi cluster or asking an unrelated question about 802.11s mesh clusters... :) Here are my answers for both: Pi cluster: meshing won't save any additional wires here since there is already a wireless bridge set up (between the black Mikrotik device pictured, and an off-camera Mikrotik wireless router.) I chose to wire the cluster simply to guarantee a reliable connection within the cluster. You can absolutely use a Raspberry Pi over WiFi without an Ethernet cable, it would just be a less reliable connection (jitter, congestion), and then of course it would have to be powered conventionally rather than via PoE. General 802.11s meshing: If all you want is fewer network cabling in your home specifically between switches and access points, then it could certainly help. However, the best choice network-wise is almost always a proper deployment with a central router and several APs (broadcasting the same SSID) wired back to it.
  - hiq 6y ago
    I guess I was curious about both, thanks for yours answers! I hadn't thought about PoE, having never used it myself, so indeed at one cable per device there's not much more to do.
jacquesm 6y ago
That looks gorgeous.
moray 6y ago
Nice project! I just started researching the pieces I need to build something similar. What kind of PoE hat did you use? I was also thinking of having one of the PIs have also an SSD with bootp/dhcp and let the others boot with PXE. I had been thinking of building a PIs cluster for a while now and I was looking with interest at the turingpi board project [1] they are taking preorders for the second batch if anyone is interested. But I would like nodes with more than 1GB of RAM so I guess I'll have to wait for that. [1] https://turingpi.com https://turingpi.com
- herpderperator 6y ago
  This is the PoE hat I used - I specifically chose it because of its low profile and fanless design: https://www.amazon.com/gp/product/B07WD7HXSQ/ https://www.amazon.com/gp/product/B07WD7HXSQ/

youreanidi0t 6y ago

You spent at least $720 on the Pis not including case, network and console cables, storage, PSU and PDU harness, switch, (missing) thermal management and a management system. Congrats on not saving any money (or power usage) over buying a Ryzen 5 desktop, which would absolutely murder the Pi4 cluster performance-wise. Better memory bandwidth, AES-NI...for about the same wattage. Seriously, go look at the benchmarks. For compression, a Ryzen 3600 is TWENTY times faster than a Pi4, just to give one example. On top of performance you wouldn't have to deal with hardware consoles, slow disk IO, the network config would be virtual and have a fraction of the latency with much greater bandwidth...and 8x8GB of ram is far less flexible than 1x64GB of ram. You could put the whole thing into sleep/suspend mode when you aren't using it, too. Pi4s are cool but I'm so sick of people using Pi's because they're trendy and will get them attention, instead of them actually making sense for the application.

danmur 6y ago
Burn :P One small thing they're nice for is if you want to play with a cluster (e.g. nomad, kubernetes, whatever) on a budget. Of course it's not going to be great bang for buck just for performance but if you're trying out some distributed computing it's nice to be working to real limitations, network and RAM and particular.
hddherman 6y ago
I'm sure that if their goal was a fast server with a modern CPU, then they would have gone for it. I see their project as a fun hobby. It's very fun to put a lot of tiny low-power computers together in a cluster and learn as you go. Sure, you should take into account the limitations, but you should not feel discouraged if all you want to do is to have fun, learn and build something that is quite unique.
panpanna 6y ago
Most pi clusters are created for learning devops theologies. Speaking of that, happy to see LXD on the front page. Its not as hyped as Docker but I really like it.
herpderperator 6y ago
As others have correctly pointed out, this is a fun, ongoing project on server clusters built using real hardware with real limitations. My laptop from 2018 has 64GB RAM... and would it even matter to say that I already have several high-performance machines? I am not sure what your point is. I did not buy Raspberry Pis to save money, I bought them to have a bunch of small physical servers to play around with. I could have easily started 8 VMs bridged to the network in the same way with virt-manager/QEMU and got 8 much more powerful Linux instances for free if that's what I wanted.
- fapjacks 6y ago
  Like looking at a jeweler's workshop and asking "Why all these shitty little hammers? Don't you know how well a sledge works?"
oarsinsync 6y ago
In terms of raw compute, you're absolutely right. In terms of understanding distributed computing and the failure modes that can occur, the GP's approach is definitely better. If you build distributed systems only testing them through virtualisation, you'll run into all sorts of issues that come from adding real world constraints that are otherwise masked through the virtualised abstraction.
jacquesm 6y ago
And I'm sick of people pissing on other people's nice projects out of envy or malice. Why don't you simply fuck off? Flagged.
barumi 6y ago
This post just adds noise to an otherwise interesting discussion while missing the whole point of learning how to set up a cluster. Even the tone itself is non-constructive, all wrong and off-putting. I'm sure you can understand that someone who bolts on computers over WiFi and praises a 100MB/s connection is not aiming for performance. This is not what HN is all about.

hiq 6y ago

Have you tried to experiment with 802.11s and mesh networks to remove some of these Ethernet cables by any chance? I'm looking into some "advanced" router features these days, and I was surprised that there is a standard for this, although it seems badly supported. That would seem like a good option for a more flexible (in terms of networking) cluster if the nodes are that close to each other that wireless networking shortcomings aren't much of an issue. Of course that's only the theory...

herpderperator 6y ago
I couldn't tell by your question if you were referring to the Pi cluster or asking an unrelated question about 802.11s mesh clusters... :) Here are my answers for both: Pi cluster: meshing won't save any additional wires here since there is already a wireless bridge set up (between the black Mikrotik device pictured, and an off-camera Mikrotik wireless router.) I chose to wire the cluster simply to guarantee a reliable connection within the cluster. You can absolutely use a Raspberry Pi over WiFi without an Ethernet cable, it would just be a less reliable connection (jitter, congestion), and then of course it would have to be powered conventionally rather than via PoE. General 802.11s meshing: If all you want is fewer network cabling in your home specifically between switches and access points, then it could certainly help. However, the best choice network-wise is almost always a proper deployment with a central router and several APs (broadcasting the same SSID) wired back to it.
- hiq 6y ago
  I guess I was curious about both, thanks for yours answers! I hadn't thought about PoE, having never used it myself, so indeed at one cable per device there's not much more to do.

jacquesm 6y ago

That looks gorgeous.

moray 6y ago

Nice project! I just started researching the pieces I need to build something similar. What kind of PoE hat did you use? I was also thinking of having one of the PIs have also an SSD with bootp/dhcp and let the others boot with PXE. I had been thinking of building a PIs cluster for a while now and I was looking with interest at the turingpi board project [1] they are taking preorders for the second batch if anyone is interested. But I would like nodes with more than 1GB of RAM so I guess I'll have to wait for that. [1] https://turingpi.com https://turingpi.com

herpderperator 6y ago
This is the PoE hat I used - I specifically chose it because of its low profile and fanless design: https://www.amazon.com/gp/product/B07WD7HXSQ/ https://www.amazon.com/gp/product/B07WD7HXSQ/