4 ms·

Running a 6 node cluster of Pi4s at home now. Did some benchmarks. Long story short. USB thumb drive boot disks are better performance and cheaper than SD card

Running a 6 node cluster of Pi4s at home now. Did some benchmarks. Long story short. USB thumb drive boot disks are better performance and cheaper than SD cards. Fans are completely unnecessary on a Pi4 with an aluminum heat shield style "case". They do run hotter than with a fan, but they will not be heat throttled with recent firmware.

thetanil 6y ago
Example case which does away with the need for fans is: https://www.amazon.com/Geekworm-Raspberry-Computer-Aluminum-Compatible/dp/B07VD568FB https://www.amazon.com/Geekworm-Raspberry-Computer-Aluminum-...
- morganvachon 6y ago
  I like this one, a little more expensive but makes the Pi into a NUC style SFFPC: https://www.waveshare.com/pi4-case-argon-one.htm https://www.waveshare.com/pi4-case-argon-one.htm Edit to add the fan is optional, so it can run fanless with no throttling.
  - thetanil 6y ago
    looks cool, and the breakout board is nice, but they don't show how it attaches to the pi. it's aluminum yeah, but does it actually do anything for heat dissipation?
    - thinkmassive 6y ago
      Scroll down that page, there’s a photo and description... “The Extension Pillars Makes The Aluminum Case Become A Big Heatsink”
  - NegativeLatency 6y ago
    That’s good value with the 2 hdmi adapters
- NortySpock 6y ago
  I've been using the FLIRC case, which appears to be die-cast and looks a little bit more family friendly sitting next to the TV on your entertainment center. https://flirc.tv/more/raspberry-pi-4-case https://flirc.tv/more/raspberry-pi-4-case https://flirc.tv/more/raspberry-pi-case https://flirc.tv/more/raspberry-pi-case (for RPi 2 and 3)
- simonblack 6y ago
  I use one of those for my headless dedicated Pi4B website server plus UPS controller. Admittedly, that's a simple load, but my Pi never gets over 42 degrees C, in a tropical area, averaging around 30 degrees C.
ArtWomb 6y ago
Yeah, I recently spied the 4 node Jetson Mate carrier board on CNX that facilitates building your own Multi GPU cluster ;) https://www.seeedstudio.com/Jetson-Mate-Cooling-Kit-p-4784.html https://www.seeedstudio.com/Jetson-Mate-Cooling-Kit-p-4784.h... They recommend Kubernetes for private cloud workloads. But I am wondering if there isn't room in the ecosystem for a new virtualization / containerization technology to orchestrate tasks on multi gpus? Have heard NVidia is developing its own OS, or rather two, one for Cloud AI and one for real time robotics, but its still research phase...
- thetanil 6y ago
  yeah, I'm excited to get my hands on some Jetsons too. the 4gb memory on the pi is really a big pro though. when one of these SBCs finally comes with 8GB and GPU acceleration for < 100$ we will see some real progress I think.
  - epmaybe 6y ago
    Pi comes with 8gb ram now
  - rualca 6y ago
    > when one of these SBCs finally comes with 8GB and GPU acceleration for < 100$ (...) It might have GPU acceleration but the RPi already comes with 8GB of RAM for about 80$. https://www.raspberrypi.org/blog/8gb-raspberry-pi-4-on-sale-now-at-75/ https://www.raspberrypi.org/blog/8gb-raspberry-pi-4-on-sale-...
- SloopJon 6y ago
  That's a slick cluster in a box. I'm not sure which modules I'd put on it, though. Unless you harvest them from the 2 GB dev kits, the total price pushes up against a single Jetson Xavier AGX, which is a better value (and has an M.2 slot for an SSD).
studius 6y ago
Which USB thumb drives and which SD cards? Do you overclock? If you ran at 99-100% CPU on average, would you still not need fans?
- thetanil 6y ago
  any usb 3.0 drive works, but I am using SanDisk Ultra Fits. price per byte is less and performance is similar or slightly better for reads and way better for sustained writes. No Overclock. 99-100% CPU on all 4 cores for hours and no throttling. they do run at 75-78C instead of 65C, but I could not get them to throttle. Even with the cheapo 3 piece heat sinks there was almost no throttling, but they did throttle occasionally after sustained stress.
  - deleted 6y ago
    [deleted]
- thetanil 6y ago
  missed the SD cards. I am benchmarking against the best SD cards I can get because that is always the bottleneck on the Pi to my experience. The bandwidth and clock rate on USB3 vs sdcard bridge is just faster, but the media is also cheaper in USB3 format in general.
jwillmer 6y ago
I used the following case/setup and can confirm. No fans are required and booting is lightning fast. https://www.prusaprinters.org/prints/40852-pi4-case-stand https://www.prusaprinters.org/prints/40852-pi4-case-stand
hedora 6y ago
Some usb drives are much slower than a decent sd card. Also, the ultra has cooling issues. However, there are usb3 nvme enclosures that are incredibly cheap, and blow both out of the water: https://www.jeffgeerling.com/blog/2020/fastest-usb-storage-options-raspberry-pi https://www.jeffgeerling.com/blog/2020/fastest-usb-storage-o... For a build like the one in the article, I imagine something that puts two nvme sticks beside each other, sandwiched between every other rpi, with passive aluminum heat shields in the remainder of the spaces between the rpi’s. I bet it could be even denser (more pis and more storage) and quieter. I wonder if space can be saved by making a heat shield heatsink that has a pi on both sides, and fins in the middle. Super cool article.
Daho0n 6y ago
>Fans are completely unnecessary on a Pi4 with an aluminum heat shield style "case". They do run hotter than with a fan, but they will not be heat throttled with recent firmware. 1 x 4k@60hz with no case and alu heat shield and my RPI4 does throttle. Latest firmware. Seems you don't punish it enough.
- halfdan 6y ago
  That's the GPU which I would expect to be fairly idle on a cluster of any sort unless you're doing GPU computing (and then why are you using a Pi?)
dheera 6y ago
USB isn't very good for high vibration environments (e.g. inside robots) though :-/ Worst abomination of a connector design I've seen in a couple decades. I wish they had locking industrial connectors on the USB ports on the Pi ... although a SATA/m.2 would be nicer.
- flemhans 6y ago
  Does a little drop of glue fix the issue, or is there something else?
  - dheera 6y ago
    Not really, because that makes it really hard to unplug/replug for servicing. I wish USB was designed to at least snugly support housing to housing of connectors and not have the entire connector supported by a flimsy piece of sheet metal soldered to a PCB.
- scld 6y ago
  I guess that's why you'd move to the compute module?

thetanil 6y ago

Example case which does away with the need for fans is: https://www.amazon.com/Geekworm-Raspberry-Computer-Aluminum-Compatible/dp/B07VD568FB https://www.amazon.com/Geekworm-Raspberry-Computer-Aluminum-...

morganvachon 6y ago
I like this one, a little more expensive but makes the Pi into a NUC style SFFPC: https://www.waveshare.com/pi4-case-argon-one.htm https://www.waveshare.com/pi4-case-argon-one.htm Edit to add the fan is optional, so it can run fanless with no throttling.
- thetanil 6y ago
  looks cool, and the breakout board is nice, but they don't show how it attaches to the pi. it's aluminum yeah, but does it actually do anything for heat dissipation?
  - thinkmassive 6y ago
    Scroll down that page, there’s a photo and description... “The Extension Pillars Makes The Aluminum Case Become A Big Heatsink”
- NegativeLatency 6y ago
  That’s good value with the 2 hdmi adapters
NortySpock 6y ago
I've been using the FLIRC case, which appears to be die-cast and looks a little bit more family friendly sitting next to the TV on your entertainment center. https://flirc.tv/more/raspberry-pi-4-case https://flirc.tv/more/raspberry-pi-4-case https://flirc.tv/more/raspberry-pi-case https://flirc.tv/more/raspberry-pi-case (for RPi 2 and 3)
simonblack 6y ago
I use one of those for my headless dedicated Pi4B website server plus UPS controller. Admittedly, that's a simple load, but my Pi never gets over 42 degrees C, in a tropical area, averaging around 30 degrees C.

ArtWomb 6y ago

Yeah, I recently spied the 4 node Jetson Mate carrier board on CNX that facilitates building your own Multi GPU cluster ;) https://www.seeedstudio.com/Jetson-Mate-Cooling-Kit-p-4784.html https://www.seeedstudio.com/Jetson-Mate-Cooling-Kit-p-4784.h... They recommend Kubernetes for private cloud workloads. But I am wondering if there isn't room in the ecosystem for a new virtualization / containerization technology to orchestrate tasks on multi gpus? Have heard NVidia is developing its own OS, or rather two, one for Cloud AI and one for real time robotics, but its still research phase...

thetanil 6y ago
yeah, I'm excited to get my hands on some Jetsons too. the 4gb memory on the pi is really a big pro though. when one of these SBCs finally comes with 8GB and GPU acceleration for < 100$ we will see some real progress I think.
- epmaybe 6y ago
  Pi comes with 8gb ram now
- rualca 6y ago
  > when one of these SBCs finally comes with 8GB and GPU acceleration for < 100$ (...) It might have GPU acceleration but the RPi already comes with 8GB of RAM for about 80$. https://www.raspberrypi.org/blog/8gb-raspberry-pi-4-on-sale-now-at-75/ https://www.raspberrypi.org/blog/8gb-raspberry-pi-4-on-sale-...
SloopJon 6y ago
That's a slick cluster in a box. I'm not sure which modules I'd put on it, though. Unless you harvest them from the 2 GB dev kits, the total price pushes up against a single Jetson Xavier AGX, which is a better value (and has an M.2 slot for an SSD).

studius 6y ago

Which USB thumb drives and which SD cards? Do you overclock? If you ran at 99-100% CPU on average, would you still not need fans?

thetanil 6y ago
any usb 3.0 drive works, but I am using SanDisk Ultra Fits. price per byte is less and performance is similar or slightly better for reads and way better for sustained writes. No Overclock. 99-100% CPU on all 4 cores for hours and no throttling. they do run at 75-78C instead of 65C, but I could not get them to throttle. Even with the cheapo 3 piece heat sinks there was almost no throttling, but they did throttle occasionally after sustained stress.
- deleted 6y ago
  [deleted]
thetanil 6y ago
missed the SD cards. I am benchmarking against the best SD cards I can get because that is always the bottleneck on the Pi to my experience. The bandwidth and clock rate on USB3 vs sdcard bridge is just faster, but the media is also cheaper in USB3 format in general.

jwillmer 6y ago

I used the following case/setup and can confirm. No fans are required and booting is lightning fast. https://www.prusaprinters.org/prints/40852-pi4-case-stand https://www.prusaprinters.org/prints/40852-pi4-case-stand

hedora 6y ago

Some usb drives are much slower than a decent sd card. Also, the ultra has cooling issues. However, there are usb3 nvme enclosures that are incredibly cheap, and blow both out of the water: https://www.jeffgeerling.com/blog/2020/fastest-usb-storage-options-raspberry-pi https://www.jeffgeerling.com/blog/2020/fastest-usb-storage-o... For a build like the one in the article, I imagine something that puts two nvme sticks beside each other, sandwiched between every other rpi, with passive aluminum heat shields in the remainder of the spaces between the rpi’s. I bet it could be even denser (more pis and more storage) and quieter. I wonder if space can be saved by making a heat shield heatsink that has a pi on both sides, and fins in the middle. Super cool article.

Daho0n 6y ago

>Fans are completely unnecessary on a Pi4 with an aluminum heat shield style "case". They do run hotter than with a fan, but they will not be heat throttled with recent firmware. 1 x 4k@60hz with no case and alu heat shield and my RPI4 does throttle. Latest firmware. Seems you don't punish it enough.

halfdan 6y ago
That's the GPU which I would expect to be fairly idle on a cluster of any sort unless you're doing GPU computing (and then why are you using a Pi?)

dheera 6y ago

USB isn't very good for high vibration environments (e.g. inside robots) though :-/ Worst abomination of a connector design I've seen in a couple decades. I wish they had locking industrial connectors on the USB ports on the Pi ... although a SATA/m.2 would be nicer.

flemhans 6y ago
Does a little drop of glue fix the issue, or is there something else?
- dheera 6y ago
  Not really, because that makes it really hard to unplug/replug for servicing. I wish USB was designed to at least snugly support housing to housing of connectors and not have the entire connector supported by a flimsy piece of sheet metal soldered to a PCB.
scld 6y ago
I guess that's why you'd move to the compute module?