4 ms·

Setting Up a Cluster of Tiny PCs for Parallel Computing

62 points by speckx 10mo ago

sparcpile 10mo ago
So someone re-implemented the Beowulf cluster. I guess the other Slashdot memes are ready to come back.
- mikestorrent 10mo ago
  Hot Grits as a Service
- TacoCommander 10mo ago
  Score: 1 (Insightful)
  - bigwheels 10mo ago
    Don't forget the ultimate /. achievement! Score: 5 (Troll)
    - dapperdrake 10mo ago
      Score: 6 (LLM)
      - bigwheels 10mo ago
        More like -6 =D
  - phendrenad2 10mo ago
    I love that you created this account just for this joke. Now we just need Natalie Portman to join.
- esseph 10mo ago
  Can it play Crysis/Doom?
  - vee-kay 10mo ago
    Someone needs to submit it to this awesome archive/gallery of Doom ports: https://canitrundoom.org/ https://canitrundoom.org/
- TacticalCoder 10mo ago
  > So someone re-implemented the Beowulf cluster. Darn already. For I was thinking to myself: "Imagine a Beowulf cluster of these!".
dapperdrake 10mo ago
Congratulations on learning about distributed electronic computers. (This is worth tinkering with. This is how people actually get good at HPC.) Pay attention to your SRAM (L3 unified cache), DRAM and swap space tilings. [Snark] In practice: With memory access latency depending on both the square root of the memory size and the physical lengths of the wires in your cluster this sounds like a case for Adam Drake: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
dapperdrake 10mo ago
Kudos for Rscript.
MarsIronPI 10mo ago
I'm planning to set up something similar (but for compiling code). The difference is that my systems came without storage, so I intend to netboot them, which adds a whole other level of complication. I'm planning to use NixOS, with something like nixos-netboot-serve[0]. https://github.com/DeterminateSystems/nix-netboot-serve https://github.com/DeterminateSystems/nix-netboot-serve
yjftsjthsd-h 10mo ago
Something I've been playing with: what's the cheapest semi-reasonable computer that I could make a toy cluster from? Currently eyeing buying a bunch of Raspberry Pi Zeros but I suspect there are cheaper options (maybe some tiny openwrt thing). Too bad ESP32s don't have an MMU:D
- drum55 10mo ago
  You can get intel NUCs way cheaper if you look around, or the Lenovo mini PCs. Small clusters will never beat a decent CPU but you can probably make a cluster of old mini PCs for less than the price of one.
  - yjftsjthsd-h 10mo ago
    So obviously it depends on goals; I absolutely agree that up to a surprisingly high point, the best way to actually get the most compute is to go buy the most powerful single machine that will do what you want off eBay or whatever. My goal is the opposite; I want to specifically get my hands dirty figuring out how to actually provision and manage and operate a large number of hosts, even if they're so low resource as to be effectively worthless individually and are still weak clustered. To that end specifically, digikey claims to be willing to sell me a rather large number of pi zero 2w at $15/each - https://www.digikey.com/en/products/detail/raspberry-pi/SC1176/15298147?src=raspberrypi https://www.digikey.com/en/products/detail/raspberry-pi/SC11... - and even cheap used boxes on eBay seem to start at double that. Obviously you need a little bit more to actually run them, but I believe the only thing you actually need is a USB cable and then a single host computer can provide power and boot them over usb.
    - drum55 10mo ago
      Just be aware that you’ll be stuck with wifi, or spending far more on ethernet adapters for them. If you want cheap with ethernet there’s other devices supported by Arabian in the same price range which have ethernet. It’s fun for sure to have a huge number of very cheap machines, I have an old arm cluster made of Odroid boards for that purpose.
dent9 10mo ago
I appreciate the author's work in doing this and writing it all up so nicely. However every time I see someone doing this, I cannot help but wonder why they are not just using SLURM + Nextflow. SLURM can easily cluster the separate computers as worker nodes, and Nextflow can orchestrate the submission of batch jobs to SLURM in a managed pipeline of tasks. The individual tasks to submit to SLURM would be the users's own R scripts (or any script they have). Combine this with Docker containers to execute on the nodes to manage dependencies needed for task execution. And possibly Ansible for the management of the nodes themselves to install the SLURM daemons and packages etc.. Taken together this creates a FAR more portable and system-agnostic and language-agnostic data analysis workflow that can seamlessly scale over as many nodes and data sets as you can shove into it. This is a LOT better than trying to write all this code in R itself that will do the communication and data passing between nodes directly. Its not clear to me that the author actually needs anything like that, and whats worse, I have seen other authors write exactly that in R and end up re-inventing the wheel of implementing parallel compute tasks (in R). Its really not that complicated. 1) write R script that takes a chunk of your data as input, processes it, writes output to some file, 2) use a workflow manager to pass in chunks up the data to discrete parallel task instances of your script / program and submit the tasks as jobs to 3) a hardware-agnostic job scheduler running on your local hardware and/or cloud resources. This is basically the backbone of HPC but it seems like a lot of people "forget" about the 'job scheduler' and 'workflow manager' parts and jump straight to glueing data-analysis code to hardware. Also important to note that most all robust workflow managers such as Nextflow also already include the parts such as "report task completion", "collect task success / failure logs", "report task CPU / memory resource usages", etc.. So that you, the end user, only need to write the parts that implement your data analysis.
jmward01 10mo ago
Something like proxmox [1] could make wiping everything and restarting a lot easier. There really isn't a huge penalty between bare metal and a VM now so you get the ability to create/deploy/monitor all from a reasonable interface. If the standard clustering stuff isn't enough their datacenter version looks like it is designed more for this. Never used it though. (no ties to proxmox here, just a guy that likes it!) [1] https://www.proxmox.com/en/products/proxmox-virtual-environment/overview https://www.proxmox.com/en/products/proxmox-virtual-environm...
mianos 10mo ago
If you want to do the same thing, build a cluster of complete machines, in a ligh-weight manner, just for the learning experience, you can use incus to create containers in the same manner. As they are all complete machines and you bridge them, you get DHCP allocation identical to running a little PC on the local LAN. If you have ssh set up in the image, as described, you can run the identical scripts. As a plus, if you run them on ZFS with de-dup, even the disk cost for new machines is miniscule.
- macintux 10mo ago
  https://linuxcontainers.org/incus/ https://linuxcontainers.org/incus/ apparently
- abrookewood 10mo ago
  Just started working with Incus and it is great. Have you tried IncusOS yet?
  - mianos 10mo ago
    No I have not. Changing my base image is a hassle but next time I set up another NUC or such, I will for sure.
kingaillas 10mo ago
If the author had googled better they might have discovered https://www.learnpdc.org/ https://www.learnpdc.org/
infinet 10mo ago
Very interesting! For completeness, the job in the article is not strictly parallel, instead, they are separable tasks, which fall under high-throughput computing. HTCondor is a popular choice and is widely used in physics. If the cluster has a shared filesystem such as NFS, it does not even need ssh to each node. Moreover, nodes can share the same R installation on NFS, remove the complexity of installing same set of R packages on all nodes. HTCondor is available on Ubuntu.

62 points by speckx 10mo ago

sparcpile 10mo ago

So someone re-implemented the Beowulf cluster. I guess the other Slashdot memes are ready to come back.

mikestorrent 10mo ago
Hot Grits as a Service
TacoCommander 10mo ago
Score: 1 (Insightful)
- bigwheels 10mo ago
  Don't forget the ultimate /. achievement! Score: 5 (Troll)
  - dapperdrake 10mo ago
    Score: 6 (LLM)
    - bigwheels 10mo ago
      More like -6 =D
- phendrenad2 10mo ago
  I love that you created this account just for this joke. Now we just need Natalie Portman to join.
esseph 10mo ago
Can it play Crysis/Doom?
- vee-kay 10mo ago
  Someone needs to submit it to this awesome archive/gallery of Doom ports: https://canitrundoom.org/ https://canitrundoom.org/
TacticalCoder 10mo ago
> So someone re-implemented the Beowulf cluster. Darn already. For I was thinking to myself: "Imagine a Beowulf cluster of these!".

dapperdrake 10mo ago

Congratulations on learning about distributed electronic computers. (This is worth tinkering with. This is how people actually get good at HPC.) Pay attention to your SRAM (L3 unified cache), DRAM and swap space tilings. [Snark] In practice: With memory access latency depending on both the square root of the memory size and the physical lengths of the wires in your cluster this sounds like a case for Adam Drake: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...

dapperdrake 10mo ago

Kudos for Rscript.

MarsIronPI 10mo ago

I'm planning to set up something similar (but for compiling code). The difference is that my systems came without storage, so I intend to netboot them, which adds a whole other level of complication. I'm planning to use NixOS, with something like nixos-netboot-serve[0]. https://github.com/DeterminateSystems/nix-netboot-serve https://github.com/DeterminateSystems/nix-netboot-serve

yjftsjthsd-h 10mo ago

Something I've been playing with: what's the cheapest semi-reasonable computer that I could make a toy cluster from? Currently eyeing buying a bunch of Raspberry Pi Zeros but I suspect there are cheaper options (maybe some tiny openwrt thing). Too bad ESP32s don't have an MMU:D

drum55 10mo ago
You can get intel NUCs way cheaper if you look around, or the Lenovo mini PCs. Small clusters will never beat a decent CPU but you can probably make a cluster of old mini PCs for less than the price of one.
- yjftsjthsd-h 10mo ago
  So obviously it depends on goals; I absolutely agree that up to a surprisingly high point, the best way to actually get the most compute is to go buy the most powerful single machine that will do what you want off eBay or whatever. My goal is the opposite; I want to specifically get my hands dirty figuring out how to actually provision and manage and operate a large number of hosts, even if they're so low resource as to be effectively worthless individually and are still weak clustered. To that end specifically, digikey claims to be willing to sell me a rather large number of pi zero 2w at $15/each - https://www.digikey.com/en/products/detail/raspberry-pi/SC1176/15298147?src=raspberrypi https://www.digikey.com/en/products/detail/raspberry-pi/SC11... - and even cheap used boxes on eBay seem to start at double that. Obviously you need a little bit more to actually run them, but I believe the only thing you actually need is a USB cable and then a single host computer can provide power and boot them over usb.
  - drum55 10mo ago
    Just be aware that you’ll be stuck with wifi, or spending far more on ethernet adapters for them. If you want cheap with ethernet there’s other devices supported by Arabian in the same price range which have ethernet. It’s fun for sure to have a huge number of very cheap machines, I have an old arm cluster made of Odroid boards for that purpose.

dent9 10mo ago

I appreciate the author's work in doing this and writing it all up so nicely. However every time I see someone doing this, I cannot help but wonder why they are not just using SLURM + Nextflow. SLURM can easily cluster the separate computers as worker nodes, and Nextflow can orchestrate the submission of batch jobs to SLURM in a managed pipeline of tasks. The individual tasks to submit to SLURM would be the users's own R scripts (or any script they have). Combine this with Docker containers to execute on the nodes to manage dependencies needed for task execution. And possibly Ansible for the management of the nodes themselves to install the SLURM daemons and packages etc.. Taken together this creates a FAR more portable and system-agnostic and language-agnostic data analysis workflow that can seamlessly scale over as many nodes and data sets as you can shove into it. This is a LOT better than trying to write all this code in R itself that will do the communication and data passing between nodes directly. Its not clear to me that the author actually needs anything like that, and whats worse, I have seen other authors write exactly that in R and end up re-inventing the wheel of implementing parallel compute tasks (in R). Its really not that complicated. 1) write R script that takes a chunk of your data as input, processes it, writes output to some file, 2) use a workflow manager to pass in chunks up the data to discrete parallel task instances of your script / program and submit the tasks as jobs to 3) a hardware-agnostic job scheduler running on your local hardware and/or cloud resources. This is basically the backbone of HPC but it seems like a lot of people "forget" about the 'job scheduler' and 'workflow manager' parts and jump straight to glueing data-analysis code to hardware. Also important to note that most all robust workflow managers such as Nextflow also already include the parts such as "report task completion", "collect task success / failure logs", "report task CPU / memory resource usages", etc.. So that you, the end user, only need to write the parts that implement your data analysis.

jmward01 10mo ago

Something like proxmox [1] could make wiping everything and restarting a lot easier. There really isn't a huge penalty between bare metal and a VM now so you get the ability to create/deploy/monitor all from a reasonable interface. If the standard clustering stuff isn't enough their datacenter version looks like it is designed more for this. Never used it though. (no ties to proxmox here, just a guy that likes it!) [1] https://www.proxmox.com/en/products/proxmox-virtual-environment/overview https://www.proxmox.com/en/products/proxmox-virtual-environm...

mianos 10mo ago

If you want to do the same thing, build a cluster of complete machines, in a ligh-weight manner, just for the learning experience, you can use incus to create containers in the same manner. As they are all complete machines and you bridge them, you get DHCP allocation identical to running a little PC on the local LAN. If you have ssh set up in the image, as described, you can run the identical scripts. As a plus, if you run them on ZFS with de-dup, even the disk cost for new machines is miniscule.

macintux 10mo ago
https://linuxcontainers.org/incus/ https://linuxcontainers.org/incus/ apparently
abrookewood 10mo ago
Just started working with Incus and it is great. Have you tried IncusOS yet?
- mianos 10mo ago
  No I have not. Changing my base image is a hassle but next time I set up another NUC or such, I will for sure.

kingaillas 10mo ago

If the author had googled better they might have discovered https://www.learnpdc.org/ https://www.learnpdc.org/

infinet 10mo ago

Very interesting! For completeness, the job in the article is not strictly parallel, instead, they are separable tasks, which fall under high-throughput computing. HTCondor is a popular choice and is widely used in physics. If the cluster has a shared filesystem such as NFS, it does not even need ssh to each node. Moreover, nodes can share the same R installation on NFS, remove the complexity of installing same set of R packages on all nodes. HTCondor is available on Ubuntu.