5 ms·

Raspberry Pi 2 Cluster Assembly Tutorial

78 points by stkim1 11y ago

rcarmo 11y ago
Neat. Anyone else at HN did something like this? Mine is at https://github.com/rcarmo/raspi-cluster https://github.com/rcarmo/raspi-cluster (configs and some utilities included)
- stkim1 11y ago
  Matthew Saunders also designed a very nice cluster stack. https://learnaddict.com/2015/08/03/raspberry-pi-stack-a-platform-for-learning-about-iot/ https://learnaddict.com/2015/08/03/raspberry-pi-stack-a-plat...
- geerlingguy 11y ago
  I have an educational site running on my cluster of 6 Pi 2s in my basement: http://www.pidramble.com/ http://www.pidramble.com/ Code, tutorial, parts list, etc. here: https://github.com/geerlingguy/raspberry-pi-dramble https://github.com/geerlingguy/raspberry-pi-dramble
- tracker1 11y ago
  I always thought the cassandra multi datacenter cluster on rpis was interesting. http://www.datastax.com/dev/blog/32-node-raspberry-pi-cassandra-cluster http://www.datastax.com/dev/blog/32-node-raspberry-pi-cassan...
leoedin 11y ago
What sort of cost to performance do these give? Is it actually more cost effective to purchase and run 6 RPis compared to commodity x86 hardware? How about compared to the cost of computing your job on one of the numerous cloud based offerings? I've never really been faced with a compute job that takes long enough to be worth investigating parallelisation, so I'm fairly ignorant in that regard. This is certainly a neat project (and perhaps provides a nice talking point), but is it more than that?
- stkim1 11y ago
  In terms of performance per cost, the cluster isn't on par with x86 system. Probably, not even close. Nonetheless, the network cost is real, and you get to have a first hand experience of how your algorithm performs.
- geerlingguy 11y ago
  A cluster of six Raspberry Pi 2s running Drupal (a PHP CMS) is about 20% slower than a single i7 laptop with an SSD for general usage, and can be _much_ slower if your usage requires lots of disk or network I/O. The 4-core ARMv7 chip in the Pi 2 is way faster than the chip in previous Pis, but it's not a speed demon by any means. And the network I/O (and all I/O) being tied down to a slowish USB 2.0 bus means you're limited to ~90 Mbps through LAN, or ~200 Mbps if you use a USB 3.0 network adapter. Even trying a fast SSD plugged into one of the USB ports (rather than a comparatively slow 'Extreme Pro' microSD card), you won't get a huge performance boost because of limited I/O bandwidth. See many, many more benchmarks on this wiki: https://github.com/geerlingguy/raspberry-pi-dramble/wiki https://github.com/geerlingguy/raspberry-pi-dramble/wiki P.S. I have a cluster of six Pi 2s running http://www.pidramble.com/ http://www.pidramble.com/, and I'm actually considering moving more of my hobby/very-low-traffic sites over to it, since it's a sunk cost of $400 whereas many of these small sites are currently on $5 or $10/month servers.
  - kiddico 11y ago
    What's this about using a USB 3 ethernet adapter? Does the RPI have usb 3 ports? If so I've been missing out...
    - geerlingguy 11y ago
      No, but you can only buy USB 3.0 adapters with gigabit ratings (all the USB 2.0 adapters max out at 10/100). See: http://www.midwesternmac.com/blogs/jeff-geerling/getting-gigabit-networking http://www.midwesternmac.com/blogs/jeff-geerling/getting-gig...
      - 11y ago
weatherlight 11y ago
In terms of performance per cost, the cluster can't compete in terms of raw speed of a x86 system. If its for educational purposes (looking into how algorithms work across distributed systems) why not just use a parallella? It costs $99.
- ajford 11y ago
  I haven't looked very closely at the Parallella, but I don't think it's a direct comparison to this cluster. Parallella contains multiple cores on a single board, which seem to share common resources. There are two different forms of parallel processing, one for multi-core on a single node (i.e. shared RAM/peripherals), and one for multi-node (i.e. networked servers), where it can be highly impractical to share data due to the interconnections (usually Ethernet/Infiniband). In most modern clusters, you have to use both approaches. RPi 2 is nice since you have multiple cores, and the cluster shown here allows you to play with the other side of the equation, multi-node parallelism. EDIT: I looked into the parallella a bit. Still can't tell how well it represents multi-node parallelism, but it claims to have "a fast on chip network within a distributed shared memory architecture". So theoretically, this can simulate the interconnection of a large scale cluster, but I have no real knowledge of this board.
  - chadzawistowski 11y ago
    The other problem with the parallela is its weird architecture. Its master processor is a dual-core ARM, but the 16x "epiphany" cores only have a C++ compiler. It would not be possible to program the epiphany cores with Erlang, for instance.
    - weatherlight 11y ago
      no, but you can send processes to each of the cores. https://www.parallella.org/2015/03/20/erlang-otp-and-the-parallella-board/ https://www.parallella.org/2015/03/20/erlang-otp-and-the-par...
- rebootthesystem 11y ago
pnathan 11y ago
I am slouching my way towards getting one of these sorts of things set up as a "DevOps" system - cluster management tooling and so forth.
- ris 11y ago
  Why? Simulate it in software. Save yourself a lot of time and shelf space. There is nothing additional this hardware teaches you other than plugging and unplugging ethernet cables.
  - rcarmo 11y ago
    Actually, there is. Thriftiness, for one, and better planning skills. Testing anything on ARM devices requires a fair amount of patience, skill, and tolerance for when things are different from, say, x64 Linux.
  - pnathan 11y ago
    Simulation is not reality. A better way is to develop a multi-continent cluster, which will more adequately stress things like race conditions and jitter. But that's some $60/mo to have enough hosted VM nodes worldwide, and RPis win on cheapness pretty quick there.
bjd2385 11y ago
I think this would be a cool project for someone who wants to fiddle with a desktop "super"-computer with multiple nodes. I've been interested in getting started or exploring with this kind of distributed computation, and I'm assuming that starting off I wouldn't be totally concerned with what it's capable of in comparison to a regular full-scale desktop but rather if I can get it to work. Beyond that, I would extend my sights to getting a couple desktops and trying it on a large system.
ajw7 11y ago
Would this be called a bakery?

78 points by stkim1 11y ago

Neat. Anyone else at HN did something like this? Mine is at https://github.com/rcarmo/raspi-cluster https://github.com/rcarmo/raspi-cluster (configs and some utilities included)

stkim1 11y ago
Matthew Saunders also designed a very nice cluster stack. https://learnaddict.com/2015/08/03/raspberry-pi-stack-a-platform-for-learning-about-iot/ https://learnaddict.com/2015/08/03/raspberry-pi-stack-a-plat...
geerlingguy 11y ago
I have an educational site running on my cluster of 6 Pi 2s in my basement: http://www.pidramble.com/ http://www.pidramble.com/ Code, tutorial, parts list, etc. here: https://github.com/geerlingguy/raspberry-pi-dramble https://github.com/geerlingguy/raspberry-pi-dramble
tracker1 11y ago
I always thought the cassandra multi datacenter cluster on rpis was interesting. http://www.datastax.com/dev/blog/32-node-raspberry-pi-cassandra-cluster http://www.datastax.com/dev/blog/32-node-raspberry-pi-cassan...

leoedin 11y ago

What sort of cost to performance do these give? Is it actually more cost effective to purchase and run 6 RPis compared to commodity x86 hardware? How about compared to the cost of computing your job on one of the numerous cloud based offerings? I've never really been faced with a compute job that takes long enough to be worth investigating parallelisation, so I'm fairly ignorant in that regard. This is certainly a neat project (and perhaps provides a nice talking point), but is it more than that?

stkim1 11y ago
In terms of performance per cost, the cluster isn't on par with x86 system. Probably, not even close. Nonetheless, the network cost is real, and you get to have a first hand experience of how your algorithm performs.
geerlingguy 11y ago
A cluster of six Raspberry Pi 2s running Drupal (a PHP CMS) is about 20% slower than a single i7 laptop with an SSD for general usage, and can be _much_ slower if your usage requires lots of disk or network I/O. The 4-core ARMv7 chip in the Pi 2 is way faster than the chip in previous Pis, but it's not a speed demon by any means. And the network I/O (and all I/O) being tied down to a slowish USB 2.0 bus means you're limited to ~90 Mbps through LAN, or ~200 Mbps if you use a USB 3.0 network adapter. Even trying a fast SSD plugged into one of the USB ports (rather than a comparatively slow 'Extreme Pro' microSD card), you won't get a huge performance boost because of limited I/O bandwidth. See many, many more benchmarks on this wiki: https://github.com/geerlingguy/raspberry-pi-dramble/wiki https://github.com/geerlingguy/raspberry-pi-dramble/wiki P.S. I have a cluster of six Pi 2s running http://www.pidramble.com/ http://www.pidramble.com/, and I'm actually considering moving more of my hobby/very-low-traffic sites over to it, since it's a sunk cost of $400 whereas many of these small sites are currently on $5 or $10/month servers.
- kiddico 11y ago
  What's this about using a USB 3 ethernet adapter? Does the RPI have usb 3 ports? If so I've been missing out...
  - geerlingguy 11y ago
    No, but you can only buy USB 3.0 adapters with gigabit ratings (all the USB 2.0 adapters max out at 10/100). See: http://www.midwesternmac.com/blogs/jeff-geerling/getting-gigabit-networking http://www.midwesternmac.com/blogs/jeff-geerling/getting-gig...
    - 11y ago

weatherlight 11y ago

In terms of performance per cost, the cluster can't compete in terms of raw speed of a x86 system. If its for educational purposes (looking into how algorithms work across distributed systems) why not just use a parallella? It costs $99.

ajford 11y ago
I haven't looked very closely at the Parallella, but I don't think it's a direct comparison to this cluster. Parallella contains multiple cores on a single board, which seem to share common resources. There are two different forms of parallel processing, one for multi-core on a single node (i.e. shared RAM/peripherals), and one for multi-node (i.e. networked servers), where it can be highly impractical to share data due to the interconnections (usually Ethernet/Infiniband). In most modern clusters, you have to use both approaches. RPi 2 is nice since you have multiple cores, and the cluster shown here allows you to play with the other side of the equation, multi-node parallelism. EDIT: I looked into the parallella a bit. Still can't tell how well it represents multi-node parallelism, but it claims to have "a fast on chip network within a distributed shared memory architecture". So theoretically, this can simulate the interconnection of a large scale cluster, but I have no real knowledge of this board.
- chadzawistowski 11y ago
  The other problem with the parallela is its weird architecture. Its master processor is a dual-core ARM, but the 16x "epiphany" cores only have a C++ compiler. It would not be possible to program the epiphany cores with Erlang, for instance.
  - weatherlight 11y ago
    no, but you can send processes to each of the cores. https://www.parallella.org/2015/03/20/erlang-otp-and-the-parallella-board/ https://www.parallella.org/2015/03/20/erlang-otp-and-the-par...
rebootthesystem 11y ago

pnathan 11y ago

I am slouching my way towards getting one of these sorts of things set up as a "DevOps" system - cluster management tooling and so forth.

ris 11y ago
Why? Simulate it in software. Save yourself a lot of time and shelf space. There is nothing additional this hardware teaches you other than plugging and unplugging ethernet cables.
- rcarmo 11y ago
  Actually, there is. Thriftiness, for one, and better planning skills. Testing anything on ARM devices requires a fair amount of patience, skill, and tolerance for when things are different from, say, x64 Linux.
- pnathan 11y ago
  Simulation is not reality. A better way is to develop a multi-continent cluster, which will more adequately stress things like race conditions and jitter. But that's some $60/mo to have enough hosted VM nodes worldwide, and RPis win on cheapness pretty quick there.

bjd2385 11y ago

I think this would be a cool project for someone who wants to fiddle with a desktop "super"-computer with multiple nodes. I've been interested in getting started or exploring with this kind of distributed computation, and I'm assuming that starting off I wouldn't be totally concerned with what it's capable of in comparison to a regular full-scale desktop but rather if I can get it to work. Beyond that, I would extend my sights to getting a couple desktops and trying it on a large system.

ajw7 11y ago

Would this be called a bakery?