7 ms·
Raspberry Pi 2 Cluster Case, Part 2
- Quequau 11y agoSometimes I get the feeling that I'm the only person who feels like these single board computers that use ARM SoCs designed for mobile devices have insufficient interconnect capability. The networking that these devices have always seems like an afterthought. For example a lot of these SoCs have the nic hanging off an internal USB port, which is a fairly convoluted way to go about things. I really wish there were single board RPi / ODROID like devices available that sported support for a really performant and efficient interconnect fabric... like RapidI/O or something similar.
- legulere 11y agoStorage also is also such an issue. You also usually only get sata via USB. SATA and/or PCIe would be really nice to have
- stkim1 11y agoIn fact, storage is a big issue. RPI tends to destroy SD card in a way that a cluster like mine isn't exactly safe unless you put high-grade SD card and very reliable power source. Those contribute somewhat higher operation cost which violate the purpose of these single board computer.
- icebraining 11y agoAre you storing the results of computation on the SD cards? Personally, I've mounted every directory that needs to be written to a tmpfs, USB disk or network mount as appropriate, and I've configured fstab to mount the root fs in readonly mode. You can still write by just doing "mount -o remount, rw /" beforehand.
- stkim1 11y agoYes I do. Since Apache Spark is chewing up memory like nothing, I cannot spare any portion of 1G mem for something else. This could lead to a shorter lifespan of SD cards though.
- paulmd 11y agoI've been really unsuccessful at getting a Pi to not destroy SD cards. Good quality cards, good quality power supplies, no swapping - it still eats them up. The best way to fix this is to put the actual OS on an external disk. You absolutely must bootstrap from an SD card, there's no way around that. But that can be entirely read-only and then you bounce to the external drive. Or maybe you could do something like a PXE boot instead - copy a system image down from a server and boot that. Personal opinion, this is the only way I would try using Pis on a long-term basis. The Jetson TK-1 board from NVIDIA includes a real SATA connector. Because it's GPGPU-capable it's also vastly more powerful than even a Pi2. It also has USB 3.0 so running swap on a memory stick would be much more performant.
- stkim1 11y agoPXE boot is something I thought about, but I don't know if it is possible on RPI. Do you have any link where I can just go read up about that?
- dharma1 11y agoJetson is nice, if a bit pricey. Looking forward to X1 boards from NVIDIA
- Quequau 11y agoI don't suppose you've seen any hints of release dates have you?
- sdenton4 11y agoI generally set mine up to use the sd just for boot, and have the actual file system on a usb thumb drive. Works great, and haven't had any sd card corruption issues since I started doing it.
- simfoo 11y agoHave a look at the Banana Pi or even the Banana Pro
- Quequau 11y agoI agree and SATA via USB is just as suboptimal as GigE via USB (2.0). I'm still sort of aggrieved that Solid State via DDR got caught up in lawsuits and is still commanding exploitative pricing. SATA for Solid State isn't exactly optimal either, though obviously not nearly as terrible as USB - SATA - SSD. SSDs with PCIe interfaces are so common now, it's shame not to see wider support for it. A little tidbit I saw in HP's "The Machine" promotional talks was the idea of "fabric attached memory". If ARM 64 & RapidI/O ever make it into widespread use, it would be great if there were various forms of memory with RapidI/O interfaces commonly available.
- slug 11y agoParallella board ( http://www.parallella.org/board/ http://www.parallella.org/board/ ) with Gbps FPGA Mezzanine connectors ( https://en.wikipedia.org/wiki/FPGA_Mezzanine_Card https://en.wikipedia.org/wiki/FPGA_Mezzanine_Card ). Not in the price range of RPi but much more powerful. You can also try the Beaglebone Black... Also, from the ODROID block diagram, the Ethernet PHY is connected directly to the CPU MAC, afaik the issue is the bad (cost?) choice of the Raspberry Pi...
- paulmd 11y agoI was going to say, Pi isn't really the optimal choice here to begin with. For the same wattage, the Parallela is two orders of magnitude more powerful, but it's a more unusual processor to compute on. The Jetson TK-1 is the other option - an NVIDIA dev board for the Tegra K-1 SoC. It's also not entirely similar to CPU programming, but CUDA GPGPU is a more well-trodden path for HPC. I don't know of any actual switched-fabric mezzanine cards available for the Parallela. If you have a link I'd be very interested. A similar approach would be to exploit a mini PCI-Express connector like the one on the Jetson. The Jetson also has a daughterboard connector available, but my understanding is that it's proprietary and I don't think NVIDIA offers a switched-fabric card.
- Quequau 11y agoMan, I was fully onboard with the Parallella back when it was on Kickstarter but I backed out towards the end once it became clear that they weren't going to be shipping the 64 core devices. To my knowledge there still is no way to connect them together except for the Nic that is integrated into the Zynq SoC. I own a couple of ODROID boards and while the network performance is better than what you'll see on the RPi, it's still nothing to write home about. And that's sorta my whole point... if you're working with large enough problem using these low power boards that you've got to offload compute work to multiple devices, the interconnect fabric has got to be much better than what is commonly available right now.
- detaro 11y agoWhat would be problems where multiple Raspi-like systems are be better than a single more powerful system?
- Vexs 11y agoThere was some server company I can't remember the name of right now (bare metal or something?)that offered roughly rasppi sized servers for rent from them- they weren't offering it yet, but they said you should be able to buy them within a year or so from them. Also, don't forget that USB is just another serial protocol, if it wasn't USB it would just be some other serial protocol. It may not be the best option, but with stuff like the rasspi I imagine price is a bigger concern. Edit: found them https://www.scaleway.com/ https://www.scaleway.com/
- INTPenis 11y agoSwedish FSdata does that. https://fsdata.se/server/raspberry-pi-colocation/ https://fsdata.se/server/raspberry-pi-colocation/ Edit: They did that, apparently the service is "on pause".
- Alupis 11y agoThis is the service mentioned on the Raspberry Pi Foundation's website: http://raspberrycolocation.com/ http://raspberrycolocation.com/ For 36 Euro's (39.54 USD) per year, they will host your RPi with a 100Mbit pipe and 500GB of traffic.
- kickscondor 11y agoThe ESP8266 is generally pretty good this way - where many people use mqtt (over wifi). However, you're still right - would be nice to have something much lighter done well. The next Espressif chip will be SoC with onboard bluetooth LE.
- ericmo 11y agoI see @stkim1 is also the author of the blog. Performance-wise, how does this RPi cluster compare to a more standard platform? Did you make it only as a development platform or do you intend to have production code running in it?
- stkim1 11y agoI've never imagined the RPI cluster to be taken seriously for production purpose. It only serves two purposes; education and developmental staging. For those two, you have a real hardware cluster that serves you superbly showing every nature you can experience with big-ass datacenter cluster.
- geerlingguy 11y agoAs someone else who's also built a cluster of Pi 2s, I would agree that the primary purpose is education and fun/tinkering, not performance. That being said, for light web hosting purposes, a cluster of six Pi 2s performs roughly 70% as well as a similar cluster of 6 VMs on Digital Ocean. See more on this project's wiki, in the performance/benchmarking section: https://github.com/geerlingguy/raspberry-pi-dramble https://github.com/geerlingguy/raspberry-pi-dramble
- kcbanner 11y agoWhat is the use case for clustering Raspberry PIs? Why not build a more powerful computer within that same footprint?
- bodyfour 11y agoAs a productive tool, I can't think of a use at all. For learning distributed computing, I think has some advantages over just running all of the nodes on a single powerful computer. It means you can't hide from doing things scalably -- i.e. network costs between your processes are real; if your load isn't well-distributed across your nodes the OS scheduler can't save you; etc.
- sdenton4 11y agoThere was a great article a while back about a guy running a warehouse-sized bitcoin mining operation, using RPi's to coordinate the many thousands of custom chips doing the mining. The Pi was the cheapest full linux a system available at the time, and performance wasn't really an issue.
- aesthetics1 11y agoIt's uhh.. cool. Haha. I can't think of a good reason either, but we're all hobbyists here, right? It looks like a fun weekend project.
- stkim1 11y agoIt only serves two fronts; education and development. Education-wise, you need something dirt-cheap to play with. For developer, you need some intermediate stage to deploy your work to test before going after a big cluster.
- rco8786 11y agoFun and learning. Is that so bad?
- rwmj 11y agoThose little aluminium pillars are brilliant. You can see my (real PC) cluster here that I built using bare motherboards and pillars: https://rwmj.wordpress.com/2014/04/28/caseless-virtualization-cluster-part-5/#content https://rwmj.wordpress.com/2014/04/28/caseless-virtualizatio...
- stkim1 11y agoYou are da man. This is really cool! I can only imagine how much you've been through!
- pedjak 11y agoMore Raspberry clusters http://www.datastax.com/dev/blog/32-node-raspberry-pi-cassandra-cluster http://www.datastax.com/dev/blog/32-node-raspberry-pi-cassan...
- stkim1 11y agoDavid Guill is another deadly serious RPI cluster builder. I don't know what he's running, but I can only admire what he had done. http://likemagicappears.com/projects/raspberry-pi-cluster/ http://likemagicappears.com/projects/raspberry-pi-cluster/
- seynb 11y agoHere's a Beowulf Cluster with 2^5 Raspberry Pi's running Arch Linux (for ARM) http://coen.boisestate.edu/ece/files/2013/05/Creating.a.Raspberry.Pi-Based.Beowulf.Cluster_v2.pdf http://coen.boisestate.edu/ece/files/2013/05/Creating.a.Rasp...
- ansible 11y agoI've been tempted several times to build a small cluster of low-power boards for a compute cluster. Not that I have much direct use for that at the moment. I recently got a new laptop with 16GB of RAM, so just running 8 VMs with 1GB of RAM each still leaves plenty left for development. With the VMs, I can just run regular 64-bit Linux, so that makes software and setup a non-issue. I've got so many other projects on the back-burner I shouldn't even be reading about this sort of thing. But it does look fun!
- icpmacdo 11y agoWith 1GB of ram there are no problems running 64bit linux? I thought you needed 4+.
- rco8786 11y agoStartech.com sells 15cm micro USB cables, they work great with my Pis.
- stkim1 11y agoThank you very much! I'll look into that!
- systemz 11y agoRaspberry Pi cluster? Average PC with virtualization would be better and probably cheaper choice.