13 ms·
Let's Build a 28-Core Raspberry Pi Cluster
- WaxProlix 6y agoSeems down, here's a google cache of it https://webcache.googleusercontent.com/search?q=cache:lIePhGOXF9AJ:https://ikarus.sg/how-i-built-kraken/+&cd=1&hl=en&ct=clnk&gl=us https://webcache.googleusercontent.com/search?q=cache:lIePhG...
- kanobo 6y agoThanks, I wonder if the site was hosted on the Pi Cluster?
- sedatk 6y agoIt's back up now.
- ikaruswill 6y agoYes the site is indeed hosted on the cluster.
- justinclift 6y agoSeems to be down again now. Guess it'll be spotty until traffic settles down. Maybe the traffic will provide an interesting exercise in optimisation? eg move from wordpress to static content or something, maybe add some CDN pieces, then whatever else turns out to be needed?
- daneel_w 6y agoNeat setup. An alternative would perhaps be a Pine64 Clusterboard with 7 SOPINE modules: https://store.pine64.org/product/clusterboard-with-7-sopine-compute-module-slots/ https://store.pine64.org/product/clusterboard-with-7-sopine-...
- justinclift 6y agoBe careful of that. Those Clusterboards seem like a nifty idea, but looking through their forum they seem to have a tonne of problems:. :( http://forum.pine64.org/forumdisplay.php?fid=91 http://forum.pine64.org/forumdisplay.php?fid=91 That being said, I haven't personally used one. (But might get around to it one day out of curiosity.)
- Legogris 6y agoI've been a bit curious about this and other multi-board CoM carrier boards like Turing Pi[0]. Have you used it and what's your experience? Like the author mentions, pre-4 Raspberry Pis gives sad throughput due to the shared bus; Rpi 4 would be much more suitable. There's also Odroid C4, Rockpi 4, Nanopi M4V2. [0]: https://www.jeffgeerling.com/blog/2020/raspberry-pi-cluster-episode-6-turing-pi-review https://www.jeffgeerling.com/blog/2020/raspberry-pi-cluster-...
- daneel_w 6y agoNo practical experience with any of these "cluster boards" yet, I'm afraid.
- pengaru 6y agoI cannot recommend doing business with pine64, their customer service is nonexistent. If you cancel your order, or need to get ahold of someone @ sales@pine64.org for any exception, in my experience you may as well shout at the nearest wall. - Someone out $165 spent on a cancelled pinephone order, no phone and no refund, complete radio silence.
- mdoms 6y agoAm I understanding that he is storing web data on the SD cards of the Pis? Is that normal? SD storage is notoriously unreliable, especially in write-heavy applications.
- LeoPanthera 6y ago"Endurance" microSD cards, normally sold for use in dashcams, are much more reliable in the Pi. However I switched to netbooting my Pis, no local storage on the Pi at all. This does require you to have a netboot server already, of course.
- ikaruswill 6y agoIndeed, though at the point of building this cluster, "Endurance" microSD cards were not in the market yet, hence I didn't go for it.
- megous 6y agoBrand SD cards are quite reliable. If you configure your SW to avoid sources of needless constant write churn (eg. like postgresql stats collector), and use good power supply, they will last years.
- ikaruswill 6y agoYou're absolutely right. I've running these SD cards for about 2 years now. I use log2ram to stream logs to a ramdisk and then sync that at regular intervals to disk, to extend the life of the SD cards.
- redis_mlc 6y agoIf you use Postgres as a single master on any storage device, you will also want to disable the slave writes, which seem to be enabled by default. If your device crashes with those enabled, Postgres may not start up again without manual intervention. Also, use the most current version since Postgres has had a brutal number of CVEs in the past few years which are in metasploit.
- 6y ago
- magicalhippo 6y agoSo I was toying with the idea of something similar. I'm a programmer and not a sysadmin, but I've dabbled with Docker and had a specific use-case. But so far I've not come further than this: how to deal with the host OS? Like, I don't want to log into N computers and manually run "sudo apt update" etc. I want them to run the same OS but with slightly different configurations. Network booting via PXE seems to be the best alternative, but not sure how to prepare the images for distribution. Anyone got any pointers on how this is best done? edit: I should mention I just found http://cobbler.github.io/ http://cobbler.github.io/ but haven't had time to investigate it. A viable choice?
- chabad360 6y agoIt seems to be primarily for initially provisioning servers, and doesn't really seem to be targeted at their upkeep. There is Rundeck but it's far too resource heavy to be useful for rPi. Shameless plug: After a lot of research in this, I came to the conclusion that I'd just have to make it myself, hence Covey (https://github.com/chabad360/covey https://github.com/chabad360/covey) a project that's written in Go, and currently in an early alpha stage of development, but it works. More details are in the readme, feel free to email me (its on my GH profile) if you want more details than the readme provides. EDIT: seems like your looking for something that's primarily for provisioning anyways. Whoops, I seem to have missed the middle paragraph... EDIT 2: PXE booting may not be the best option for a relatively low ram machine like the Pi (unless you have the storage somewhere else, but that's its own can of worms), even the minimal image required to run k3s would be at least 400mb (I think I'm underestimating) there are ways to deal with that (using a compressed squashfs). But none will beat running directly on the SD card. I'd suggest following the sibling comment, and using overlayfs on top of the main fs to ensure that the image doesn't get tarnished.
- jvilalta 6y agoAnsible works great on PIs
- dehrmann 6y agoYeah, if you're concern is managing a lot of machines, Salt, Puppet, Ansible, or Chef are the right tool; this isn't really a Docker thing. The PXE boot idea is interesting, and you can even do that diskless.
- abraae 6y agoI have to admire someone who spends their first paycheck on a pi cluster: > The total damage my wallet sustained from Kraken was S$626.63 or in $449.73 USD, which was virtually all I had left for the month after subtracting expenses and education loan repayments.
- panpanna 6y agoAll power to him. Nobody got successful by playing it safe.
- deleted 6y ago[deleted]
- quickthrower2 6y agoThis is cash for joy!
- golergka 6y agoI think one could easily calculate ROI in incoming recruiter messages after reaching first page of HN with personal blog, if nothing else.
- wy35 6y agoThis is the part of the article that stood out to me most. I definitely don't have the confidence to spend my entire paycheck on a just-for-fun project, especially considering how many projects I planned on starting but never did. Kudos on the author for being this confident and actually going through with everything.
- kristianp 6y ago"I started noticing the performance bottlenecks started when I tried to run Wordpress on it, where a single page load of a newly installed Wordpress blog takes about 10 seconds!" 10 seconds is probably not bad for an uncached wordpress page.
- chillfox 6y ago10 seconds is terrible, a WordPress page should render in under 2 seconds.
- ikaruswill 6y agoHello, original author here. I saw a bunch of traffic from Hacker News and decided to drop by. Thanks for sharing it here! I'd be happy to answer any questions.
- gchamonlive 6y agoHey there. Thanks for the article! I wanted to ask you if it is worth it implementing something similar in the cloud to learn kubernetes. The hardware is prohibitively expensive on Brazil, so maybe a cluster of t2.micro on AWS running for the total amount of 750 hours could serve the learning purpose for free
- ikaruswill 6y agoThanks for reading! It's probably worth implementing it in the cloud using the free credits if hardware is prohibitively expensive, at least for the purposes of learning. If you do intend to self-host some services in the longer term, the cost of running it on the cloud will catch up with the hardware cost in no time at all.
- nl 6y agoIf you have a PC, then just run it on VM(s) on that. That's what AWS is giving you anyway. Here's an example: https://kubernetes.io/blog/2019/03/28/running-kubernetes-locally-on-linux-with-minikube-now-with-kubernetes-1.14-support/ https://kubernetes.io/blog/2019/03/28/running-kubernetes-loc...
- me551ah 6y agoOr you could also checkout lowendbox.com (not affiliated with them). They provide really cheap vps servers.
- tinco 6y agoI don't know how it is in Brazil but in Europe and the US offices frequently go out of business that have dozens if not hundreds tiny desktop computers in their bankruptcy auctions. They've got i5 cpu's 4-8gb of ram and 128gb SSD's and go for (much) less than a hundred bucks. Stack 3 of them and you have a kubernetes cluster that you could carry in one hand and I think outperforms the 7 raspberry Pi 3's. Servethehome has a series of articles about the machines, though they're focusing on the slightly higher end ones.
- sxp 6y agoA slightly cheaper option would be to get a bunch of Pi Zero W devices. They're $10 + SDcards. They would rely on WiFi which might be acceptable if the use case allows having a few dozen radios chattering next to each other. If not, then a $5 Pi Zero + $5-10 ethernet adapter might work. Since the goal isn't to achieve maximum performance, the slow CPU & RAM (comparable to a 20th century desktop) might not be an issue. And the lower price would allow more devices so you could test more complex configurations and server management options.
- numpad0 6y agoZero/Zero W at $5/$10 are promotional price, for first unit per person only. For second unit and after you have to order case kits or other high(positive) margin options.
- ikaruswill 6y agoI actually have had a cluster of Raspberry Pi 1Bs before this current cluster. Though the specs are very much similar to a Zero W, the performance wasn't great. I agree that it would cost much less, but I would also get much less utility out of it. At this point, the cluster is doing quite a lot, even the site itself is hosted on it. My point here is that having room for future utility at not much additional cost, would be best.
- rcarmo 6y agoZeros are borderline unusable for actual compute due to having a single core and only half a gig or RAM. They make for great beefy microcontrollers and single-purpose boards, but are not the kind of thing where you want to run various network services. (I have a cluster running Pi 2s, which have 4 cores apiece, but are still too short on RAM)
- deleted 6y ago[deleted]
- tobmlt 6y agoNice! I tried this out a while back as well, though I did nothing nearly as snazzy. Southampton had instructions here, I believe it was: http://www.southampton.ac.uk/~sjc/raspberrypi/ http://www.southampton.ac.uk/~sjc/raspberrypi/ (Maybe still useful if you want to do without docker?)
- throwawaynothx 6y agoPersonally I would not use pi's for this. Massive waste of money, the experience is worth it but other than that..
- ikaruswill 6y agoWhat would you use then?
- aaronax 6y agoMaybe used micro desktops? https://www.servethehome.com/introducing-project-tinyminimicro-home-lab-revolution/ https://www.servethehome.com/introducing-project-tinyminimic...
- rumanator 6y ago600$ buys you a considerable amount of time in a cloud vendor. For example you can have a 1vCPU instance in Hetzner for 3€/month, which means 600$ can buy you about a year's worth of 20 instances running 24/7. If you don't need 99.9999 availability of your cluster and are only interested in firing up the cluster from time to time then those same 600$ can buy you a few years of cluster time
- wahern 6y agoThe 3€/month is a single thread (not even a core), and it's not even dedicated. The lowest dedicated instance is 2 vCPU (1 core) for 23€/month with 8GB or RAM. That's 276€/year. 8 of those instances is equivalent to an 8 core (16 thread) chip with 64GB of memory. Using, e.g., Supermicro AMD EPYC 3251 + parts, you could build an entire server for less than 1/2 that price. You could have twice the compute for the same price the first year. Servers easily last 3 years minimum, so Hetzner has a 6x markup at the low end. Admittedly, building your own server doesn't include transit, but transit is cheap. Assuming you have a fast home connection (a big if for many people), the hard part is access to static IP addresses. There are dedicated services that lease IPs over a VPN tunnel, or you can setup your own VPN into, e.g., Hetzner or Vultr using a handful of the cheapest instances. (Not sure about Hetzner, but Vultr let's you lease and attach 2 additional IPs to the same instance for $2/month/IPv4 address.) Now, the above setup is probably far too much trouble for most people, and it has many downsides[1]. But the point is that if we're just talking about CPU time, not even Hetzner or some other cheap VPS provider can come close to the cost of running your own equipment. [1] Power and cooling are two downsides, but the 3251 has a TDP of 50W. Granted, it doesn't have the highest clock speeds, but there's a reason cloud providers don't normally advertise Ghz. Anyhow, rather than build 2x 3251's you could build a single 16 core rig using a higher-clocked and newer EPYC or Xeon for about the same money (definitely at or under 2200€ all-in).
- deleted 6y ago[deleted]
- ArchD 6y agoWhat's the actual use case for rpi clusters like this? If the goal is to learn docker or k8s using a realistic setup involving multiple machines, can't it be achieved by using multiple VMs on one physical desktop instead of using the cluster? The 7-node cluster has only 7GB RAM in total, which is not too much to ask on a desktop. I don't know what people do with rpi clusters and I'm honestly puzzled why people bother building them. If you get a beefy expensive server, I can understand -- you care about performance. If you get a single rpi to serve files & media in a home LAN it makes sense too -- low footprint, low power and low cost. But I don't know what a rpi cluster excels at. Is it just a toy? Or a learning tool to make computer science & networking concepts concrete and tangible? Or just to get a computer system on a shoestring budget, much cheaper than even a single desktop or laptop? BTW, computer hardware in Singapore is overpriced. $140 for a switch in Singapore vs $73 for a similarly-capable switch on Amazon. https://www.amazon.com/TP-Link-Ethernet-Unmanaged-Rackmount-TL-SG1016D/dp/B004ELA0SS https://www.amazon.com/TP-Link-Ethernet-Unmanaged-Rackmount-...
- tra3 6y agoThe use case is tinkering. I think it’s pretty sweet to be able to build a super low cost cluster like this. Play with docker, k8s etc.
- fock 6y agoif you only want to play with software, VMs are a lot easier though (and you got to learn the basics of the VMs)
- justinclift 6y agoGenerally, the point of a cluster like this is to also gain insight into things that involve physical components as well. It's a different type of desired end result / learning than a purely software approach. :)
- grep_it 6y agoI think Jeff Geerling answers this question succinctly in this video https://youtu.be/kgVz4-SEhbE?t=307 https://youtu.be/kgVz4-SEhbE?t=307 (starting at the 5min mark). TL;DW - It's fun and you gain the added skills of managing bare metal with physical IO limitations.
- daniel-s 6y agoI think that there are a lot of use cases of Pis for which online cloud providers might be a better option. Where does it become a better idea to just rent a few instances in Azure/EC2/Google Cloud?
- ulucs 6y agoReminds me of moot "revealing" the "datacenter" that hosted 4chan: https://imgur.com/sBLRuZY https://imgur.com/sBLRuZY I'm putting the air-quotes because I still sincerely hope that it was a joke
- saagarjha 6y agohttps://news.ycombinator.com/item?id=22767439 https://news.ycombinator.com/item?id=22767439
- swebs 6y agoHeh, it can get worse. >Wiichan it's hosted on a real Wii and it's served throught a shitty Mexican ISP. https://no.wiichan.uwu.mx/ https://no.wiichan.uwu.mx/
- swebs 6y agoAnd photos of the lovely server https://no.wiichan.uwu.mx/meta/res/51.html https://no.wiichan.uwu.mx/meta/res/51.html
- alias_neo 6y agoHow did that load so fast??
- dylz 6y agoThe host backing it is a ramnode vps in NL?
- alias_neo 6y agoOh, my bad, I thought it was hosted on the Wii.
- olabyne 6y agoBecause that seems to be the mirror. The mexican url is https://wiichan.uwu.mx/meta/res/51.html https://wiichan.uwu.mx/meta/res/51.html (proof : https://iplookup.flagfox.net/?ip=201.128.144.213&host=wiichan.uwu.mx https://iplookup.flagfox.net/?ip=201.128.144.213&host=wiicha...) It's really cool tho
- moonbug 6y agoThere's a new carrier board with integrated switching for Pi clustering: https://turingpi.com/ https://turingpi.com/
- fock 6y ago"This build is for you if you:" ahem, for 600 dollars I get a fine computer which can host all the kubernets and docker I ever want in a nice formfactor... I mean, it's cool and all, but effectively utterly useless except for the blinking lights.
- dumb1224 6y agoVery cool work! I wonder the same, for a 'cluster' I'm expecting performance and large memory for example for scientific computing. Other than that I couldn't think of any other use cases. If anyone could enlighten me? Cheers
- noir_lord 6y agoThey built a multi-thousand pi cluster to test the SMP and configuration stuff before running it on a super computer. Virtualising thousands of CPUs on a single machine is still not trivial. Never mind simulating real world network issues. https://www.bitscope.com/blog/FM/?p=GF19A https://www.bitscope.com/blog/FM/?p=GF19A
- fock 6y agoyes, if you want to optimize a worst-case MPI-cluster, then a Pi (4) might be optimal for you (because sadly, 4 measly ARM cores with 100MBit/s is a some magnitudes removed from 100 cores and 100GBit/s Infiniband). But then you can also use a stack of old desktops, which is cheaper and you can just throw in a standard image and everything (including CUDA and MKL) can work.
- noir_lord 6y ago> But then you can also use a stack of old desktops, which is cheaper If electricity and cooling is free.
- mysterydip 6y agoI made a cluster of 9 pi's for a work project, not for the computing power but for the segmentation. I had 8 pieces of equipment that needed to be monitored while undergoing various physical tests. A simple app was developed that would run on each piece being tested and communicate over the ethernet port each millisecond back to a pi, which then forwarded the on/off status to a central pc connected to the 9th pi acting as a wifi access point. This computer served as the monitor/display and logger of the system so we could see at a glance if everything was green and reporting or not. It was important that each piece be physically isolated so that we knew the whole path was operational and nothing was short circuiting via something else's network connection. You could have accomplished something similar with VLANs and some routes, but this allowed for easier troubleshooting and to explain to the powers that be how things were guaranteed isolated because they could see it. Plus it looked cool.
- mvanbaak 6y agoPeople wanting to torture themselves with k8s at home ... I still dont understand why.
- sdgjhfdlkgjhdf 6y agoI actually have a k3s cluster running my blog/a few services. It just works, really, and so far it has been zero maintenance, zero downtime, apart from the large upfront investment (one weekend). Most importantly it was a great learning experience.
- aaronax 6y agoImagine that the jobs you are interested in want to see some k8s experience. Either you teach it to yourself at home or you justify implementing it at your current job (resume driven development basically).
- mvanbaak 6y agoI have played around and learned the basics of docker etc without never ever wanting to run my home stuff on it. Learning and experimenting is not the same as using it as the primary tool to run your 'production' services at home.
- adsjhdashkj 6y agoI'd want to - or rather, the idea of managing a series of applications in a distributed fashion is interesting to me. HOWEVER, i like to write distributed software that i want "uninformed" users to run. Aka not run on nice, automated k8 deployments - but on dirty, random computers. My current idea is to run this software on all my personal PCs, but also a couple RaspPis over wifi in random locations. Slow, poor connectivity machines sounds like a decent and predictably bad user environment.
- bullen 6y agoIf you want to get more performance out of a cluster like this you can look at my distributed HTTP app. server + JSON database solution: https://github.com/tinspin/rupy https://github.com/tinspin/rupy It's actually usable for live services on RPi 2 and on RPi 4 it beats Intel on float ops./joules at 100% utilization. And if you use multicast UDP you can make these behave cooperatively to scale horizontally beyond what's possible with a vertical solution at a fraction of the purchase/maintenance cost. I think we are peaking at 2 Gflops/watt for CPU, so if you need silence (passive cooling) and small size this is as good as it will get, probably ever: http://talk.binarytask.com/task?id=8015986770003767235 http://talk.binarytask.com/task?id=8015986770003767235 I'm building a RPi 4 cluster with these heatsinks.
- Merocle 6y agoHello! I did it like this: https://www.instagram.com/p/CC76bkADL-c/ https://www.instagram.com/p/CC76bkADL-c/ It costed about 3k euro for me. And this one special for k8s: https://www.instagram.com/p/CAqtrRBpNRH/ https://www.instagram.com/p/CAqtrRBpNRH/
- rdc12 6y agoIs the design files available for the rack mounted version (in particular the clip/holder parts). Been quite keen to build something similar (albeit less nodes) for a little while. I find the homebrew blade style cluster amusing for some reason.
- john_alan 6y agoUntil Raspberry Pi has native hardware crypto instructions (AES) it’s useless for real work.
- qubex 6y agoHow timely... just today I received my CloverPi which I backed on Kickstarter (no other affiliation), a “back panel to enable 4 RaspberryPis to be clustered”. Finally I can earn my “My other computer is a Beowulf Cluster” bumper sticker from what feels like aeons ago (courtesy of the now-defunct ThinkGeek.com).
- frandroid 6y ago> now-defunct ThinkGeek.com Noooo!! They had the best toys. Unfortunately their shipping to Canada made everything double the price for me...
- cptskippy 6y ago> The maximum number of ports on a consumer-grade network switch was 8, enough for 7 nodes and a cable to the router. Are there laws in Singapore that delineate between consumer and business grade networking equipment based on port numbers or something? Here in Yankey Land you can pickup a consumer grade 16-port unmanaged gigabit switch for $50 at the local Wally World. The going rate on switch ports is $3-4 per port.
- rtkwe 6y agoIt looks like it was more a question of price in their market. In the post he mentions a 16 port switch but it's 10x the price of an 8 port. I bet it's a question of there being fewer people in the Singapore market looking for an >8 port switch on the low end of the market.
- cptskippy 6y agoThe switches he references in his post are from two entirely different product verticals. One is from the unmanaged vertical, the other is from the managed PoE vertical. Not only is he ignoring that a 16 port switch exists in the unmanaged vertical, he's ignored an entire vertical of managed switches without PoE. TP-Link makes a $40 unmanaged 16-port gigabit switch and a $80 managed 16-port gigabit switch. He's created a False dilemma/dichotomy by pretending that other options don't exist. This is like saying "I'd like a Toyota automobile larger than the Yaris but I can't afford the Lexus LS".
- rtkwe 6y agoIt's entirely possible that those options weren't readily available in the local store he's using. I'm not familiar with Singapore in particular but a lot of Asian countries don't have the huge electronics box stores that are common in the US and Europe. I've seen quite a few videos where the best electronic source the person had was what would be a small sized store in a US mall (20 feet wide, one isle with a counter running down one side for example). Online shopping does exist but shipping times and cost can also be pretty large.
- deleted 6y ago[deleted]
- alibarber 6y agoI wonder if there is something to be said for not using Ethernet in something like this? Perhaps it is possible to make a kind of IP over GPIO interface, or attach them all directly to a USB bus? It just seems to me like RasPi -> USB -> Ethernet (for less than 1m) -> USB -> RasPi has a layer or two more than necessary (and I believe the on-board ethernets on RasPi are USB interfaces to the chip too). Could be an interesting project...
- Accujack 6y agoIt would be educational. I'd say the most interesting way to approach this at the moment would be to use RasPi 4s and link them via the PCI express bus. None of this stuff is more than educational, though. I admire this build because it's almost a rite of passage for nerds to build a cluster, but when you're all done and the cool wears off you've basically got a pile of obsolete hardware for which you need to find a home.
- klickitat 6y agoWow, its wonderful.