8 ms·
I thought the conclusion should have been obvious: A cluster of Raspberry Pi units is an expensive nerd indulgence for fun, not an actual pathway to high perfor
by Aurornis 1y ago
I thought the conclusion should have been obvious: A cluster of Raspberry Pi units is an expensive nerd indulgence for fun, not an actual pathway to high performance compute. I don’t know if anyone building a Pi cluster actually goes into it thinking it’s going to be a cost effective endeavor, do they? Maybe this is just YouTube-style headline writing spilling over to the blog for the clicks.
If your goal is to play with or learn on a cluster of Linux machines, the cost effective way to do it is to buy a desktop consumer CPU, install a hypervisor, and create a lot of VMs. It’s not as satisfying as plugging cables into different Raspberry Pi units and connecting them all together if that’s your thing, but once you’re in the terminal the desktop CPU, RAM, and flexibility of the system will be appreciated.
- wccrawford 1y agoGeerling's titles have been increasingly click-bait for a while now. It's pretty sad, because I like his content, but hate the click-bait BS.
- jonathanlydall 1y agoIf it makes an appreciable difference to how much money he makes on YouTube then I can’t begrudge him for doing it. Don’t hate the player, hate the game.
- geerlingguy 1y agoJust to add context — I've been experimenting on my 2nd channel (Level 2 Jeff) with titles that are straight/barebones exactly describing the content of the video, vs a slight bit of clickbait (never untrue, but certainly more intriguing and not describing the exact topic of the video). The ones that are dead straight with no clickbait are 10/10 (the worst performers), and usually by a massive margin. Even with the same thumbnail. The sad fact is, if you want your work seen on YouTube, you can't just say "I built a 10 node Raspberry Pi blade cluster and ran HPL and LLMs on it". Some people are fine with a limited audience. And that's fine too! I don't have to write on my blog at all—I earn negative income from that, since I pay for hosting and a domain, but I hope some people enjoy the content in text form like I do.
- mrguyorama 1y agoBlame Youtube. They are the ones that run a purposely zero sum and adversarial system for directing attention at your videos. If he doesn't have a high enough click rate on his videos, Youtube will literally stop showing them to people, even subscribers. Youtube demonstrably wants clickbait titles and thumbnails. They built tooling to automatically A/B test titles and thumbnails for you. Youtube could fix this and stop it if they want, but that might lose them 1% of business so they never will. They love that you blame creators for this market dynamic instead of the people who literally create the market dynamic.
- bunderbunder 1y agoThe cost effective way to do it is in the cloud. Because there's a very good chance you'll learn everything you intended to learn and then get bored with it long before your cloud compute bill reaches the price of a desktop with even fairly modest specs for this purpose.
- aprdm 1y agoThat really depends on what you want to learn and how deep. If you're automating things before the hypervisor comes online or there's an OS running (e.g: working on datacenter automation, bare metal as a service) you will have many gaps
- leoc 1y agoIf you want to run something like GNS3 network simulation on a hosting service's hardware you'll either have to deal with hiring a bare-metal server or deal with nested virtualisation on other people's VM setups. Network simulation absolutely drinks RAM, too, so just filling an old Xeon with RAM starts to look very attractive in comparison to cloud providers who treat it an expensive upsell.
- Almondsetat 1y agoI can get a Xeon E5-2690V4 with 28 threads and 64GB of RAM for about $150. If you need cores and memory to make a lot of VMs you can do it extremely cheaply
- nine_k 1y agoIt will probably consume $150 worth of electricity in less than a month, even sitting idle :-\
- blobbers 1y agoThe internet says 100W idle, so maybe more like $40-50 electricity, depending on where you live could be cheaper could be more expensive. Makes me wonder if I should unplug more stuff when on vacation.
- glitchc 1y agoI did some calculations on this. Procuring a Mac Studio with the latest Mx Ultra processor and maxing out the memory seems to be the most cost effective way to break into 100b+ parameter model space.
- Palomides 1y agoeven a single new mac mini will beat this cluster on any metric, including cost
- randomgermanguy 1y agoDepends on how heavy one wants to go with the quants (for Q6-Q4 the AMD Ryzen AI MAX chips seem better/cheaper way to get started). Also the Mac Studio is a bit hampered by its low compute-power, meaning you really can't use a 100b+ dense model, only MoE feasibly without getting multi minute prompt-processing times (assuming 500+ tokens etc.)
- GeekyBear 1y agoGiven the RAM limitations of the first gen Ryzen AI MAX, you have no choice but to go heavy on the quantization of the larger LLMs on that hardware.
- mercutio2 1y agoHuh? My maxed out Mac Studio gets 60-100 tokens per second on 120B models, with latency on the order of 2 seconds. It was expensive, but slow it is not for small queries. Now, if I want to bump the context window to something huge, it does take 10-20 seconds to respond for agent tasks, but it’s only 2-3x slower than paid cloud models, in my experience. Still a little annoying, and the models aren’t as good, but the gap isn’t nearly as big as you imply, at least for me.
- zargon 1y agoGPT OSS 120B only has 5B active parameters. GP specifically said dense models, not MoE.
- deleted 1y ago
- moduspol 1y agoAlso cost effective is to buy used rack mount servers from Amazon. They may be out of warranty but you get a lot more horsepower for your buck, and now your VMs don’t have to be small.
- Aurornis 1y agoPutting a retired datacenter rack mount server in your house is a great way to learn how unbearably loud a real rack mount datacenter server is.
- tempest_ 1y agoahah and pricey power wise. Currently the cloud providers are dumping second gen xeon scalables and those things are pigs when it comes to power use. Sound wise its like someone running a hair dryer at full speed all the time and it can be louder under load.
- Tsiklon 1y agoTo quote @swiftonsecurity - https://x.com/swiftonsecurity/status/1650223598903382016 https://x.com/swiftonsecurity/status/1650223598903382016 ; > DO NOT TAKE HOME THE FREE 1U SERVER YOU DO NOT WANT THAT ANYWHERE A CLOSET DOOR WILL NOT STOP ITS BANSHEE WAIL TO THE DARK LORD AN UNHOLY CONDUIT TO THE DEPTHS OF INSOMNIA BINDING DARKNESS TO EVEN THE DAY
- buildbot 1y agoThis 1000%; and some 1us are extra 666. I had a sparc t2000 at one point, it was so much louder than a 1u Supermicro. Or whatever was in Microsoft HW labs, those you could hear from multiple hallways over… There were non optional earplugs at the doors.
- _boffin_ 1y agoNot true. Have one running in the closet and never hear it.
- ComputerGuru 1y ago
- vlovich123 1y agoI’d say it’s inconclusive. For traditional compute it wins on power and cost (it’ll always lose on space). The inference is noted to not be able to use the GPU due to llama.cpp’s vulkan backend AND that clustering software in llama.cpp is bad. I’d say it’s probably still going to be worse for AI but it’s inconclusive where it could be due to the software immaturity (ie not worth it today but could be with better software)
- tracker1 1y agoBut will there be a CM6 while you're waiting for the software to improve?
- llm_nerd 1y agoIf you assume that the author did this to have content for his blog and his YouTube channel, it makes much more sense. Going back to the well with a "I regret" entry allows for extra exploiting of a pretty dubious venture. YouTube is absolute jam packed full of people pitching home "lab" sort of AI buildouts that are just catastrophically ill-advised, but it yields content that seems to be a big draw. For instance Alex Ziskind's content. I worry that people are actually dumping thousands to have poor performing ultra-quantized local AIs that will have zero comparative value.
- philipwhiuk 1y agoI doubt anyone does this seriously.
- nerdsniper 1y agoI sure hope no one does this seriously expecting to save some money. I enjoy the videos on "catastrophically ill-advised" build-outs. My primary curiosities that get satisfied by them are: 1) How much worse / more expensive are they than a conventional solution? 2) What kinds of weird esoteric issues pop up and how do they get solved (e.g. the resizable BAR issue for GPU's attached to RPi's PCIe slot)
- ww520 1y agoNow. Imagine a Beawulf of these...
- TZubiri 1y agoFun fact, a raspberry pi does not have a built in Real Time Clock with its own battery, so it relies on network clocks to keep the time. Another fun fact, the network module of the pi is actually connected to the USB bus, so there's some overhead as well as a throughput limitation. Fun fact, the Pi does not have a power button, relying on software to shut down cleanly. If you lose access to the machine, it's not possible to avoid corrupted states on the disk. Despite all of this, if you want to self host some website, the raspberry pi is still an amazingly cost effective choice, from anywhere between 2 to 20000 monthly users, one pi will be overprovisioned. And you can even get an absolutely overkill redundant pi as a failover, but still a single pi can reach 365 days of uptime with no problem, and as long as you don't reboot or lose power or lose internet, you can achieve more than a couple of nines of reliability. But if you are thinking of a third, much less a 10th raspberry pi, you are probably scaling the wrong way, way before you reach the point where a quantity matters ( a third machine), it becomes cost effective to upgrade the quality of your one or two machines. On the embedded side it's the same story, these are great for prototyping, but you are not going to order 10k and sell them in production, maybe a small 100 test batch? But you will optimize and make your own PCB before a mass batch.
- stuxnet79 1y ago> Fun fact, a raspberry pi does not have a built in Real Time Clock with its own battery, so it relies on network clocks to keep the time. > Another fun fact, the network module of the pi is actually connected to the USB bus, so there's some overhead as well as a throughput limitation. > Fun fact, the Pi does not have a power button, relying on software to shut down cleanly. If you lose access to the machine, it's not possible to avoid corrupted states on the disk. With all these caveats in mind, a raspberry pi seems to be an incredibly poor choice for distributed computing
- CamperBob2 1y agoWith all these caveats in mind, a raspberry pi seems to be an incredibly poor choice for distributed computing Exactly. This build sounds like the proverbial "1024 chickens" in Seymour Cray's famous analogy. If nothing else, the communications overhead will eat you alive.
- kolbe 1y agoThe author, Jeff Geerling, is a very intelligent person. He has more experience with using niche hardware than almost anyone on earth. If he does something, there's usually a good a priori rationale for it.
- buildbot 1y agoJeff is a good person/blogger and does interesting projects but more experience with niche hardware than literally anyone is a stretch. Like what about the people who maintain the alpha/sparc/parisc linux kernels? Or the designers behind idk tilera or tenstorrent hardware.
- phatfish 1y agoYoutubers have armies of sycophants (check their video comments if you dare). Not saying they even court them, something to do with video building a stronger parasocial relationship than a text blog I think.
- geerlingguy 1y agoI was just at VCF Midwest this past weekend, and I can assure you I am on some of the lower echelons of people who know about niche hardware. I do get to see and play with a lot of interesting systems, but for most of them, I only get to go just under surface-level. It's a lot different seeing someone who's reverse engineered every aspect of an IBM PC110, or someone who's restored an entire old mainframe that was in storage for years... or the group of people who built an entire functional telephone exchange with equipment spread over 50 years (including a cell network, a billing system, etc.).
- kolbe 1y ago> more experience with niche hardware than literally anyone is a stretch. This is why I said "almost anyone." If I changed your words, I could disagree with you as well.
- amelius 1y agoIs a Pi still considered "niche" hardware?
- randomNumber7 1y agoWhat I think is strange with stuff like this that you should be able to come to that conclusion without technical knowledge. Just the fact that everyone runs AI on GPUs and NVIDIAs stock skyrocketed since the AI boom should tell you s.th.. Did OP really think his fellow humans are that moronic that they just didn't find out you can plug in together a cuple of rasperri pis?
- rustyminnow 1y agoNobody thought an RPI cluster would ever be competitive, and Geerling never expected anybody would. But it's fun to play "what if" and then make the thing just to see how it stacks up and that's his job. Any implication or suggestion of this being a good idea is just part of the story telling.
- Asraelite 1y ago> I don’t know if anyone building a Pi cluster actually goes into it thinking it’s going to be a cost effective endeavor, do they? Some Raspberry Pi products are sold at a loss, so I could see how it's in the realm of possibility.