14 ms·
I regret building this $3000 Pi AI cluster
- elzbardico 1y agoFrankly, always thought about Pi Clusters as a nerd indulgence, something to play, not to do serious work.
- devmor 1y agoAfter a few years of experience with them I agree for the most part. They are great for individual projects and even as individual servers for certain loads, but once you start clustering them you will probably get better results from a purpose built computer in the same price range as multiple pis.
- randomgermanguy 1y agoI think the only exception is specifically for studying network/communciation-topologies. I've seen a couple clusters (ca. 10-50 Pi's) in universities for both research and teaching.
- ofrzeta 1y agoThere are so many network emulators you can use, such as Mininet or GNS3.
- JambalayaJimbo 1y agoI'm sure pedagogically speaking it's better to use physical devices
- NitpickLawyer 1y agoIt reminds me of the Beowulf clusters of the 90s-2000s, that were all the rage at some point, then slowly lost ground... I remember many friends tinkering with some variant of those, we had one in Uni, and there were even some linux distros dedicated to the concept.
- gary_0 1y agoOh yeah, the "imagine a Beowulf cluster of these" Slashdot meme! I miss those days. At least the "can it run Doom?" meme is still alive and kicking.
- unregistereddev 1y agoDitto! It reminded me of the time in college when I built a Beowulf cluster from recently-retired Pentium II desktops. Was it fast? No. But that wasn't the point. I was learning about distributed computing.
- dekhn 1y agoBeowulf style clusters went on to dominate supercomputing.
- fidotron 1y agoIf Pi Clusters were actually cost competitive for performance there would be data centres full of them.
- phoronixrly 1y agoIf they were cost competitive for ... anything at all really...
- wltr 1y agoWell I have a Pi as a home server, and it’s very energy efficient, while doing what I want. Since I don’t need latest and greatest (I don’t see any difference with a modern PC for my use case), it’s very competitive for me. No need for any cooling is bonus.
- Waraqa 1y ago>very energy efficient If your server has a lot of idle time, ARM will always win.
- wltr 1y agoMy home server mostly not used. It triggers some simple bash scripts each hour, or each night. It serves some simple personal web pages. I access them a couple of times a day, each time randomly. I have a Raspberry Pi 1B, and it looks like despite being massively underpowered, it’s the most energy efficient. And in my use-case, I think of using it for the task. One day my primary Raspberry Pi broke (turned out to be a PSU issue), and I thought of having an old laptop running 24/7 as a home server. While being not very power hungry, it’s still wants much more energy (plus it has fans). For a casual usage (I forgot to mention Pi-Hole) it feels like an overkill. So, while a Raspberry Pi isn’t the best, it has its niche, and I’m happy of having one (actually, a few).
- ACCount37 1y agoPrototyping and low volume. They're good for long as the development costs dominate the total costs.
- dbg31415 1y agoWas it cost effective? Meh. Was it a learning experience? More importantly, did you have some fun? Just a little? (=
- phoronixrly 1y ago> Was it a learning experience? Also no. The guy's a youtuber On the other hand, will this make him 100+k views? Yes. It's bait - the perfect combo to attract both the AI crowd and the 'homelab' enthusiasts (of which the bulk are yet to find any use for their raspberry devices)...
- aprdm 1y agoHe is not a YouTuber. And even if he was - what's the problem ? Jeff has many useful OSS software used by many companies around the world daily - including mine. What have you created ?
- phoronixrly 1y ago> What have you created ? Nothing that is not AGPL-licensed, so you and your company haven't taken advantage of it. I am not sure how this relates to my comment though.
- vel0city 1y agoHe may also be a good OSS contributor and writer, but he is also a Youtuber. Over 500 videos posted, 175M views, nearly a million subscribers. Not that its a problem, I don't see why it would inherently be a negative thing. Dude seems to make some good content across a lot of different mediums. Cheers to Jeff.
- Coffeewine 1y agoIt's a pretty rough headline, clearly the author had fun performing the test and constructing the thing. I would be pretty regretful of just the first sentence in the article, though: > I ordered a set of 10 Compute Blades in April 2023 (two years ago), and they just arrived a few weeks ago. That's rough.
- geerlingguy 1y agoThat's the biggest regret; but I've backed 6 Kickstarter projects over the years. Median time to deliver is 1 year. Somehow I've actually gotten every item I backed shipped at some point (which is unexpected). Hardware startups are _hard_, and after interacting with a number of them (usually one or two people with a neat idea in an underserved market), it seems like more than half fail before delivering their first retail product. Some at least make it through delivering prototypes/crowdfunded boards, but they're already in complete disarray by the end of the shipping/logistics nightmares.
- maartin0 1y agoNot completely related, but do you know if hardware kickstarters typically have any IP protection? I'm surprised there haven't been any cases of large companies creating patents for ideas from kickstarter at least that I've seen
- ssl-3 1y agoOne cannot (or at least, one is not supposed to be able to) patent someone else's invention.
- privatelypublic 1y agoYou theoretically have a year before you even have to apply- but patents are expressly "first to file."
- oasisaimlessly 1y ago
- bravetraveler 1y agoWow that's a lot of scratch for... scratch. Pays for itself, I'm sure: effective bait :) 'Worth it any more'? At this size, never. A Pi is a Pi is a Pi! A few are fine for toying around; beyond that, hah. Price:perf is rough, does not improve with multiplication [of units, cost, or complexity].
- xnx 1y agoFun project. Was the author hoping for cost effective performance?! I assumed this was a novelty, like building a RAID array out of floppy drives.
- leptons 1y agoA lot of people don't understand the performance limits of the Raspberry Pi. It's a great little platform for some things, but it isn't really fit for half the use cases I've seen.
- Our_Benefactors 1y agoThis was my impression as well, the bit about GPU incompatibility with llama.cpp made me think he was in over his head.
- LTL_FTC 1y agoThe author is a YouTuber and projects like these pay for themselves with the views they garner. Even the title is designed for engagement.
- kirito1337 1y ago[dead]
- lumost 1y agoI don’t really get why anyone would be buying ai compute unless A) to your goal is to rent out the compute B) no vendor can rent you enough compute when you need it C) you have an exotic funding arrangement that makes compute capex cheap and opex expensive. Unless you can keep your compute at 70% average utilization for 5 years - you will never save money purchasing your hardware compared to renting it.
- 2OEH8eoCRo0 1y agoI don't get why anyone would hack on and have fun with unique hardware either /s
- seanw444 1y agoIt's also not always just about fun or cost effectiveness. Taking the infrastructure into your own hands is a nice way to know that you're not being taken advantage of, and you only have yourself to rely on to make the thing work. Freedom and self-reliance, in short.
- HenryMulligan 1y agoData privacy and security don't matter? My secondhand RTX 3060 would buy a lot of cloud credits, but I don't want tons of highly personal data sent to the cloud. I can't imagine how it would be for healthcare and finance, at least if they properly shepherded their data.
- tern 1y agoFor most people, no, privacy does not matter in this sense, and "security" would only be a relevant term if there was a pre-existing adversarial situation
- justinrubek 1y agoAt some point, the work has to actually be done rather than shuffling the details off to someone else.
- 1y ago
- bunderbunder 1y agoReminds me a bit of one of my favorite NormConf sessions, "Just use one big machine for model training and inference." https://youtu.be/9BXMWDXiugg?si=4MnGtOSwx45KQqoP https://youtu.be/9BXMWDXiugg?si=4MnGtOSwx45KQqoP Or the oldie-but-goodie paper "Scalability! But at what COST?": https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf https://www.usenix.org/system/files/conference/hotos15/hotos... Long story short, performance considerations with parallelism go way beyond Amdahl's Law, because supporting scale-out also introduces a bunch of additional work that simply doesn't exist in a single node implementation. (And, for that matter, multithreading also introduces work that doesn't exist for a sequential implementation.) And the real deep down black art secret to computing performance is that the fastest operations are the ones you don't perform.
- Aurornis 1y agoI thought the conclusion should have been obvious: A cluster of Raspberry Pi units is an expensive nerd indulgence for fun, not an actual pathway to high performance compute. I don’t know if anyone building a Pi cluster actually goes into it thinking it’s going to be a cost effective endeavor, do they? Maybe this is just YouTube-style headline writing spilling over to the blog for the clicks. If your goal is to play with or learn on a cluster of Linux machines, the cost effective way to do it is to buy a desktop consumer CPU, install a hypervisor, and create a lot of VMs. It’s not as satisfying as plugging cables into different Raspberry Pi units and connecting them all together if that’s your thing, but once you’re in the terminal the desktop CPU, RAM, and flexibility of the system will be appreciated.
- wccrawford 1y agoGeerling's titles have been increasingly click-bait for a while now. It's pretty sad, because I like his content, but hate the click-bait BS.
- jonathanlydall 1y agoIf it makes an appreciable difference to how much money he makes on YouTube then I can’t begrudge him for doing it. Don’t hate the player, hate the game.
- geerlingguy 1y agoJust to add context — I've been experimenting on my 2nd channel (Level 2 Jeff) with titles that are straight/barebones exactly describing the content of the video, vs a slight bit of clickbait (never untrue, but certainly more intriguing and not describing the exact topic of the video). The ones that are dead straight with no clickbait are 10/10 (the worst performers), and usually by a massive margin. Even with the same thumbnail. The sad fact is, if you want your work seen on YouTube, you can't just say "I built a 10 node Raspberry Pi blade cluster and ran HPL and LLMs on it". Some people are fine with a limited audience. And that's fine too! I don't have to write on my blog at all—I earn negative income from that, since I pay for hosting and a domain, but I hope some people enjoy the content in text form like I do.
- aprdm 1y agoLove Jeff's ansible roles/playbooks and his cluster building ! Quite interesting, I should reserve some time to play with a Pi cluster and ansible, sounds fun
- noelwelsh 1y agoI'd love to understand the economics here. $3000 purely for fun seems like a lot. $3000 for promotion of a channel? consulting? seems reasonable.
- philipwhiuk 1y agoJeff has a million YouTube subscribers, gets $2000 a month from Patreon and has 200 GitHub sponsors. The economics of spending $3,000 on a video probably work out fine.
- geerlingguy 1y agoIt's definitely a stretch for my per-video budget, but I did want to have a 'maxed out' Pi cluster for future testing as well. A lot of people (here, Reddit, elsewhere) speculate about how good/bad a certain platform or idea is. Since I have the means to actually test how good or bad something is, I try to justify the hardware costs for it. Similar to testing various graphics cards on Pis, I've probably spent a good $10,000 on those projects over the past few years, but now I have a version of every major GPU from the past 3 generations to test on, not only on Pi, but other Arm platforms like Ampere and Snapdragon. Which is fun, but also educational; I've learned a lot about inference, GPU memory access, cache coherency, the PCIe bus... So a lot of intangibles, many of which never make it directly into a blog post or video. (Similar story with my time experiments).
- markx2 1y ago> "But if you're on the blog, you're probably not the type to sit through a video anyway. So moving on..." Thank you!
- AlfredBarnes 1y agoMy pi's are just an easy onramp for me to have a functional NAS, PIHole, and webcam security. Not at all the best, but they were cheap. If i WANTED the best or reliable, i'd actually buy real products.
- hn_throw_250915 1y agoI read through it and it’s amusing but along with the title being something I’d receive in email from a newsletter mailing list I’ve never subscribed to (hoping it has an unsubscribe link at the bottom), there’s nothing really of hacker curiosity here to keep me hooked. It’s shallow and appeals to some LCD “I did the thing with the stuff and the results will shock you because of how obvious they are now click here” mentality. Vainposting at its most average. The Mac restoration video was somewhat easier to sit through if only because the picture quality beats out a handful of other YT videos doing the exactly same thing as I’m holding back a jaw grating wince of watching someone butchering a board with poor knowledge of soldering iron practice, so YMMV? Back to hackaday for me I think. I’m not here to read submarine resumes of people applying to work at Linus Tech Tips.
- paxys 1y agoThis is just the evolution of clickbait titles. The only thing missing is a thumbnail of an AI generated raspberry pi cluster with a massive arrow pointing to it and the words "not worth it!!"
- Joker_vD 1y agoThere also needs to be a Face Screaming in Fear Emoji plastered on the other side of it.
- imtringued 1y agoOh come on Jeff, you forgot to buy GPUs for your AI cluster. Such a beginner mistake. All you needed to do is buy 4x xtx 7900 used on ebay and build a four node raspberry pi cluster using the external GPU setup you've come up with in one of your previous blog posts [0]. [0] https://www.jeffgeerling.com/blog/2024/use-external-gpu-on-raspberry-pi-5-4k-gaming https://www.jeffgeerling.com/blog/2024/use-external-gpu-on-r...
- geerlingguy 1y agoMore on that soon... ;)
- cosarara 1y ago> Compared to the $8,000 Framework Cluster I benchmarked last month, this cluster is about 4 times faster: Slower. 4 times slower.
- teleforce 1y agoThat's definitely a typo because I've to read the sentence 3 times from the article still cannot make a sense until I saw the figure. TL;DR, just buy one framework desktop and it's better than the Pi AI cluster of the OP in every single performance metrics including cost, performance, efficiency, headache, etc.
- geerlingguy 1y agoOops, fixed the typo! Thanks. And regarding efficiency, in CPU-bound tasks, the Pi cluster is slightly more efficient. (Even A76 cores on a 16nm node still do well there, depending on the code being run).
- drillsteps5 1y agoIf he was building compute device for LLM inference specifically it would help to check in advance what that would entail. Like GPU requirement. Which putting bunch of RPis in the cluster doesn't help one bit. Maybe I'm missing something.
- deadbabe 1y agoI really don’t understand the hype over raspberry Pi. It’s an overrated, overhyped little computer. Like ok it’s small I guess but why is it the default that everyone wants to build something new on? Because it’s cheap? Whatever happened to buy once, cry once? Why not just build an actual powerful rig? For your NAS? For your firewalls? For security cameras? For your local AI agents?
- theultdev 1y agoI use mine for a plex server. I don't need to transcode + I need something I can leave on that draws little power. I have a powerful rig, but the one time I get to turn it off is when I'd need the media server lol. There's a lot of scenarios where power usage comes into play. These clusters don't make much sense to me though.
- deadbabe 1y agoThat’s insane, drawing very little power from an always on server is a solved problem.
- geerlingguy 1y agoWhat's your idea of very little power, though? I know for many who run SBCs (RK3588, Pi, etc.), very little is 1-2W idle, which is almost nothing (and doesn't even need a heatsink if you can stand some throttling from time to time). Most of the Intel Mini PCs (which are about the same price, with a little more performance) idle at 4-6W, or more.
- lostmsu 1y agoNot to mention there's more cost, effort, and energy effective compute from old laptops. Unless you have a robot body for your potential RPi, don't buy one.
- theultdev 1y agohow many can run on batteries? it's nice to take it on road trips / into hotels. can't really imagine hauling a server around. we probably have different definitions of "very little power".
- hendersoon 1y agoI mean, obviously it isn't practical, he got a couple of videos out of it.
- nromiun 1y agoThere is a reason all the big supercomputers have started using GPUs in the last decade. They are much more efficient. If you want 32bit parallel performance just buy some consumer GPUs and hook them up. If you need 64bit buy some prosumer GPUs like the RTX 6000 Pro and you are done. Nobody is really building CPU clusters these days.
- ted_dunning 1y agoWell, El Capitan uses AMD CPUs (which have integrated GPU capabilities) and it is right on top of the rankings lately. Frontier is right behind it with the same arrangement. Having honest to god dedicated GPUs on their own data bus with their own memory isn't necessarily the fastest way to roll.
- nromiun 1y agoThey do not. The CPUs are only there to support and push data to the GPUs. Much like Nvidia GH200 systems. Nobody buys these APU chips for their CPU parts. For comparison there are 9,988,224 GPU compute units in El Capitan and only 1,051,392 CPU cores. Roughly one CPU core to push data to 10 GPU CUs.
- anematode 1y agoUnfortunately even the RTX 6000 Pro has nerfed double-precision throughput at about 2 TFLOPS, 64x slower than single precision. For comparison an EPYC 9755 does ~10 TFLOPS, while drawing less power. An A100 -- if you can find one -- is in the same ballpark. The best option for DP throughput for hobbyists interested in HPC might be old AMD cards from before they, too, realized that scientific folks would pay up the nose for higher precision.
- deater 1y agoas someone who has built various raspberry pi clusters over the years (I even got an academic paper out of one) the big shame is that as far as I know it's still virtually impossible to use the fairly powerful GPUs they have for GPGPU work
- fgbarben 1y agosub hundred gigaflop counts as "fairly powerful" now?
- zamadatix 1y agoThe article focuses on compute performance but I wonder if that was ever the bottleneck considering the memory bandwidth involved.
- amelius 1y agoOk, what are the back-of-the-envelope computations that he should have done before starting to build this?
- geerlingguy 1y agoPi memory bandwidth is less than 10 GB/sec, so AI use cases will be extremely limited. Network I/O is maximum of 1 Gbps (or more if you do some unholy thing with M.2 NICs), so that also limits maximum networked performance. But still can be decent for HPC learning, CI testing, or isolated multi-node smaller-app performance.
- bearjaws 1y agoAm I the only one who looks at both the Pi Cluster and the Framework PC and wonders how they are both slower and less cost effective than a MacBook Pro M4 Max? 88 token/s on a 2.3b model is not exactly great, most likely you will want a 32 or 70b model.
- Drblessing 1y agoThe bee-link AI max+ is the best value AI pc right now.
- stirfish 1y agoI came here to ask about these. You like yours?
- Drblessing 1y agoYes. Incredible form factor.
- pluto_modadic 1y ago[flagged]
- pstuart 1y agoEwww. Religion is a hell of a drug.
- traverseda 1y agoA specific women? Women in general? Is vague-posting back in vogue?
- titaniumtown 1y agoCan you elaborate? I don't know what you are referring to.
- stanac 1y agoQuick google search, Jeff has a blog post about abortion, I haven't read it, but it starts with a warning: > This post is more than 10 years old, I do not delete posts... https://www.jeffgeerling.com/articles/religion/abortion-cases-of-rape https://www.jeffgeerling.com/articles/religion/abortion-case... I guess GP is referring to that post.
- IAmBroom 1y agoI'd support this idea if you provided some measure of evidence. As it is, pure ad hominem.
- Havoc 1y agoHave a bunch of Pis too, but realized I can use them to create a high availability control plane for a k8s cluster. Pi4s are entirely adequate for that
- phito 1y agoCould you please elaborate? I find this interesting as I'm building a k8s cluster and also have a bunch of RPIs laying around...
- Havoc 1y ago3x Pi 4s booting off ssd running Talos with virtual IP enabled You know in k8s you've got worker nodes and control plane nodes? The control planes don't need much horsepower, but they're what you need to be online all to communicate with the cluster. Pis work just fine for that.
- WhitneyLand 1y ago“with 160 GB of total RAM, shared by the CPU and iGPU, this could be a small, efficient AI Cluster, right? Well, you'd think.” No, I wouldn’t think.
- owaislone 1y agoWhat is the ideal or realistic setup for a home lab to power big enough models running locally and service ~2-3 users at a time?
- hamonrye 1y agoStrange quirks for language syntax. I read about a parsing system that was distilled towards being-towards this particular species of silver-tail that existed below Golden Gate bridge. Godwin's Law?
- bee_rider 1y ago> The first benchmark I ran was my top500 High Performance Linpack cluster benchmark. This is my favorite cluster benchmark, because it's the traditional benchmark they'd run on massive supercomputers to get on the top500 supercomputer list. […] > After fixing the thermals, the cluster did not throttle, and used around 130W. At full power, I got 325 Gflops I was sort of surprised to find that the top500 list on their website only goes back to 1993. I was hoping to find some ancient 70’s version of the list where his ridiculous Pi cluster could sneak on. Oh well, might as well take a look… I’ll pull from the sub-lists of https://www.top500.org/lists/top500/ https://www.top500.org/lists/top500/ They give the top 10 immediately. First list (June 1993): placement name RPEAK (GFlop/s) 1 CM-5/1024 131.00 10 Y-MP C916/16256 15.24 Last list he wins, I think (June 1996): 1 SR2201/1024 307.20 10 SX-4/32 64.00 First list he’s bumped out of the top 10 (November 1997): 1 ASCI Red 1,830.40 10 T3E 326.40 I think he gets bumped off the full top500 list around 2002-2003. Unfortunately I made the mistake of going by Rpeak here, but they sort by Rmax, and I don’t want to go through the whole list. Apologies for any transcription errors. Actually, pretty good showing for such a silly cluster. I think I’ve been primed by stuff like “your watch has more compute power than the Apollo guidance computer” or whatever to expect this sort of thing to go way, way back, instead of just to the 90’s.
- deleted 1y ago[deleted]
- m3kw9 1y agowhen i saw your tok/s on a shtty model, i said yah.
- yalogin 1y agoCan the LLMs be deployed split across multiple Pis? I thought it was not possible, may be I am not caught up.
- bityard 1y agoI can think of SOME decent--if suboptimal--reasons to build an RPi cluster, but it never would have occurred to me to try running an LLM on one. Even if you could cluster hundreds or thousands of Pis, it's still just entirely the wrong architecture for running an AI model as they are currently built.
- asimpleusecase 1y agoGreat write up! This is nice nerd catnip. And sharing a failed project teaches so much. Please, more should share the projects that absorbed tons of time yet never really delivered.
- curtisszmania 1y ago[dead]
- numpad0 1y ago> goes round-robin style asking each node to perform its prompt processing, then token generation. Yeah, this is a now-long-wide-known issue with LLM processing. This can be remediated so that all nodes split computation, but then you'll come back to classical supercomputing problem of node interconnect latency/bandwidth bottlenecks. It looks to me that many such interconnect simulate Ethernet cards. I wonder if it can be recreated using the M.2 slot rather than using that slot for node-local data, and cost effectively so(like cheaper than bunch of 10GE cards and short DACs).
- wcchandler 1y agoI was just exploring Pi’s and AI hats, so this post is appreciatively timely. I’m finally at the point where I can dedicate time for building an AI with a specific use case in mind. I play competitive paintball and would like to utilize AI for a handful of things. Specifically hit detections in video streams. Pi’s were my natural choice simply because of low cost of entry and wide range of supported products to get a PoV running. I even thought about reaching out to Jeff and asking his input. This post didn’t change my direction too much, but it did help level set some realistic expectations. So thanks for sharing.
- ComputerGuru 1y agoGuys, don’t take the claim so literally. He’s a successful tech poster. He makes good money showing off his purchases the good money complaining about how expensive they were. But certainly don’t imitate his choices, his economics aren’t your economics!
- deleted 1y ago[deleted]
- system2 1y agoI totally agree. A person with that kind of builder knowledge already knows a decent GPU could 10x that compute power.
- esskay 1y agoThats pretty much a given, but what the real takeaway should be from most of the content is that whatever you are doing, these days the answer in all likelihood is not to buy a raspberry pi. It's specs to price just do not add up at all anymore, and it's looking like a pretty damn stagnant place these days.
- Computer0 1y agoWhat if you need ARM? What is the best sub $100 sbc that I am missing? Orange Pi hardware always looks good but I hear a lot of negativity about the software that I don't really experience with Raspbian.
- michaelt 1y agoThen you should take a good hard look at older, much cheaper Raspberry Pis. Then look at Apple’s ARM offerings, and AWS Graviton if you need ARM with raw power. If you need embedded/GPIO you should consider an Arduino, or clone. If you need GPIOs and Internet connectivity, look at an ESP32. GPIOs, ARM and wired ethernet? Consdier the the STM32H. Robotics/machine vision applications, needing IO and lots of compute power? Consider a regular PC with an embedded processor on serial or USB. Or nvidia jetson if you want to run CUDA stuff. And take a good hard look at your assumptions, as mini PCs using the Intel N100 CPU are very competitive with modern Pis.
- iLoveOncall 1y ago> With 160 GB of total RAM, shared by the CPU and iGPU, this could be a small, efficient AI Cluster, right? Well, you'd think. I don't know anyone who would think this actually.
- geerlingguy 1y agoMost people who haven't actually self-hosted AI models would think this—for some reason people still think RAM is RAM, and they don't think about specs like memory speed, whether the full amount of RAM is shared with the GPU, etc. You'd be surprised by the number of emails, Instagram DMs, YouTube comments, etc. I get—even after explicitly showing how bad a system is at a certain task—asking if a Pi would be good for X, or if they could run ChatGPT on their laptop...
- WhereIsTheTruth 1y agobrand fanboyism, and vendor locking, the bane of our society the common denominator is always capital gain capitalism is the reason why we haven't been able to go back to the moon and build bases there
- pmarreck 1y agoand yet spaceX is for-profit and is pushing the boundaries again, after NASA (a not-for-profit government-created monopoly) stagnated. blanket-blaming capitalism without good reasoning is becoming the new red-flag of "can't think critically"
- WhereIsTheTruth 1y agoNASA didn't stagnate, NASA landed humans on the Moon with 1960s technology private space companies, despite decades of hype and funding, have stagnated by comparison the fact that SpaceX depends heavily on government contracts just to function is yet another proof: their "innovation" isn't self sustaining, it's underwritten by taxpayer money are you denying that NASA landed on the Moon? Elon psyop doesn't work on me, i know who is behind it all, they need a charismatic sales man for the masses, just like Ford, Disney, Reagan and all, masking structural power with a digestible story for the masses > blanket-blaming capitalism without good reasoning is becoming the new red-flag of "can't think critically" it's quite the opposite, people unable to take criticism of capitalism, talk about "critical thinking", how is China doing?
- system2 1y agoLook at it this way, you had fun playing with an expensive toy. A single $1000 GPU could 10x that, even when you were building those (not today's GPUs). You probably already knew it. But I gotta admit, the rig looks very nice.
- densh 1y agoFor anyone interested in playing with distributed systems, I'd really recommend getting a single machine with latest 16-core CPU from AMD and just running 8 virtual machines on it. 8 virtual machines, with 4 hyper threads pinned per machine, and 1/8 of total RAM per machine. Create a network between them virtually within your virtualization software of choice (such as Proxmox). And suddenly you can start playing with distributed software, even though it's running on a single machine. For resiliency tests you can unplug one machine at a time with a single click. It will annihilate a Pi cluster in Perf/W as well, and you don't have to assemble a complex web of components to make it work. Just a single CPU, motherboard, m.2 SSD, and two sticks of RAM. Naturally, using a high core count machine without virtualization will get you best overall Perf/W in most benchmarks. What's also important but often not highlighted in benchmarks in Idle W if you'd like to keep your cluster running, and only use it occasionally.
- bee_rider 1y agoTangentially related: I really expected running old MPI programs on stuff like the AMD multi-chip workstation packages to become a bigger thing.
- le-mark 1y agoI actually worked with some MPI code way back. What MPI programs are you referring to?
- MathMonkeyMan 1y agoI don't know, but when I was playing with finite difference code as an undergrad in Physics, all of the docs I could find (it was a while ago, though) assumed that I was going to use MPI to run a distributed workload across the university's supercomputer. My needs were less, so I just ran my Boost.Thread code on the four cores of one node. What if you had a single server with a zillion cores in it? Maybe you could take some 15 year old MPI code and run it locally -- it'd be like a mini supercomputer with an impossibly fast network.
- bee_rider 1y ago
- teaearlgraycold 1y agoHaving seen someone else build and tinker with a 7 node Pi cluster it seems like an absolute waste of time. 1Gb/s networking, PCIe 3.0 x1, slow RAM, slow CPU. And with all the hats and accessories needed it’s not that good of a deal. Getting some NUC-like machines makes a lot more sense to me. You’ll get 2.5Gb/s Ethernet at the least and way more FlOPS as well.
- nathan_douglas 1y agoYou might really dislike the ESP32 cluster I'm planning right now :P
- leakycap 1y agoI dipped my toe into cluster computing when rendering a MAXON Cinema 4D project across a lab of ~30 dual G5 Power Macs. Quickly learned that there is so much more to manage when you split a task up across systems, even when the system (like Cinema 4D) is designed for it.
- nodesocket 1y agoI feel like AI is not the right workload for this. I run a 4x node Kubernetes cluster (1x control and 3x workers) and it’s surprising good. While performance is certainly not state of the art by any stretch of the imagination it works great for HomeLab’in. Portainer, Glance, OpenSpeed Test, Semaphore UI, Uptime Kuma, and Home Assistant. Many small and lightweight containers is where a cluster such as this can shine.
- djhworld 1y agoI watched the video and enjoyed it, I think the most interesting part to me was running the distributed Llama.cpp, Jeff mentioned it seems to work in a linear fashion where processing would hop between nodes. Which got me thinking about how do these frontier AI models work when you (as a user) run a query. Does your query just go to one big box with lots of GPUs attached and it runs in a similar way, but much faster? Do these AI companies write about how their infra works?
- geerlingguy 1y agoServeTheHome has a few videos covering AI servers and interconnects. And yes, they basically have 1 Tbps+ interconnects and throw tens or hundreds of GPUs at queries. Nvidia was wise to invest so much in their networking side—they have massive bandwidth between machines and shared memory, so they can run massive models with tons of cards, with minimal latency. It's still not as good as tons of GPU attached to tons of memory on _one_ machine, but it's better than 10, 25, or 40 Gbps networking that most small homelabs would run.
- MomsAVoxell 1y agoI have a SOPINE cluster board that I'd quite like to get booted up and pressed into some sort of useful service. I think the biggest problem with cluster products is that they just don't work out of the box. Vendors haven't really done the "last 2%" of development required to make them viable - its left to us purchasers to get the final bits in place. Still, it'll make a fun distributed computing experimental platform some day. Just like the Inmos Transputer I've got somewhere, sitting in a box, waiting for a power supply ..
- dragontamer 1y agoWhy build a cluster? I believe the Rasp Pi cluster is one of the cheapest multi node / MPI machines you can buy. That's useful even if it is t fast. You need to practice the programming interfaces, not necessarily make a fast computer. However, NUMA is also a big deal. The various AMD Threadrippers with multi-die memory controllers are better on this regards. Maybe the aging Threadrippers 1950x, yes it's much slower than modern chips but the NUMA issues are exaggerated (especially poor) on this old architecture. That exaggerates the effects of good NUMA and now you as a programmer can get more NUMA skills. Of course, the best plan is to spend $20,000,000++ on your own custom NUMA nodes cluster out of EPYCs or something. ------- But no. The best supercomputers are your local supercomputers that you should rent some time from. You need a local box to see various issues and learn to practice programming.
- zeristor 1y agoIf it worked out well he would be trumpeting the glory. Not so good, and this is the sort of title. you need to bring the punters in for YouTube. I don't mean to sound too cynical, I appreciate Jeff's videos, just wanted to point out that if you've spent money and time on content you can either ditch it or make a regret video. Just so long as the thumbnails don't have an arrow on them I'm happy.
- computersuck 1y agoRPIs have always had shit cpu performance - the initial idea for the pi was to use very low cost broadcom crap chips that no one would buy and market it for "educational" use. They are essentially for kids to play around with learning computers by blinking LEDs and integrating with circuit boards. The idea of building a high performance cluster with pis is dumb from day one
- trhway 1y ago>They are essentially for kids to play around with learning computers by blinking LEDs and integrating with circuit boards. Where a "kid" may be a 53 years old with 30+ years softdev experience who ultimately got to get to the stuff he wanted to for quite some time, and the "blinking LEDs" are a bunch of servos programmatically controlled based on input from a bunch of sensors. While there are definitely better alternatives based on various narrow metrics, especially when it may come to actual productization, the ease (and cheapness, so you don't think much about that spending) of starting with all those easily available for RPi servo array drive boards and various IO ports array boards and all the available software - it is hard to imagine how it can be more easy/cheaper/available than it already is with all that actual compute power and full-featured Linux environment.
- yellowapple 1y agoIncredible to me that you can buy a 10-pack of 256GB NVMe drives for less than $200. Seems like yesterday when those ran for $200 each.
- OrangeMusic 1y agoImagine a Beowulf cluster of these!