9 ms·
I always wanted such thing for various "plumbing" services (DHCP/DNS/wifi controller etc) but lack of ECC and OOB management kinda disqualifies it for anything
by ilyt 4y ago
I always wanted such thing for various "plumbing" services (DHCP/DNS/wifi controller etc) but lack of ECC and OOB management kinda disqualifies it for anything serious.
>He's running forty Blades in 2U. That's:
>
> 160 ARM cores
> 320 GB of RAM
> (up to) 320 terabytes of flash storage
>
>...in 2U of rackspace.
Yay that's like... almost as much as normal 1U server can do
Edit: I give up, HN formatting is idiotic
- mkl 4y agoFrom the FAQ: https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc
- xattt 4y agoBut does anyone remember the Beowulf trope(1) from Slashdot? Am I a greybeard now? (1) https://hardware.slashdot.org/story/01/07/14/0748215/can-you-imagine-a-beowulf-cluster-of-these https://hardware.slashdot.org/story/01/07/14/0748215/can-you...
- pjmlp 4y agoI do. And cool research OSes that did process migration.
- nine_k 4y agoAh, Plan 9.
- zozbot234 4y agoI'm not sure that Plan 9 does process migration out of the box. It does have complete "containerization" by default, i.e. user-controlled namespacing of all OS resources - so snapshotting and migration could be a feasible addition to it. Distributed shared memory is another intriguing possibility, particularly since large address spaces are now basically ubiquitous. It would allow users to seamlessly extend multi-threaded workloads to run on a cluster; the OS would essentially have to implement memory-coherence protocols over the network.
- nine_k 4y agoIf not Plan 9, then likely Inferno. (A pretty different system, of course.)
- wiredfool 4y agoAnd God help us, OS2/Warp.
- pjmlp 4y agoWith much better tooling for OO ABI than COM/WinRT will ever get (SOM).
- infinite8s 4y agoIt's too bad Apple never bought Gerry Popek's LOCUS (https://en.wikipedia.org/wiki/LOCUS https://en.wikipedia.org/wiki/LOCUS), which could do process migration between heterogeneous hardware!
- Quequau 4y agoI do but in all fairness, I have an entirely grey beard.
- KingOfCoders 4y agoGrey! Mine is white.
- neilv 4y agoTo go along with "Imagine a Beowulf cluster of those!", don't forget "Take my money!"
- Koshkin 4y agoYou can get off my lawn now.
- edoloughlin 4y agoI do and you are. I’m also imagining one covered in hot grits…
- pbronez 4y agoNew to me - found the source article in the Wayback Machine: https://web.archive.org/web/20010715201416/http://www.scientificamerican.com/2001/0801issue/0801hargrove.html https://web.archive.org/web/20010715201416/http://www.scient...
- pbronez 4y agoFound the definition: Sterling and his Goddard colleague Donald J. Becker connected 16 PCs, each containing an Intel 486 microprocessor, using Linux and a standard Ethernet network. For scientific applications, the PC cluster delivered sustained performance of 70 megaflops--that is, 70 million floating-point operations per second. Though modest by today's standards, this speed was not much lower than that of some smaller commercial supercomputers available at the time. And the cluster was built for only $40,000, or about one tenth the price of a comparable commercial machine in 1994. NASA researchers named their cluster Beowulf, after the lean, mean hero of medieval legend who defeated the giant monster Grendel by ripping off one of the creature's arms. Since then, the name has been widely adopted to refer to any low-cost cluster constructed from commercially available PCs.
- iamflimflam1 4y agoBut does it run Doom?
- faichai 4y agoNatalie Portman says yes, and instructs you to put some hot grits down your pants.
- trollied 4y agoDon't forget the Hot Grits & Natalie Portman.
- pdpi 4y agoBeowulf clusters were those lame things that didn’t have wireless, and had less space than a nomad, right?
- wrldos 4y agoIt remember building a 4 node Beowulf cluster out of discarded compaq desktops and then having no idea what to do with it.
- mkj 4y ago75mhz, yeah! Stacked on top of each other! With 10mbit ethernet! I think we got OpenMosix going even. But then 5 years later I was working on them for a living in HPC, but they were no longer called Beowulf Clusters then.
- red-iron-pine 4y agoDid kinda the same thing but with Raspberry Pis. Neat, a cluster of r-pi's... now what?
- ipsin 4y agoIf you want to continue the chain of specific goals in service of no specific purpose: run Kubernetes on it.
- Sebb767 4y agoThen add ArgoCD for deployment and istio for a service mesh! While you are at it, also setup Longhorn for storage. With that solved, you might as well start hosting Gitea and DroneCI on the cluster, plus an extra helm- and docker repo for good measure. And in no time you will have a full modern CI/CD setup to do nothing but updates on! :-) Seriously, though, you will learn a lot of things in the process and get a bottom up view of current stacks, which is definitely helpful.
- wrldos 4y agoI did this. I am still dead inside. Thank goodness all my production shit has a managed control plane and network.
- worthless-trash 4y ago
- flyinghamster 4y agoYou and me both. The funny thing is, I wound up writing a program that would benefit from clustering, and felt my way around setting up MPICH on my zoo. I laughed out loud when I realized that, after all these years, I'd built an impromptu Beowulf cluster, even though the machines are scattered around the house. Installing MPICH from source instead of from your distribution is best if you can't have all your cluster members running the same version of the same distro and/or have multiple architectures to contend with. But it takes forever to compile, even on a fast machine.
- ilyt 4y agowell since you already have a cluster you could distcc the compilation. I remember doing that on a bunch of machines when we were building gentoo
- dredmorbius 4y agoThat's more a COW, than a (Beo)wulf, no? <https://www.oreilly.com/library/view/high-performance-linux/0596005709/ch01.html https://www.oreilly.com/library/view/high-performance-linux/...> A cluster of workstations (COW) is usually opportunistic exploiting existing systems, and lower density than a dedicated (usually rack-based or datacentre-based) cluster. In practice, COWs usually turn out to be not especially useful, though there are exceptions.
- lemper 4y agoyeah, wanted to replicate something like that by proposing a hardware vendor who visited my uni. decades ago. didn't go nowhere because i was intimidated by the red-tapes.
- mometsi 4y agoI like user big.ears' speculation on what someone could possibly do with that much parallel compute: I don't think there's any theoretical reason someone couldn't build a fairly realistic highly-complex "brain" using, say, 100,000,000 simplified neural units (I've heard of a guy in Japan who is doing such a thing), but I don't really know what it would do, or if it would teach us anything that is interesting.
- b33j0r 4y agoSure do! It is interesting that these technologies evolve more slowly than it seems, sometimes. On the graybearding of the cohort, here’s a weird one to me. These days, I mention slashdot and get more of a response from peers than mentioning digg! In 2005, I totally thought digg would be around forever as the slashdot successor, but it’s almost like it never happened (to software professionals… er, graybeards)
- bradleyy 4y agoSo, I once met a guy named Don. We were hanging out in the garage of a mutual friend, chatting. Got to the "what do you do" section of the conversation, and he says he works in massively parallel stuff at XYZ corp. Something something, GPUs. I make the obvious "can you make a Beowulf cluster?" joke, to which he responds (after a pregnant pause), "you... do know who I am?" Yep. Donald Becker. A slightly awkward moment, I'll cherish forever.
- rfrey 4y agoEw. "Do you have any idea who I am?" is never a good look.
- deleted 4y ago[deleted]
- zaarn 4y agoThe 1U server is however likely to use more than 200 Watts of power that the 40 Blade 2U setup would use.
- logifail 4y ago> The 1U server is however likely to use more than 200 Watts of power Q: Why would a 1U server need more than 200W if you're doing nothing more than basic network services? I have mini tower servers that draw a fraction of that at idle.
- bluedino 4y agoI have some idle Dell R650's that draw 384W. A couple drives, buncha RAM, two power supplies, 2 CPU's (Xeon 8358)
- logifail 4y ago> Dell R650's that draw 384W Umm, I'm not sure I can afford the electricity to run kit like that :) I'm currently awaiting delivery of an Asus PN41 (w/ Celeron N5100) to use as yet another home server, after a recommendation from a friend. Be interesting to see how much it draws at idle!
- livueta 4y agoHrm, interesting to see how the TDP of those 8358s drives overall power consumption. I'm looking at the idrac consoles of a couple R720XDs with 12 3.5" hdds, >128gb ram, two E5-2665s per, and they're all currently sipping ~150W at < 1 load average. The E5s have a TDP of 115W to the 8358's 250W, so I assume that's what's most of it. I admittedly do some special IPMI fan stuff, but that only shaves off tens of watts.
- zaarn 4y agoThe Pi's will be using those 200Watts at near full tilt. The main use here would be larger computational tasks that you can easily split up among the blades. Or you run a very hardware-failure tolerant software service on top.
- guntherhermann 4y agoIt's ~~four~~ two spaces to get the "code block" style. like this and asterisk for italics (I don't think there is a 'quote' available, and I'm not sure how they play together. * does this work? * Edit: No! Haha *how* *about* *this* Edit: No, no joy there either. I agree, it's not the most intuitive formatting syntax I've come across :) I guess we're stuck with BEGIN_QUOTE and END_QUOTE blocks!
- teddyh 4y agoIt’s two spaces. https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc
- ilyt 4y agoThe code block was workaround I wanted to do this > * list element 1 > * list element 2 without the indent. I don't get why it just doesn't use common mark or something, it's just some inept, half-assed clone of it anyway.
- guntherhermann 4y ago> Yay that's like... almost as much as normal 1U server can do What about cost, and other metrics around cost (power usage, reliability)? If space is the only factor we care about then it seems like a loss.
- betaby 4y agoWhat about them? 1U servers from vendors are reliable and efficient - people use them in production for years. As for the cost, those hobby-style board are very expensive for dollars/performance. Indeed I'm not getting why would one want a cluster of expensive, low spec nodes?
- sys42590 4y agoIndeed, that box here next to my desk draws 50W of electricity continuously despite being mostly idle. Why? Because it has ECC. Having some affordable low power device with ECC would be a game changer for me. I added affordable to exclude expensive (and noisy) workstation class laptops with ECC RAM.
- aidenn0 4y agoXeon-D series?
- walterbell 4y agoEpyc Embedded and possibly some Ryzen Embedded devices.
- nsteel 4y ago> Why? Because it has ECC Sorry if I am missing the obvious here, but why would ECC consume so much power?
- growse 4y agoIt's not that ECC consumes power, it's that systems that support ECC tend to consume more idle power (because they're larger etc.)
- Maakuth 4y agoThere are Intel Atom CPUs that support ECC. I had a Supermicro motherboard with a quad core part like that and I used it as a NAS. It was not that fast, but the power consumption was very low.
- smartbit 4y agoDo you remember how many Watts it was using with idle disks?
- Maakuth 4y agoIt was this board: https://www.supermicro.com/en/products/motherboard/a2sdi-2c-hln4f https://www.supermicro.com/en/products/motherboard/a2sdi-2c-... I think it was idling at something like 30-40W with four HDDs and a UPS. I didn't have an especially efficient PSU and the UPS must have taken some power too. The motherboard alone would draw as little as 15W, I suppose.
- LeonM 4y ago> I always wanted such thing for various "plumbing" services (DHCP/DNS/wifi controller etc) You don't need a cluster for that, even a 1st gen Pi can run those services without any problem.
- guntherhermann 4y agoI can only speak for Raspi 3B+, but I agree. I have multiple services running on it (including pihole, qbittorrent, vpn) and it's at about 40% mem usage right now.
- ilyt 4y agoI need multiple nodes for redundancy, not because it can't fit on one. And preferably at least 3 for quorum too.
- marginalia_nu 4y agoI do think this is sort of fool's gold in terms of actual performance. Even though the core count and RAM size is impressive, those cores are talking over ethernet rather than system bus. Latency and bandwidth is atrocious in comparison, and you're going to run into problems like no individual memory allocation being able to exceed 8 Gb. Like for running a hundred truly independent jobs then sure, maybe you'll get equivalent performance, but that's a very unique scenario that is rare in the real world.
- varispeed 4y agoI built such a toy cluster once to see for my self and gave up. It is too slow to do anything serious. You can be much better off by just buying older post lease server. Sure it will consume more power, but conversely you will finish more tasks in shorter time, so advantage of using ARM in that case may be negligible. If it was Apple's M1 or M2, that would have been a different story though. RPi4 and clones are not there yet.
- marginalia_nu 4y agoI overall think people tend to underestimate the overhead of clustering. It's always significantly faster to run a computation on one machine than spread over N machines with hardware of (1/N) power. That's not always a viable option because of hardware costs, and sometimes you want redundancy, but those concerns are on an orthogonal axis to performance.
- jbverschoor 4y agoWell, a complete M1 board, which is basically about as large as half an iPhone mini, is fast enough. It's also super efficient. So I'm still waiting for Apple to announce their cloud. They're currently putting Mx chips in every device they have, even the monitors. It'll be the base system for any electric device. I'm sure we'll see more specialized devices for different applications, because at this point, the hardware is compact, fast, and secure enough for anything, as well as the software stack. Hello Apple Fridge
- imtringued 4y agoDidn't AMD announce a 96 core processor with dual socket support? As usual this is either done for entertainment value or to simulate physical networks (not clusters).
- adrian_b 4y agoIntel also has now up to 480 cores in an 8-socket server with 60 cores per socket, though Sapphire Rapids is handicapped in comparison with AMD Genoa by much lower clock frequencies and cache memory sizes. However, while the high-core-count CPUs have excellent performance per occupied volume and per watt, they all have extremely low performance per dollar, unless you are able to negotiate huge discounts, when buying them by the thousands. Using multiple servers with Ryzen 9 7950X can provide a performance per dollar many times higher than that of any current server CPU, i.e. six 16-core 7950X with a total of 384 GB of unbuffered ECC DDR5-4800 will be both much faster and much cheaper than one 96-core Genoa with 384 GB of buffered ECC DDR5-4800. Nevertheless, the variant with multiple 7950X is limited for many applications by either the relatively low amount of memory per node or by the higher communication latency between nodes. Still, for a small business it can provide much more bang for the buck, when the applications are suitable for being distributed over multiple nodes (e.g. code compilation).
- cjbgkagh 4y agoThis is the exact space I’m in, high cpu low network. By my estimates it’s about 1/4 the cost per CPU operation to use consumer hardware instead of enterprise. The extra computers allow for application level redundancy so the other components can be cheaper as well.
- bee_rider 4y agoOne problem with 480 cores in single node: 480 cores is a shitload of cores, who needs more than a single node at this point? The MPI programmer inside me is having an existential breakdown.
- goodpoint 4y ago> Yay that's like... almost as much as normal 1U server can do ...but the normal server is much cheaper.
- 2OEH8eoCRo0 4y ago> Yay that's like... almost as much as normal 1U server can do Hyperscale in your Homelab. Something to hack on, learn, host things like Jellyfin, and have fun with.
- jeffbee 4y agoI agree but can't you get the same effect with VMWare ESXi? If I just wanted to "have fun" managing scores of tiny computers, and I emphasize that this sounds like the least amount of fun anyone could have, I can have as many virtual machines as I want.
- fishtacos 4y agoI can understand why some people want something physical/tangible while testing or playing in their hobby environment. I'm still a fan of virtualization - passmark scores for an RPi4 (entire SOC/quad core) are 21 times less than a per-single-core comparison in a 14-core 15-13600k (as a point of reference, my current system) and while am running 64GB RAM, can easily upgrade to 128GB or more on a single DDR4 node. Hard to see to an advantage given obvious limitations, although it may make it more fun to work within latency and memory constrictions, I guess.
- gaudat 4y agoHaha jellyfin would eat through all your memory and cpu time transcoding or remuxing on a SBC. I'm seriously thinking of getting another home server just to run that.
- FlyingAvatar 4y agoI think the hardware isolation would be a selling point in some cases. Granted, it's niche.
- ilyt 4y agoIn most cases it is wrong choice unless your business is selling raspberry pi hosting I guess
- metalspot 4y agoits a nice hobby project, but of course a commercial blade system will have far higher compute density. supermicro can do 20 epyc nodes in 8u, which at 64 cores per node is 1280 cores in 8u, or 160 in 1u, so double the core density, and far more powerful cores, so way higher effective compute density.
- PragmaticPulp 4y ago> but lack of ECC and OOB management kinda disqualifies it for anything serious. > Yay that's like... almost as much as normal 1U server can do It’s a fun toy. Obviously it isn’t the best or most efficient way to get any job done. That’s not the point. Enjoy it for the fun experiment that it is.
- deleted 4y ago[deleted]
- singron 4y agoIt's actually 3U since the 2U of 40 pis will need almost an entire 1U 48 port PoE switch instead of plugging into the TOR. The switch will use 35-100W for itself depending on features and conversion losses. If each pi uses more than 8-9W or so under load, then you might actually need a second PoE switch. If you are building full racks, it probably makes more sense to use ordinary systems, but if you want to have a lot of actual hardware isolation at a smaller scale, it could make sense. In some colos, they don't give you enough power to fill up your racks, so the low energy density wouldn't be such a bummer there.
- bee_rider 4y agoJust the Pi’s are $35 a pop, right? So that’s $1400 of Pi’s, on top of whatever the rest of the stuff costs. Wonder how it compares to, I guess, a whatever the price equivalent AMD workstation chip is…
- timerol 4y agoAlso not noted: 320 TB in 40 M.2 drives will be extremely expensive. Newegg doesn't have any 8 TB M.2 SSDs under $1000. $0.12/GB is about twice as expensive as more normally-sized drives, to say nothing of the price of spinning rust.
- mayli 4y agoThat would be 40x (Rpi4 8GB $75 + 8TB nvme $1200 + psu and others) ~ $51000.
- MuffinFlavored 4y ago> lack of ECC and OOB management kinda disqualifies it Can you expand on this please?
- geerlingguy 4y agoECC RAM is more robust (fewer crashes due to random bit flips), and OOB management means if a server has issues, you can remotely view it as if you were jacked in, and force reboot, among other things (like installing an OS remotely).
- actually_a_dog 4y agoWhat are you talking about "lack of ECC?" The Pi4 has ECC.
- ilyt 4y agoDidn't knew that! Would still prefer one that actually reports the errors tho