5 ms·
I’m hoping this isn’t as attractive as it sounds for non-hobbyists because the performance won’t scale well to parallel workloads or even context processing, wh
by andy99 10mo ago
I’m hoping this isn’t as attractive as it sounds for non-hobbyists because the performance won’t scale well to parallel workloads or even context processing, where parallelism can be better used.
Hopefully this makes it really nice for people that want the experiment with LLMs and have a local model but means well funded companies won’t have any reason to grab them all vs GPUs.
- codazoda 10mo agoI haven’t looked yet but I might be a candidate for something like this, maybe. I’m RAM constrained and, to a lesser extent, CPU constrained. It would be nice to offload some of that. That said, I don’t think I would buy a cluster of Macs for that. I’d probably buy a machine that can take a GPU.
- ChrisMarshallNY 10mo agoI’m not particularly interested in training models, but it would be nice to have eGPUs again. When Apple Silicon came out, support for them dried up. I sold my old BlackMagic eGPU. That said, the need for them also faded. The new chips have performance every bit as good as the eGPU-enhanced Intel chips.
- andy_ppp 10mo agoeGPU with an Apple accelerator with a bunch or RAM and GPU cores could be really interesting honestly. I’m pretty sure they are capable of designing something very competitive especially in terms of performance per watt.
- sroussey 10mo agoReally, that’s a place for the MacPro: slide in SoC with ram modules / blades. Put 4, 8, 16 Ultra chips in one machine.
- andy_ppp 10mo agoYou honestly don’t need extra CPUs in this system at some point do you?
- sroussey 10mo agoThey are inseparable for Apple. CPUS/GPUs/memory. They can use chipsets to tweak ratios, but I doubt they will change the underlying module format—everything together. My suggestion is to accept that format and just provide a way to network them at a low level via pci or better.
- bigyabai 10mo agoThe lack of official Linux/BSD support is enough to make it DOA for any serious large-scale deployment. Until Apple figures out what they're doing on that front, you've got nothing to worry about.
- Eggpants 10mo agoNot sure I understand, Mac OS is BSD based. https://en.wikipedia.org/wiki/Darwin_(operating_system) https://en.wikipedia.org/wiki/Darwin_(operating_system)
- bigyabai 10mo agomacOS is XNU-based. There is BSD code that runs in the microkernel level and BSD tools in the userland, but the kernel does not resemble BSD's architecture or adopt BSD's license. This is an issue for some industry-standard software like CUDA, which does provide BSD drivers with ARM support that just never get adopted by Apple: https://www.nvidia.com/en-us/drivers/unix/ https://www.nvidia.com/en-us/drivers/unix/
- api 10mo agoNo way buying a bunch of minis could be as efficient as much denser GPU racks. You have to consider all the logistics and power draw, and high end nVidia stuff and probably even AMD stuff is faster than M series GPUs. What this does offer is a good alternative to GPUs for smaller scale use and research. At small scale it’s probably competitive. Apple wants to dominate the pro and serious amateur niches. Feels like they’re realizing that local LLMs and AI research is part of that, is the kind of thing end users would want big machines to do.
- gumboshoes 10mo agoExactly: The AI appliance market. A new kind of home or small-business server.
- jabbywocker 10mo agoI’m expecting Apple to release a new Mac Pro in the next couple years who’s main marketing angle is exactly this
- firecall 10mo agoSeems like it could be a thing. Also, I’m curious and in case anyone that knows reads this comment: Apple say they can’t get the performance they want out of discreet GPUs. Fair enough. But yet nVidia becomes the most valuable company in the world selling GPUs. So… Now I get that Apples use case is essentially sealed consumer devices built with power consumption and performance tradeoffs in mind. But could Apple use its Apple Silicon tech to build a Mac Pro with its own expandable GPU options? Or even other brand GPUs knowing they would be used for AI research etc…. If Apple ever make friends with nVidia again of course :-/ What we know of Tim Cooks Apple is that it doesn’t like to leave money on the table, and clearly they are right now!
- jabbywocker 10mo agoThere’s been rumors of Apple working on M-chips that have the GPU and CPU as discrete chiplets. The original rumor said this would happen with the M5 Pro, so it’s potentially on the roadmap. Theoretically they could farm out the GPU to another company but it seems like they’re set on owning all of the hardware designs.
- willtemperley 10mo agoI think it’s going to be great for smaller shops that want on premise private cloud. I’m hoping this will be a win for in-memory analytics on macOS.