3 ms·
Silicon Mac’s are great for running LLMs. Unified memory and memory bandwidth of the Max and Ultra processors is very useful in doing inference locally.
by seanmcdirmid 1y ago
Silicon Mac’s are great for running LLMs. Unified memory and memory bandwidth of the Max and Ultra processors is very useful in doing inference locally.
- bigyabai 1y agoGreat news, but entirely lost on commercial hyperscalers and much of the PC market. Apple's recalcitrance towards supporting Nvidia drivers basically killed their last shot at real-world rackmount deployment of Apple Silicon. Now you can go buy an ARM Grace CPU that does the same thing, but cheaper and with better software support.
- seanmcdirmid 1y agoYou really can’t. NVIDIA’s arm chip still looks nerfed compared to apple’s offering, and…I can run 40GB sized LLMs on the plane with no internet…it’s not something that you can do with any other platform.
- pstuart 1y agoI imagine they recognize the need for increased memory bandwidth and will bump that up significantly -- they are well positioned to do so.