3 ms·

Local models running on Apple hardware are definitely a thing, a low-hanging fruit they are harvesting right now while every pure-play AI company is losing mone

• bigyabai 2mo ago
  It's not a huge thing outside the consumer/novelty sphere. Apple missed the train on the trillion-dollar ARM datacenter rollout, and will suck those lemons for the next decade.
• DwarvenEngineer 2mo ago
  Ironically for home-lab users (ie those who primarily have access to consumer hardware), apple silicon is probably the best value (and also the most obtainable for running some of the smaller sized models. You can get unified memory of up to 192 GB in the mac studios. The equivalent for the "open market" is probably AMD's halo strix which offers up to 128GB unified memory, and the speed of the memory is much slower. I guess the Nvidia Jetson AGX Thor also exists, but good luck getting your hands on that... and regardless that's also 128GB. Put it this way, with 192 GB VRAM you can run deepseekv4... with 128gb only with lots of quantization Edit: WTH! I was curious and i went to apple's website to price out a high end unit just to price comparison, and it looks like the high ram units are gone now, and they max out at 96GB?? Guess i missed that...
  • freediddy 2mo ago
    Not supporting FP8 is a pain in the butt though. It feels like there are more compromises using Apple Silicon than just going with nVidia and CUDA.
  • deleted 2mo ago
    [deleted]