4 ms·

Apple has a huge opportunity here to lead the market for machines to run local models if they step into it. Their stuff is already better than what nVidia is of

by api 1mo ago

Apple has a huge opportunity here to lead the market for machines to run local models if they step into it. Their stuff is already better than what nVidia is offering with stuff like the DGX Spark. It's a niche market but it's a market that overlaps heavily with professionals in the AI space and lead developers, so it's a market that gets them customers in those roles. If I were running Apple I'd call the RAM price bubble for what it is and temporarily eat some margin to offer machines with more RAM than competitors, especially these models that are great for edge AI, and capture market share.

bigyabai 1mo ago
Apple doesn't design GPUs on-par with Nvidia's efficiency yet. They need an architectural overhaul to be a serious competitor, which is what I'm expecting is queued up for M7. Nvidia has CUDA, AMD has CDNA, and Apple has... compute shaders, I guess?
- nicce 1mo ago
  > Apple doesn't design GPUs on-par with Nvidia's efficiency yet How much it matters in inference? Most GPUs have enough computing for that and the bottleneck is the RAM speed and size. And M5 Ultra is becoming to challenge this.
  - bigyabai 1mo ago
    For prefill, it's basically all that matters. Long-horizon agent tasks, session compaction, file reads and context manipulation will all hit the compute bottleneck in regular usage, incurring several minutes of latency on most Apple Silicon chips, regardless of RAM. It's kinda why memory bandwidth is an enormous red herring, even for datacenter applications. Nvidia's huge advantage is a compute-optimized GPU architecture and their Infiniband networking, their memory controllers aren't really the star of the show.
- newsclues 1mo ago
  Apple has metal and mlx
- Danox 1mo ago
  Probably two generations away. I’m more interested in how much uplift/speed and more importantly what is the power usage is required for the new computers Apple is shipping particularly for the Studio versions.
Danox 1mo ago
Apple isn’t the company that eats margins but they are company that would design around the problem and I think that’s what they will do after all, they have the design and engineering and plenty of money because they didn’t burn it on AI models or data centers.
- api 1mo ago
  With AI Apple is doing their classic strategy I guess: wait and then follow with something that learns all the lessons the pioneers learned. They're letting everyone else pay for model training, with the obvious end of the road being open weights. They're letting everyone else overspend on first-generation AI data centers before the chip industry has truly optimized for today's AI designs (which will reduce data center footprint). They're letting everyone else play with and pioneer UI ideas. Meanwhile all they've done is put a few toes in the water: "Apple Intelligence" which is barely anything, and adding AI acceleration to their GPUs and doing a bit of up-market marketing to AI. So yeah they'll probably follow with a second generation Apple Intelligence that incorporates everything everyone else pioneered that worked, and an M7 or M8 line of chips with a GPU augmented with whatever approaches the other chip companies found worked best for running models... and built around the state of the art model architecture the industry finally converges on. By then RAM will probably be cheap again, so you'll get a 14" MacBook Pro with an M8 with a tensor-GPU and 512GiB of RAM.
mwcampbell 1mo ago
> Their stuff is already better than what nVidia is offering with stuff like the DGX Spark. How so? In tokens per second when running major open-weights models, or something else?
haskman 1mo ago
A Mac Mini with M5 Pro, half the memory of a DGX Spark, half the disk of DGX Spark, and no CUDA, is already more expensive than a spark. The Spark runs Linux natively, has a tiny form factor, and is usable as a general purpose computer. I use a DGX Spark with NixOS on it as my daily driver and it's fantastic.

by api 1mo ago

bigyabai 1mo ago

Apple doesn't design GPUs on-par with Nvidia's efficiency yet. They need an architectural overhaul to be a serious competitor, which is what I'm expecting is queued up for M7. Nvidia has CUDA, AMD has CDNA, and Apple has... compute shaders, I guess?

nicce 1mo ago
> Apple doesn't design GPUs on-par with Nvidia's efficiency yet How much it matters in inference? Most GPUs have enough computing for that and the bottleneck is the RAM speed and size. And M5 Ultra is becoming to challenge this.
- bigyabai 1mo ago
  For prefill, it's basically all that matters. Long-horizon agent tasks, session compaction, file reads and context manipulation will all hit the compute bottleneck in regular usage, incurring several minutes of latency on most Apple Silicon chips, regardless of RAM. It's kinda why memory bandwidth is an enormous red herring, even for datacenter applications. Nvidia's huge advantage is a compute-optimized GPU architecture and their Infiniband networking, their memory controllers aren't really the star of the show.
newsclues 1mo ago
Apple has metal and mlx
Danox 1mo ago
Probably two generations away. I’m more interested in how much uplift/speed and more importantly what is the power usage is required for the new computers Apple is shipping particularly for the Studio versions.

Danox 1mo ago

Apple isn’t the company that eats margins but they are company that would design around the problem and I think that’s what they will do after all, they have the design and engineering and plenty of money because they didn’t burn it on AI models or data centers.

api 1mo ago
With AI Apple is doing their classic strategy I guess: wait and then follow with something that learns all the lessons the pioneers learned. They're letting everyone else pay for model training, with the obvious end of the road being open weights. They're letting everyone else overspend on first-generation AI data centers before the chip industry has truly optimized for today's AI designs (which will reduce data center footprint). They're letting everyone else play with and pioneer UI ideas. Meanwhile all they've done is put a few toes in the water: "Apple Intelligence" which is barely anything, and adding AI acceleration to their GPUs and doing a bit of up-market marketing to AI. So yeah they'll probably follow with a second generation Apple Intelligence that incorporates everything everyone else pioneered that worked, and an M7 or M8 line of chips with a GPU augmented with whatever approaches the other chip companies found worked best for running models... and built around the state of the art model architecture the industry finally converges on. By then RAM will probably be cheap again, so you'll get a 14" MacBook Pro with an M8 with a tensor-GPU and 512GiB of RAM.

mwcampbell 1mo ago

> Their stuff is already better than what nVidia is offering with stuff like the DGX Spark. How so? In tokens per second when running major open-weights models, or something else?

haskman 1mo ago

A Mac Mini with M5 Pro, half the memory of a DGX Spark, half the disk of DGX Spark, and no CUDA, is already more expensive than a spark. The Spark runs Linux natively, has a tiny form factor, and is usable as a general purpose computer. I use a DGX Spark with NixOS on it as my daily driver and it's fantastic.