4 ms·

The best bang for the buck on VRAM is a maxed out Mac Studio.

fabiensanglard 2y ago
Absolutely! I have been playing with Ollama on a Macbook Pro 192 GiB RAM and it is able to run most models whereas my 3090 runs our of RAM.
- sepositus 2y ago
  Do you mean 128GB? Not aware of any variant of the Macbook Pro with that much RAM.
  - simonw 2y ago
    192GB is available for the M2 Mac Studio.
    - mdaniel 2y ago
      I was curious "how bad is it?" and it seems $5500-ish https://www.ebay.com/sch/i.html?_nkw=192gb+studio&_sop=15 https://www.ebay.com/sch/i.html?_nkw=192gb+studio&_sop=15
      - fiberhood 2y ago
        $6500 depending on VAT. But 10-12 times M4 Mac mini's with 100 Gbps networking gives you triple the cores and 160 GB with 2.5 times the memeory bandwith if the sharding of the NN layers is done right.
        fennecfoxy 2y ago
        $6500!! You may as well buy 5x3090s for $1000 each for 120GB ram, spend the extra $1500 on the sundries. Like, I'm sure Nvidia is aware of Apple's "unified memory" as an alternative to their cards and yet...they aren't offering >24GB consumer cards yet, so clearly they don't feel threatened. Don't get me wrong, I've always disliked Apple as a company, but the M series chips are brilliant, I'm writing this on one right now. But people seem to think that Apple will be able to get the same perf increases yoy when they're really stretching process limits by dumping everything onto the same die like that - where do they go from here? That said Nvidia is using HBM so it does make me wonder why they aren't also doing memory on package with HBM, I think SK Hynix et al were looking at making this possible. I'm glad we're headed in the direction of 3d silicon though, always seemed like we may as well scale in z, I imagine they can stack silicon/cooling/silicon/cooling etc. I'm sure they can use lithography to create cooling dies to sandwich between everything else. Then just pass connections/coolant through those.
    - qingcharles 2y ago
      Hoping that M4 Ultra Mac Pros will bump this again.
washadjeffmad 2y ago
No one who claims this ever posts a benchmark Prompt eval is slow, inference for large models at high context is slow, training is limited and slow. It's better than not having anything, but we got rid of our M1 Max 192GBs after about a year.
- morphle 2y ago
  What will you pay me for the benchmarks, for the professional knowledge and analysis? I can post benchmarks for these Mac machines and clusters of Studios and M4 Mac Minis (see my other HN posts last month, the largest Mac cluster I can benchmark for you has 4 TB of ultrafast unified memory and around 9216 M4 cores).
  - digdugdirk 2y ago
    I mean, I can't pay you anything, but that sounds interesting as hell. Are there any interesting use cases to massive amounts of memory outside of training?
    - morphle 2y ago
      > No one who claims this ever posts a benchmark I meant to explain why no one ever posts a benchmark, it's expensive as hell to do a professional benchmark against accepted standards. Its several days work, very expensive rental of several pieces of $10K hardware, etc. You don't often hand that over for free. With my benchmark results some companies can save millions if they take my advice. >any interesting use cases to massive amounts of memory outside of training? Dozens, hundreds. Almost anything you use databases, CPUs, GPUs or TPUs for. 90% of computing is done on the wrong hardware, not just datacenter hardware. The interesting use case we discussed here on HN last week was running full DeepSeek-R1 LLms on 778 GB fast DRAM computers locally. I benchmarked getting hundreds of tokens per second on a cluster of M4 Mac minis or a cluster of M2 Mac Studio Ultras where others reported 0.015 or 6 tokens per second on single machines. I just heard of a Brazilian man who build a 256 Mac Mini cluster at double the cost that I would. He leaves $600K value on the table because he won't reverse engineer the instruction set, rewrite his software or even call Apple to negotiated a low price. HN votes me down for commenting that I, a supercomputer builder for 43 years, can build better cheaper faster low power supercomputers from Mac Mini's and FPGA's than from any Nvidia, AMD or Intel state of the art hardware, it even beats the fastest supercomputer of the moment or the Cerebras wafer engine V3 (on energy. coding cost and performance per watt per dollar). I design and build wafer scale 2 million core reconfigurable supercomputers for $30K a piece that cost $150-$300 million to mass produce. That's why I know how to benchmark M2 Ultra and M4 Macs, as they are the second best chip a.t.m. that we need to compete against. As a consulting job I do benchmarks or build your on-prem hardware or datacenter. This job consists mainly teaching the customer's programming staf how to program massively parallel software or convincing the CEO not to rent cloud hardware but buy on-prem hardware. OP at Fly.io should have hired me, then he wouldn't have needed to write his blog post. I replied to your comment in hope of someone hiring me when they read this.

Absolutely! I have been playing with Ollama on a Macbook Pro 192 GiB RAM and it is able to run most models whereas my 3090 runs our of RAM.

sepositus 2y ago
Do you mean 128GB? Not aware of any variant of the Macbook Pro with that much RAM.
- simonw 2y ago
  192GB is available for the M2 Mac Studio.
  - mdaniel 2y ago
    I was curious "how bad is it?" and it seems $5500-ish https://www.ebay.com/sch/i.html?_nkw=192gb+studio&_sop=15 https://www.ebay.com/sch/i.html?_nkw=192gb+studio&_sop=15
    - fiberhood 2y ago
      $6500 depending on VAT. But 10-12 times M4 Mac mini's with 100 Gbps networking gives you triple the cores and 160 GB with 2.5 times the memeory bandwith if the sharding of the NN layers is done right.
      - fennecfoxy 2y ago
        $6500!! You may as well buy 5x3090s for $1000 each for 120GB ram, spend the extra $1500 on the sundries. Like, I'm sure Nvidia is aware of Apple's "unified memory" as an alternative to their cards and yet...they aren't offering >24GB consumer cards yet, so clearly they don't feel threatened. Don't get me wrong, I've always disliked Apple as a company, but the M series chips are brilliant, I'm writing this on one right now. But people seem to think that Apple will be able to get the same perf increases yoy when they're really stretching process limits by dumping everything onto the same die like that - where do they go from here? That said Nvidia is using HBM so it does make me wonder why they aren't also doing memory on package with HBM, I think SK Hynix et al were looking at making this possible. I'm glad we're headed in the direction of 3d silicon though, always seemed like we may as well scale in z, I imagine they can stack silicon/cooling/silicon/cooling etc. I'm sure they can use lithography to create cooling dies to sandwich between everything else. Then just pass connections/coolant through those.
  - qingcharles 2y ago
    Hoping that M4 Ultra Mac Pros will bump this again.

washadjeffmad 2y ago

No one who claims this ever posts a benchmark Prompt eval is slow, inference for large models at high context is slow, training is limited and slow. It's better than not having anything, but we got rid of our M1 Max 192GBs after about a year.

morphle 2y ago
What will you pay me for the benchmarks, for the professional knowledge and analysis? I can post benchmarks for these Mac machines and clusters of Studios and M4 Mac Minis (see my other HN posts last month, the largest Mac cluster I can benchmark for you has 4 TB of ultrafast unified memory and around 9216 M4 cores).
- digdugdirk 2y ago
  I mean, I can't pay you anything, but that sounds interesting as hell. Are there any interesting use cases to massive amounts of memory outside of training?
  - morphle 2y ago
    > No one who claims this ever posts a benchmark I meant to explain why no one ever posts a benchmark, it's expensive as hell to do a professional benchmark against accepted standards. Its several days work, very expensive rental of several pieces of $10K hardware, etc. You don't often hand that over for free. With my benchmark results some companies can save millions if they take my advice. >any interesting use cases to massive amounts of memory outside of training? Dozens, hundreds. Almost anything you use databases, CPUs, GPUs or TPUs for. 90% of computing is done on the wrong hardware, not just datacenter hardware. The interesting use case we discussed here on HN last week was running full DeepSeek-R1 LLms on 778 GB fast DRAM computers locally. I benchmarked getting hundreds of tokens per second on a cluster of M4 Mac minis or a cluster of M2 Mac Studio Ultras where others reported 0.015 or 6 tokens per second on single machines. I just heard of a Brazilian man who build a 256 Mac Mini cluster at double the cost that I would. He leaves $600K value on the table because he won't reverse engineer the instruction set, rewrite his software or even call Apple to negotiated a low price. HN votes me down for commenting that I, a supercomputer builder for 43 years, can build better cheaper faster low power supercomputers from Mac Mini's and FPGA's than from any Nvidia, AMD or Intel state of the art hardware, it even beats the fastest supercomputer of the moment or the Cerebras wafer engine V3 (on energy. coding cost and performance per watt per dollar). I design and build wafer scale 2 million core reconfigurable supercomputers for $30K a piece that cost $150-$300 million to mass produce. That's why I know how to benchmark M2 Ultra and M4 Macs, as they are the second best chip a.t.m. that we need to compete against. As a consulting job I do benchmarks or build your on-prem hardware or datacenter. This job consists mainly teaching the customer's programming staf how to program massively parallel software or convincing the CEO not to rent cloud hardware but buy on-prem hardware. OP at Fly.io should have hired me, then he wouldn't have needed to write his blog post. I replied to your comment in hope of someone hiring me when they read this.