4 ms·

An interesting difference with the new pro models, if you want to pay around $5k, is that you can have 128Gb unified Ram in them and run inference locally wit

by pama 3y ago

An interesting difference with the new pro models, if you want to pay around $5k, is that you can have 128Gb unified Ram in them and run inference locally with exciting LLMs.

gnicholas 3y ago
I suspect LLM performance will be a differentiator for Pro and newer Apple laptops in coming years. If Apple releases on-device AI, this will be an effective way of getting people to upgrade like they used to, but haven't had to recently. For example, I bought Pro-level computers in my younger years, but now would only consider an MBA, mini, or iMac. But they could get me to go for a Pro if it were the only way to get more RAM for better AI performance. It will also likely shorten upgrade cycles since newer computers would have the latest and greatest performance. When I bought my M2 MBA years ago I suspected it would last me a long time. Now I'm not so sure since I don't have a ton of RAM.
kjkjadksj 3y ago
5k for a computer with 128gb doesn’t seem like a great deal. You can get that much ddr5 today for like $400 sometimes less than that on sale.
- theGnuMe 3y ago
  yeah but you can't get an nvidia gpu with at least 128gb (H200) for less than $40k
  - kjkjadksj 3y ago
    You can set how much of your system ram your gpu gets in windows and linux
    - xcv123 3y ago
      Useless for LLMs. Limited system memory bandwidth kills it. GPU will be mostly idle waiting for memory access. The H200 has 4.8 TB/s bandwidth. System memory is below 90 GB/s
      - kjkjadksj 3y ago
        If you have a use case for an h200 class gpu you are probably making enough money off of it to not care about $40k and the 900gb/s bandwidth mentioned in another comment would seem absolutely pedestrian in comparison. If these things were really that compelling of a performance per dollar ratio you wouldn’t be able to buy them. They’d be sold out because people would be filling datacenters with them.
        xcv123 3y ago
        M2 Ultra sits between a lower end Intel setup and an H200 for this purpose. I reject this all-or-nothing black and white childish framing of it.
- xcv123 3y ago
  > You can get that much ddr5 today for like $400 Not comparable. M2 Ultra with 128GB RAM has 800 GB/s bandwidth. Maximum bandwidth for a DDR5 Intel 14900K system is 89.6 GB/s. https://www.intel.com/content/www/us/en/products/sku/236773/intel-core-i9-processor-14900k-36m-cache-up-to-6-00-ghz/specifications.html https://www.intel.com/content/www/us/en/products/sku/236773/...
  - kjkjadksj 3y ago
    You can only get 2tb of storage max i believe, you are probably bottlenecked elsewhere if you are actually needing to move that much data that fast into memory in practice. I wonder what the temps would even be doing something that maxes that ram bandwidth
    - xcv123 3y ago
      > I wonder what the temps would even be doing something that maxes that ram bandwidth The temps will be completely normal since that is normal operation. TDP is only 80 watts for M2 Ultra. > You can only get 2tb of storage max i believe, you are probably bottlenecked elsewhere No. Just no. This is not the issue. The models are loaded into GPU memory.
skybrian 3y ago
It seems like a pretty speculative use. That much money would go a long way on a $20/month subscription, and the field is changing rapidly. Unless you’re an AI researcher, might as well wait a year or two and see what changes?
- hnfong 3y ago
  Using M1/2/3 Max for LLM inference isn't at all "speculative", it's a thing today and high end Apple Silicon being an option for LLM inference is becoming general knowledge among the local inference community. The original author of llama.cpp (one of the leading LLM inference projects) developed it on a Mac and it has full Metal acceleration support. The $20/month subscription is going to give you access to commercial models, but generally you have to run the open weight models yourself. With the unified RAM you can trivially run the larger 70B+ models. AI researchers generally have to use CUDA due to how the ecosystem is still mostly CUDA-only for training and fine tuning, but those who need to occasionally use custom/local models for inference will likely find high end Macs being a good fit for their use cases.
  - skybrian 3y ago
    Okay, but this is rather low-level. Most users aren’t going to care about “running an LLM.” They want to ask an AI chatbot questions or get code autocompletion or something like that. What applications are there that need a local LLM and who is using them? What do they do with them? Personally, I’m reasonably happy with GPT4 and Github Copilot, and I’ve sometimes used Midjourney, though I cancelled my subscription since I’m not currently generating any images. Are there important apps that I’m missing?
    - hnfong 3y ago
      If you're happy with the commercial offerings there isn't a very compelling reason to use local models. The local ones are not "better" in general. But sometimes people have reasons to use local models, eg. privacy, customization, control, etc. I personally use it mostly to keep tabs on the latest models released on huggingface. There has been a lot of interesting developments since last year, and models have become more and more powerful.
fennecbutt 3y ago
What a waste. Build a 4090 desktop for that price, sheesh. Or spend the money renting spot 4090 instances whenever you need to via a MB air or whatever other laptop. And you'd probably still not end up spending the full amount.

by pama 3y ago

An interesting difference with the new pro models, if you want to pay around $5k, is that you can have 128Gb unified Ram in them and run inference locally with exciting LLMs.

gnicholas 3y ago

I suspect LLM performance will be a differentiator for Pro and newer Apple laptops in coming years. If Apple releases on-device AI, this will be an effective way of getting people to upgrade like they used to, but haven't had to recently. For example, I bought Pro-level computers in my younger years, but now would only consider an MBA, mini, or iMac. But they could get me to go for a Pro if it were the only way to get more RAM for better AI performance. It will also likely shorten upgrade cycles since newer computers would have the latest and greatest performance. When I bought my M2 MBA years ago I suspected it would last me a long time. Now I'm not so sure since I don't have a ton of RAM.

kjkjadksj 3y ago

5k for a computer with 128gb doesn’t seem like a great deal. You can get that much ddr5 today for like $400 sometimes less than that on sale.

theGnuMe 3y ago
yeah but you can't get an nvidia gpu with at least 128gb (H200) for less than $40k
- kjkjadksj 3y ago
  You can set how much of your system ram your gpu gets in windows and linux
  - xcv123 3y ago
    Useless for LLMs. Limited system memory bandwidth kills it. GPU will be mostly idle waiting for memory access. The H200 has 4.8 TB/s bandwidth. System memory is below 90 GB/s
    - kjkjadksj 3y ago
      If you have a use case for an h200 class gpu you are probably making enough money off of it to not care about $40k and the 900gb/s bandwidth mentioned in another comment would seem absolutely pedestrian in comparison. If these things were really that compelling of a performance per dollar ratio you wouldn’t be able to buy them. They’d be sold out because people would be filling datacenters with them.
      - xcv123 3y ago
        M2 Ultra sits between a lower end Intel setup and an H200 for this purpose. I reject this all-or-nothing black and white childish framing of it.
xcv123 3y ago
> You can get that much ddr5 today for like $400 Not comparable. M2 Ultra with 128GB RAM has 800 GB/s bandwidth. Maximum bandwidth for a DDR5 Intel 14900K system is 89.6 GB/s. https://www.intel.com/content/www/us/en/products/sku/236773/intel-core-i9-processor-14900k-36m-cache-up-to-6-00-ghz/specifications.html https://www.intel.com/content/www/us/en/products/sku/236773/...
- kjkjadksj 3y ago
  You can only get 2tb of storage max i believe, you are probably bottlenecked elsewhere if you are actually needing to move that much data that fast into memory in practice. I wonder what the temps would even be doing something that maxes that ram bandwidth
  - xcv123 3y ago
    > I wonder what the temps would even be doing something that maxes that ram bandwidth The temps will be completely normal since that is normal operation. TDP is only 80 watts for M2 Ultra. > You can only get 2tb of storage max i believe, you are probably bottlenecked elsewhere No. Just no. This is not the issue. The models are loaded into GPU memory.

skybrian 3y ago

It seems like a pretty speculative use. That much money would go a long way on a $20/month subscription, and the field is changing rapidly. Unless you’re an AI researcher, might as well wait a year or two and see what changes?

hnfong 3y ago
Using M1/2/3 Max for LLM inference isn't at all "speculative", it's a thing today and high end Apple Silicon being an option for LLM inference is becoming general knowledge among the local inference community. The original author of llama.cpp (one of the leading LLM inference projects) developed it on a Mac and it has full Metal acceleration support. The $20/month subscription is going to give you access to commercial models, but generally you have to run the open weight models yourself. With the unified RAM you can trivially run the larger 70B+ models. AI researchers generally have to use CUDA due to how the ecosystem is still mostly CUDA-only for training and fine tuning, but those who need to occasionally use custom/local models for inference will likely find high end Macs being a good fit for their use cases.
- skybrian 3y ago
  Okay, but this is rather low-level. Most users aren’t going to care about “running an LLM.” They want to ask an AI chatbot questions or get code autocompletion or something like that. What applications are there that need a local LLM and who is using them? What do they do with them? Personally, I’m reasonably happy with GPT4 and Github Copilot, and I’ve sometimes used Midjourney, though I cancelled my subscription since I’m not currently generating any images. Are there important apps that I’m missing?
  - hnfong 3y ago
    If you're happy with the commercial offerings there isn't a very compelling reason to use local models. The local ones are not "better" in general. But sometimes people have reasons to use local models, eg. privacy, customization, control, etc. I personally use it mostly to keep tabs on the latest models released on huggingface. There has been a lot of interesting developments since last year, and models have become more and more powerful.

fennecbutt 3y ago

What a waste. Build a 4090 desktop for that price, sheesh. Or spend the money renting spot 4090 instances whenever you need to via a MB air or whatever other laptop. And you'd probably still not end up spending the full amount.