6 ms·

Apple has good uniform hardware to enable this, but they are a product company and an “AI studio” would not fit their usual definition of a product. I do hope

Apple has good uniform hardware to enable this, but they are a product company and an “AI studio” would not fit their usual definition of a product. I do hope they are considering going in that direction though.

TheRoque 3y ago
That's the weird thing. They seem to have the best hardware for the price, for individuals to develop and use local LLMs, but so far they have been pretty quiet on all this.
- diffeomorphism 3y ago
  Highly depends on the price. Currently 1600€ gets you a laptop with 8gb shared memory, which surely is a nice device for other use cases but for developing local LLMs that seems awfully limiting. With upgraded ram (for just 230€ for each 8GB) and storage (just over 1000€ for 2tb; a samsung 990 pro is like 170€) that might be another story, but "check your specs before downloading this app" seems very un-appley. Also, no cuda support etc. Maybe if they make it exclusive to the mac studio?
  - kolinko 3y ago
    Cuda would be pointless for Apple, because cuda is for Nvidia only, no? As for the price - when you get to 32/64/96GB levels of RAM it’s the cheapest setup on the market that can get you this much VRAM. At least that’s what it was half a year ago when I checked the last time
    - diffeomorphism 3y ago
      > Cuda would be pointless for Apple, because cuda is for Nvidia only, no? That is the point. Cuda is widely used in particular for training as opposed to tuning and is just flat out not available. So you buy your nice $6000 machine and it just does not work for its intended purpose. > it’s the cheapest setup on the market that can get you this much VRAM. shared VRAM is not everything.
      - jeroenhd 3y ago
        I think it's clear that GPU manufacturers are scamming their customers more than Apple does with their ridiculous prices. There's an 80GB GPU out there that costs about six times as much as an equivalent Mac with 96 GB of shared RAM. Nvidia intentionally nerfs the amount of VRAM in their consumer cards so you need to buy their ridiculously overpriced enterprise cards. I think it's fair to say Apple's lineup is the cheapest way to get >64GB of VRAM-ish memory and that's definitely not because Apple prices their products so fairly.
        hu3 3y ago
        > There's an 80GB GPU out there that costs about six times as much as an equivalent Mac with 96 GB of shared RAM. Those are not equivalent in speed. macs RAM are much slower than these GPUs.
        jeroenhd 3y ago
        There is no Apple-to-apples comparison between a Macbook and a GPU. However, the problem with running most models is the lack of simultaneous RAM. You can swap memory back and forth between RAM and VRAM (with a huge performance penalty) of course, but that's not exactly usable or comparable to what Apple's VRAM sharing setup allows. Nvidia doesn't sell a nice-but-not-amazing GPU equivalent to Apple's processing power and memory bandwidth that's also capable of operating on >80GB of VRAM at once. Apple's SoC is kind of an oddball in that regard. I suppose you could take a regular old iGPU (for AMD, Intel, probably also Qualcom/Mediatek) and use its shared memory capabilities as a comparison. However, iGPUs are terrible at machine learning tasks, they don't come close to what Apple can do with their dedicated accelerators. The best middle ground may be the laptop GPUs with both dedicated RAM and shared RAM, but those will start swapping memory back and forth like crazy running large ML workloads so they're not really comparable. If you want to run a model that operates on a huge amount of memory at once, I don't think there is a desktop option that can do what Apple does without going for the massive overkill GPUs that will crush the Macbook in terms of performance (at great cost). Perhaps you know a GPU or iGPU that's capable of running 80GB VRAM workloads at comparable speeds? Because I don't.
- capableweb 3y ago
  It's not weird, is typical Apple playbook. Year 0 of cool thing - Nothing and silence Year 1-3 of cool thing - Maybe, if you're lucky, a mention on some hardware thing related to it Year 3-5 of cool thing - Hardware or software launches that uses thing, no mentions of this besides the earlier one if any. Year 5-6 of cool thing - Next part of hardware or software launches that uses previous launch, no mentions of this besides the earlier one if any. Obviously, the time-frames differ, but that's generally how they do things.
- fragmede 3y ago
  https://github.com/ml-explore/mlx-examples/ https://github.com/ml-explore/mlx-examples/ is Apple engineers working to get ML models going on Apple Silicon's Metal hardware.
lostmsu 3y ago
Nobody is going to train or even fine-tune large models on Apple hardware. It is too slow for that purpose.
m3kw9 3y ago
They do have such tools for developers like CreateML to train your own models, pretty sure they will have one for LLM.
paradite 3y ago
Apple has it: https://github.com/ml-explore/mlx https://github.com/ml-explore/mlx
- reacharavindh 3y ago
  Two tangential thoughts about this. 1. Why would they miss the opportunity to go all in and make use of the Neural engines for this? Or do they already, and I just don’t know how to interpret it? 2. At what point is Apple going to think - hmm, we have a kick ass processor on our hands. What if we run our server fleet - the ones that serve iCloud, Apple Store, all Apple services, databases etc on M* processors? It sure would help even better the economies of scale for Apple to go to TSMC and say - here is our new CPU design for servers, iPhone, iPad, watch and whatever VR thing. Why no love for the server side that must be orders of magnitude power hungrier today?

TheRoque 3y ago

That's the weird thing. They seem to have the best hardware for the price, for individuals to develop and use local LLMs, but so far they have been pretty quiet on all this.

diffeomorphism 3y ago
Highly depends on the price. Currently 1600€ gets you a laptop with 8gb shared memory, which surely is a nice device for other use cases but for developing local LLMs that seems awfully limiting. With upgraded ram (for just 230€ for each 8GB) and storage (just over 1000€ for 2tb; a samsung 990 pro is like 170€) that might be another story, but "check your specs before downloading this app" seems very un-appley. Also, no cuda support etc. Maybe if they make it exclusive to the mac studio?
- kolinko 3y ago
  Cuda would be pointless for Apple, because cuda is for Nvidia only, no? As for the price - when you get to 32/64/96GB levels of RAM it’s the cheapest setup on the market that can get you this much VRAM. At least that’s what it was half a year ago when I checked the last time
  - diffeomorphism 3y ago
    > Cuda would be pointless for Apple, because cuda is for Nvidia only, no? That is the point. Cuda is widely used in particular for training as opposed to tuning and is just flat out not available. So you buy your nice $6000 machine and it just does not work for its intended purpose. > it’s the cheapest setup on the market that can get you this much VRAM. shared VRAM is not everything.
    - jeroenhd 3y ago
      I think it's clear that GPU manufacturers are scamming their customers more than Apple does with their ridiculous prices. There's an 80GB GPU out there that costs about six times as much as an equivalent Mac with 96 GB of shared RAM. Nvidia intentionally nerfs the amount of VRAM in their consumer cards so you need to buy their ridiculously overpriced enterprise cards. I think it's fair to say Apple's lineup is the cheapest way to get >64GB of VRAM-ish memory and that's definitely not because Apple prices their products so fairly.
      - hu3 3y ago
        > There's an 80GB GPU out there that costs about six times as much as an equivalent Mac with 96 GB of shared RAM. Those are not equivalent in speed. macs RAM are much slower than these GPUs.
        jeroenhd 3y ago
        There is no Apple-to-apples comparison between a Macbook and a GPU. However, the problem with running most models is the lack of simultaneous RAM. You can swap memory back and forth between RAM and VRAM (with a huge performance penalty) of course, but that's not exactly usable or comparable to what Apple's VRAM sharing setup allows. Nvidia doesn't sell a nice-but-not-amazing GPU equivalent to Apple's processing power and memory bandwidth that's also capable of operating on >80GB of VRAM at once. Apple's SoC is kind of an oddball in that regard. I suppose you could take a regular old iGPU (for AMD, Intel, probably also Qualcom/Mediatek) and use its shared memory capabilities as a comparison. However, iGPUs are terrible at machine learning tasks, they don't come close to what Apple can do with their dedicated accelerators. The best middle ground may be the laptop GPUs with both dedicated RAM and shared RAM, but those will start swapping memory back and forth like crazy running large ML workloads so they're not really comparable. If you want to run a model that operates on a huge amount of memory at once, I don't think there is a desktop option that can do what Apple does without going for the massive overkill GPUs that will crush the Macbook in terms of performance (at great cost). Perhaps you know a GPU or iGPU that's capable of running 80GB VRAM workloads at comparable speeds? Because I don't.
capableweb 3y ago
It's not weird, is typical Apple playbook. Year 0 of cool thing - Nothing and silence Year 1-3 of cool thing - Maybe, if you're lucky, a mention on some hardware thing related to it Year 3-5 of cool thing - Hardware or software launches that uses thing, no mentions of this besides the earlier one if any. Year 5-6 of cool thing - Next part of hardware or software launches that uses previous launch, no mentions of this besides the earlier one if any. Obviously, the time-frames differ, but that's generally how they do things.
fragmede 3y ago
https://github.com/ml-explore/mlx-examples/ https://github.com/ml-explore/mlx-examples/ is Apple engineers working to get ML models going on Apple Silicon's Metal hardware.

lostmsu 3y ago

Nobody is going to train or even fine-tune large models on Apple hardware. It is too slow for that purpose.

m3kw9 3y ago

They do have such tools for developers like CreateML to train your own models, pretty sure they will have one for LLM.

paradite 3y ago

Apple has it: https://github.com/ml-explore/mlx https://github.com/ml-explore/mlx

reacharavindh 3y ago
Two tangential thoughts about this. 1. Why would they miss the opportunity to go all in and make use of the Neural engines for this? Or do they already, and I just don’t know how to interpret it? 2. At what point is Apple going to think - hmm, we have a kick ass processor on our hands. What if we run our server fleet - the ones that serve iCloud, Apple Store, all Apple services, databases etc on M* processors? It sure would help even better the economies of scale for Apple to go to TSMC and say - here is our new CPU design for servers, iPhone, iPad, watch and whatever VR thing. Why no love for the server side that must be orders of magnitude power hungrier today?