14 ms·

>n case it is not abundantly clear by now: Apple's AI strategy is to put inference (and longer term even learning) I'm curious: is anyone seriously using apple

>n case it is not abundantly clear by now: Apple's AI strategy is to put inference (and longer term even learning) I'm curious: is anyone seriously using apple hardware to train Ai models at the moment? Obviously not the big players, but I imagine it might be a viable option for Ai engineers in smaller, less ambitious companies.

andrewmcwatters 2y ago
Yes, it can be more cost effective for smaller businesses to do all their work on Mac Studios, versus having a dedicated Nvidia rig plus Apple or Linux hardware for your workstation. Honestly, you can train basic models just fine on M-Series Max MacBook Pros.
- nightski 2y ago
  A decked out Mac Studio is like $7k for far less GPU power. I find that highly unlikely.
  - inciampati 2y ago
    But you get access to a very large amount of RAM for that price.
  - softfalcon 2y ago
    Don't attack me, I'm not disagreeing with you that an nVidia GPU is far superior at that price point. I simply want to point out that these folks don't really care about that. They want a Mac for more reasons than "performance per watt/dollar" and if it's "good enough", they'll pay that Apple tax. Yes, yes, I know, it's frustrating and they could get better Linux + GPU goodness with an nVidia PC running Ubuntu/Arch/Debian, but macOS is painless for the average science AI/ML training person to set up and work with. There are also known enterprise OS management solutions that business folks will happily sign off on. Also, $7000 is chump change in the land of "can I get this AI/ML dev to just get to work on my GPT model I'm using to convince some VC's to give me $25-500 million?" tldr; they're gonna buy a Mac cause it's a Mac and they want a Mac and their business uses Mac's. No amount of "but my nVidia GPU = better" is ever going to convince them otherwise as long as there is a "sort of" reasonable price point inside Apple's ecosystem.
    - brookst 2y ago
      What Linux setup do you recommend for 128GB of GPU memory?
  - TylerE 2y ago
    A non-decked out Mac Studio is a hell of a machine for $1999. Do you also compare cars by looking at only the super expensive limited editions, with every single option box ticked? I'd also point out that said 3 year old $1999 Mac Studio that I'm typing this on already runs ML models usefully, maybe 40-50% of the old 3000-series Nvidia machine it replaces, while using literally less than 10% of the power and making a tiny tiny fraction of the noise. Oh, and it was cheaper. And not running Windows.
    - bee_rider 2y ago
      They are talking about training models, though. Run is a bit ambiguous, is that also what you mean?
      - TylerE 2y ago
        No. For training the Macs do have some interesting advantages due to the unified memory. The GPU cores have access to all of system RAM (and also the system RAM is ridiculously fast - 400GB/sec when DDR4 is barely 30GB/sec, which has a lot of little fringe benefits of it's own, part of why the Studio feels like an even more powerful machine than it actually is. It's just super snappy and responsive, even under heavy load.) The largest consumer NVidia card has 22GB of useable RAM. The $1999 Mac has 32GB, and for $400 more you get 64GB. $3200 gets you 96GB, and more GPU cores. You can hit the system max of 192GB for $5500 on an Ultra, albeit it with the lessor GPU. Even the recently announced 6000-series AI-oriented NVidia cards max out at 48GB. My understanding is a that a lot of enthusiasts are using Macs for training because for certain things having more RAM is just enabling.
        Der_Einzige 2y ago
        The huge amount of optimizations available on Nvidia and not available on Apple make the reduced VRAM worth it, because even the most bloated of foundation models will have some magical 0.1bit quantization technique be invented by a turbo-nerd which only works on Nvidia. I keep hearing this meme of Mac's being a big deal in LLM training, but I have seen zero evidence of it, and I am deeply immersed in the world of LLM training, including training from scratch. Stop trying to meme apple M chips as AI accelerators. I'll believe it when unsloth starts to support a single non-nvidia chip.
  - andrewmcwatters 2y ago
    Not all of us who own small businesses are out here speccing AMD Ryzen 9s and RTX 4090s for workstations. You can't lug around a desktop workstation.
- skohan 2y ago
  > a dedicated Nvidia rig I am honestly shocked Nvidia has been allowed to maintain their moat with cuda. It seems like AMD would have a ton to gain just spending a couple million a year to implement all the relevant ML libraries with a non-cuda back-end.
  - bee_rider 2y ago
    AMD doesn’t really seem inclined toward building developer ecosystems in general. Intel seems like they could have some interesting stuff in the annoyingly named “OneAPI” suite but I ran it on my iGPU so I have no idea if it is actually good. It was easy to use, though!
    - jimmySixDOF 2y ago
      There are quite a few back and forth X/Twitter storms in teacups between George Hotz / tinygrad and the AMD management about opening up the firmware for custom ML integrations to replace CUDA but last I checked they were running into walls
      - skohan 2y ago
        I don't understand why you would need custom firmware. It seems like you could go a long way just implementing back-ends for popular ML libraries in openCL / compute shaders
- whimsicalism 2y ago
  smaller businesses have no business having a dedicated GPU rig of any kind
Q6T46nT668w6i3m 2y ago
Yes, there’re a handful of apps that use the neural engine to fine tune models to their data.
alfalfasprout 2y ago
Not really (I work on AI/ML Infrastructure at a well known tech company and talk regularly w/ our peer companies). That said, inference on apple products is a different story. There's definitely interest in inference on the edge. So far though, nearly everyone is still opting for inference in the cloud for two reasons: 1. There's a lot of extra work involved in getting ML/AI models ready for mobile inference. And this work is different for iOS vs. Android 2. You're limited on which exact device models will run the thing optimally. Most of your customers won't necessarily have that. So you need some kind of fallback. 3. You're limited on what kind of models you can actually run. You have way more flexibility running inference in the cloud.
- teaearlgraycold 2y ago
  Pytorch actually has surprisingly good support for Apple Silicon. Occasionally an operation needs to use CPU fallback but many applications are able to run inference entirely off of the CPU cores.
  - rcarmo 2y ago
    And there is a lot of work being done with mlx.
  - ein0p 2y ago
    I’ve found it to be pretty terrible compared to CUDA, especially with Huggingface transformers. There’s no technical reason why it has to be terrible there though. Apple should fix that.
    - teaearlgraycold 2y ago
      Yeah. It’s good with YOLO and Dino though. My M2 Max can compute Dino embeddings faster than a T4 (which is the GPU in AWS’s g4dn instance type).
      - ein0p 2y ago
        MLX will probably be even faster than that, if the model is already ported. Faster startup time too. That’s my main pet peeve though: there’s no technical reason why PyTorch couldn’t be just as good. It’s just underfunding and neglect
deanishe 2y ago
Isn't Apple hardware too expensive to make that worthwhile?
- brookst 2y ago
  For business-scale model work, sure. But you can get an M2 Ultra with 192GB of UMA for $6k or so. It's very hard to get that much GPU memory at all, let alone at that price. Of course the GPU processing power is anemic compared to a DGX Station 100 cluster, but the mac is $143,000 less.
- MBCook 2y ago
  You want to buy a bunch of new equipment to do training? Yeah Mac’s aren’t going to make sense. You want your developers to be able to do training locally and they already use Macs? Maybe an upgrade would make business sense. Even if you have beefy servers or the cloud for large jobs.
cafed00d 2y ago
I like to think back to 2011 and paraphrase what people were saying: "Is anyone seriously using gpu hardware to write nl translation software at the moment?" "No, we should be use cheap commodity abundantly available cpus and orchestrate then behind cloud magic to write our nl translation apps" or maybe "no we should build purpose built high performance computing hardware to write our nl translation apps" Or perhaps in the early 70s "is anyone seriously considering personal computer hardware to ...". "no, we should just buy IBM mainframes ..." I don't know. Im probably super biased. I like the idea of all this training work breaking the shackles of cloud/mainframe/servers/off-end-user-device and migrating to run on peoples devices. It feels "democratic".
- seanmcdirmid 2y ago
  I remember having lunch with a speech recognition researcher who was using GPUs to train DNNs to do speech recognition in 2011. It really was thought of as niche back then. But the writing was on the wall I guess in the results they were getting.
  - diego_sandoval 2y ago
    AMD didn't read the wall, unfortunately.
- fennecfoxy 2y ago
  I don't think of examples really apply, because it's more a question of being on "cutting edge" vs personal hardware. For example, running a local model and access to the features of a larger more capable/cloud model are two completely different features therefore there is no "no we should do x instead". I'd imagine that a dumber local model runs and defers to cloud model when it needs to/if user has allowed it to go to cloud. Apple could not compete on "our models run locally privacy is a bankable feature" alone imo, TikTok install base has shown us enough that users prefer content/features over privacy, they'll definitely still need SoA cloud based models to compete.
dylan604 2y ago
Does one need to train an AI model on specific hardware, or can a model be trained in one place and then used somewhere else? Seems like Apple could just run their fine tuned model called Siri on each device. Seems to me like asking for training on Apple devices is missing the strategy. Unless of course, it's just for purely scientific $reasons like "why install Doom on the toaster?" vs doing it for a purpose.
- xanderlewis 2y ago
  It doesn’t require specific hardware; you can train a neural net with pencil and paper if you have enough time. Of course, some pieces of hardware are more efficient than others for this.
avianlyric 2y ago
Apple are. Their “Personal Voice” feature fine tunes a voice model on device using recordings of your own voice. An older example is the “Hey Siri” model, which is fine tuned to your specific voice. But with regards to on device training, I don’t think anyone is seriously looking at training a model from scratch on device, that doesn’t make much sense. But taking models and fine tuning them to specific users makes a whole ton of sense, and an obvious approach to producing “personal” AI assistants. [1] https://support.apple.com/en-us/104993 https://support.apple.com/en-us/104993
- MBCook 2y ago
  They already do some “simple” training on device. The example I can think of is photo recognition in the photo library. It likely builds on something else but being able to identify which phase is your grandma versus your neighbor is not done in Apple‘s cloud. It’s done when your devices are idle and plugged into power. A few years ago it wasn’t shared between devices so each device had to do it themselves. I don’t know if it’s shared at this point. I agree you’re not going to be training an LLM or anything. But smaller tasks limited and scope may prove a good fit.
robbomacrae 2y ago
I don't think this is what you meant but it matches the spec: federated learning is being used by Apple to train models for various applications and some of that happens on device (iphones/ipads) with your personal data before its hashed and sent up to the mothership model anonymously. https://www.technologyreview.com/2019/12/11/131629/apple-ai-personalizes-siri-federated-learning/ https://www.technologyreview.com/2019/12/11/131629/apple-ai-...

andrewmcwatters 2y ago

Yes, it can be more cost effective for smaller businesses to do all their work on Mac Studios, versus having a dedicated Nvidia rig plus Apple or Linux hardware for your workstation. Honestly, you can train basic models just fine on M-Series Max MacBook Pros.

nightski 2y ago
A decked out Mac Studio is like $7k for far less GPU power. I find that highly unlikely.
- inciampati 2y ago
  But you get access to a very large amount of RAM for that price.
- softfalcon 2y ago
  Don't attack me, I'm not disagreeing with you that an nVidia GPU is far superior at that price point. I simply want to point out that these folks don't really care about that. They want a Mac for more reasons than "performance per watt/dollar" and if it's "good enough", they'll pay that Apple tax. Yes, yes, I know, it's frustrating and they could get better Linux + GPU goodness with an nVidia PC running Ubuntu/Arch/Debian, but macOS is painless for the average science AI/ML training person to set up and work with. There are also known enterprise OS management solutions that business folks will happily sign off on. Also, $7000 is chump change in the land of "can I get this AI/ML dev to just get to work on my GPT model I'm using to convince some VC's to give me $25-500 million?" tldr; they're gonna buy a Mac cause it's a Mac and they want a Mac and their business uses Mac's. No amount of "but my nVidia GPU = better" is ever going to convince them otherwise as long as there is a "sort of" reasonable price point inside Apple's ecosystem.
  - brookst 2y ago
    What Linux setup do you recommend for 128GB of GPU memory?
- TylerE 2y ago
  A non-decked out Mac Studio is a hell of a machine for $1999. Do you also compare cars by looking at only the super expensive limited editions, with every single option box ticked? I'd also point out that said 3 year old $1999 Mac Studio that I'm typing this on already runs ML models usefully, maybe 40-50% of the old 3000-series Nvidia machine it replaces, while using literally less than 10% of the power and making a tiny tiny fraction of the noise. Oh, and it was cheaper. And not running Windows.
  - bee_rider 2y ago
    They are talking about training models, though. Run is a bit ambiguous, is that also what you mean?
    - TylerE 2y ago
      No. For training the Macs do have some interesting advantages due to the unified memory. The GPU cores have access to all of system RAM (and also the system RAM is ridiculously fast - 400GB/sec when DDR4 is barely 30GB/sec, which has a lot of little fringe benefits of it's own, part of why the Studio feels like an even more powerful machine than it actually is. It's just super snappy and responsive, even under heavy load.) The largest consumer NVidia card has 22GB of useable RAM. The $1999 Mac has 32GB, and for $400 more you get 64GB. $3200 gets you 96GB, and more GPU cores. You can hit the system max of 192GB for $5500 on an Ultra, albeit it with the lessor GPU. Even the recently announced 6000-series AI-oriented NVidia cards max out at 48GB. My understanding is a that a lot of enthusiasts are using Macs for training because for certain things having more RAM is just enabling.
      - Der_Einzige 2y ago
        The huge amount of optimizations available on Nvidia and not available on Apple make the reduced VRAM worth it, because even the most bloated of foundation models will have some magical 0.1bit quantization technique be invented by a turbo-nerd which only works on Nvidia. I keep hearing this meme of Mac's being a big deal in LLM training, but I have seen zero evidence of it, and I am deeply immersed in the world of LLM training, including training from scratch. Stop trying to meme apple M chips as AI accelerators. I'll believe it when unsloth starts to support a single non-nvidia chip.
- andrewmcwatters 2y ago
  Not all of us who own small businesses are out here speccing AMD Ryzen 9s and RTX 4090s for workstations. You can't lug around a desktop workstation.
skohan 2y ago
> a dedicated Nvidia rig I am honestly shocked Nvidia has been allowed to maintain their moat with cuda. It seems like AMD would have a ton to gain just spending a couple million a year to implement all the relevant ML libraries with a non-cuda back-end.
- bee_rider 2y ago
  AMD doesn’t really seem inclined toward building developer ecosystems in general. Intel seems like they could have some interesting stuff in the annoyingly named “OneAPI” suite but I ran it on my iGPU so I have no idea if it is actually good. It was easy to use, though!
  - jimmySixDOF 2y ago
    There are quite a few back and forth X/Twitter storms in teacups between George Hotz / tinygrad and the AMD management about opening up the firmware for custom ML integrations to replace CUDA but last I checked they were running into walls
    - skohan 2y ago
      I don't understand why you would need custom firmware. It seems like you could go a long way just implementing back-ends for popular ML libraries in openCL / compute shaders
whimsicalism 2y ago
smaller businesses have no business having a dedicated GPU rig of any kind

Q6T46nT668w6i3m 2y ago

Yes, there’re a handful of apps that use the neural engine to fine tune models to their data.

alfalfasprout 2y ago

Not really (I work on AI/ML Infrastructure at a well known tech company and talk regularly w/ our peer companies). That said, inference on apple products is a different story. There's definitely interest in inference on the edge. So far though, nearly everyone is still opting for inference in the cloud for two reasons: 1. There's a lot of extra work involved in getting ML/AI models ready for mobile inference. And this work is different for iOS vs. Android 2. You're limited on which exact device models will run the thing optimally. Most of your customers won't necessarily have that. So you need some kind of fallback. 3. You're limited on what kind of models you can actually run. You have way more flexibility running inference in the cloud.

teaearlgraycold 2y ago
Pytorch actually has surprisingly good support for Apple Silicon. Occasionally an operation needs to use CPU fallback but many applications are able to run inference entirely off of the CPU cores.
- rcarmo 2y ago
  And there is a lot of work being done with mlx.
- ein0p 2y ago
  I’ve found it to be pretty terrible compared to CUDA, especially with Huggingface transformers. There’s no technical reason why it has to be terrible there though. Apple should fix that.
  - teaearlgraycold 2y ago
    Yeah. It’s good with YOLO and Dino though. My M2 Max can compute Dino embeddings faster than a T4 (which is the GPU in AWS’s g4dn instance type).
    - ein0p 2y ago
      MLX will probably be even faster than that, if the model is already ported. Faster startup time too. That’s my main pet peeve though: there’s no technical reason why PyTorch couldn’t be just as good. It’s just underfunding and neglect

deanishe 2y ago

Isn't Apple hardware too expensive to make that worthwhile?

brookst 2y ago
For business-scale model work, sure. But you can get an M2 Ultra with 192GB of UMA for $6k or so. It's very hard to get that much GPU memory at all, let alone at that price. Of course the GPU processing power is anemic compared to a DGX Station 100 cluster, but the mac is $143,000 less.
MBCook 2y ago
You want to buy a bunch of new equipment to do training? Yeah Mac’s aren’t going to make sense. You want your developers to be able to do training locally and they already use Macs? Maybe an upgrade would make business sense. Even if you have beefy servers or the cloud for large jobs.

cafed00d 2y ago

I like to think back to 2011 and paraphrase what people were saying: "Is anyone seriously using gpu hardware to write nl translation software at the moment?" "No, we should be use cheap commodity abundantly available cpus and orchestrate then behind cloud magic to write our nl translation apps" or maybe "no we should build purpose built high performance computing hardware to write our nl translation apps" Or perhaps in the early 70s "is anyone seriously considering personal computer hardware to ...". "no, we should just buy IBM mainframes ..." I don't know. Im probably super biased. I like the idea of all this training work breaking the shackles of cloud/mainframe/servers/off-end-user-device and migrating to run on peoples devices. It feels "democratic".

seanmcdirmid 2y ago
I remember having lunch with a speech recognition researcher who was using GPUs to train DNNs to do speech recognition in 2011. It really was thought of as niche back then. But the writing was on the wall I guess in the results they were getting.
- diego_sandoval 2y ago
  AMD didn't read the wall, unfortunately.
fennecfoxy 2y ago
I don't think of examples really apply, because it's more a question of being on "cutting edge" vs personal hardware. For example, running a local model and access to the features of a larger more capable/cloud model are two completely different features therefore there is no "no we should do x instead". I'd imagine that a dumber local model runs and defers to cloud model when it needs to/if user has allowed it to go to cloud. Apple could not compete on "our models run locally privacy is a bankable feature" alone imo, TikTok install base has shown us enough that users prefer content/features over privacy, they'll definitely still need SoA cloud based models to compete.

dylan604 2y ago

Does one need to train an AI model on specific hardware, or can a model be trained in one place and then used somewhere else? Seems like Apple could just run their fine tuned model called Siri on each device. Seems to me like asking for training on Apple devices is missing the strategy. Unless of course, it's just for purely scientific $reasons like "why install Doom on the toaster?" vs doing it for a purpose.

xanderlewis 2y ago
It doesn’t require specific hardware; you can train a neural net with pencil and paper if you have enough time. Of course, some pieces of hardware are more efficient than others for this.

avianlyric 2y ago

Apple are. Their “Personal Voice” feature fine tunes a voice model on device using recordings of your own voice. An older example is the “Hey Siri” model, which is fine tuned to your specific voice. But with regards to on device training, I don’t think anyone is seriously looking at training a model from scratch on device, that doesn’t make much sense. But taking models and fine tuning them to specific users makes a whole ton of sense, and an obvious approach to producing “personal” AI assistants. [1] https://support.apple.com/en-us/104993 https://support.apple.com/en-us/104993

MBCook 2y ago
They already do some “simple” training on device. The example I can think of is photo recognition in the photo library. It likely builds on something else but being able to identify which phase is your grandma versus your neighbor is not done in Apple‘s cloud. It’s done when your devices are idle and plugged into power. A few years ago it wasn’t shared between devices so each device had to do it themselves. I don’t know if it’s shared at this point. I agree you’re not going to be training an LLM or anything. But smaller tasks limited and scope may prove a good fit.

robbomacrae 2y ago

I don't think this is what you meant but it matches the spec: federated learning is being used by Apple to train models for various applications and some of that happens on device (iphones/ipads) with your personal data before its hashed and sent up to the mothership model anonymously. https://www.technologyreview.com/2019/12/11/131629/apple-ai-personalizes-siri-federated-learning/ https://www.technologyreview.com/2019/12/11/131629/apple-ai-...