4 ms·
Some people have been fine-tuning mistral 7B and phi-2 on their high-end macs. Unified memory is a hell of a thing. The resulting model here is not spectacular
by rgbrgb 3y ago
Some people have been fine-tuning mistral 7B and phi-2 on their high-end macs. Unified memory is a hell of a thing. The resulting model here is not spectacular but as a proof of concept it's pretty exciting what you get in 3.5 hours on a consumer machine.
- Apple M2 Max 64GB shared RAM
- Apple Metal (GPU), 8 threads
- 1152 iterations (3 epochs), batch size 6, trained over 3 hours 24 minutes
https://www.reddit.com/r/LocalLLaMA/comments/18ujt0n/using_gpus_on_a_mac_m2_max_via_mlx_update_on/ https://www.reddit.com/r/LocalLLaMA/comments/18ujt0n/using_g...
- bearjaws 3y agoWatch out for the memory bandwidth limit on M3, you have to buy the top of the line 16c CPU to get the same memory speed as an M2 Pro.
- justanotheratom 3y agoWhich one would you recommend between these two for fine-tuning OSS LLMs? M2 Ultra Mac Studio vs M3 Max Macbook Pro https://www.apple.com/shop/buy-mac/mac-studio https://www.apple.com/shop/buy-mac/mac-studio https://www.apple.com/shop/buy-mac/macbook-pro/16-inch https://www.apple.com/shop/buy-mac/macbook-pro/16-inch
- petercooper 3y agoThere's little between the M2 and M3 at this point with regards to LLM work, and given no M3 Ultra exists yet, the M2 Ultra would be the way to go considering this task in isolation, and indeed the only option if you wanted to go up to 192GB RAM. If you needed to be portable or had budgetary concerns, then your choice may differ.
- myself248 3y ago> Unified memory is a hell of a thing. Does that also apply to stuff like AMD APUs? Honestly asking, I have no clue.
- jeswin 3y agoNo, they have far less bandwidth. However, I am not sure if getting Apple CPUs would be a good idea. You can get some toy projects running with them, but a pay-as-you-go cloud solution would have been cheaper and faster. I don't find my M1 Max to be particularly competitive.
- brcmthrowaway 3y agoWhats the best cloud lab available for Joe Blow LLM learner?
- kube-system 3y agoAMD and Intel allow their customers to choose lots of different memory configurations and so they prioritize compatibility over bandwidth. Apple's architecture has no support for external memory, they put their memory directly on package and can prioritize for bandwidth.
- ProllyInfamous 3y agoWhat can a bone-stock (16GB) M2Pro Mini do? Would it be correct to assume this takes 8x longer (on account of 1/4 memory and 1/2 GPU), or is training not even possible?
- rgbrgb 3y agoi'm not sure, but I bet you could get an answer to a more specific hardware question like this over at /r/locallama