14 ms·
If I may tack on a question as someone with zero clue of ML: when, if ever, will someone like me be able to run this on a Mac Studio with a M1 Ultra and 128GB o
by joker99 4y ago
If I may tack on a question as someone with zero clue of ML: when, if ever, will someone like me be able to run this on a Mac Studio with a M1 Ultra and 128GB of ram?
- terafo 4y agoAs far as I can tell you can do it right now, at least for small 13B model, not sure about bigger models.
- eightysixfour 4y agoI don't believe they could, need CUDA and more VRAM...
- terafo 4y ago128 gigs is more than enough to load 13B model into. Pytorch has M1 support for some time now so CUDA isn't required.
- eightysixfour 4y agoDoes M1 use system memory as VRAM as well?
- jml7c5 4y agoYes, it's all unified memory. As much as I dislike the loss of socketed RAM, I have to say it's working out well for Apple users so far. I wonder if CXL will change the situation for consumer devices or if it will only be useful at the scale of a server rack.
- MacsHeadroom 4y ago13B fits on a single 3090 (24GB) in int8. 128 gigs might even be enough for 65B, if slowly.
- TaylorAlexander 4y agoThe 13B parameter model is two 13.5GB chunks, but the 7B parameter model is one 13.5GB chunk so that one might be possible. EDIT: There seems to be a lot of overhead. Here someone struggles to fit the 7B parameter model (13.5GB checkpoint) on a 24GB card: https://github.com/facebookresearch/llama/issues/55 https://github.com/facebookresearch/llama/issues/55 EDIT2: actually using the tip in that link I got it to run on a P5000 with 16GB of ram! It just barely fits so I had to log out of gnome and run it terminal only.
- MacsHeadroom 4y agoYou can run 7B (equal to GPT-3 175B), 13B (better than GPT-3 175B), or 30B (better than anything else publicly available) but probably not 65B with that much RAM on an M1. That would be using the CPU, as the M1 GPU is not yet supported.