4 ms·
As far as I can tell you can do it right now, at least for small 13B model, not sure about bigger models.
by terafo 4y ago
As far as I can tell you can do it right now, at least for small 13B model, not sure about bigger models.
- eightysixfour 4y agoI don't believe they could, need CUDA and more VRAM...
- terafo 4y ago128 gigs is more than enough to load 13B model into. Pytorch has M1 support for some time now so CUDA isn't required.
- eightysixfour 4y agoDoes M1 use system memory as VRAM as well?
- jml7c5 4y agoYes, it's all unified memory. As much as I dislike the loss of socketed RAM, I have to say it's working out well for Apple users so far. I wonder if CXL will change the situation for consumer devices or if it will only be useful at the scale of a server rack.
- MacsHeadroom 4y ago13B fits on a single 3090 (24GB) in int8. 128 gigs might even be enough for 65B, if slowly.
- TaylorAlexander 4y agoThe 13B parameter model is two 13.5GB chunks, but the 7B parameter model is one 13.5GB chunk so that one might be possible. EDIT: There seems to be a lot of overhead. Here someone struggles to fit the 7B parameter model (13.5GB checkpoint) on a 24GB card: https://github.com/facebookresearch/llama/issues/55 https://github.com/facebookresearch/llama/issues/55 EDIT2: actually using the tip in that link I got it to run on a P5000 with 16GB of ram! It just barely fits so I had to log out of gnome and run it terminal only.