3 ms·
128 gigs is more than enough to load 13B model into. Pytorch has M1 support for some time now so CUDA isn't required.
by terafo 4y ago
128 gigs is more than enough to load 13B model into. Pytorch has M1 support for some time now so CUDA isn't required.
- eightysixfour 4y agoDoes M1 use system memory as VRAM as well?
- jml7c5 4y agoYes, it's all unified memory. As much as I dislike the loss of socketed RAM, I have to say it's working out well for Apple users so far. I wonder if CXL will change the situation for consumer devices or if it will only be useful at the scale of a server rack.
- MacsHeadroom 4y ago13B fits on a single 3090 (24GB) in int8. 128 gigs might even be enough for 65B, if slowly.