19 ms·

I don't believe they could, need CUDA and more VRAM...

• terafo 4y ago
  128 gigs is more than enough to load 13B model into. Pytorch has M1 support for some time now so CUDA isn't required.
  • eightysixfour 4y ago
    Does M1 use system memory as VRAM as well?
    • jml7c5 4y ago
      Yes, it's all unified memory. As much as I dislike the loss of socketed RAM, I have to say it's working out well for Apple users so far. I wonder if CXL will change the situation for consumer devices or if it will only be useful at the scale of a server rack.
  • MacsHeadroom 4y ago
    13B fits on a single 3090 (24GB) in int8. 128 gigs might even be enough for 65B, if slowly.