4 ms·
> Then I realized that the 4070 has more vram than most consumers do The M1 and M2 Apple processors might change that equation. The 32gb and 64gb MacBooks can
by RcouF1uZ4gsC 3y ago
> Then I realized that the 4070 has more vram than most consumers do
The M1 and M2 Apple processors might change that equation.
The 32gb and 64gb MacBooks can run inference on a lot of these big models.
They might be quite a bit slower in TFLOPS than a 4070, but if they can run the model while the 4070 can't run it due to limited RAM, then they come out the winner
- pram 3y agoYes I could run llama 65b on my M1 Max with 64GB of ram.
- washadjeffmad 3y agoThat's true, and you do make a disclaimer about performance, but often when this is brought up, people either haven't actually tried MPS or are unaware that a GPU isn't required for inference. The long and short of it is, right now, CUDA beats TFLOPS. Now for training, $10K for an M2 with 192GB memory could break even somewhere around a dozen models with current costs, iff the compute is not saturated. The limit of memory bandwidth is only a function of the compute; it doesn't scale infinitely.
- lukevp 3y agoI do also have a MacBook Pro with 32 gigs of Ram and an M2 Studio with 64 gigs, so if a great model comes out that supports unified memory, I should be good there!