2 ms·
I run LLaMA 70B and 120B (frankenmerges) locally on a 2022 Mac Studio with M1 Ultra and 12Gb RAM. It gives ~7 tok/s for 120B and ~9.5 tok/s for 70B. Note that
by int_19h 3y ago
I run LLaMA 70B and 120B (frankenmerges) locally on a 2022 Mac Studio with M1 Ultra and 12Gb RAM. It gives ~7 tok/s for 120B and ~9.5 tok/s for 70B.
Note that M1/M2 Ultra is quite a bit faster than M3 Max, mostly due to 800 Gb/s vs 400 Gb/s memory