2 ms·
9.66 token/s inference on M2 Ultra with 4bit quants, ~90gb RAM used: https://twitter.com/awnihannun/status/1778054275152937130 https://twitter.com/awnihannun/st
by tosh 3y ago
9.66 token/s inference on M2 Ultra with 4bit quants, ~90gb RAM used: https://twitter.com/awnihannun/status/1778054275152937130 https://twitter.com/awnihannun/status/1778054275152937130