3 ms·
With 16 threads, about 140ms per token for 30B, 300ms per token for 65B I should also mention that 65B should be able to run on 64GB systems. Total system memo
by minxomat 4y ago
With 16 threads, about 140ms per token for 30B, 300ms per token for 65B
I should also mention that 65B should be able to run on 64GB systems. Total system memory consumption on M1 Ultra is about 67GB when running nothing else.