3 ms·
What's the tokens/s on those?
by detrites 4y ago
What's the tokens/s on those?
- minxomat 4y agoWith 16 threads, about 140ms per token for 30B, 300ms per token for 65B I should also mention that 65B should be able to run on 64GB systems. Total system memory consumption on M1 Ultra is about 67GB when running nothing else.