3 ms·
I have 64gb on my 5 year old thinkpad. What kind of performance (tokens per sec) I could expect on that nowadays for a 70B model?
by beefield 3y ago
I have 64gb on my 5 year old thinkpad. What kind of performance (tokens per sec) I could expect on that nowadays for a 70B model?
- jerrygenser 3y agoLlama cpp speed is dramatically improved by avx instructions. If your CPU has those it would be much faster than not. And if it doesn't you need to do some workarounds with compiling and it gets a bit harder to run.