3 ms·
4.3 tokens/s (on M2 Max 96GB Ram) with the 4bit quantisation. But for chat questions i am getting mostly usless answers, duplicates and senseless chatter.
by rspoerri 3y ago
4.3 tokens/s (on M2 Max 96GB Ram) with the 4bit quantisation.
But for chat questions i am getting mostly usless answers, duplicates and senseless chatter.
- tosh 3y agoafaiu it is a base-model, need to wait for an instruct fine tune or prompt in a different way.