2 ms·
It depends on model size I think, but yeah, from my understanding at ~30B and below Q6 or even Q4 will get you 95%+ of the way there
by mdgld 3mo ago
It depends on model size I think, but yeah, from my understanding at ~30B and below Q6 or even Q4 will get you 95%+ of the way there