3 ms·
What is a decent tok/s? Your best bet is to run a quantized 7b model using LMStudio or Ollamma on your M1 Mac, like neural chat v3.1 from Mistral/Intel.
by brianjking 3y ago
What is a decent tok/s?
Your best bet is to run a quantized 7b model using LMStudio or Ollamma on your M1 Mac, like neural chat v3.1 from Mistral/Intel.