4 ms·
for 3.8? I can get 40 tok/sec on it (M3 Max 64GB), but I don't use it because I can get 90 tok/sec with 3.6 MoE (MTP + 6 bit quant), and I don't notice any qual
by seanmcdirmid 2mo ago
for 3.8? I can get 40 tok/sec on it (M3 Max 64GB), but I don't use it because I can get 90 tok/sec with 3.6 MoE (MTP + 6 bit quant), and I don't notice any quality improvements for my tasks using a dense model.
Did you ask Gemini or DeepSeek to look at your oMLX server log to see what was going on? This can help a lot if it is just a misconfiguration.
- robotresearcher 2mo agoJust did the fun thought experiment of playing back this comment thread in my head ten years in the past and it's amusingly incomprehensible.
- dcastm 2mo agoWhat’s the context size?
- seanmcdirmid 2mo ago128k, 256k is also possible, but my tok/s drops off and the performance isn’t good.