4 ms·
Mixtral 8x22B on MLX
- deleted 3y ago[deleted]
- tosh 3y ago9.66 token/s inference on M2 Ultra with 4bit quants, ~90gb RAM used: https://twitter.com/awnihannun/status/1778054275152937130 https://twitter.com/awnihannun/status/1778054275152937130
- rspoerri 3y ago4.3 tokens/s (on M2 Max 96GB Ram) with the 4bit quantisation. But for chat questions i am getting mostly usless answers, duplicates and senseless chatter.
- tosh 3y agoafaiu it is a base-model, need to wait for an instruct fine tune or prompt in a different way.