2 ms·
I usually get mlx versions of models and didn't realize I was using non-mlx - misattributing the performance boost to speculative decoding. With dflash-mlx lib
by jasonjmcghee 3mo ago
I usually get mlx versions of models and didn't realize I was using non-mlx - misattributing the performance boost to speculative decoding.
With dflash-mlx library, with dflash disabled 3.6-35b-a3b mlx model I'm getting ~60 t/s on low power and >100 t/s on high power.
Compared with previous message being official qwen huggingface release (non-mlx) using lm studio.