3 ms·
Surely the fact that IFW uses batching makes it apples to oranges? The MLX-enabled version didn’t batch, did it? That fundamentally changes the nature of the op
by darkteflon 3y ago
Surely the fact that IFW uses batching makes it apples to oranges? The MLX-enabled version didn’t batch, did it? That fundamentally changes the nature of the operation. Wouldn’t the better comparison be faster-whisper?
- modeless 3y agoI don't know exactly what the MLX version did, but you're probably right. I'd love to see the MLX side optimized to the max as well. I'm confident that it would not reach the performance of the 4090, but it might do better. That said, for practical purposes, the ready availability of Nvidia-optimized versions of every ML system is a big advantage in itself.
- darkteflon 3y agoYeah, I think everyone knows that Nvidia is doing a cracker job. But it is good to just be specific about these benchmarks because numbers get thrown around and it turns out people are testing different things. The other thing is that Apple is extracting this performance on a laptop, at ~1/8 the power draw of the desktop Nvidia card. In any event, it’s super cool to see such huge leaps just in the past year on how easy it is to run this stuff locally. Certainly looking very promising.
- modeless 3y agoThe M2 Ultra that got the best numbers that I was comparing to is not in a laptop. Regardless, you're probably right that the power consumption is significantly lower per unit time. However, is it lower per unit work done? It would be interesting to see a benchmark optimized for power. Nvidia's power consumption can usually be significantly reduced without much performance cost. Also, the price difference between a prebuilt 4090 PC and a M2 Ultra Mac Studio can buy a lot of kilowatt hours.