3 ms·
Running a 125M model on-device at that speed is impressive. How much did you have to optimize the model to get that performance on an iPhone?
by gaya3bollineni 2mo ago
Running a 125M model on-device at that speed is impressive. How much did you have to optimize the model to get that performance on an iPhone?
- simedw 2mo agoThe biggest speed improvement came from changing the note representation when I switched to compound note events: roughly 5× fewer autoregressive passes per note. For the current model I’m using Core ML, which optimizes the kernels the first time you run it. I haven’t actually spent that much time tuning performance beyond that.
- Naitik88 2mo agoThe answer about changing the note representation was interesting. Sometimes a change in how the problem is represented ends up giving a much bigger improvement than trying to optimize the model itself.