4 ms·
You don't even need to train the model just to see if you can infer it at the claimed speed
by lostmsu 5mo ago
You don't even need to train the model just to see if you can infer it at the claimed speed
- gaeld 5mo agoTrue, and for third-party models we'll just re-use their public open weights. There is a time-consuming part, though, that is performed manually by our (human) team: implement the logic of the model in C++ and assembly code in a super-optimized way, co-designed for each specific hardware card. This can take months. We hope to accelerate the process with AI agents, but we're not there yet.