4 ms·
Could be amazing, but it's hard to judge if it will really work with say a 27 B model or larger. We can already get pretty good speed with a 2B model.
by ilaksh 5mo ago
Could be amazing, but it's hard to judge if it will really work with say a 27 B model or larger. We can already get pretty good speed with a 2B model.
- gaeld 5mo agothanks! we explain how it scales to larger models in the last section the OP blog post
- bcjdjsndon 5mo agoShame you stopped short of actually benchmarking that scale though, eh?
- gaeld 5mo agowill do - we are a small team and it takes time to implement and optimize a new model, whatever the size.
- lostmsu 5mo agoYou don't even need to train the model just to see if you can infer it at the claimed speed
- gaeld 5mo agoTrue, and for third-party models we'll just re-use their public open weights. There is a time-consuming part, though, that is performed manually by our (human) team: implement the logic of the model in C++ and assembly code in a super-optimized way, co-designed for each specific hardware card. This can take months. We hope to accelerate the process with AI agents, but we're not there yet.
- bcjdjsndon 5mo agoOh