2 ms·
I honestly don't see the problem. "just to serve a single model" could be easily fixed by adding a single LPDDR4 channel per LPU. Then you can reload the model
by imtringued 3y ago
I honestly don't see the problem.
"just to serve a single model" could be easily fixed by adding a single LPDDR4 channel per LPU. Then you can reload the model sixty times per second and serve 60 different models per second.
- treesciencebot 3y agoper-chip compute is not the main thing this chip innovates for fast inference, it is the extremely fast memory bandwith. when you do that, you'll loose all of that and will be much worse off than any off the shelf accelerators.
- QuadmasterXLII 3y agoload model, compute a 1k token response (ie, do a thousand forward passes in sequence, one per token), load a different model, compute a response, I would expect the model loading to take basically zero percent of the time in the above workflow