3 ms·
load model, compute a 1k token response (ie, do a thousand forward passes in sequence, one per token), load a different model, compute a response, I would expe
by QuadmasterXLII 3y ago
load model, compute a 1k token response (ie, do a thousand forward passes in sequence, one per token), load a different model, compute a response,
I would expect the model loading to take basically zero percent of the time in the above workflow