3 ms·
Running a model is a lot less resource intensive than training it, how many times do you have to loop through the model in order to get an output? If you've got
by mikewarot 4y ago
Running a model is a lot less resource intensive than training it, how many times do you have to loop through the model in order to get an output? If you've got the model with all the weights stored on SSD, is it possible to simply iterate through the layers without keeping it in RAM, using the CPU?
StableDiffusion needed to be in RAM because you were iterating over all the pixels, but if it's just a token at a time coming out of ChatGPT, it might just be possible, right?