3 ms·

Noob question - why is it must to load the whole model all at once in VRAM? Why it can't be streamed from disk layer by layer, a sliding window of it, computed

by wg0 3y ago

Noob question - why is it must to load the whole model all at once in VRAM? Why it can't be streamed from disk layer by layer, a sliding window of it, computed, temporary results held, offloaded it back and load the next window. Repeat till whole inference is done. Also, if these are so much repetitive calculations in nature that you need CUDA cores to compute then why the inference can't be streamed and spread across a cluster of machines each having multiple commodity GPUs where one central "conductor/orchestrator" machines collects all the results from all the cluster participants?

SCHiM 3y ago
That is actually possible. For example, someone wrote python code to do this for the massive open source model BLOOM. However, it's still slow as tar. When I was running the BLOOM model I think my inference time was 1 token / m. See: https://towardsdatascience.com/run-bloom-the-largest-open-access-ai-model-on-your-desktop-computer-f48e1e2a9a32 https://towardsdatascience.com/run-bloom-the-largest-open-ac...
foxes 3y ago
No, its not necessary to load it in all at once, you could in theory make it stream in chunks of layers.
p1esk 3y ago
Yes, it can be done and is done when needed, but your life will be simpler if you can fit your models in GPU memory. Even better if you can fit it in a single GPU memory.
swatcoder 3y ago
It's not a formal must, it's a practical "must". Paging and streaming and distribution (and collection) all bring their own overhead, and given the architecture of most of these systems, the overhead of those techniques is very large. So for researchers who can access a system with enough memory, which is most of the people professionally exploring these models, nobody needs to invest much preparatory effort into those other techniques or reducing their overhead. For them, there's basically no ROI for it. But a lot of that work to optimize for those other techniques is gradually being accomplished in the open source community, where people don't have access to expensive clouds and lab systems. It takes time though, and can still only achieve so much.

by wg0 3y ago

SCHiM 3y ago

That is actually possible. For example, someone wrote python code to do this for the massive open source model BLOOM. However, it's still slow as tar. When I was running the BLOOM model I think my inference time was 1 token / m. See: https://towardsdatascience.com/run-bloom-the-largest-open-access-ai-model-on-your-desktop-computer-f48e1e2a9a32 https://towardsdatascience.com/run-bloom-the-largest-open-ac...

foxes 3y ago

No, its not necessary to load it in all at once, you could in theory make it stream in chunks of layers.

p1esk 3y ago

Yes, it can be done and is done when needed, but your life will be simpler if you can fit your models in GPU memory. Even better if you can fit it in a single GPU memory.

swatcoder 3y ago

It's not a formal must, it's a practical "must". Paging and streaming and distribution (and collection) all bring their own overhead, and given the architecture of most of these systems, the overhead of those techniques is very large. So for researchers who can access a system with enough memory, which is most of the people professionally exploring these models, nobody needs to invest much preparatory effort into those other techniques or reducing their overhead. For them, there's basically no ROI for it. But a lot of that work to optimize for those other techniques is gradually being accomplished in the open source community, where people don't have access to expensive clouds and lab systems. It takes time though, and can still only achieve so much.