3 ms·
Yep. The model is too big to fit on the two GPUs I have (even though each one is 48GB). So instead it gets split between the GPUs, CPU RAM, and some parts even
by moyix 4y ago
Yep. The model is too big to fit on the two GPUs I have (even though each one is 48GB). So instead it gets split between the GPUs, CPU RAM, and some parts even have to sit on disk (SSD). This makes it much slower than if everything were on the GPU.
I believe there are performance enhancements planned in Accelerate that will allow layers to be preloaded in parallel as other layers are executing inference, which should make things faster: https://github.com/huggingface/accelerate/issues/512#issuecomment-1182480255 https://github.com/huggingface/accelerate/issues/512#issueco...
- zxcb1 4y agoWould NVLink and more RAM improve performance?
- dekhn 4y agoI wasn't going to ask about this until I had it running on my machine but... its very interesting that you can do out of core models with weights in various memory hierarchies. It feels almost like swap/paging. This is exciting!
- lostmsu 4y agoSimple swap via memory-mapped files would have worked probably. The issue is the lack of prefetch.