3 ms·
So NVLink/NVSwitch pools multiple GPU resources on a single (very expensive) system. A cheaper alternative to that is "offloading", which is a technique that sp
by bogwog 4y ago
So NVLink/NVSwitch pools multiple GPU resources on a single (very expensive) system. A cheaper alternative to that is "offloading", which is a technique that splits the inference process into smaller steps, so it can run on systems with much less resources available... and Petals is a 10x faster alternative to that.
Did I get that right?
This AI stuff is moving very fast and it's hard to keep up, but it's all fascinating.
- parentheses 4y agoMatches my understanding also. Can someone in the know confirm?
- nmitchko 4y agoYour understanding is correct, but I can't vouch for the claim's accuracy. This could make the execution of models much more accessible to people who don't have a 4x / RTX3090 or better in a ML or mining rig ..
- cardine 4y agoOffloading is when the computation is done on the CPU instead of the GPU. DeepSpeed is an example of this.
- borzunov 4y agoIn case of offloading, the computations are usually still performed on GPU, but the model is hosted in RAM/SSD instead of the GPU memory (and its chunks are copied to the GPU memory when necessary).
- cardine 4y agoA lot of computation is offloaded to the CPU, such as gradients and optimizer states. You are right though that quite a bit of computation is still done on the GPU.
- rolenthedeep 4y agoI remember when GPUs were starting to support arbitrary computation and offloading meant shifting work away from the CPU.
- borzunov 4y agoYou're right. This comment explains offloading in more detail: https://news.ycombinator.com/item?id=34216213 https://news.ycombinator.com/item?id=34216213