3 ms·
Interesting, how does that work with the multiple GPUs? I'm not familiar with the internal workings of these models, is there anywhere where I can get a brief r
by thot_experiment 4y ago
Interesting, how does that work with the multiple GPUs? I'm not familiar with the internal workings of these models, is there anywhere where I can get a brief rundown of how the processing is split. I imagine there can't me much swapping between GPUs as that seems prohibitively slow? How is the model split such that it can be worked on in parallel by multiple GPUs w/o being bottlenecked by IO?
- borzunov 4y agoI think this is a relevant link for you: https://huggingface.co/transformers/v4.9.0/parallelism.html https://huggingface.co/transformers/v4.9.0/parallelism.html For large LMs, people usually use tensor-parallelism (TP) or pipeline-parallelism (PP). TP involves lots of communication, but uses all GPUs 100% of the time and works faster. PP requires much less communication, but may keep some GPUs idle while they are waiting for data from others. Usually, TP is used when you have good communication channels between GPUs (e.g., they are in one data center and connected with NVLink), while PP is used when communication is a bottleneck (like in Petals, where the data is sent over the Internet, which is much slower than NVLink).
- zone411 4y agoYou can read all the gory details here: https://arxiv.org/pdf/2207.00032.pdf https://arxiv.org/pdf/2207.00032.pdf
- nmitchko 4y agoYou can split the model across devices with huggingface accelerate library. Check out the infer_auto_memory_map metho which will optimize the model for your configuration (multi gpu, ram, nvme) and then run dispatch model on with that memory map.