3 ms·
Can you actually use multiple videocards easily with existing AI model tools?
by holoduke 2y ago
Can you actually use multiple videocards easily with existing AI model tools?
- jbarrow 2y agoYes, though how you do it depends on what you're doing. I do a lot of training of encoders, multimodal, and vision models, which are typically small enough to fit on a single GPU; multiple GPUs enables data parallelism, where the data is spread to an independent copy of each model. Occasionally fine-tuning large models and need to use model-parallelism, where the model is split across GPUs. This is also necessary for inference of the really big models, as well. But most tooling for training/inference of all kinds of models supports using multiple cards pretty easily.
- benob 2y agoYes, multi-GPU on the same machine is pretty straightforward. For example ollama uses all GPUs out of the box. If you are into training, the huggingface ecosystem supports it and you can always go the manual route to put tensors on their own GPUs with toolkits like pytorch.
- qingcharles 2y agoYes. Depends what software you're using. Some will use more than one (e.g. llama.cpp), some commercial software won't bother.
- dpeterson 2y agoI just made a video on this very thing: https://youtu.be/JtbyA94gffc https://youtu.be/JtbyA94gffc