2 ms·
The models are too large to fit on a desktop GPU's VRAM. Progress would either require smaller models (MoE might help here? not sure) or bigger VRAM. For exampl
by HPsquared 2y ago
The models are too large to fit on a desktop GPU's VRAM. Progress would either require smaller models (MoE might help here? not sure) or bigger VRAM. For example training a 70 billion parameter model would require at least 140GB of VRAM in each system, whereas a large desktop GPU (4090) has only 24GB.
You need enough memory to run the unquantized model for training, then stream the training data through - that part is what is done in parallel, farming out different bits of training data to each machine.
- mr_toad 2y agoData parallel training is not the only approach. Sometimes the model itself needs to be distributed across multiple GPU. https://www.microsoft.com/en-us/research/blog/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters/ https://www.microsoft.com/en-us/research/blog/zero-deepspeed... The communications overhead of doing this over the internet might be unworkable though.
- htrp 2y agoor if the internet became significantly faster fiber connections