3 ms·
even if their chips weren't as good, what stops a company from training a large model for a very long time in less capable hardware? Is there a way to overcome
by seydor 1y ago
even if their chips weren't as good, what stops a company from training a large model for a very long time in less capable hardware? Is there a way to overcome memory limitations somehow?
- umgefahren 1y agoIn general yes, you can (and do) shard the model over multiple GPUs. If you want to do that yourself look at DeepSpeed or FSDP . There is a communication overhead though and the speed at which the GPUs can communicate is key. Thats where NVLink comes in btw. So yes, it’s actually what you can and do do. However this limits your ability to iterate on the models quickly and from what I‘ve read a lot of times the foundational labs throw out their models because by the time they are done training they are already outdated.
- checker659 1y agoNVLink isn’t magic
- rfoo 1y ago> the speed at which the GPUs can communicate is key Guess what a telco equipment company is good at :p