4 ms·
Training foundation models is very expensive. Easily a few million per model with no guarantee of success.
by spacebanana7 3y ago
Training foundation models is very expensive. Easily a few million per model with no guarantee of success.
- wslh 3y agoAgain, we can assume a distribute training if that is feasible.
- spacebanana7 3y agoUnfortunately, I don’t think distributed training is feasible. My understanding is that these models are trained using some form of gradient descent. This requires knowledge of the model’s prior state and current training error to adjust parameter values. With distributed training it’s difficult, to point of impossible, to communicate the state of the model across large distances because these models are so large. Take my comment with a grain of salt because I’m not an expert in this area. I’d very much enjoy being wrong about this because distributed training would unlock so much potential.