3 ms·
I'd say GPU and people trained in knowing what to do while training. There is a lot of money in the world, as we see Microsoft and other companies handing it ou
by pixl97 3y ago
I'd say GPU and people trained in knowing what to do while training. There is a lot of money in the world, as we see Microsoft and other companies handing it out. Power even, if you build in the right place can be cheap, see bitcoin mining.
- patapong 3y agoVery much agree... I think people underestimate how finicky these models are to train. So much so that one of the big announcement around GPT-4 was the fact that OpenAI found a way to make it smoother and more predictable. As evidence by this 114 page log of how engineers resolved problems that came up during the training of OPT-175, which lasted for several months. https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/OPT175B_Logbook.pdf https://github.com/facebookresearch/metaseq/blob/main/projec...