Training GPT-3 costs only a few million dollars. Scaling up is still pretty cheap. I wouldn't be surprised if we see a quadrillion parameter model in 5 years, g

Training GPT-3 costs only a few million dollars. Scaling up is still pretty cheap. I wouldn't be surprised if we see a quadrillion parameter model in 5 years, given the potential value. Google has actually already trained a trillion parameter model IIUC [1], though that was a Mixture of Experts so was way cheaper to train. [1] https://twitter.com/lepikhin/status/1278174444528132098 https://twitter.com/lepikhin/status/1278174444528132098