This article assumes the cost of training GPT-3 will remain high. I think it will depend on how quickly we can make these models more efficient vs how much bigg

This article assumes the cost of training GPT-3 will remain high. I think it will depend on how quickly we can make these models more efficient vs how much bigger we should make them to keep improving the quality of output. At some point, the quality will plateau, and the engineering improvements should catch up. For example, the 2013 VGG model has 160M parameters, and achieves the same Imagenet accuracy as a 16M parameter 2020 model (or maybe even 1.6M parameter one, I haven't checked the state of the art in efficiency). The algorithmic improvements will be combined with hardware improvements. Once the cost of training falls below some threshold, people will apply this technology to various domains, and entirely new applications will appear, creating a lot of room for new startups.