I keep wanting to write a long explanation of just why this is so... silly? to read? But Gwern has already done the hard work. [0] The only other bit I'd like

I keep wanting to write a long explanation of just why this is so... silly? to read? But Gwern has already done the hard work. [0] The only other bit I'd like to mention is that GPT-3 uses exactly none of the new techniques that have been coming out in the last two years that would have significant impact on text generation. From working methods to apply GANs to text, to far more efficient transformer models that can handle longer sequences. For instance [1] [2] [3] for better direction, or [4] [5] [6] for efficiency. Or perhaps the outside view might help. After seeing GPT-2 last year, did you expect GPT-3 would work as well as it does after just naively scaling up the number of parameters with nothing else? [0https://www.gwern.net/newsletter/2020/05#gpt-3 https://www.gwern.net/newsletter/2020/05#gpt-3 [1 ] http://arxiv.org/abs/1905.09922 http://arxiv.org/abs/1905.09922 [2] https://github.com/anonymous1100/D_Improves_G_without_Updating_G https://github.com/anonymous1100/D_Improves_G_without_Updati... [3] http://arxiv.org/abs/2006.04643 http://arxiv.org/abs/2006.04643 [4] http://arxiv.org/abs/2007.14062 http://arxiv.org/abs/2007.14062 [5] http://arxiv.org/abs/2006.04768 http://arxiv.org/abs/2006.04768 [6] http://arxiv.org/abs/2002.05645 http://arxiv.org/abs/2002.05645