Here's a random collection of thoughts that I have about the progress of Artificial Intelligence (AI): Machine learning (ML) and Deep Learning (DL) in particul

Here's a random collection of thoughts that I have about the progress of Artificial Intelligence (AI): Machine learning (ML) and Deep Learning (DL) in particular benefit from fast computer chips. The most impressive gains in AI in the 2010s (Computer Vision & Natural Language Processing) were made thanks to DL. There's a famous post by AI researcher Rich Sutton, which summarises this fairly neatly [1]. Now, this connection between DL and fast chips would tie the progress of AI pretty tightly to the progress of Moore's Law [2]. There's some compelling evidence that Moore's Law is at least slowing down [3]. On the other hand, there are industry experts like Jim Keller who pretty strongly disagree with these assessments [4] and even TSMC seems to be bullish on being able to keep up with Moore's Law [5]. Some estimates of GPT-3s training cost put it in the range of ~5-10 million dollars [6]. It's hard to say how big of an impact on the economy GPT-3 will have. It's probably safe to assume though, that OpenAI is already working on GPT-4. The jump in parameter sizes from GPT-2 to 3 was roughly 100x (1.5 billion vs. 175 billion) and I'm assuming the price of training the model increased in roughly the same proportion (I might be wrong here and if anyone can point me to evidence on this, it would be much appreciated). With these assumptions, and provided that GPT-4 won't be affected by diminishing returns of adding more parameters (big if), the price for it would be somewhere between 500 million and a billion dollars. That's still not an insane amount of money to put into R&D, but you'd probably want it to be at least be somehow economically viable to be able to justify putting a hundred billion dollars into GPT-5. All this is to say, that I find making predictions of the progress of AI really hard due to the large amount of uncertainty related to the field and the underlying technologies (mainly the hardware). [1]: http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html [2]: https://arxiv.org/pdf/2007.05558.pdf https://arxiv.org/pdf/2007.05558.pdf [3]: https://p4.org/assets/P4WS_2019/Speaker_Slides/9_2.05pm_John_Hennessey.pdf https://p4.org/assets/P4WS_2019/Speaker_Slides/9_2.05pm_John... [4]: https://www.youtube.com/watch?v=oIG9ztQw2Gc https://www.youtube.com/watch?v=oIG9ztQw2Gc [5]: https://www.nextplatform.com/2019/09/13/tsmc-thinks-it-can-uphold-moores-law-for-decades/ https://www.nextplatform.com/2019/09/13/tsmc-thinks-it-can-u... [6]: https://venturebeat.com/2020/06/11/openai-launches-an-api-to-commercialize-its-research/ https://venturebeat.com/2020/06/11/openai-launches-an-api-to...