> Another big difference is that fundamental AI insights are what drives these AI innovations. Bigger models, more compute, more data, better compute have driv

> Another big difference is that fundamental AI insights are what drives these AI innovations. Bigger models, more compute, more data, better compute have driven a lot of the performance of ML post-2013. For ChatGPT specifically, the secret sauce is the addition of human-vetted conversations [1]. In general fundamental AI insights come when - (1) somebody does something that works incredibly well and others try to explain it. Transformers are arguably an example of this although there’s a ton of attention literature before that. - (2) a ton of people make incremental steps towards a goal and at some point it becomes actually useful. ChatGPT is an example of this one. The point I’m trying to make is fundamental AI is boring, slow, rarely drives AI innovations and that’s not a bad thing. What drives AI innovations is really good UI and good data both of which are hard to nail down. [1] https://openai.com/blog/chatgpt/#methods https://openai.com/blog/chatgpt/#methods