I think from the perspective of a Google researcher/engineer, it must be alarming to see the crazy explosion going on w/ LLM development. We've gone from just o

I think from the perspective of a Google researcher/engineer, it must be alarming to see the crazy explosion going on w/ LLM development. We've gone from just one or two weirdos implementing papers (eg https://github.com/lucidrains?tab=repositories https://github.com/lucidrains?tab=repositories who's amazing) to now an explosion where basically every dev and PhD student is hacking on neat new things and having a field day and "lapping" (eg productizing) what Google Research was previously holding back. And we're also seeing amazing fine-tunes/distillations of very useful/capable smaller models - there's no denying that things have gotten better and more importantly, cheaper way faster than anyone expected. That being said, most of these are being trained with the help of GPT-4, and so far nothing I've seen being done publicly (and I've been spending a lot of time tracking these https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYpb63e1ZR3aePczz3zlbJW-Y4/edit?usp=sharing https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp...) gets close in quality/capabilities to GPT-4. I'm always rooting for the open source camp, but I think the flip-side is that there are still only a handful of organizations in the world that can train a >SoTA foundational model, and that having a mega-model is probably a huge force multiplier if you know how to take advantage of it (eg, I can't imagine that OpenAI has been able to release software at the pace they have been without leveraging GPT-4 for co-development; also can you distill or develop capable smaller models without a more capable foundational model to leverage?). Anthropic for example has recently taken the flip side of the "no moat" argument, arguing that there is a potential winner-take-all scenario where the lead may become insurmountable if one group gets too far ahead in the next couple years. I guess what we'll just have to see, but my suspicion, is that the crux to the "moat" question is going to be whether the open source approach can actually train a GPT-n++ system.