Historically, datasets have been a moat. Google had a massive head start from having a massive search index and user data. Then access to compute became the moa

Historically, datasets have been a moat. Google had a massive head start from having a massive search index and user data. Then access to compute became the moat - fully training a trillion-parameter language model has only been in reach for megacorps. But now, there's a ton of publicly-available datasets, and LLaMA showed that you don't need massive numbers of parameters.