The value of data decreases. There are diminishing returns to dataset sizes (specifically, for GPT, it's a power law). The more public data there is, and the mo

The value of data decreases. There are diminishing returns to dataset sizes (specifically, for GPT, it's a power law). The more public data there is, and the more data in general, the less any additional piece of private data is worth. Why would any tech company with access to lots of private data, like Google, risk the enormous backlash and legal penalties when they can so easily scrape terabytes and terabytes of text from public datasets to train uncannily intelligent models like GPT-3? Even hobbyists can make the necessary datasets from Common Crawl et al (see EleutherAI's terabyte+ of clean text in The Pile dataset). This is in addition to the ever greater sample-efficiency of bigger models, which learn eerily fast from just a handful of datapoints, rendering the supposed 'moats' of big giant proprietary datasets ever more moot...