So many interesting take-aways here. * The "bigger models" line of research is probably exhausted for now: "insofar as we trust our equation, this entire line

So many interesting take-aways here. * The "bigger models" line of research is probably exhausted for now: "insofar as we trust our equation, this entire line of research -- which includes GPT-3, LaMDA, Gopher, Jurassic, and MT-NLG -- could never have beaten Chinchilla, no matter how big the models got" * We don't really know how much written data is available. Even Google - which has access to data repositories like scanned books that they cannot share - seems to have trouble getting consistently sized datasets for unknown reasons. * There seems to be as much (more?) written English available in books than on the entire web. MassiveText "scrape" = 506B tokens, MassiveText "books" = 560B tokens