The parameter/dataset arms race producing emergent intelligence rather than just bigger models is a fantasy. However without a good/better explanation in the m

The parameter/dataset arms race producing emergent intelligence rather than just bigger models is a fantasy. However without a good/better explanation in the meantime , really interesting capabilities emerge at higher parameters Specifically to your point of arithmetic— here is GPT3 performing “zero shot” (ie no examples) arithmetic: https://photos.app.goo.gl/pf6fHcrio7QFtU4S8 https://photos.app.goo.gl/pf6fHcrio7QFtU4S8 The “completion” includes an answer to the problem but also provides a natural language summary (GPT3) GPT3 has ~170billion parameters (GPT2 ~1.5B) & somehow is capable of many-digit arithmetic w/o fine-tuning or task-specific training. (Its training data was “messy” from Common Crawl like for search engines) Microsoft’s Turing-NLG has ~17billion parameters & was successful only in 1 of every 2 attempts of 2-digit arithmetic and only successful less than 10% of the time for higher numbers of digits.