4 ms·
> The ability of complex reasoning with chain-of-thought is likely to be a magical side product of training on code Seems reasonable, and magical indeed. Can a
by tunnuz 4y ago
> The ability of complex reasoning with chain-of-thought is likely to be a magical side product of training on code
Seems reasonable, and magical indeed. Can any expert on the topic comment on this hypothesis?
- keepquestioning 4y agoThis would win the Nobel Prize if true
- pmayrgundter 4y agoSimilar question, tho by my read, chain-of-thought seems to be standalone and the primary breakthrough. I asked the author[1]. I've worked in AI&ML since the 90s and both of these do seem much more intuitively promising than simply making larger nets. This seems borne out by the discussion in his previous post, that in Oct '21 OpenAI ran GPT-3 on it, solving 35% and they estimated: "it appears likely that the 175 [billion link neural network] model would require at least two additional orders of magnitude of training data to reach an 80% solve rate.” But instead 85% was achieved within a year using chain-of-thought, not a larger model.[2] [1]https://twitter.com/PMayrgundter/status/1603224294124920832 https://twitter.com/PMayrgundter/status/1603224294124920832 [2]https://yaofu.notion.site/A-Closer-Look-at-Large-Language-Models-Emergent-Abilities-493876b55df5479d80686f68a1abd72f https://yaofu.notion.site/A-Closer-Look-at-Large-Language-Mo...