5 ms·
Not just connect something. For example, at a certain size, GPT models start to "learn" how to do basic arithmetic (addition), even for numbers with multiple d
by spion 4y ago
Not just connect something.
For example, at a certain size, GPT models start to "learn" how to do basic arithmetic (addition), even for numbers with multiple digits they've never encountered before.
It might look like a small thing, what with computers being quite able to do arithmetic at the base level. But this is a language model, so its a bit different. It learns how to add numbers without "carrying the 1" first, then at a certain larger size also learns to carry the 1, then when even larger it learns to do that across multiple digits... So its not just blindly guessing, its learning the rules of the game (and in some cases some quite complex rules) by building a model of the world made of words.
And the model of digits and addition is just one small bit most likely, as the training space doesn't contain much of that - writing about adding numbers is pretty boring after all. The full model must encode rules and knowledge about a variety of complex things to be able to make reliable predictions. It probably also contains true generalizations that humanity hasn't thought of before, as well as specializations of those generalizations that could be immensely useful.
- LakshyAAAgrawal 4y agoHey! That's a very interesting explanation, could you please provide any references for further/detailed reading on model's abilities to learn to add numbers, with carry and finally with carry across digits?
- spion 4y agoThere is some commentary in the GPT-3 paper https://arxiv.org/pdf/2005.14165.pdf https://arxiv.org/pdf/2005.14165.pdf (figure 3.10 and table 3.10) note: I may have extrapolated a bit more than strictly correct from those two bits, but the accuracy being about 50% indicates carrying is the issue. GPT-3 is at around carrying twice, but hasn't generalized full carrying yet.
- ChatGTP 4y agoThe only issue with the math example would be that we don't do complex math in English. Of course it can learn other "languages" but as you said there is less "written" complex math It probably also contains true generalizations that humanity hasn't thought of before, as well as specializations of those generalizations that could be immensely useful. I thought this is quite exciting, I actually think this will become a field of research in itself, and that might spur completely novel new industries and field of technology, research etc. I'm being optimistic here but I hope we're on the cusp of a greater future, rather than a dystopian one brought on by terminators and AI that replaces all work :)
- hprotagonist 4y agoBorges, of course, kind of skewered your hopes about 80 years ago.
- ChatGTP 4y agoBit of context might be nice.
- zerocrates 4y agoI assume they're referring to Jorge Luis Borges's short story "The Library of Babel," which imagines an enormous library that has a book for every possible sequence of letters and punctuation, all of a certain length (a few hundred pages). The library therefore contains all useful books on any topic in any language (as long as it's covered by the alphabet), but also all useless or inaccurate ones, and of course a vast sea of gibberish. All the knowledge you could want is there, yet unattainable.
- ChatGTP 4y agoThank you :)
- phire 4y agoWorth pointing out that it's not 100% accurate at arithmetic, that becomes quite clear when you ask it to calculate large numbers. It's result is usually close, but not perfect. So it hasn't learned the exact rules for arithmetic, it's learned rules for approximating arithmetic to a decent level of accuracy. Similar to how humans can know the approximate result of an equation before doing the actual math (though GPT is way more precise than humans)
- mirekrusin 4y agoThey need to add support for co-brains - functions that are available for weightware brain to use. Ie if could and knew how to use wolfram alpha - it’d boost performance dramatically.
- rvba 4y agoI wonder if someone could make some standardized way to describe/write mathematical proofs - in a very formal language and then train a model that will try to find more proofs for open questions.
- theturtletalks 4y agoWhat if it solves the P versus NP problem?
- angelbar 4y agoWe go home to rest.... no more working for us....
- krainboltgreene 4y agoOkay but that's not "learning" how to do basic math in the same way I can't "learn" japanese just by mimicking the mouth noises. Yeah I'll get some of the pronunciation right sometime, maybe even get it in the right order, but only for the listener. To me, it's still just mouth noises.
- qingdao99 4y agoIf you were given the task "mimic the sound of Japanese as best you can", at first, you would just learn the basic phonology and just try and mimic the general sounds of the language. You would get good at that, and eventually you would be perfectly mimicking the pronunciation of Japanese phonemes. At some point, it would become worth your time to actually learn how different sounds are put together, e.g. Japanese can't have an S sound followed by a T sound. After that, you may start to learn how the different syllables interact. And at some point, you would learn how words are put together to form sentences. At a certain point, there really is no difference between "being really good at mimicking the sound of Japanese" and actually knowing it, because in order to mimic it to a high level you will have to actually know it. "Mimic the sound of Japanese" is the equivalent task here to "predict the next token in this text".
- Retric 4y agoWhere that argument falls apart is when you’re talking about something that’s still bad a mimicry. ChatGPT still completely fails the touring test it’s not even vaguely close to that threshold. For now it’s slightly better than fake crowd noise you’re hearing in movies, but still frequently just gibberish.
- Kiro 4y agoChatGPT passes the Turing test and is not even the first bot to have done so. No idea why you're so keen on downplaying it but soon there won't be anyone left that you will be able to convince that this is no big deal. You're fighting an uphill battle.