16 ms·

I've seen this objection raised a lot, but I think it betrays a misunderstanding of what GPT-3 is capable of doing. The best, in fact the only way to generate

I've seen this objection raised a lot, but I think it betrays a misunderstanding of what GPT-3 is capable of doing. The best, in fact the only way to generate truly convincing text output on most subjects is to understand, on some level, what you're writing about. In other words, to create a higher level abstraction than simply "statistically speaking, this word seems to follow that one". Once you start to encode that words map to concepts, you can use the resulting conceptual model to create output which is conceptually consistent, then map it backwards to words. There is what humans do with sensory data, and there is good evidence that GPT-3 is doing this too, to some degree. Take simple arithmetic, such as adding two and three digit numbers. GPT-2 could not do this very successfully. It did indeed look like it was treating it as a "find the textual pattern" problem. But GPT-3 is much more successful, including at giving correct answers to arithmetic problems that weren't in its training set. So what changed? We aren't sure, but the speculation is that in the process of training, GPT-3 found that the best strategy to correctly predicting the continuation of arithmetic expressions was to figure out the rules of basic arithmetic and encode them in some portion of its neural network, then apply them whenever the prompt suggested to do so. If this is the case, and it remains speculation at this point, would you still argue that GPT-3 doesn't "understand" arithmetic, on some level? I would argue that this abstraction, this mapping of words onto higher-level concepts, which can then be manipulated to solve more complex problems, is exactly what intelligence is, once you strip away biologically-biased assumptions. Certainly, at this point GPT-3's conceptual understanding remains somewhat primitive and unstable, but the fact that it exhibits it at all, and sometimes in spookily impressive ways, is what has people excited and worried. We have produced AIs that can perhaps think conceptually about relatively narrow topics like playing Go, but we have never before created one that can do so one such a wide range of topics. And there is no suggestion that GPT-3's level of ability represents a maximum. GPT-4 and beyond will be more powerful, meaning that it can mine more and more powerful conceptual understanding from their training data.

simonh 6y ago
The example of simple arithmetic is interesting. I think you might be right, that is evidence that GPT-3 might be generating what we might consider models of it's input data. Very simple, primitive and fragile models, but yes that's a start. Thank you.
- klipt 6y ago
  A disembodied AI with a really good model might be able to do good theoretical science, but it would still need a way of acting in the physical world to do experimental science.
  - Veedrac 6y ago
    With a sufficiently effective language model it would be fairly easy to bridge this model, by letting the text direct humans on the other side. Hypothesis: <AI writes this> Results: <human observations> <repeat>
    - endgame 6y ago
      That seems like a _very bad_ habit to get humans into.
  - falcor84 6y ago
    Well, we already have tons of examples of computers telling humans what to do, e.g. autogenerated emails alerting a human to handle an issue. The novel Manna explores where this can lead quite nicely - http://www.marshallbrain.com/manna1.htm http://www.marshallbrain.com/manna1.htm
Nasrudith 6y ago
I wonder how well it would perform in accuracy if given a large number of simple but lengthy sums like 13453 + 53521. Increased set size would move it beyond simple input/output memorization. Although if it recurses properly and carries the digit it could be text parsing and have an accurate but probably very inefficently written math parser.
- dwohnitmok 6y ago
  > Although if it recurses properly and carries the digit it could be text parsing and have an accurate but probably very inefficently written math parser. I suspect this is how many humans do arithmetic (especially considering how many people conflate numbers with their representation as digits). So if GPT-3 is doing that, that's pretty impressive.
- ja3k 6y ago
  You don't have to wonder. In their paper: https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165 they state it has 0.7% accuracy on zero shot 5 digit addition problems and 9.3% accuracy on few shot 5 digit addition problems.
  - gwern 6y ago
    You do have to wonder, because as that section states, the BPEs may impede arithmetic, and as we've found using the API, if you use commas, the accuracy (zero and few-shot) goes way up.
  - ralfd 6y ago
    By the way: Arithmetic accuracy is better if dollar sign and commas are added (financial data in the training set): http://gptprompts.wikidot.com/logic:math http://gptprompts.wikidot.com/logic:math
yters 6y ago
Solmonoff induction would imply the algorithm that learns the rules of arithmetic will have the most concise model for the data. But, it is unclear these gpt-3 type algorithms are solomonoff learners.
_greim_ 6y ago
This strikes me as very similar to the debate around the Chinese Room. https://plato.stanford.edu/entries/chinese-room/ https://plato.stanford.edu/entries/chinese-room/
- slowmovintarget 6y ago
  Peter Watts explores this in the novel Blindsight. I don't want to give the plot away, but the main idea is really interesting, and relevant to this discussion.
  - Baeocystin 6y ago
    I'm posting to second the recommendation for the novel. It is the most interesting exploration of the Mind's I (not a typo) that I've come across in modern sci-fi. It can be read in its entirety at the author's site: https://rifters.com/real/Blindsight.htm https://rifters.com/real/Blindsight.htm
    - tinmandespot 6y ago
      Threads like these are the reason why I keep coming back to HN !
  - _greim_ 6y ago
    Recently finished my second read of Blindsight. Enjoyed it more the second time than the first.
- joefourier 6y ago
  I would love to talk to someone who actually believes in the Chinese Room argument. To me it seems to be ignoring the existence of emergent behavior, and the same argument could prove that a human Chinese speaker doesn't understand Chinese either: his neurons are just reacting to produce answers depending on the input and their current state (e.g. neurotransmitters and action potentials).
  - 6y ago
Jack000 6y ago
I don't think training on language by itself is enough. Consider for example, if we found extraterrestrial transmissions from an alien civilization. We don't know what they look like, what they're made of or if they even have corporeal form. All we have is a large quantity of sequential tokens from their communications. It's possible to train GPT3 to produce a facsimile of these transmissions, but doing so does not let us learn anything at all about these aliens, beyond statistical correlations like ⊑⏃⟒⍀ often occurring in close proximity to ⋏⟒⍙⌇ (what do they represent - who knows?). Just having the text is not enough, because we have no understanding of the underlying processes that produced the text. That said, this is only a limitation of language models as they currently exist. I imagine it would be possible to train a ML model that encodes more of the human experience via video/audio/proprioception data.
- Veedrac 6y ago
  This is as true of any information channel, including your eyes and ears.
  - Jack000 6y ago
    That kind of gets into "what is it like to be a bat" territory. The more imminent question is more of engineering than philosophy - what does it take for GPT-3 to not make the mistakes it does? This would require it to have some internal model for why humans generate text (persuasion, entertainment, etc.) as well as the social context in which that human generated the text. On a lower level it also needs to know about cognitive shortcuts that humans take for granted (object permanence, gravity) Basically, some degree of human subjective experience must be encoded and fed to the model. That's a difficult problem, but not an intractable one.
- hackinthebochs 6y ago
  I wouldn't be so sure we couldn't decode the meaning of an alien language given enough sample text. There have been some advances[1] towards learning a translation between two human languages in an unsupervised manner, meaning without any (language1, language2) sentence pairs to serve as the ground truth for building a translation. Essentially it independently learns abstract representations of the two languages from written text in each language, all the while nudging these abstract representations towards identical feature spaces. The result is a strong translation model trained without utilizing any upfront translations as training data. The intuition behind this idea is that the structure inherent in a language is dependent upon features of the world being described by that language to some degree. If we can abstract out the details of the language and get at the underlying structure the language is describing, then this latent structure should be language-independent. But then translation turns out to simply be a matter of decoding and encoding a language to this latent structure. One limitation of this idea is that it depends on there being some shared structure that underlies the languages we're attempting to model and translate. It's easy to imagine this constraint holds in the real world as human contexts are very similar regardless of language spoken. The basic units and concepts that feature in our lives are more-or-less universally shared and so this shared structure provides a meaningful pathway to translation. We might even expect the world of intelligent aliens to share enough latent structure from which to build a translation given enough source text. The laws of physics and mathematics are universal after all. [1] https://openreview.net/pdf?id=rkYTTf-AZ https://openreview.net/pdf?id=rkYTTf-AZ
ben_w 6y ago
I certainly agree that GPT-3 appears to be learning how to do mathematics. I suspect that if you gave it enough it might perhaps even learn the maths of physics. I suspect that if it did that, it would be able to write a very convincing fake paper about how it designed and tested an Alcubierre drive, and that the main clue about the paper being fake being a sentence such as “we dismantled Jupiter for use as a radiation shield against the issue raised by McMonigal et al, 2012”. Or, to put it another way, the hardest of hard SciFi, but still SciFi, not science.
YeGoblynQueenne 6y ago
>> But GPT-3 is much more successful, including at giving correct answers to arithmetic problems that weren't in its training set. That's not exactly what the GPT-3 paper [1] claims. The paper claims that a search of the training dataset for instances of, very specifically, three-digit addition, returned no matches. That doesn't mean there weren't any instances, it only means the search didn't find any. It also doesn't say anything about the existence of instances of other arithmetic operations in GPT-3's training set (and the absence of "spot checks" for such instances of other operations suggests they were, actually, found- but not reported, in time-honoured fashion of not reporting negative results). So at best we can conclude that GPT-3 gave correct answers to three-digit addition problems that weren't in its training set and then again, only the 2000 or so problems that were specifically searched for. In general, the paper tested GPT-3's arithmetic abilities with addition and subtraction between one to five digit numbers and multiplication between two-digit numbers. They also tested a composite task of one-digit expressions, e.g. "6+(4*8)" etc. No division was attempted at all (or no results were reported). Of the attempted tasks, all than addition and subtraction between one to three digit numbers had accuracy below 20%. In other words, the only tasks that were at all successful were exactly those tasks that were the most likely to be found in a corpus of text, rather than a corpus of arithmetic expressions. The results indicate that GPT-3 cannot "perform arithmetic" despite the paper's claims to the contrary. They are precisely the results one should expect to see if GPT-3 was simply memorising examples of arithmetic in its training corpus. >> So what changed? We aren't sure, but the speculation is that in the process of training, GPT-3 found that the best strategy to correctly predicting the continuation of arithmetic expressions was to figure out the rules of basic arithmetic and encode them in some portion of its neural network, then apply them whenever the prompt suggested to do so. There is no reason why a language model should be able to "figure out the rules of basic arithmetic" so this "speculation" is tantamount to invoking magick. Additionally, language models and neural networks in general are not capable of representing the rules of arithmetic because they are incapable of representing recursion and universally quantified variables, both of which are necessary to express the rules of arithmetic. In any case, if GPT-3 had "figure(d) out the rules of basic arithmetic", why stop at addition, subtraction and multiplication between one to five digit numbers? Why was it not able to use those learned rules to perform the same operations with more digits? Why was it not capable of performing division (i.e. the opposite of multiplication)? A very simple asnwer is: GPT-3 did not learn the rules of arithmetic. _________ [1] https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165
bo1024 6y ago
> We aren't sure, but the speculation is that in the process of training, GPT-3 found that the best strategy to correctly predicting the continuation of arithmetic expressions was to figure out the rules of basic arithmetic and encode them in some portion of its neural network I don't mean to attack you personally, but this is a perfect example of what I feel is wrong with so much neural network research. (And I understand that you are just commenting in a discussion, not conducting research.) In a word, it's baloney. And it's a really common pattern in neural networks' recent history: "How did they perform reasonably well on this task? We aren't sure, but the speculation is that they magically solved artificial general intelligence under the hood." Usually this is followed up by "I don't know how it works, but let's see if a bigger network can make even prettier text." Meanwhile, "it's funny how our image classifiers grossly misperform if you rotate the images a little or add some noise." A rigorous scientific approach would be aimed at actually figuring out what these models can do, why, and how they work. Rather than just assuming the most optimistic possible explanation for what's happening -- that's antithetical to science.
- atomicity 6y ago
  I don't know if this means something is truly wrong. AI is a mix of engineering and scientific research, just like most CS subfields. Recently, the emphasis has shifted towards engineering, as the applications of neural nets have skyrocketed after a few breakthroughs in performance. It's similar to computer systems research. For example, a research paper on filesystems might tell us a simple trick which leads to better performance on NVMM. The paper may go into why the trick works, but it doesn't (and shouldn't need to) generalize and try to improve our general understanding of how to design filesystems on different hardware. We've been designing filesystems to this day and well, we are always still guessing about which approaches to use and hoping for the best. In the same vein, we don't even have a widely-accepted theory of how to use data structures yet. So, I don't think that neural nets aren't scientific enough means that it's all BS. We have gaps in understanding, but the power of the models warrants a lot of continued work on finding useful applications. Doesn't mean I don't think AI is over-hyped/overfunded though...
  - dragongod2718 6y ago
confuseshrink 6y ago
> So what changed? We aren't sure, but the speculation is that in the process of training, GPT-3 found that the best strategy to correctly predicting the continuation of arithmetic expressions was to figure out the rules of basic arithmetic and encode them in some portion of its neural network, then apply them whenever the prompt suggested to do so. I saw a lot of basic arithmetic in the thousands range where it failed. If we have to keep scaling it quadratically for it to learn log n scale arithmetic then we're doing it wrong. I'm surprised you think it learned some basic rules around arithmetic. A lot of simple rules extrapolate very well, into all number ranges. To me it seems like it's just making things up as it goes along. I'll grant you this though, it can make for a convincing illusion at times.
- coryfklein 6y ago
  > To me it seems like it's just making things up as it goes along. Oh, aren’t we all?
darepublic 6y ago
I dunno to me it seems clear that there is nothing of what we call intelligence in these neural networks. And I think we could have a general AI that can problem solve in the world but have zero of what we know of as understanding and sel awareness
vivekkalyan 6y ago
> We aren't sure, but the speculation is that in the process of training, GPT-3 found that the best strategy to correctly predicting the continuation of arithmetic expressions was to figure out the rules of basic arithmetic and encode them in some portion of its neural network, then apply them whenever the prompt suggested to do so. I strongly disagree. GPT-3 has 100% accuracy on 2-digit addition, 80% on 3-digit addition, 25% on 4-digit addition and 9% on 5-digit addition. If it could indeed "understand arithmetic" the increase in number of digits should not affect its accuracy. My perspective as an ML practitioner is that the cool part of GPT-3 is storing information effectively and it is able to decode queries easier than before to get the information that is required. Yet with things like arithmetic, the most efficient way would be to understand the rules of addition but the internal structure is too rigid to encode those rules atm.
TomSwirly 6y ago
Nothing you say convinces me that GPT-3 is exhibiting any conceptual understanding. Imitating existing texts better is not conceptual understanding. "Understanding" means you can explain why you made a decision. It means there exists a model with conceptual entities that you can access and make available to others. What GPT-3 does is this: "I am given many answers to similar questions, and I build up a huge model that reflects these answers. If I'm given a new question, I come up with a response that's probably right, based on the previous answers, but there's no explanation possible." Don't get me wrong - it's amazing! But it's not understanding anything yet. Even humans have skills that we know but do not understand - like "walking" for most of us! But on abstract question, we almost always have access to a complete set of reasons. "Why did you go back to the store?" "I left my bag there." "Why did you talk to that man?" "I know he's the manager, I'm a regular." "Why were you happy?" "I had my bag." (Indeed, this is so common that people often "backdate" reasons for actions that didn't really have any reason at the time. But I digress.)

simonh 6y ago

The example of simple arithmetic is interesting. I think you might be right, that is evidence that GPT-3 might be generating what we might consider models of it's input data. Very simple, primitive and fragile models, but yes that's a start. Thank you.

klipt 6y ago
A disembodied AI with a really good model might be able to do good theoretical science, but it would still need a way of acting in the physical world to do experimental science.
- Veedrac 6y ago
  With a sufficiently effective language model it would be fairly easy to bridge this model, by letting the text direct humans on the other side. Hypothesis: <AI writes this> Results: <human observations> <repeat>
  - endgame 6y ago
    That seems like a _very bad_ habit to get humans into.
- falcor84 6y ago
  Well, we already have tons of examples of computers telling humans what to do, e.g. autogenerated emails alerting a human to handle an issue. The novel Manna explores where this can lead quite nicely - http://www.marshallbrain.com/manna1.htm http://www.marshallbrain.com/manna1.htm

Nasrudith 6y ago

I wonder how well it would perform in accuracy if given a large number of simple but lengthy sums like 13453 + 53521. Increased set size would move it beyond simple input/output memorization. Although if it recurses properly and carries the digit it could be text parsing and have an accurate but probably very inefficently written math parser.

dwohnitmok 6y ago
> Although if it recurses properly and carries the digit it could be text parsing and have an accurate but probably very inefficently written math parser. I suspect this is how many humans do arithmetic (especially considering how many people conflate numbers with their representation as digits). So if GPT-3 is doing that, that's pretty impressive.
ja3k 6y ago
You don't have to wonder. In their paper: https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165 they state it has 0.7% accuracy on zero shot 5 digit addition problems and 9.3% accuracy on few shot 5 digit addition problems.
- gwern 6y ago
  You do have to wonder, because as that section states, the BPEs may impede arithmetic, and as we've found using the API, if you use commas, the accuracy (zero and few-shot) goes way up.
- ralfd 6y ago
  By the way: Arithmetic accuracy is better if dollar sign and commas are added (financial data in the training set): http://gptprompts.wikidot.com/logic:math http://gptprompts.wikidot.com/logic:math

yters 6y ago

Solmonoff induction would imply the algorithm that learns the rules of arithmetic will have the most concise model for the data. But, it is unclear these gpt-3 type algorithms are solomonoff learners.

_greim_ 6y ago

This strikes me as very similar to the debate around the Chinese Room. https://plato.stanford.edu/entries/chinese-room/ https://plato.stanford.edu/entries/chinese-room/

slowmovintarget 6y ago
Peter Watts explores this in the novel Blindsight. I don't want to give the plot away, but the main idea is really interesting, and relevant to this discussion.
- Baeocystin 6y ago
  I'm posting to second the recommendation for the novel. It is the most interesting exploration of the Mind's I (not a typo) that I've come across in modern sci-fi. It can be read in its entirety at the author's site: https://rifters.com/real/Blindsight.htm https://rifters.com/real/Blindsight.htm
  - tinmandespot 6y ago
    Threads like these are the reason why I keep coming back to HN !
- _greim_ 6y ago
  Recently finished my second read of Blindsight. Enjoyed it more the second time than the first.
joefourier 6y ago
I would love to talk to someone who actually believes in the Chinese Room argument. To me it seems to be ignoring the existence of emergent behavior, and the same argument could prove that a human Chinese speaker doesn't understand Chinese either: his neurons are just reacting to produce answers depending on the input and their current state (e.g. neurotransmitters and action potentials).
- 6y ago

Jack000 6y ago

I don't think training on language by itself is enough. Consider for example, if we found extraterrestrial transmissions from an alien civilization. We don't know what they look like, what they're made of or if they even have corporeal form. All we have is a large quantity of sequential tokens from their communications. It's possible to train GPT3 to produce a facsimile of these transmissions, but doing so does not let us learn anything at all about these aliens, beyond statistical correlations like ⊑⏃⟒⍀ often occurring in close proximity to ⋏⟒⍙⌇ (what do they represent - who knows?). Just having the text is not enough, because we have no understanding of the underlying processes that produced the text. That said, this is only a limitation of language models as they currently exist. I imagine it would be possible to train a ML model that encodes more of the human experience via video/audio/proprioception data.

Veedrac 6y ago
This is as true of any information channel, including your eyes and ears.
- Jack000 6y ago
  That kind of gets into "what is it like to be a bat" territory. The more imminent question is more of engineering than philosophy - what does it take for GPT-3 to not make the mistakes it does? This would require it to have some internal model for why humans generate text (persuasion, entertainment, etc.) as well as the social context in which that human generated the text. On a lower level it also needs to know about cognitive shortcuts that humans take for granted (object permanence, gravity) Basically, some degree of human subjective experience must be encoded and fed to the model. That's a difficult problem, but not an intractable one.
hackinthebochs 6y ago
I wouldn't be so sure we couldn't decode the meaning of an alien language given enough sample text. There have been some advances[1] towards learning a translation between two human languages in an unsupervised manner, meaning without any (language1, language2) sentence pairs to serve as the ground truth for building a translation. Essentially it independently learns abstract representations of the two languages from written text in each language, all the while nudging these abstract representations towards identical feature spaces. The result is a strong translation model trained without utilizing any upfront translations as training data. The intuition behind this idea is that the structure inherent in a language is dependent upon features of the world being described by that language to some degree. If we can abstract out the details of the language and get at the underlying structure the language is describing, then this latent structure should be language-independent. But then translation turns out to simply be a matter of decoding and encoding a language to this latent structure. One limitation of this idea is that it depends on there being some shared structure that underlies the languages we're attempting to model and translate. It's easy to imagine this constraint holds in the real world as human contexts are very similar regardless of language spoken. The basic units and concepts that feature in our lives are more-or-less universally shared and so this shared structure provides a meaningful pathway to translation. We might even expect the world of intelligent aliens to share enough latent structure from which to build a translation given enough source text. The laws of physics and mathematics are universal after all. [1] https://openreview.net/pdf?id=rkYTTf-AZ https://openreview.net/pdf?id=rkYTTf-AZ

ben_w 6y ago

I certainly agree that GPT-3 appears to be learning how to do mathematics. I suspect that if you gave it enough it might perhaps even learn the maths of physics. I suspect that if it did that, it would be able to write a very convincing fake paper about how it designed and tested an Alcubierre drive, and that the main clue about the paper being fake being a sentence such as “we dismantled Jupiter for use as a radiation shield against the issue raised by McMonigal et al, 2012”. Or, to put it another way, the hardest of hard SciFi, but still SciFi, not science.

YeGoblynQueenne 6y ago

>> But GPT-3 is much more successful, including at giving correct answers to arithmetic problems that weren't in its training set. That's not exactly what the GPT-3 paper [1] claims. The paper claims that a search of the training dataset for instances of, very specifically, three-digit addition, returned no matches. That doesn't mean there weren't any instances, it only means the search didn't find any. It also doesn't say anything about the existence of instances of other arithmetic operations in GPT-3's training set (and the absence of "spot checks" for such instances of other operations suggests they were, actually, found- but not reported, in time-honoured fashion of not reporting negative results). So at best we can conclude that GPT-3 gave correct answers to three-digit addition problems that weren't in its training set and then again, only the 2000 or so problems that were specifically searched for. In general, the paper tested GPT-3's arithmetic abilities with addition and subtraction between one to five digit numbers and multiplication between two-digit numbers. They also tested a composite task of one-digit expressions, e.g. "6+(4*8)" etc. No division was attempted at all (or no results were reported). Of the attempted tasks, all than addition and subtraction between one to three digit numbers had accuracy below 20%. In other words, the only tasks that were at all successful were exactly those tasks that were the most likely to be found in a corpus of text, rather than a corpus of arithmetic expressions. The results indicate that GPT-3 cannot "perform arithmetic" despite the paper's claims to the contrary. They are precisely the results one should expect to see if GPT-3 was simply memorising examples of arithmetic in its training corpus. >> So what changed? We aren't sure, but the speculation is that in the process of training, GPT-3 found that the best strategy to correctly predicting the continuation of arithmetic expressions was to figure out the rules of basic arithmetic and encode them in some portion of its neural network, then apply them whenever the prompt suggested to do so. There is no reason why a language model should be able to "figure out the rules of basic arithmetic" so this "speculation" is tantamount to invoking magick. Additionally, language models and neural networks in general are not capable of representing the rules of arithmetic because they are incapable of representing recursion and universally quantified variables, both of which are necessary to express the rules of arithmetic. In any case, if GPT-3 had "figure(d) out the rules of basic arithmetic", why stop at addition, subtraction and multiplication between one to five digit numbers? Why was it not able to use those learned rules to perform the same operations with more digits? Why was it not capable of performing division (i.e. the opposite of multiplication)? A very simple asnwer is: GPT-3 did not learn the rules of arithmetic. _________ [1] https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165

bo1024 6y ago

> We aren't sure, but the speculation is that in the process of training, GPT-3 found that the best strategy to correctly predicting the continuation of arithmetic expressions was to figure out the rules of basic arithmetic and encode them in some portion of its neural network I don't mean to attack you personally, but this is a perfect example of what I feel is wrong with so much neural network research. (And I understand that you are just commenting in a discussion, not conducting research.) In a word, it's baloney. And it's a really common pattern in neural networks' recent history: "How did they perform reasonably well on this task? We aren't sure, but the speculation is that they magically solved artificial general intelligence under the hood." Usually this is followed up by "I don't know how it works, but let's see if a bigger network can make even prettier text." Meanwhile, "it's funny how our image classifiers grossly misperform if you rotate the images a little or add some noise." A rigorous scientific approach would be aimed at actually figuring out what these models can do, why, and how they work. Rather than just assuming the most optimistic possible explanation for what's happening -- that's antithetical to science.

atomicity 6y ago
I don't know if this means something is truly wrong. AI is a mix of engineering and scientific research, just like most CS subfields. Recently, the emphasis has shifted towards engineering, as the applications of neural nets have skyrocketed after a few breakthroughs in performance. It's similar to computer systems research. For example, a research paper on filesystems might tell us a simple trick which leads to better performance on NVMM. The paper may go into why the trick works, but it doesn't (and shouldn't need to) generalize and try to improve our general understanding of how to design filesystems on different hardware. We've been designing filesystems to this day and well, we are always still guessing about which approaches to use and hoping for the best. In the same vein, we don't even have a widely-accepted theory of how to use data structures yet. So, I don't think that neural nets aren't scientific enough means that it's all BS. We have gaps in understanding, but the power of the models warrants a lot of continued work on finding useful applications. Doesn't mean I don't think AI is over-hyped/overfunded though...
- dragongod2718 6y ago

confuseshrink 6y ago

> So what changed? We aren't sure, but the speculation is that in the process of training, GPT-3 found that the best strategy to correctly predicting the continuation of arithmetic expressions was to figure out the rules of basic arithmetic and encode them in some portion of its neural network, then apply them whenever the prompt suggested to do so. I saw a lot of basic arithmetic in the thousands range where it failed. If we have to keep scaling it quadratically for it to learn log n scale arithmetic then we're doing it wrong. I'm surprised you think it learned some basic rules around arithmetic. A lot of simple rules extrapolate very well, into all number ranges. To me it seems like it's just making things up as it goes along. I'll grant you this though, it can make for a convincing illusion at times.

coryfklein 6y ago
> To me it seems like it's just making things up as it goes along. Oh, aren’t we all?

darepublic 6y ago

I dunno to me it seems clear that there is nothing of what we call intelligence in these neural networks. And I think we could have a general AI that can problem solve in the world but have zero of what we know of as understanding and sel awareness

vivekkalyan 6y ago

> We aren't sure, but the speculation is that in the process of training, GPT-3 found that the best strategy to correctly predicting the continuation of arithmetic expressions was to figure out the rules of basic arithmetic and encode them in some portion of its neural network, then apply them whenever the prompt suggested to do so. I strongly disagree. GPT-3 has 100% accuracy on 2-digit addition, 80% on 3-digit addition, 25% on 4-digit addition and 9% on 5-digit addition. If it could indeed "understand arithmetic" the increase in number of digits should not affect its accuracy. My perspective as an ML practitioner is that the cool part of GPT-3 is storing information effectively and it is able to decode queries easier than before to get the information that is required. Yet with things like arithmetic, the most efficient way would be to understand the rules of addition but the internal structure is too rigid to encode those rules atm.

TomSwirly 6y ago

Nothing you say convinces me that GPT-3 is exhibiting any conceptual understanding. Imitating existing texts better is not conceptual understanding. "Understanding" means you can explain why you made a decision. It means there exists a model with conceptual entities that you can access and make available to others. What GPT-3 does is this: "I am given many answers to similar questions, and I build up a huge model that reflects these answers. If I'm given a new question, I come up with a response that's probably right, based on the previous answers, but there's no explanation possible." Don't get me wrong - it's amazing! But it's not understanding anything yet. Even humans have skills that we know but do not understand - like "walking" for most of us! But on abstract question, we almost always have access to a complete set of reasons. "Why did you go back to the store?" "I left my bag there." "Why did you talk to that man?" "I know he's the manager, I'm a regular." "Why were you happy?" "I had my bag." (Indeed, this is so common that people often "backdate" reasons for actions that didn't really have any reason at the time. But I digress.)