5 ms·

> We aren't sure, but the speculation is that in the process of training, GPT-3 found that the best strategy to correctly predicting the continuation of arithme

by bo1024 6y ago

> We aren't sure, but the speculation is that in the process of training, GPT-3 found that the best strategy to correctly predicting the continuation of arithmetic expressions was to figure out the rules of basic arithmetic and encode them in some portion of its neural network I don't mean to attack you personally, but this is a perfect example of what I feel is wrong with so much neural network research. (And I understand that you are just commenting in a discussion, not conducting research.) In a word, it's baloney. And it's a really common pattern in neural networks' recent history: "How did they perform reasonably well on this task? We aren't sure, but the speculation is that they magically solved artificial general intelligence under the hood." Usually this is followed up by "I don't know how it works, but let's see if a bigger network can make even prettier text." Meanwhile, "it's funny how our image classifiers grossly misperform if you rotate the images a little or add some noise." A rigorous scientific approach would be aimed at actually figuring out what these models can do, why, and how they work. Rather than just assuming the most optimistic possible explanation for what's happening -- that's antithetical to science.

atomicity 6y ago
I don't know if this means something is truly wrong. AI is a mix of engineering and scientific research, just like most CS subfields. Recently, the emphasis has shifted towards engineering, as the applications of neural nets have skyrocketed after a few breakthroughs in performance. It's similar to computer systems research. For example, a research paper on filesystems might tell us a simple trick which leads to better performance on NVMM. The paper may go into why the trick works, but it doesn't (and shouldn't need to) generalize and try to improve our general understanding of how to design filesystems on different hardware. We've been designing filesystems to this day and well, we are always still guessing about which approaches to use and hoping for the best. In the same vein, we don't even have a widely-accepted theory of how to use data structures yet. So, I don't think that neural nets aren't scientific enough means that it's all BS. We have gaps in understanding, but the power of the models warrants a lot of continued work on finding useful applications. Doesn't mean I don't think AI is over-hyped/overfunded though...
- dragongod2718 6y ago
  Wait, you think AI is overfunded?
- bo1024 6y ago
  I agree with a lot of this, but I think there is a consistent pattern of AI announcements playing on humans' intuitions to create the impression that much more has been achieved than can actually be proven -- in fact, not even trying to prove anything. Part of this is that the researchers are humans too and may be misled themselves. But a rigorous research process would at least try to prevent that. For example, people once thought playing chess was hard. So they thought that if a computer could beat the world champion, then computers would probably also be able to replace every job and so on. If you sent Deep Blue back in time to the 1960s, they wouldn't understand how it works so they'd probably assume that it since it could beat Petrosian in chess, it could probably drive cars and treat disease. But then we built Deep Blue and realized that you don't need AGI to play chess; a very specialized algorithm will do it. So we're like people in the 70s who've been handed Deep Blue. It's irresponsible, in my opinion, to over-hype it when we have no idea how it works.
glenstein 6y ago
>In a word, it's baloney. This is where you lost me. They included important caveats to indicate not being sure, which is important to me as an indication of healthy skepticism. And you substituted a specific example: making inferences about arithmetic, for an more expansive, uncharitable, easy-to-caricature claim of "gee we must have solved general AI!" which is much easier to attack. And, unlike your counterpart, who hedged, you just went ahead and categorically declared it to be baloney, making you the only person to take a definitive side on an unsettled question before the data is in. This is a perfect example of the anti-scientific attitude exhibited in Overconfident Pessimism [0]. I don't think it's known how GPT-3 got so much better at answering math questions it wasn't trained on, I do think the explanation that it made inferences about arithmetic is reasonable, I think the commenter added all the qualifiers you could reasonably ask them to make before suggesting the idea, and frankly I would disagree that there's some sort of obvious history of parallels that GPT-3 can be compared to. There is an interesting conversation to be had here, and there probably is much more to learn about why GPT-3 probably isn't quite as advanced as it may immediately appear to be to those who want to believe in it. But I think a huge wrench is thrown in that whole conversation with the total lack of humility required to confidently declare it 'baloney', which is the thing that sticks out to me as antithetical to science. 0: https://www.lesswrong.com/posts/gvdYK8sEFqHqHLRqN/overconfident-pessimism https://www.lesswrong.com/posts/gvdYK8sEFqHqHLRqN/overconfid...
- bo1024 6y ago
  Thanks for your reply. A couple responses to advance the conversation. As a side note, it's worth mentioning that apparently, from other responses, it seems we have little idea how much arithmetic GPT-3 has learned, and it may not be much. Anyway, I think the important distinction between my perspective and Overconfident Pessimism, which you attribute to me, is that I'm not talking about (im)possibility of achievement, I'm talking about scientific methodology or lack thereof. In other words, I'm not saying (here) that some NLP achievements are impossible. I'm saying that we are not rigorously testing, measuring, and verifying what we are even achieving. Instead we throw out superficially impressive examples of results and invite, or provoke, speculation about how much achievement probably must have maybe happened somewhere in order to produce them. We have seen several years of this pattern, so this is not a GPT-3 specific criticism; it's just that particular quote so neatly captured patterns of lack of scientific rigour that we have seen repeatedly at this point. Probably the first example was image recognition. Everyone was amazed by how well neural nets could classify images. There was a ton of analogous speculation -- along the lines of 'we're not sure, but the speculation is the networks figured out what it really means to be a panda or a stop sign and encoded it in their weights.' The terms "near-human performance" and then "human-level performance" were thrown around a lot. Then we found adversarial examples and realized that e.g. if you rotate the turtle image slightly, the model becomes extremely confident that it's a rifle. So, obviously it has no understand of what a turtle or a rifle is. And obviously, we as researchers don't understand what those neural nets were doing under the hood, and that speculation was extremely over-optimistic. Engineering cool things can absolutely be a part of a scientific process. But we have seen countless repetitions of this pattern (especially since GANs): press releases and impressive-looking examples without rigorous evaluation of what the models are doing or how; invitations to speculate on the best-possible interpretation; and announcing that the next step is to make it bigger. I think this approach is both anti-science and misleading to readers.
JoshuaDavid 6y ago
> And it's a really common pattern in neural networks' recent history: "How did they perform reasonably well on this task? We aren't sure, but the speculation is that they magically solved artificial general intelligence under the hood." Usually this is followed up by "I don't know how it works, but let's see if a bigger network can make even prettier text." Layperson here, but my impression is that "let's see if a bigger network can make even prettier text" has _worked_ far beyond the point most people expected it would stop working. Also my layperson impression: most "researchers" that are on the cutting edge of cool things are more interested in seeing what cool things they can do than on doing rigorous science (which makes sense -- if you optimize for rigorous science, your stuff probably isn't as flashy as the stuff produced by people optimizing for flash).
- lowdose 6y ago
  > what cool things they can do than on doing rigorous science (which makes sense -- if you optimize for rigorous science, your stuff probably isn't as flashy as the stuff produced by people optimizing for flash. Is this a new iteration on that zigzag quote? > Zak phases of the bulk bands and the winding number associated with the bulk Hamiltonian, and verified it through four typical ribbon boundaries, i.e. zigzag, bearded zigzag, armchair, and bearded armchair. From "The existence of topological edge states in honeycomb plasmonic lattices" https://iopscience.iop.org/article/10.1088/1367-2630/18/10/103029 https://iopscience.iop.org/article/10.1088/1367-2630/18/10/1...
cma 6y ago
> Meanwhile, "it's funny how our image classifiers grossly misperform if you rotate the images a little or add some noise." Same thing arguably happens with humans with rotation. Our eyes even rotate in the roll axis to keep gravity aligned things upright. Most people can draw faces more accurately when copying from an upside down face than a right side up one.
- deleted 6y ago
  [deleted]

by bo1024 6y ago

atomicity 6y ago

I don't know if this means something is truly wrong. AI is a mix of engineering and scientific research, just like most CS subfields. Recently, the emphasis has shifted towards engineering, as the applications of neural nets have skyrocketed after a few breakthroughs in performance. It's similar to computer systems research. For example, a research paper on filesystems might tell us a simple trick which leads to better performance on NVMM. The paper may go into why the trick works, but it doesn't (and shouldn't need to) generalize and try to improve our general understanding of how to design filesystems on different hardware. We've been designing filesystems to this day and well, we are always still guessing about which approaches to use and hoping for the best. In the same vein, we don't even have a widely-accepted theory of how to use data structures yet. So, I don't think that neural nets aren't scientific enough means that it's all BS. We have gaps in understanding, but the power of the models warrants a lot of continued work on finding useful applications. Doesn't mean I don't think AI is over-hyped/overfunded though...

dragongod2718 6y ago
Wait, you think AI is overfunded?
bo1024 6y ago
I agree with a lot of this, but I think there is a consistent pattern of AI announcements playing on humans' intuitions to create the impression that much more has been achieved than can actually be proven -- in fact, not even trying to prove anything. Part of this is that the researchers are humans too and may be misled themselves. But a rigorous research process would at least try to prevent that. For example, people once thought playing chess was hard. So they thought that if a computer could beat the world champion, then computers would probably also be able to replace every job and so on. If you sent Deep Blue back in time to the 1960s, they wouldn't understand how it works so they'd probably assume that it since it could beat Petrosian in chess, it could probably drive cars and treat disease. But then we built Deep Blue and realized that you don't need AGI to play chess; a very specialized algorithm will do it. So we're like people in the 70s who've been handed Deep Blue. It's irresponsible, in my opinion, to over-hype it when we have no idea how it works.

glenstein 6y ago

>In a word, it's baloney. This is where you lost me. They included important caveats to indicate not being sure, which is important to me as an indication of healthy skepticism. And you substituted a specific example: making inferences about arithmetic, for an more expansive, uncharitable, easy-to-caricature claim of "gee we must have solved general AI!" which is much easier to attack. And, unlike your counterpart, who hedged, you just went ahead and categorically declared it to be baloney, making you the only person to take a definitive side on an unsettled question before the data is in. This is a perfect example of the anti-scientific attitude exhibited in Overconfident Pessimism [0]. I don't think it's known how GPT-3 got so much better at answering math questions it wasn't trained on, I do think the explanation that it made inferences about arithmetic is reasonable, I think the commenter added all the qualifiers you could reasonably ask them to make before suggesting the idea, and frankly I would disagree that there's some sort of obvious history of parallels that GPT-3 can be compared to. There is an interesting conversation to be had here, and there probably is much more to learn about why GPT-3 probably isn't quite as advanced as it may immediately appear to be to those who want to believe in it. But I think a huge wrench is thrown in that whole conversation with the total lack of humility required to confidently declare it 'baloney', which is the thing that sticks out to me as antithetical to science. 0: https://www.lesswrong.com/posts/gvdYK8sEFqHqHLRqN/overconfident-pessimism https://www.lesswrong.com/posts/gvdYK8sEFqHqHLRqN/overconfid...

bo1024 6y ago
Thanks for your reply. A couple responses to advance the conversation. As a side note, it's worth mentioning that apparently, from other responses, it seems we have little idea how much arithmetic GPT-3 has learned, and it may not be much. Anyway, I think the important distinction between my perspective and Overconfident Pessimism, which you attribute to me, is that I'm not talking about (im)possibility of achievement, I'm talking about scientific methodology or lack thereof. In other words, I'm not saying (here) that some NLP achievements are impossible. I'm saying that we are not rigorously testing, measuring, and verifying what we are even achieving. Instead we throw out superficially impressive examples of results and invite, or provoke, speculation about how much achievement probably must have maybe happened somewhere in order to produce them. We have seen several years of this pattern, so this is not a GPT-3 specific criticism; it's just that particular quote so neatly captured patterns of lack of scientific rigour that we have seen repeatedly at this point. Probably the first example was image recognition. Everyone was amazed by how well neural nets could classify images. There was a ton of analogous speculation -- along the lines of 'we're not sure, but the speculation is the networks figured out what it really means to be a panda or a stop sign and encoded it in their weights.' The terms "near-human performance" and then "human-level performance" were thrown around a lot. Then we found adversarial examples and realized that e.g. if you rotate the turtle image slightly, the model becomes extremely confident that it's a rifle. So, obviously it has no understand of what a turtle or a rifle is. And obviously, we as researchers don't understand what those neural nets were doing under the hood, and that speculation was extremely over-optimistic. Engineering cool things can absolutely be a part of a scientific process. But we have seen countless repetitions of this pattern (especially since GANs): press releases and impressive-looking examples without rigorous evaluation of what the models are doing or how; invitations to speculate on the best-possible interpretation; and announcing that the next step is to make it bigger. I think this approach is both anti-science and misleading to readers.

JoshuaDavid 6y ago

> And it's a really common pattern in neural networks' recent history: "How did they perform reasonably well on this task? We aren't sure, but the speculation is that they magically solved artificial general intelligence under the hood." Usually this is followed up by "I don't know how it works, but let's see if a bigger network can make even prettier text." Layperson here, but my impression is that "let's see if a bigger network can make even prettier text" has _worked_ far beyond the point most people expected it would stop working. Also my layperson impression: most "researchers" that are on the cutting edge of cool things are more interested in seeing what cool things they can do than on doing rigorous science (which makes sense -- if you optimize for rigorous science, your stuff probably isn't as flashy as the stuff produced by people optimizing for flash).

lowdose 6y ago
> what cool things they can do than on doing rigorous science (which makes sense -- if you optimize for rigorous science, your stuff probably isn't as flashy as the stuff produced by people optimizing for flash. Is this a new iteration on that zigzag quote? > Zak phases of the bulk bands and the winding number associated with the bulk Hamiltonian, and verified it through four typical ribbon boundaries, i.e. zigzag, bearded zigzag, armchair, and bearded armchair. From "The existence of topological edge states in honeycomb plasmonic lattices" https://iopscience.iop.org/article/10.1088/1367-2630/18/10/103029 https://iopscience.iop.org/article/10.1088/1367-2630/18/10/1...

cma 6y ago

> Meanwhile, "it's funny how our image classifiers grossly misperform if you rotate the images a little or add some noise." Same thing arguably happens with humans with rotation. Our eyes even rotate in the roll axis to keep gravity aligned things upright. Most people can draw faces more accurately when copying from an upside down face than a right side up one.

deleted 6y ago
[deleted]