5 ms·

I think you are underestimating what an advance these models are over previous NLP models in terms of quality. Before GPT-2 we didn't even have models that coul

I think you are underestimating what an advance these models are over previous NLP models in terms of quality. Before GPT-2 we didn't even have models that could reliably generate grammatical sentences. Now we have things that generate coherent (if not beautiful) paragraphs. It seems easy in retrospect, but some of the smartest people around have been working on this for decades.

abernard1 6y ago
> I think you are underestimating what an advance these models are over previous NLP models in terms of quality. Yeah I mean, I agree. But in my opinion, it's a case of "doing the wrong thing right" instead of a more useful "doing the right thing wrong." I grant that these automated models are useful for low-value classification/generation tasks at high-frequency scale. I don't think that in any way is related to intelligence though, and the only reason I think they've been pursued is because of immediate economic usefulness _shrug_. When high-value, low-frequency tasks begin to be reproduced by software without intervention, I think we'll be closer to intelligence. This is just mimicry. Change the parameters even in the slightest (e.g. have this algorithm try to "learn" the article it created to actually do something in the world) and it all falls down.
- natch 6y ago
  This kind of facile moving the goalposts is imho a cheap shot and (not imho, fact) is a recurring phenomenon as we make incremental progress toward AI. Progress is often made with steps that would have been astonishing a few years ago. And every time the bar is raised higher. Rightly so, but characterizing this as doing the wrong thing is missing the point of what we, and the system, are learning. Yes it's not intelligence. But then, it's not even clear that we ourselves can define intelligence at all… not all philosophers agree on this. Daniel Dennett (philosopher and computer scientist) for example thinks that consciousness may be just a collection of illusions and tricks a mind plays with itself as it models different facets of and lenses into what it stores and perceives.
  - abernard1 6y ago
    > This kind of facile moving the goalposts is imho a cheap shot and (not imho, fact) is a recurring phenomenon as we make incremental progress toward AI. I think you missed my point. I think we're going in the wrong direction for AI entirely, and these "advances" are fundamentally misguided. OpenAI is explicitly about "intelligence," and so we should question if this is in fact that. It's clear that humans have fundamental intelligence much better than all of this stuff with 6 orders of magnitude less input (at least of the same data sort) on a problem. Perhaps it would be better to say, "I think the ML winter is just around the corner" as opposed to "the AI winter is just around the corner." That said, this really is math, and these algos still don't actually do anything resembling true intelligence.
    - canjobear 6y ago
      > think we're going in the wrong direction for AI entirely This direction has produced results that eluded 30+ years of research. What is the evidence that this is the wrong direction?
    - a9h74j 6y ago
      > 6 orders of magnitude less input IIRC the following is attributable to either Margaret Atwood or Iris Murdoch: "A writer should be able to look into a room [full of people] and understand [in breadth] what is going on."
    - natch 6y ago
      But we’re not done yet. Give it time. We can go in plenty of directions. Just because you don’t think the current direction is right, that doesn’t rule out other directions happening. And who’s to say this stuff won’t end up being somehow useful? There’s a great talk on how trying to make progress toward an objective in evolutionary algorithms is not a good way to get there. https://www.youtube.com/watch?v=dXQPL9GooyI https://www.youtube.com/watch?v=dXQPL9GooyI Of course evolutionary algorithms are just one direction as well. But that doesn’t mean that nothing else is happening.
    - natch 6y ago
      It’s actually about AI which is distinct from intelligence. >6 orders of magnitude less input That is utterly mistaken. We have the input of millions of generations of evolution which have shaped our brains and given us a lot of instinctive knowledge that we do not need to learn from environmental input that happens during our lifetime. Instead it was learned over the course of billions of years, during the lifetimes of other organisms that preceded us. Our brain structure was developed and tuned by all these inputs to have some built in pretrained models. That’s what instincts are. Billions of years in the making. Millions, at the very least, if you want to restrict it to recent primates, although doing so is nonsensical.
enjoylife 6y ago
I marvel at the jump in BLEU or other measures but I'll second the sentiment, it alone is not showing we are making leaps toward what we need. Yes its a large gradient step minimizing our error, but is it really in the right direction? However I will admit GPT-3 being directed by some yet to be invented causal or counterfactual inference model might be the something which defies my expectations.
pgt 6y ago
Is there a term for the casual dismissal of breakthrough technologies and ever-moving goalposts for what is considered impressive?
- pornel 6y ago
  It's simply calling things "A.I." Seriously, a few years ago recognizing if there's a bird in a photo was an example of a "virtually impossible" task: https://xkcd.com/1425/ https://xkcd.com/1425/
- EvgeniyZh 6y ago
  https://en.wikipedia.org/wiki/AI_effect https://en.wikipedia.org/wiki/AI_effect
  - pgt 6y ago
    Ehahaha, thank you!
- maest 6y ago
  "God of the gaps"? The original usage is in theology, but the idea is the same.

abernard1 6y ago

> I think you are underestimating what an advance these models are over previous NLP models in terms of quality. Yeah I mean, I agree. But in my opinion, it's a case of "doing the wrong thing right" instead of a more useful "doing the right thing wrong." I grant that these automated models are useful for low-value classification/generation tasks at high-frequency scale. I don't think that in any way is related to intelligence though, and the only reason I think they've been pursued is because of immediate economic usefulness _shrug_. When high-value, low-frequency tasks begin to be reproduced by software without intervention, I think we'll be closer to intelligence. This is just mimicry. Change the parameters even in the slightest (e.g. have this algorithm try to "learn" the article it created to actually do something in the world) and it all falls down.

natch 6y ago
This kind of facile moving the goalposts is imho a cheap shot and (not imho, fact) is a recurring phenomenon as we make incremental progress toward AI. Progress is often made with steps that would have been astonishing a few years ago. And every time the bar is raised higher. Rightly so, but characterizing this as doing the wrong thing is missing the point of what we, and the system, are learning. Yes it's not intelligence. But then, it's not even clear that we ourselves can define intelligence at all… not all philosophers agree on this. Daniel Dennett (philosopher and computer scientist) for example thinks that consciousness may be just a collection of illusions and tricks a mind plays with itself as it models different facets of and lenses into what it stores and perceives.
- abernard1 6y ago
  > This kind of facile moving the goalposts is imho a cheap shot and (not imho, fact) is a recurring phenomenon as we make incremental progress toward AI. I think you missed my point. I think we're going in the wrong direction for AI entirely, and these "advances" are fundamentally misguided. OpenAI is explicitly about "intelligence," and so we should question if this is in fact that. It's clear that humans have fundamental intelligence much better than all of this stuff with 6 orders of magnitude less input (at least of the same data sort) on a problem. Perhaps it would be better to say, "I think the ML winter is just around the corner" as opposed to "the AI winter is just around the corner." That said, this really is math, and these algos still don't actually do anything resembling true intelligence.
  - canjobear 6y ago
    > think we're going in the wrong direction for AI entirely This direction has produced results that eluded 30+ years of research. What is the evidence that this is the wrong direction?
  - a9h74j 6y ago
    > 6 orders of magnitude less input IIRC the following is attributable to either Margaret Atwood or Iris Murdoch: "A writer should be able to look into a room [full of people] and understand [in breadth] what is going on."
  - natch 6y ago
    But we’re not done yet. Give it time. We can go in plenty of directions. Just because you don’t think the current direction is right, that doesn’t rule out other directions happening. And who’s to say this stuff won’t end up being somehow useful? There’s a great talk on how trying to make progress toward an objective in evolutionary algorithms is not a good way to get there. https://www.youtube.com/watch?v=dXQPL9GooyI https://www.youtube.com/watch?v=dXQPL9GooyI Of course evolutionary algorithms are just one direction as well. But that doesn’t mean that nothing else is happening.
  - natch 6y ago
    It’s actually about AI which is distinct from intelligence. >6 orders of magnitude less input That is utterly mistaken. We have the input of millions of generations of evolution which have shaped our brains and given us a lot of instinctive knowledge that we do not need to learn from environmental input that happens during our lifetime. Instead it was learned over the course of billions of years, during the lifetimes of other organisms that preceded us. Our brain structure was developed and tuned by all these inputs to have some built in pretrained models. That’s what instincts are. Billions of years in the making. Millions, at the very least, if you want to restrict it to recent primates, although doing so is nonsensical.

enjoylife 6y ago

I marvel at the jump in BLEU or other measures but I'll second the sentiment, it alone is not showing we are making leaps toward what we need. Yes its a large gradient step minimizing our error, but is it really in the right direction? However I will admit GPT-3 being directed by some yet to be invented causal or counterfactual inference model might be the something which defies my expectations.

pgt 6y ago

Is there a term for the casual dismissal of breakthrough technologies and ever-moving goalposts for what is considered impressive?

pornel 6y ago
It's simply calling things "A.I." Seriously, a few years ago recognizing if there's a bird in a photo was an example of a "virtually impossible" task: https://xkcd.com/1425/ https://xkcd.com/1425/
EvgeniyZh 6y ago
https://en.wikipedia.org/wiki/AI_effect https://en.wikipedia.org/wiki/AI_effect
- pgt 6y ago
  Ehahaha, thank you!
maest 6y ago
"God of the gaps"? The original usage is in theology, but the idea is the same.