8 ms·

This quote is the first thing I've seen that really makes me worried. I don't think of ChatGPT as being "smart" at all, and comparing it to a human seems nonse

This quote is the first thing I've seen that really makes me worried. I don't think of ChatGPT as being "smart" at all, and comparing it to a human seems nonsensical to me. Yet here is a Turing award winning preeminent expert in the field telling me that AI smarter than humans is less (implied: much less) than 30 years away and quitting his job due to the ramifications.

Version467 4y ago
He is far from the only one. If you're interested in exploring this further I can really recommend taking a look at some of the papers that explore GPT-4's capabilities. Most prominent among them are the "Sparks of AGI" paper from Microsoft, as well as the technical report from openai. Both of them are obviously to be taken with a grain of salt, but they serve as a pretty good jumping off point. There are some pretty good Videos on Youtube exploring these papers if you don't want to read them yourself. Also take a look at the stuff that Rob Miles has published over on Computerphile, as well as his own channel. He's an Alignment Researcher with a knack for explaining. He covers not just the theoretical dangers, but also real examples of misaligned ai, that alignment researchers have predicted would occur as capabilities grow. Also I think it's important to mention that just a short while ago virtually no-one thought that shoving more layers into an llm would be enough to reach AGI. It's still unclear that it will get us all the way there, but recent developments have made a lot of ai researchers rethink that possibility, with many of them significantly shortening their own estimates as to when and how we will get there. It's very unusual that the people that are better informed and closer to the research are more worried than the rest of the world and it's worth keeping this in mind as you explore the topic.
- defgeneric 4y ago
  > Also I think it's important to mention that just a short while ago virtually no-one thought that shoving more layers into an llm would be enough to reach AGI. This was basically the strategy of the OpenAI team if I understand them correctly. Most researchers in the field looked down on LLMs and it was a big surprise when they turned out to perform so well. It also seems to be the reason the big players are playing catch up right now.
  - whimsicalism 4y ago
    I think it was a surprise the behaviors that were unlocked at different perplexity levels, but I don't really agree that LLMs were "looked down on."
    - TeMPOraL 4y ago
      Maybe not "looked down on", but more of "looked at as a promising avenue". I mean, 2-3 years ago, it felt LLMs are going to be nice storytellers at best. These days, we're wondering just how much of the overall process of "understanding" and "reasoning" can be reduced to adjacency search in sufficiently absurdly high-dimensional vector space.
      - whimsicalism 4y ago
        People certainly knew that language modeling was a key unsupervised objective to unlock inference on language. I agree that I think they underestimated quite how useful a product could be built around just the language modeling objective, but it's still been critical for most NLP advances of the last ~6+ years.
- nradov 4y ago
  I read that pre-print Microsoft paper. Despite the title, it doesn't actually show any real "sparks" of AGI (in the sense of something that could eventually pass a rigorous Turing test). What the paper actually shows is that even intelligent people have a bias towards perceiving patterns in randomness; our brains seem to be wired that way and this is likely the source of most superstition. https://arxiv.org/abs/2303.12712 https://arxiv.org/abs/2303.12712 While there is no scientific evidence that LLMs can reach AGI, they will still be practically useful for many other tasks. A human mind paired with an LLM is a powerful combination.
  - mirker 4y ago
    Agreed. Here’s the thing: the authors of that paper got early access to GPT-4 and ran a bunch of tests on it. The important bit is that MSR does not see into OpenAI’s sausage making. Now imagine if you were a peasant from 1000 AD who was given a car or TV to examine. Could you really be confident you understood how it worked by just running experiments on it as a black box? If you give a non-programmer the linux kernel, will he/she think it’s magical? Things look like magic especially when you can’t look under the hood. The story of the Mechanical Turk is one example of that.
    - ux-app 4y ago
      >Could you really be confident you understood how it worked by just running experiments on it as a black box the human brain is a black box, we can certainly learn a lot about it by prodding and poking it. >Things look like magic especially when you can’t look under the hood. imagine we had a 100% complete understanding of the mechanical/chemical/electrical functioning of the human brain. Would knowing the magic make it any less magical? in some sense, yes (the mystique would be gone, bye bye dualism), but in a practical sense, not really. It's still an astonishingly useful piece of grey matter.
  - ux-app 4y ago
    >What the paper actually shows is that even intelligent people have a bias towards perceiving patterns in randomness I'm not saying that you're wrong, but... you'd have to provide a more rigorous rebuttal to be taken seriously. AGI can exist without sapience and intelligence is a continuum. you can't just hand wave away GPT's capabilities which is why the sharpest minds on the planet are poking this new machine to work out wtf is going on. human intelligence is a black box. we judge it by its outputs from given inputs. GPT is already producing human-like outputs. a common rebuttal is: "but it doesn't *really* think/understand/feel", to which my response is: ...and? ¯\_(ツ)_/¯ what does that even mean?
AndrewKemendo 4y ago
I had lunch with Yoshua Bengio at the AGI 2014 conference in Laval, CA. This was just before his talk on pathways to AGI via neural networks. Everyone at that conference, including myself, have assumed we will eventually create smarter than human computers and beyond. So it’s not a new position for people who have been in AI for a long time, though generally it was seen as an outsider position until recently. There’s a ton of really great work done prior to all of this around these questions and technical approaches - I think my mentor Ben Goertzel was the pioneer here holistically, but others were doing good technical work then too.
- fatherzine 4y ago
  Possibly he estimated that AGI will come after his death. Like most of us, he was content to do his best work, knowing he will not have to personally deal with the consequences of his own creation. That he is 75 and got worried, now that's an interesting development.
  - AndrewKemendo 4y ago
    I can promise you that this is not the case. Also Yoshua is significantly younger than Geoff.
- 93po 4y ago
  Hey can I ask a question about Ben Goertzel? It's sort of hard to figure out how seriously to take anything he says. Which is maybe a mean thing to say. But his recent crypto venture sort of seems scammy and cash grabby, and the thing he's most well known for (Sophia) seems like sort of a gimmick, so I'm not really sure what to think.
maxdoop 4y ago
Every single retort of “these machines aren’t smart or intelligent” requires answering the question, “what is intelligence”? I struggle to see how GPT-4 is not intelligent by any definition that applies to a human.
- lowbloodsugar 4y ago
  Indeed. The internet and public gatherings are chock full of humans regurgitating rehashed nonsensical statements. Compared against these folks, GPT-4 is more intelligent.
- commandlinefan 4y ago
  > what is intelligence The only way anybody has ever come up with to measure it is test-taking - which machines can already do far better than we can. Real intelligence is creativity, but good luck measuring that.
  - maxdoop 4y ago
    Not sure most would agree that "creativity == intelligence", but I'll go with it: Even assuming that definition, it begs the question of, "what is creativity?"
  - winter_blue 4y ago
    > Real intelligence is creativity Well said. Even Jim Keller (a key designer involved with a lot of major CPUs, in his interview with Lex Freidman) said that there might be some sort of magic, or something magical about human consciousness / the human soul. I agree with that. That's something that a machine will never have.
    - abc_lisper 4y ago
      I think you may be in denial. Douglas Hosftadter thought very deeply about it, wrote a book(GEB) which won a pulitzer 40 years ago, about the "magic" in the brain. He has been worried about developments in AI for 5 years now.
    - 4y ago
ssnistfajen 4y ago
By saying "I no longer think that", it's not necessarily that he thinks ChatGPT is smart than humans. Google Search has been far more capable at indexing and retrieving information than humans for over two decades now. He's talking about AGI no longer being 30-50 years away but instead may arrive far sooner than society is ready to deal with.
- hammyhavoc 4y ago
  Is he? Have you asked him? Maybe somebody should ask him to clarify his statement.
- proto_lambda 4y ago
  Who are you arguing against? You are just repeating the comment you replied to.
e12e 4y ago
I think the question about llms being AGI or not (or "actually" intelligent or not) is interesting, but also somewhat beside the point. We have LLMs that can perform "read and respond", we have systems that can interpret images and sound/speech - and we have plugins that can connect generated output to api calls - that feed back in. Essentially this means that we could already go from "You are an automated home security system. From the front door camera you see someone trying to break in. What do you do?" - to actually building such a system. Maybe it will just place a 911 call, maybe it will deploy a tazer. Maybe the burglar is just a kid in a Halloween costume. The point is that just because you can chain a series of AI/autonomous systems today - with the known, gaping holes - you probably shouldn't. Ed: Crucially the technology is here (in "Lego parts") to construct systems with (for all intents and purposes) real "agency" - that interact both with the real world, and our data (think: purchase a flight based off an email sent to your inbox). I don't think it really matters if these simulacra embody AGI - as long as they already demonstrate agency. Ed2: Or demonstrate behavior so complex that it is indistinguishable to agency for us.
- physicles 4y ago
  This is also the understanding I came to a few weeks ago. LLMs themselves won’t be confused with AGI, but LLMs with tools have the potential to be more powerful than we can anticipate. No leap to “proper” AGI is required to live in a future where AGI functionally exists, and as a result the timeline is much shorter than anyone thought five years ago.
TechBro8615 4y ago
The guy is well past retirement age, so is "quitting his job" evidence of taking an unusually meaningful stance?

Version467 4y ago

He is far from the only one. If you're interested in exploring this further I can really recommend taking a look at some of the papers that explore GPT-4's capabilities. Most prominent among them are the "Sparks of AGI" paper from Microsoft, as well as the technical report from openai. Both of them are obviously to be taken with a grain of salt, but they serve as a pretty good jumping off point. There are some pretty good Videos on Youtube exploring these papers if you don't want to read them yourself. Also take a look at the stuff that Rob Miles has published over on Computerphile, as well as his own channel. He's an Alignment Researcher with a knack for explaining. He covers not just the theoretical dangers, but also real examples of misaligned ai, that alignment researchers have predicted would occur as capabilities grow. Also I think it's important to mention that just a short while ago virtually no-one thought that shoving more layers into an llm would be enough to reach AGI. It's still unclear that it will get us all the way there, but recent developments have made a lot of ai researchers rethink that possibility, with many of them significantly shortening their own estimates as to when and how we will get there. It's very unusual that the people that are better informed and closer to the research are more worried than the rest of the world and it's worth keeping this in mind as you explore the topic.

defgeneric 4y ago
> Also I think it's important to mention that just a short while ago virtually no-one thought that shoving more layers into an llm would be enough to reach AGI. This was basically the strategy of the OpenAI team if I understand them correctly. Most researchers in the field looked down on LLMs and it was a big surprise when they turned out to perform so well. It also seems to be the reason the big players are playing catch up right now.
- whimsicalism 4y ago
  I think it was a surprise the behaviors that were unlocked at different perplexity levels, but I don't really agree that LLMs were "looked down on."
  - TeMPOraL 4y ago
    Maybe not "looked down on", but more of "looked at as a promising avenue". I mean, 2-3 years ago, it felt LLMs are going to be nice storytellers at best. These days, we're wondering just how much of the overall process of "understanding" and "reasoning" can be reduced to adjacency search in sufficiently absurdly high-dimensional vector space.
    - whimsicalism 4y ago
      People certainly knew that language modeling was a key unsupervised objective to unlock inference on language. I agree that I think they underestimated quite how useful a product could be built around just the language modeling objective, but it's still been critical for most NLP advances of the last ~6+ years.
nradov 4y ago
I read that pre-print Microsoft paper. Despite the title, it doesn't actually show any real "sparks" of AGI (in the sense of something that could eventually pass a rigorous Turing test). What the paper actually shows is that even intelligent people have a bias towards perceiving patterns in randomness; our brains seem to be wired that way and this is likely the source of most superstition. https://arxiv.org/abs/2303.12712 https://arxiv.org/abs/2303.12712 While there is no scientific evidence that LLMs can reach AGI, they will still be practically useful for many other tasks. A human mind paired with an LLM is a powerful combination.
- mirker 4y ago
  Agreed. Here’s the thing: the authors of that paper got early access to GPT-4 and ran a bunch of tests on it. The important bit is that MSR does not see into OpenAI’s sausage making. Now imagine if you were a peasant from 1000 AD who was given a car or TV to examine. Could you really be confident you understood how it worked by just running experiments on it as a black box? If you give a non-programmer the linux kernel, will he/she think it’s magical? Things look like magic especially when you can’t look under the hood. The story of the Mechanical Turk is one example of that.
  - ux-app 4y ago
    >Could you really be confident you understood how it worked by just running experiments on it as a black box the human brain is a black box, we can certainly learn a lot about it by prodding and poking it. >Things look like magic especially when you can’t look under the hood. imagine we had a 100% complete understanding of the mechanical/chemical/electrical functioning of the human brain. Would knowing the magic make it any less magical? in some sense, yes (the mystique would be gone, bye bye dualism), but in a practical sense, not really. It's still an astonishingly useful piece of grey matter.
- ux-app 4y ago
  >What the paper actually shows is that even intelligent people have a bias towards perceiving patterns in randomness I'm not saying that you're wrong, but... you'd have to provide a more rigorous rebuttal to be taken seriously. AGI can exist without sapience and intelligence is a continuum. you can't just hand wave away GPT's capabilities which is why the sharpest minds on the planet are poking this new machine to work out wtf is going on. human intelligence is a black box. we judge it by its outputs from given inputs. GPT is already producing human-like outputs. a common rebuttal is: "but it doesn't *really* think/understand/feel", to which my response is: ...and? ¯\_(ツ)_/¯ what does that even mean?

AndrewKemendo 4y ago

I had lunch with Yoshua Bengio at the AGI 2014 conference in Laval, CA. This was just before his talk on pathways to AGI via neural networks. Everyone at that conference, including myself, have assumed we will eventually create smarter than human computers and beyond. So it’s not a new position for people who have been in AI for a long time, though generally it was seen as an outsider position until recently. There’s a ton of really great work done prior to all of this around these questions and technical approaches - I think my mentor Ben Goertzel was the pioneer here holistically, but others were doing good technical work then too.

fatherzine 4y ago
Possibly he estimated that AGI will come after his death. Like most of us, he was content to do his best work, knowing he will not have to personally deal with the consequences of his own creation. That he is 75 and got worried, now that's an interesting development.
- AndrewKemendo 4y ago
  I can promise you that this is not the case. Also Yoshua is significantly younger than Geoff.
93po 4y ago
Hey can I ask a question about Ben Goertzel? It's sort of hard to figure out how seriously to take anything he says. Which is maybe a mean thing to say. But his recent crypto venture sort of seems scammy and cash grabby, and the thing he's most well known for (Sophia) seems like sort of a gimmick, so I'm not really sure what to think.

maxdoop 4y ago

Every single retort of “these machines aren’t smart or intelligent” requires answering the question, “what is intelligence”? I struggle to see how GPT-4 is not intelligent by any definition that applies to a human.

lowbloodsugar 4y ago
Indeed. The internet and public gatherings are chock full of humans regurgitating rehashed nonsensical statements. Compared against these folks, GPT-4 is more intelligent.
commandlinefan 4y ago
> what is intelligence The only way anybody has ever come up with to measure it is test-taking - which machines can already do far better than we can. Real intelligence is creativity, but good luck measuring that.
- maxdoop 4y ago
  Not sure most would agree that "creativity == intelligence", but I'll go with it: Even assuming that definition, it begs the question of, "what is creativity?"
- winter_blue 4y ago
  > Real intelligence is creativity Well said. Even Jim Keller (a key designer involved with a lot of major CPUs, in his interview with Lex Freidman) said that there might be some sort of magic, or something magical about human consciousness / the human soul. I agree with that. That's something that a machine will never have.
  - abc_lisper 4y ago
    I think you may be in denial. Douglas Hosftadter thought very deeply about it, wrote a book(GEB) which won a pulitzer 40 years ago, about the "magic" in the brain. He has been worried about developments in AI for 5 years now.
  - 4y ago

ssnistfajen 4y ago

By saying "I no longer think that", it's not necessarily that he thinks ChatGPT is smart than humans. Google Search has been far more capable at indexing and retrieving information than humans for over two decades now. He's talking about AGI no longer being 30-50 years away but instead may arrive far sooner than society is ready to deal with.

hammyhavoc 4y ago
Is he? Have you asked him? Maybe somebody should ask him to clarify his statement.
proto_lambda 4y ago
Who are you arguing against? You are just repeating the comment you replied to.

e12e 4y ago

I think the question about llms being AGI or not (or "actually" intelligent or not) is interesting, but also somewhat beside the point. We have LLMs that can perform "read and respond", we have systems that can interpret images and sound/speech - and we have plugins that can connect generated output to api calls - that feed back in. Essentially this means that we could already go from "You are an automated home security system. From the front door camera you see someone trying to break in. What do you do?" - to actually building such a system. Maybe it will just place a 911 call, maybe it will deploy a tazer. Maybe the burglar is just a kid in a Halloween costume. The point is that just because you can chain a series of AI/autonomous systems today - with the known, gaping holes - you probably shouldn't. Ed: Crucially the technology is here (in "Lego parts") to construct systems with (for all intents and purposes) real "agency" - that interact both with the real world, and our data (think: purchase a flight based off an email sent to your inbox). I don't think it really matters if these simulacra embody AGI - as long as they already demonstrate agency. Ed2: Or demonstrate behavior so complex that it is indistinguishable to agency for us.

physicles 4y ago
This is also the understanding I came to a few weeks ago. LLMs themselves won’t be confused with AGI, but LLMs with tools have the potential to be more powerful than we can anticipate. No leap to “proper” AGI is required to live in a future where AGI functionally exists, and as a result the timeline is much shorter than anyone thought five years ago.

TechBro8615 4y ago

The guy is well past retirement age, so is "quitting his job" evidence of taking an unusually meaningful stance?