3 ms·

I keep hearing that AGI (or any significant AI milestone) is far away and other generic claims like "it's not going to take your job just yet", but I have no id

by atleta 3y ago

I keep hearing that AGI (or any significant AI milestone) is far away and other generic claims like "it's not going to take your job just yet", but I have no idea what people mean when they say these. Like what do you mean by being quite far away? How many years? 10-20-50? On a side note, Geoffrey Hinton just told in an interview that he used to think that it's 20-50 years away (as I said I've never heard a specific number from anyone before this) and now he thinks it's less than 20. Now add to this that people working in the field tended to systematically underestimate the progress. My gut feeling is that if Hinton says less than 20 years then it can easily turn out to be 10 or less. (Also, I very much respect him for coming up with an actual number.) I think AI researchers underestimate the progress partly because they don't want to look stupid by making over-optimistic predictions. However, in this case, I'd argue that the optimistic prediction is saying that it takes longer. Because it will definitely stir up the whole society and economy at the very minimum.

galaxytachyon 3y ago
GPT is a breakthrough in many more ways than just being an advance LLM. GPT3 was released a year or so ago and technically, it was outclassed by PaLM from Google quite a bit in terms of parameter count and Chinchilla in terms of training. What was amazing was they managed to build a scalable system from it, capable of serving millions of users at the same time, and for free. The engineering and backend works must have been astounding and I argue that was the secret sauce for the success of ChatGPT. They did not need to dumb it down or cut corners anywhere. The early releases of ChatGPT and Bing Chat showed they literally put unmodified SOTA models in the hand of users with no price tag attached. These AIs were known a long time ago but only to some people, remember how a bunch of billionaires suddenly got concerned about AIs a year or two ago? I bet they got early access to these LLMs. But only by scaling it up, they can explore the deeper depths of these models and discover new emergent abilities and realize actually how much progress they had made. Before people didn't really expect an LLM to play chess and simulate world models. Now they just found out these things are probably closer to AGI than they thought and the progress bar got pushed forward. Basically my rant is that current progress was made over a long time and people just didn't really realize how far they have come until they opened it up to the public. I would not expect too many surprises in the future on the scale of ChatGPT again. If I am wrong though then we will actually start getting serious candidates for an AGI.
- whimsicalism 3y ago
  > Basically my rant is that current progress was made over a long time and people just didn't really realize how far they have come until they opened it up to the public. As someone in the field, I largely agree with this take but we're still talking about progress over the course of 5 years or so. Also, fine-tuning/RLHF considerably advanced the usability of the models by the lay public and hasn't been around for that long.
  - jmerz 3y ago
    It's crazy how fast humans normalize and adapt. Five years ago this stuff was science fiction, now we're arguing about how it's wrong sometimes. We just kind of walked straight past the Turing test and nobody cares.
    - atleta 3y ago
      I had a teacher at the university a very long time ago (well before even advanced AI computer vision was a thing, around the time DeepBlue beat Kasparov) and when he talked about AI and the future he warned us to take note that as long as machines cannot do something, we keep saying that you need to be intelligent to do it and as long as machines can do it we say that no intelligence is needed (because machines can do it). And as a result we may never admit that AI is actually intelligent or at least not until it's much better than us. On a side note, I think the idea behind the first part of the phenomenon, i.e. that we think you need intelligence (and probably general intelligence) to do most things that machines cannot is that that is the way we do it. So until we could build machines that could calculate (say multiply and divide) we though you needed actual intelligence because we didn't know any other way and that is how we did it. (I remember that calculating and calculators were an actual example in that lesson.) Same thing for chess. And yes, we were right to say that you don't need general intelligence for neither chess nor calculating because we could come up with relatively simple algorithms. But people a few years ago started to say this about go. And I do remember that after DeepBlue beating Kasparov everybody was like "yeah, but you need intelligence for go because that's a game with vastly more possible moves". And in a sense this is what we saw, because AlphaGo was indeed a kidn of AI, but still people started saying that you don't need intelligence to win go - as long as you are a machine. Now people started to make up shit about how GPT is just generating text and that's not intelligent (some people, obviously ignorant laypeople, even say that it's just copy pasting text together and other nonsense). Despite that the freaking thing passes high level exams aimed at people. (Of course, you can say that it's still not a sign of intelligence, because the exams do not measure that, we know people are more or less intelligent, the exam measures knowledge, but the amazing thing here is not the knowledge part but being able to answer the questions aimed at people.)

by atleta 3y ago

galaxytachyon 3y ago

GPT is a breakthrough in many more ways than just being an advance LLM. GPT3 was released a year or so ago and technically, it was outclassed by PaLM from Google quite a bit in terms of parameter count and Chinchilla in terms of training. What was amazing was they managed to build a scalable system from it, capable of serving millions of users at the same time, and for free. The engineering and backend works must have been astounding and I argue that was the secret sauce for the success of ChatGPT. They did not need to dumb it down or cut corners anywhere. The early releases of ChatGPT and Bing Chat showed they literally put unmodified SOTA models in the hand of users with no price tag attached. These AIs were known a long time ago but only to some people, remember how a bunch of billionaires suddenly got concerned about AIs a year or two ago? I bet they got early access to these LLMs. But only by scaling it up, they can explore the deeper depths of these models and discover new emergent abilities and realize actually how much progress they had made. Before people didn't really expect an LLM to play chess and simulate world models. Now they just found out these things are probably closer to AGI than they thought and the progress bar got pushed forward. Basically my rant is that current progress was made over a long time and people just didn't really realize how far they have come until they opened it up to the public. I would not expect too many surprises in the future on the scale of ChatGPT again. If I am wrong though then we will actually start getting serious candidates for an AGI.

whimsicalism 3y ago
> Basically my rant is that current progress was made over a long time and people just didn't really realize how far they have come until they opened it up to the public. As someone in the field, I largely agree with this take but we're still talking about progress over the course of 5 years or so. Also, fine-tuning/RLHF considerably advanced the usability of the models by the lay public and hasn't been around for that long.
- jmerz 3y ago
  It's crazy how fast humans normalize and adapt. Five years ago this stuff was science fiction, now we're arguing about how it's wrong sometimes. We just kind of walked straight past the Turing test and nobody cares.
  - atleta 3y ago
    I had a teacher at the university a very long time ago (well before even advanced AI computer vision was a thing, around the time DeepBlue beat Kasparov) and when he talked about AI and the future he warned us to take note that as long as machines cannot do something, we keep saying that you need to be intelligent to do it and as long as machines can do it we say that no intelligence is needed (because machines can do it). And as a result we may never admit that AI is actually intelligent or at least not until it's much better than us. On a side note, I think the idea behind the first part of the phenomenon, i.e. that we think you need intelligence (and probably general intelligence) to do most things that machines cannot is that that is the way we do it. So until we could build machines that could calculate (say multiply and divide) we though you needed actual intelligence because we didn't know any other way and that is how we did it. (I remember that calculating and calculators were an actual example in that lesson.) Same thing for chess. And yes, we were right to say that you don't need general intelligence for neither chess nor calculating because we could come up with relatively simple algorithms. But people a few years ago started to say this about go. And I do remember that after DeepBlue beating Kasparov everybody was like "yeah, but you need intelligence for go because that's a game with vastly more possible moves". And in a sense this is what we saw, because AlphaGo was indeed a kidn of AI, but still people started saying that you don't need intelligence to win go - as long as you are a machine. Now people started to make up shit about how GPT is just generating text and that's not intelligent (some people, obviously ignorant laypeople, even say that it's just copy pasting text together and other nonsense). Despite that the freaking thing passes high level exams aimed at people. (Of course, you can say that it's still not a sign of intelligence, because the exams do not measure that, we know people are more or less intelligent, the exam measures knowledge, but the amazing thing here is not the knowledge part but being able to answer the questions aimed at people.)