4 ms·

Uh, what? That is entirely what my question is. Whether extrapolating the current state of the art is enough to get us to "the future". If I start walking out

Uh, what? That is entirely what my question is. Whether extrapolating the current state of the art is enough to get us to "the future". If I start walking out the direction of my office door, you could predict I would reach California. But I might say that's a nonsense prediction, because it would take a ridiculous amount of time given my walking pace. So are you predicting that I will reach California by extrapolating my current position + walking velocity, or are you saying I will get on a plane and fly there? These are two very different predictions, and the Carmack interview was the first time I heard someone predict that "I would reach California by walking" (to stretch the analogy), i.e., that we would reach the singularity purely by extrapolating our current trajectory. I am asking if there are any knowledgeable people who might comment on this.

hackinthebochs 7y ago
Right, I get that was your question. But you also said that you initially ignored those raising awareness about the dangers because "there was no way we would reach there with current technologies". I'm addressing that inference in isolation. But to address your analogy, the situation with AI is more like we already know the ultimate destination is California, and you've left your house in the direction of California, so we can extrapolate that you are likely traveling to California. It should be a given that the intended goal is general intelligence. The wealth and power that will be bestowed onto the first group to develop this is unparalleled in human history. But any advance in ML can be seen as taking a step towards general AGI (the direction analogy breaks down here). I also think Carmack is wrong that current technology will get us to AGI. I follow Carmack pretty closely and he is far from knowledgable in AI. He tweeted about his first foray into developing a DNN around a year ago, for example. But to require a roadmap to know how you will get to "California" to start planning for what you will do when you get there is a serious mistake.
- dharmon 7y ago
  Ok. I see what you're saying. Thanks for clarifying. I mean, sure, feel free to wring your hands about AGI, but why worry about something that's not anywhere close to happening? I see people talking like this is keeping them up at night, which I find strange. Do you really think it will go from "we can (poorly) identify pictures of cats" to "omg here come the machines" as soon as this linchpin discovery is made? I guess in my mind there will be plenty of gap between the two to figure things out. Sort of like with self-driving cars now. And thanks for clarifying re: Carmack. He's got a formidable intellect, to say the least, so when he said that I was more inclined to take it seriously than a lot of folks in tech with a pulpit. I'd like to see him "show his work" on how he reached that conclusion, though.
  - hackinthebochs 7y ago
    > I see people talking like this is keeping them up at night, which I find strange. I also find that strange. There is more alarm than is warranted by some of the doomsday folks. But I think the time to begin the conversation is now. There's just far more near-term things that might end humanity than machines to be alarmist about it. That said, I think people like Musk for example are well positions to be "kept up at night" about it since their money and influence can go a long way towards putting us on a proper path (or blindly lead us down a dark one). >Do you really think it will go from "we can (poorly) identify pictures of cats" to "omg here come the machines" as soon as this linchpin discovery is made? I agree that this is unlikely, but not implausible. We've seen large leaps in capabilities through new algorithms and new scales of compute being thrown at the problem. I think a somewhat likely scenario is that at some point we'll have all the required ingredients and someone will happen to put them together in the right way to create an AGI. If history has shown us anything about ML, it is that theory follows the technology. So we will likely stumble upon the result before we realize what we have. In this scenario it is important that we make that discovery in a safe manner.

hackinthebochs 7y ago

Right, I get that was your question. But you also said that you initially ignored those raising awareness about the dangers because "there was no way we would reach there with current technologies". I'm addressing that inference in isolation. But to address your analogy, the situation with AI is more like we already know the ultimate destination is California, and you've left your house in the direction of California, so we can extrapolate that you are likely traveling to California. It should be a given that the intended goal is general intelligence. The wealth and power that will be bestowed onto the first group to develop this is unparalleled in human history. But any advance in ML can be seen as taking a step towards general AGI (the direction analogy breaks down here). I also think Carmack is wrong that current technology will get us to AGI. I follow Carmack pretty closely and he is far from knowledgable in AI. He tweeted about his first foray into developing a DNN around a year ago, for example. But to require a roadmap to know how you will get to "California" to start planning for what you will do when you get there is a serious mistake.

dharmon 7y ago
Ok. I see what you're saying. Thanks for clarifying. I mean, sure, feel free to wring your hands about AGI, but why worry about something that's not anywhere close to happening? I see people talking like this is keeping them up at night, which I find strange. Do you really think it will go from "we can (poorly) identify pictures of cats" to "omg here come the machines" as soon as this linchpin discovery is made? I guess in my mind there will be plenty of gap between the two to figure things out. Sort of like with self-driving cars now. And thanks for clarifying re: Carmack. He's got a formidable intellect, to say the least, so when he said that I was more inclined to take it seriously than a lot of folks in tech with a pulpit. I'd like to see him "show his work" on how he reached that conclusion, though.
- hackinthebochs 7y ago
  > I see people talking like this is keeping them up at night, which I find strange. I also find that strange. There is more alarm than is warranted by some of the doomsday folks. But I think the time to begin the conversation is now. There's just far more near-term things that might end humanity than machines to be alarmist about it. That said, I think people like Musk for example are well positions to be "kept up at night" about it since their money and influence can go a long way towards putting us on a proper path (or blindly lead us down a dark one). >Do you really think it will go from "we can (poorly) identify pictures of cats" to "omg here come the machines" as soon as this linchpin discovery is made? I agree that this is unlikely, but not implausible. We've seen large leaps in capabilities through new algorithms and new scales of compute being thrown at the problem. I think a somewhat likely scenario is that at some point we'll have all the required ingredients and someone will happen to put them together in the right way to create an AGI. If history has shown us anything about ML, it is that theory follows the technology. So we will likely stumble upon the result before we realize what we have. In this scenario it is important that we make that discovery in a safe manner.