> I see people talking like this is keeping them up at night, which I find strange. I also find that strange. There is more alarm than is warranted by some of

> I see people talking like this is keeping them up at night, which I find strange. I also find that strange. There is more alarm than is warranted by some of the doomsday folks. But I think the time to begin the conversation is now. There's just far more near-term things that might end humanity than machines to be alarmist about it. That said, I think people like Musk for example are well positions to be "kept up at night" about it since their money and influence can go a long way towards putting us on a proper path (or blindly lead us down a dark one). >Do you really think it will go from "we can (poorly) identify pictures of cats" to "omg here come the machines" as soon as this linchpin discovery is made? I agree that this is unlikely, but not implausible. We've seen large leaps in capabilities through new algorithms and new scales of compute being thrown at the problem. I think a somewhat likely scenario is that at some point we'll have all the required ingredients and someone will happen to put them together in the right way to create an AGI. If history has shown us anything about ML, it is that theory follows the technology. So we will likely stumble upon the result before we realize what we have. In this scenario it is important that we make that discovery in a safe manner.