5 ms·

I agree that the rate of progress in AI is unpredictable, which means we are probably not right on the cusp of superhuman AI. But what if we actually are on the

by rw 11y ago

I agree that the rate of progress in AI is unpredictable, which means we are probably not right on the cusp of superhuman AI. But what if we actually are on the precipice? How could you tell? You seem to be taking a bet on the following statement: "Before superhuman AI is developed, the techniques that make it possible will look dangerous." That's a risky proposition. There's so much to lose in this situation. Elon Musk, Stephen Hawking, and many other people are taking a different, more risk-averse bet: "Superhuman AI is extremely dangerous, so we need to be pessimistic about how good we are at predicting when it will happen." By that logic, general AI is to be feared and worried about right now, because our predictive abilities are imperfect. The AI trend is towards more danger, not less: there's a slight chance of a cataclysmic event happening today, and as time goes on, the likelihood of it happening will increase (due to ongoing R&D in AI). (As an aside, if you want to learn how hard the "AI Control Problem" is, I recommend the book "Superintelligence" by Nick Bostrum.)

jbhatab 11y ago
I agree with this completely and I think a simple board or coalition of companies and governments will be the answer. Regulation, policies, and awareness are the solutions to this problem, but thinking because it isn't a problem now that it won't be a potential problem at all is a scary thought.
vonnik 11y ago
The real cost of fearing an AGI now is an opportunity cost. The technology is a long ways away. Other events -- such as ecological collapse or nuclear disaster to name two -- are much more likely to irremediably affect our lives. So by hopping on Bostrom's fear wagon, you are making a bet on several levels: a) this matters now; b) this matters more than other dangers we could devote ourselves to minimizing; and c) something can actually be done about it. All three bets seem like bad ones to make to me.
- rw 11y ago
  Fair comparison, but I didn't intend any monopoly on fear. All I'm saying is that the risk of GAI should be taken seriously, calibrated along with the other problems you mention. There's a lot of room between 'irrelevant' and 'the most important issue' :-)
  - argonaut 11y ago
    Yes, and given the $1B in funding that OpenAI has, I'd seriously say that too much attention is given to the AI "risk."
    - Houshalter 11y ago
      OpenAI has nothing to do with AI risk. They want to accelerate research in AI and make it open to the public. Which is the exact opposite you would want to do to protect against superintelligent AIs. They are not focusing on solving the control problem, or anything meaningful.
      - argonaut 11y ago
        This is clearly false. OpenAI would not exist if Sam Altman and Elon Musk were not worried about AI.
      - vonnik 11y ago
        Actually, @Houshalter, OpenAI has several purposes, and one of them is to address AI risk. You may not think that is an effective approach, but that is their stated intent. They believe that the best hedge against the dangers of superintelligence is to conduct open research that encourages a multipolar world of several AGIs, rather a unipolar world where one company, such as Google, reaps the benefits and determines the course of such an advance. It's true that OpenAI accelerates AI research, which makes it paradoxical. But that's the nature of a hedge. You bet on both sides at once. I do believe that they will focus on solving meaningful problems (they are only a few weeks old...).
        Houshalter 11y ago
        >rather a unipolar world where one company, such as Google, reaps the benefits and determines the course of such an advance Yes that is their model of AI risk. However the people worried about AI risk are not worried about any single organization controlling AI. They are worried about a world where no one controls AI. Controlling AI is actually a very difficult and unsolved problem. OpenAI is not intended to solve that problem, and if anything they are making it far worse by accelerating AI research before the problem has been solved.
- Houshalter 11y ago
  I would make those bets. I'm not afraid of those other things at all. Nukes are really unlikely to be used, and even if they were it wouldn't destroy the whole world, let alone end humanity. Climate change is going to cause lots of economic damage at worst, but it isn't going to destroy the world, let alone end humanity. Same with many other proposed risks. AI is the only thing I believe has a very high probability of destroying all humanity. Probably within our lifetimes. And there is probably something we can do about it right now - working on the control problem.
  - vonnik 11y ago
    It seems to me like you're discounting real and present dangers in favor of one that's more speculative. The reason why climate change and nuclear disasters (I did not limit it to nuclear weapons, but am referring also to events like Chernobyl and Fukushima, which have poisoned significant regions of the planet for millennia to come) are real and present is because a) climate change is already under way and causing massive demographic shifts; and b) nuclear technology is already in a state to destroy large parts of the species and render much of the planet uninhabitable. We are one button click away from a bomb going off at any time, and there are lots of crazy fingers in the world itching for a button.
    - Houshalter 11y ago
      Even nuclear meltdowns aren't' enough to literally destroy the planet. In the worst case they might kill a few thousand people and cause billions in economic damage. Which sounds like a lot, but on the scale of the whole planet is almost nothing. All of them are real dangers, and present dangers at that. But they still are not existential dangers. Even in the worst case humanity wouldn't go extinct. It's unlikely even modern civilization would collapse, just a few countries. AI is one of the few things that can literally wipe out humanity, or even the entire universe. So I think it's worth special consideration. Personally I also believe it's far more likely as well. And people are already doing things about climate change and nuclear proliferation. Until very recently, there were only a few people working on AI risk, and it's still nowhere near enough.
otakucode 11y ago
I've read many arguments concerned with machine intelligence and the risks it poses, but I haven't read that book. Does it tackle explaining why we should conclude that a machine intelligence would even communicate with us, much less seize resources that we need? I rarely see it addressed, and have never seen it addressed to my own satisfaction; what it is about humans that would be worth the trouble of eliminating. I'm also really interested in the communication question. I do not see any clear reason why a machine intelligence would develop "individuals" or would develop the ability to communicate with us.
- Houshalter 11y ago
  Yes that's one of the important topics covered in the book. The AI probably would probably value resources and self preservation. Resources help it fulfil whatever it's goal is, and the more the better. That includes energy and computing power. Which it could gain a lot of by converting the mass of the solar system into a dyson swarm. Or covering the Earth with solar panels or fusion reactors and comuters. Etc. Self preservation would be optimized by destroying anything that has even a tiny chance of being a threat to it. Even if humans are harmless, we could build other AIs like it which might compete with it. It's best to just destroy us. The problem is the AI doesn't need to want to eliminate us. It will likely be entirely indifferent to us. We just need to be in it's way.
- JoshTriplett 11y ago
  > what it is about humans that would be worth the trouble of eliminating Among other reasons, we're made of matter that isn't actively contributing to its value function as efficiently as it could be. Any AI that doesn't actively value sapient beings will very likely destroy them as an unintentional side effect of whatever it intends to do.
argonaut 11y ago
The problem is you could say the exact same thing about an alien invasion (you could literally substitute appropriate analogues in your statement), but you'd be laughed out of the room.
- Houshalter 11y ago
  We've searched the skies exhaustively and found no evidence whatsoever of alien civilizations. And why is the Earth is still around after 4 billion years if aliens want it? If they didn't show up in the last few million years, they probably won't in the next few million. Same logic applies to most natural threats like meteors and volcanoes. AI is a threat that is immediate in time. There is plenty of reason to believe we will build it, and probably within our lifetimes. And unlike aliens, there might be something we can do about it. Whereas aliens would be so far beyond us technologically it's very unlikely we can do anything to stop them. I know you are just pattern matching "both of those things are from science fiction, therefore shouldn't be taken seriously". But there are actual reasons why aliens aren't taken seriously and AI is.
  - argonaut 11y ago
    We've searched the skies exhaustively and found no evidence whatsoever of intelligence * * except [human intelligence] / [human civilization]

by rw 11y ago

jbhatab 11y ago

I agree with this completely and I think a simple board or coalition of companies and governments will be the answer. Regulation, policies, and awareness are the solutions to this problem, but thinking because it isn't a problem now that it won't be a potential problem at all is a scary thought.

vonnik 11y ago

The real cost of fearing an AGI now is an opportunity cost. The technology is a long ways away. Other events -- such as ecological collapse or nuclear disaster to name two -- are much more likely to irremediably affect our lives. So by hopping on Bostrom's fear wagon, you are making a bet on several levels: a) this matters now; b) this matters more than other dangers we could devote ourselves to minimizing; and c) something can actually be done about it. All three bets seem like bad ones to make to me.

rw 11y ago
Fair comparison, but I didn't intend any monopoly on fear. All I'm saying is that the risk of GAI should be taken seriously, calibrated along with the other problems you mention. There's a lot of room between 'irrelevant' and 'the most important issue' :-)
- argonaut 11y ago
  Yes, and given the $1B in funding that OpenAI has, I'd seriously say that too much attention is given to the AI "risk."
  - Houshalter 11y ago
    OpenAI has nothing to do with AI risk. They want to accelerate research in AI and make it open to the public. Which is the exact opposite you would want to do to protect against superintelligent AIs. They are not focusing on solving the control problem, or anything meaningful.
    - argonaut 11y ago
      This is clearly false. OpenAI would not exist if Sam Altman and Elon Musk were not worried about AI.
    - vonnik 11y ago
      Actually, @Houshalter, OpenAI has several purposes, and one of them is to address AI risk. You may not think that is an effective approach, but that is their stated intent. They believe that the best hedge against the dangers of superintelligence is to conduct open research that encourages a multipolar world of several AGIs, rather a unipolar world where one company, such as Google, reaps the benefits and determines the course of such an advance. It's true that OpenAI accelerates AI research, which makes it paradoxical. But that's the nature of a hedge. You bet on both sides at once. I do believe that they will focus on solving meaningful problems (they are only a few weeks old...).
      - Houshalter 11y ago
        >rather a unipolar world where one company, such as Google, reaps the benefits and determines the course of such an advance Yes that is their model of AI risk. However the people worried about AI risk are not worried about any single organization controlling AI. They are worried about a world where no one controls AI. Controlling AI is actually a very difficult and unsolved problem. OpenAI is not intended to solve that problem, and if anything they are making it far worse by accelerating AI research before the problem has been solved.
Houshalter 11y ago
I would make those bets. I'm not afraid of those other things at all. Nukes are really unlikely to be used, and even if they were it wouldn't destroy the whole world, let alone end humanity. Climate change is going to cause lots of economic damage at worst, but it isn't going to destroy the world, let alone end humanity. Same with many other proposed risks. AI is the only thing I believe has a very high probability of destroying all humanity. Probably within our lifetimes. And there is probably something we can do about it right now - working on the control problem.
- vonnik 11y ago
  It seems to me like you're discounting real and present dangers in favor of one that's more speculative. The reason why climate change and nuclear disasters (I did not limit it to nuclear weapons, but am referring also to events like Chernobyl and Fukushima, which have poisoned significant regions of the planet for millennia to come) are real and present is because a) climate change is already under way and causing massive demographic shifts; and b) nuclear technology is already in a state to destroy large parts of the species and render much of the planet uninhabitable. We are one button click away from a bomb going off at any time, and there are lots of crazy fingers in the world itching for a button.
  - Houshalter 11y ago
    Even nuclear meltdowns aren't' enough to literally destroy the planet. In the worst case they might kill a few thousand people and cause billions in economic damage. Which sounds like a lot, but on the scale of the whole planet is almost nothing. All of them are real dangers, and present dangers at that. But they still are not existential dangers. Even in the worst case humanity wouldn't go extinct. It's unlikely even modern civilization would collapse, just a few countries. AI is one of the few things that can literally wipe out humanity, or even the entire universe. So I think it's worth special consideration. Personally I also believe it's far more likely as well. And people are already doing things about climate change and nuclear proliferation. Until very recently, there were only a few people working on AI risk, and it's still nowhere near enough.

otakucode 11y ago

I've read many arguments concerned with machine intelligence and the risks it poses, but I haven't read that book. Does it tackle explaining why we should conclude that a machine intelligence would even communicate with us, much less seize resources that we need? I rarely see it addressed, and have never seen it addressed to my own satisfaction; what it is about humans that would be worth the trouble of eliminating. I'm also really interested in the communication question. I do not see any clear reason why a machine intelligence would develop "individuals" or would develop the ability to communicate with us.

Houshalter 11y ago
Yes that's one of the important topics covered in the book. The AI probably would probably value resources and self preservation. Resources help it fulfil whatever it's goal is, and the more the better. That includes energy and computing power. Which it could gain a lot of by converting the mass of the solar system into a dyson swarm. Or covering the Earth with solar panels or fusion reactors and comuters. Etc. Self preservation would be optimized by destroying anything that has even a tiny chance of being a threat to it. Even if humans are harmless, we could build other AIs like it which might compete with it. It's best to just destroy us. The problem is the AI doesn't need to want to eliminate us. It will likely be entirely indifferent to us. We just need to be in it's way.
JoshTriplett 11y ago
> what it is about humans that would be worth the trouble of eliminating Among other reasons, we're made of matter that isn't actively contributing to its value function as efficiently as it could be. Any AI that doesn't actively value sapient beings will very likely destroy them as an unintentional side effect of whatever it intends to do.

argonaut 11y ago

The problem is you could say the exact same thing about an alien invasion (you could literally substitute appropriate analogues in your statement), but you'd be laughed out of the room.

Houshalter 11y ago
We've searched the skies exhaustively and found no evidence whatsoever of alien civilizations. And why is the Earth is still around after 4 billion years if aliens want it? If they didn't show up in the last few million years, they probably won't in the next few million. Same logic applies to most natural threats like meteors and volcanoes. AI is a threat that is immediate in time. There is plenty of reason to believe we will build it, and probably within our lifetimes. And unlike aliens, there might be something we can do about it. Whereas aliens would be so far beyond us technologically it's very unlikely we can do anything to stop them. I know you are just pattern matching "both of those things are from science fiction, therefore shouldn't be taken seriously". But there are actual reasons why aliens aren't taken seriously and AI is.
- argonaut 11y ago
  We've searched the skies exhaustively and found no evidence whatsoever of intelligence * * except [human intelligence] / [human civilization]