7 ms·

The risk of AI comes from the control problem. We have a vague idea on how to build AIs. The work is just to optimize the algorithms extensively so they can run

The risk of AI comes from the control problem. We have a vague idea on how to build AIs. The work is just to optimize the algorithms extensively so they can run in real time. But we have no idea how to control those AIs. We can push a "reward button" every time it does something we like. But we can't prevent it from killing the programmer and stealing the button for itself. It doesn't matter at all who makes AI first. Without the control problem solved, no one will be able to use it for evil ends, even if they want to. But it's still incredibly dangerous if it gets loose in some uncontrolled setting. So this project is the exact opposite of what you would want to do if you wanted to minimize AI risk. This is rushing towards the risk as fast as possible.

chei0aiV 11y ago
Lets start by figuring out how to control humans before we start on the AIs.
- _yosefk 11y ago
  Actually, unlike controlling humans, whatever that means, it's very easy to limit the impact of AI - all you have to do is treat advanced chip fabrication processes the way nuclear technology is treated, and put a legal limit on processor power so it's somewhere where it was in the late 80s. (So you have office software and email but neither YouTube nor Doom 3; a reasonable price for our survival, I find.) Fewer organizations today can manufacture chips using the latest process nodes than there are governments that can make nuclear weapons, and it's probably harder to build a chip fab in secrecy and make a large enough number of chips to power an AI than it is to build nuclear weapons in secrecy. Berglas argues for this in http://berglas.org/Articles/AIKillGrandchildren/AIKillGrandchildren.html http://berglas.org/Articles/AIKillGrandchildren/AIKillGrandc... Of course the reason governments don't do this is because almost nobody sees the risks of AI "acting on its own" (and they're probably right), nor the risks of things like rogue drones made from cheap commodity hardware (which might be sensible as well - these are more like RPGs in the wrong hands than they are like nukes in the wrong hands.)
  - sawwit 11y ago
    > The risks of things like rogue drones made from cheap commodity hardware, which might be sensible as well … In Prof. Tegmark’s recent presentation at the UN he mentioned the possibility of extremely cheap drones that approach the victim's face very quickly and pierce a bolt into their brain through one of their eyes. Such a drone wouldn't require high precision projectiles which would make it cheap to build. > Of course the reason governments don't do this is because almost nobody sees the risks of AI "acting on its own"… It is near impossible to enforce something like this globally and forever. At best, it would be a near-term solution, especially so because there is a huge military and economic interest in technology and AI. Quite possibly, the only long-term solution is solving the control problem.
    - _yosefk 11y ago
      > extremely cheap drones that approach the victim's face very quickly and pierce a bolt into their brain It's not much more of a threat to society than a handgun. A WMD it is not, unless you make a lot of these and launch them at the same time, which is probably less effective than an H-bomb. (The one major difference between such a drone and a handgun is you might be able to target politicians and other people who're hard to shoot; a somewhat higher mortality rate among politicians is hardly a huge threat to society though.) > there is a huge military and economic interest in technology and AI There's also a huge interest in nuclear energy, and it doesn't follow that a consumer should be or is able to own a nuclear reactor. If anyone took the dangers seriously, it wouldn't be that hard to at least limit what consumer devices can do. Right now a single botnet made from infected consumer PCs has more raw computing power than the biggest tech company server farm, which is ridiculous if you think of rogue AI as an existential threat. Actually it's ridiculous even if you think of "properly controlled" AI in the wrong hands as an existential threat; the botnet could run AI controlled to serve the interest of an evil programmer. Nobody cares because nobody has ever seen an AI that doesn't fail the Turing test in its first couple of minutes, much less come up with anything that would be an existential threat to any society.
      - sawwit 11y ago
- amelius 11y ago
  Let's figure out the human brain first. Let's say you have these two options: (1) A technology that will make you feel the best you possibly can, without any negative consequences, or (2) a robot that will do stuff for you, like cleaning your house, driving you to work. Now which one would you choose?
wyager 11y ago
>But we can't prevent it from killing the programmer and stealing the button for itself. Not to be condescending, but do you have any idea what practical AI actually looks like? The scenario you've imagined makes about as much sense as a laptop sprouting arms and strangling its owner.
- Houshalter 11y ago
  Yes, I'm very familiar with current AI. It's my hobby. I'm not really talking about current AI though. I'm talking about a machine with thousands of times more intelligence than humans.
  - p1esk 11y ago
    thousands of times more intelligence than humans I see this phrase thrown around a lot by Kurzweil and fans. What does it even mean? How do you measure intelligence? Smarter than whom?
    - rpglover64 11y ago
      I don't know if this is a definition other people use, but here's one possibility. Intelligence (in a domain) is measured by how well you solve problems in that domain. If problems in the domain have binary solutions and no external input, a good measure of quality is average time to solution. Sometimes, you can get a benefit by batching the problems, so lets permit that. In other cases, quality is best measured by probability of success given a certain amount of time (think winning a timed chess or go game). Sometimes instead of a binary option, we want to minimize error in a given time (like computing pi). Pick a measure appropriate to the problem. These measures require thinking of the system as a whole, so an AI is not just a program but a physical device, running a program. The domain for the unrestricted claim of intelligence is "reasonable problems". Having an AI tell you what the mass of Jupiter or find Earth-like planets is reasonable. Having it move its arms (when it doesn't have any) is not. Having it move _your_ arms is reasonable, though. The comparison is to the human who is or was most qualified to solve the problem, with the exception of people uniquely qualified to solve the problem (I'm not claiming that the AI is better than you are at moving your own arms).

chei0aiV 11y ago

Lets start by figuring out how to control humans before we start on the AIs.

_yosefk 11y ago
Actually, unlike controlling humans, whatever that means, it's very easy to limit the impact of AI - all you have to do is treat advanced chip fabrication processes the way nuclear technology is treated, and put a legal limit on processor power so it's somewhere where it was in the late 80s. (So you have office software and email but neither YouTube nor Doom 3; a reasonable price for our survival, I find.) Fewer organizations today can manufacture chips using the latest process nodes than there are governments that can make nuclear weapons, and it's probably harder to build a chip fab in secrecy and make a large enough number of chips to power an AI than it is to build nuclear weapons in secrecy. Berglas argues for this in http://berglas.org/Articles/AIKillGrandchildren/AIKillGrandchildren.html http://berglas.org/Articles/AIKillGrandchildren/AIKillGrandc... Of course the reason governments don't do this is because almost nobody sees the risks of AI "acting on its own" (and they're probably right), nor the risks of things like rogue drones made from cheap commodity hardware (which might be sensible as well - these are more like RPGs in the wrong hands than they are like nukes in the wrong hands.)
- sawwit 11y ago
  > The risks of things like rogue drones made from cheap commodity hardware, which might be sensible as well … In Prof. Tegmark’s recent presentation at the UN he mentioned the possibility of extremely cheap drones that approach the victim's face very quickly and pierce a bolt into their brain through one of their eyes. Such a drone wouldn't require high precision projectiles which would make it cheap to build. > Of course the reason governments don't do this is because almost nobody sees the risks of AI "acting on its own"… It is near impossible to enforce something like this globally and forever. At best, it would be a near-term solution, especially so because there is a huge military and economic interest in technology and AI. Quite possibly, the only long-term solution is solving the control problem.
  - _yosefk 11y ago
    > extremely cheap drones that approach the victim's face very quickly and pierce a bolt into their brain It's not much more of a threat to society than a handgun. A WMD it is not, unless you make a lot of these and launch them at the same time, which is probably less effective than an H-bomb. (The one major difference between such a drone and a handgun is you might be able to target politicians and other people who're hard to shoot; a somewhat higher mortality rate among politicians is hardly a huge threat to society though.) > there is a huge military and economic interest in technology and AI There's also a huge interest in nuclear energy, and it doesn't follow that a consumer should be or is able to own a nuclear reactor. If anyone took the dangers seriously, it wouldn't be that hard to at least limit what consumer devices can do. Right now a single botnet made from infected consumer PCs has more raw computing power than the biggest tech company server farm, which is ridiculous if you think of rogue AI as an existential threat. Actually it's ridiculous even if you think of "properly controlled" AI in the wrong hands as an existential threat; the botnet could run AI controlled to serve the interest of an evil programmer. Nobody cares because nobody has ever seen an AI that doesn't fail the Turing test in its first couple of minutes, much less come up with anything that would be an existential threat to any society.
    - sawwit 11y ago
amelius 11y ago
Let's figure out the human brain first. Let's say you have these two options: (1) A technology that will make you feel the best you possibly can, without any negative consequences, or (2) a robot that will do stuff for you, like cleaning your house, driving you to work. Now which one would you choose?

wyager 11y ago

>But we can't prevent it from killing the programmer and stealing the button for itself. Not to be condescending, but do you have any idea what practical AI actually looks like? The scenario you've imagined makes about as much sense as a laptop sprouting arms and strangling its owner.

Houshalter 11y ago
Yes, I'm very familiar with current AI. It's my hobby. I'm not really talking about current AI though. I'm talking about a machine with thousands of times more intelligence than humans.
- p1esk 11y ago
  thousands of times more intelligence than humans I see this phrase thrown around a lot by Kurzweil and fans. What does it even mean? How do you measure intelligence? Smarter than whom?
  - rpglover64 11y ago
    I don't know if this is a definition other people use, but here's one possibility. Intelligence (in a domain) is measured by how well you solve problems in that domain. If problems in the domain have binary solutions and no external input, a good measure of quality is average time to solution. Sometimes, you can get a benefit by batching the problems, so lets permit that. In other cases, quality is best measured by probability of success given a certain amount of time (think winning a timed chess or go game). Sometimes instead of a binary option, we want to minimize error in a given time (like computing pi). Pick a measure appropriate to the problem. These measures require thinking of the system as a whole, so an AI is not just a program but a physical device, running a program. The domain for the unrestricted claim of intelligence is "reasonable problems". Having an AI tell you what the mass of Jupiter or find Earth-like planets is reasonable. Having it move its arms (when it doesn't have any) is not. Having it move _your_ arms is reasonable, though. The comparison is to the human who is or was most qualified to solve the problem, with the exception of people uniquely qualified to solve the problem (I'm not claiming that the AI is better than you are at moving your own arms).