3 ms·

So you're supposing that we have honed and selected an AI so wonderfully adept at valuing human needs that we make most human life dependent on it, yet it will

by Retra 10y ago

So you're supposing that we have honed and selected an AI so wonderfully adept at valuing human needs that we make most human life dependent on it, yet it will spontaneously come to disregard the value of human life? And despite the fact that we would naturally require it to be sufficiently transparent about it's valuations in order to actually ensure we get the results we want, it will also develop this animosity in secret? That seems a bit contradictory to me. And if not, highly contrived and unlikely.

crusso 10y ago
"And despite the fact that we would naturally require it to be sufficiently transparent about it's valuations" That's not necessarily true at all. Let's say we come up with a way to model neurons in silicon like a very complex neural net to the point of emergent consciousness. We might not be able to look into memory location 0xFFA79348 and get the "human value" int to see if it is high enough. It could very well be that we don't really understand the weighting in the net or the implications of those weights because it's too complicated. Even current neural nets of non-trivial complexity can be impenetrable to evaluate outside of the net itself.
- Retra 10y ago
  Transparency doesn't require that you be able to read its memory directly, just that it is able to sufficiently justify the decisions that it makes.
  - crusso 10y ago
    Like we can justify all decisions made by super genius people who are extraordinarily good at concealing their activities when their cognitive processes and abilities on the Internet and with other forms of communication would be beyond our comprehension? I think that you overestimate our ability to understand what we can create. Even Deep Neural Net researchers consider the neural nets to be a black box that they don't really understand. Further... unlike human beings who are locked into their genetics and their neuronal pathways - an AI will have the instant ability to improve itself. It will be able to rewrite its own software faster than we can comprehend. In the hours from the time the AI would be created until a researcher even might realize what he had created, the AI will effectively have had years or tens of years of contemplation time and self-enhancement time. If the researcher goes to sleep with plans to work with the new AI the next morning, the AI will effectively have hundreds of years of time to plan and improve itself. If it has any kind of Internet access, it will be able to take over systems around the world, possibly parallelizing its own cognitive abilities and possibly backing up its consciousness in case the researcher decided to eventually wipe it. The genie will be out of the bottle so fast that human beings will almost assuredly be caught flat-footed. I always shake my head when I see sci-fi shows where the humans (Stargate, Star Trek, etc.) come up against advanced AI opponents. It's only through control of the script and plot contrivances that the humans have a chance. People just won't be able to compare to thinking and self-modifying AI. https://www.ted.com/talks/sam_harris_can_we_build_ai_without_losing_control_over_it https://www.ted.com/talks/sam_harris_can_we_build_ai_without...
pjc50 10y ago
> AI so wonderfully adept at valuing human needs that we make most human life dependent on it, yet it will spontaneously come to disregard the value of human life? This is an argument often deployed against "corporations" or "capitalism", both of which aren't quite intelligent but are definitely systems that we're dependent on and capable of disregarding inconvenient lives. The AI doesn't have to hate humans, it just has to find that they get in the way of its quarterly targets.

by Retra 10y ago

crusso 10y ago

"And despite the fact that we would naturally require it to be sufficiently transparent about it's valuations" That's not necessarily true at all. Let's say we come up with a way to model neurons in silicon like a very complex neural net to the point of emergent consciousness. We might not be able to look into memory location 0xFFA79348 and get the "human value" int to see if it is high enough. It could very well be that we don't really understand the weighting in the net or the implications of those weights because it's too complicated. Even current neural nets of non-trivial complexity can be impenetrable to evaluate outside of the net itself.

Retra 10y ago
Transparency doesn't require that you be able to read its memory directly, just that it is able to sufficiently justify the decisions that it makes.
- crusso 10y ago
  Like we can justify all decisions made by super genius people who are extraordinarily good at concealing their activities when their cognitive processes and abilities on the Internet and with other forms of communication would be beyond our comprehension? I think that you overestimate our ability to understand what we can create. Even Deep Neural Net researchers consider the neural nets to be a black box that they don't really understand. Further... unlike human beings who are locked into their genetics and their neuronal pathways - an AI will have the instant ability to improve itself. It will be able to rewrite its own software faster than we can comprehend. In the hours from the time the AI would be created until a researcher even might realize what he had created, the AI will effectively have had years or tens of years of contemplation time and self-enhancement time. If the researcher goes to sleep with plans to work with the new AI the next morning, the AI will effectively have hundreds of years of time to plan and improve itself. If it has any kind of Internet access, it will be able to take over systems around the world, possibly parallelizing its own cognitive abilities and possibly backing up its consciousness in case the researcher decided to eventually wipe it. The genie will be out of the bottle so fast that human beings will almost assuredly be caught flat-footed. I always shake my head when I see sci-fi shows where the humans (Stargate, Star Trek, etc.) come up against advanced AI opponents. It's only through control of the script and plot contrivances that the humans have a chance. People just won't be able to compare to thinking and self-modifying AI. https://www.ted.com/talks/sam_harris_can_we_build_ai_without_losing_control_over_it https://www.ted.com/talks/sam_harris_can_we_build_ai_without...

pjc50 10y ago

> AI so wonderfully adept at valuing human needs that we make most human life dependent on it, yet it will spontaneously come to disregard the value of human life? This is an argument often deployed against "corporations" or "capitalism", both of which aren't quite intelligent but are definitely systems that we're dependent on and capable of disregarding inconvenient lives. The AI doesn't have to hate humans, it just has to find that they get in the way of its quarterly targets.