4 ms·
> I mean, the paperclip example in the Medium article... did that factory ever need AI involved? That example got a bit lost in translation. The standard exam
by JoshTriplett 11y ago
> I mean, the paperclip example in the Medium article... did that factory ever need AI involved?
That example got a bit lost in translation. The standard example of the paperclipping AI is not about building an AI to run a paperclip factory. It's about providing an appropriate value system for an AI, so that it can evaluate courses of action. The standard example: you use a paperclip as an example for an AI as the smallest unit of incremental value, such that having a paperclip is epsilon better than not having a paperclip. The AI files that information away, and tiles the universe with paperclips. Related examples include using a smiling human face as an example of happiness and having the AI tile the universe with the smallest possible object that matches its "smiling human face" recognizer.
> And they also kind of assume that there's only one AI at a time
Yes. If you're going to build a strong AI, then either you got its value system right, and it will take over the world (in the good, "solves all problems at once" way), or you got its value system wrong, and it will destroy the world. There is no plausible scenario in which we build strong AI (as in, capable of self-improvement to better satisfy its value function) and it doesn't take over the world, or in which it allows other strong AIs to exist that are not effectively subroutines of itself (any AI that shares the same value system is the same AI, and any AI that doesn't share the same value system is too dangerous to exist).
> that everything is connected to the internet in some way, that humanity couldn't simply wipe out anything that poses a threat (and no, an AI in a factory environment wouldn't get access to anything that would allow for nuclear/chemical/biological weapons).
You can't keep a strong AI capable of self-improvement in a "box". Random example: a security conference this year demonstrated using nothing but DRAM to successfully transmit short-range GSM signals. That's something humans thought up and implemented.
That's leaving aside intentional human intervention or development-and-release. Not least of which because a strong AI done right provides a massive reduction in both existential risk for humanity and day-to-day risk for humans.
- Eliezer 11y agoThat DRAM thing is getting headlined on the danger of boxing if there's a source. Source?
- mietek 11y ago“GSMem: Data Exfiltration from Air-Gapped Computers over GSM Frequencies” https://www.usenix.org/conference/usenixsecurity15/technical-sessions/presentation/guri https://www.usenix.org/conference/usenixsecurity15/technical...
- Eliezer 11y agoThank you greatly!
- jimrandomh 11y agoThe DRAM thing isn't a great good example, because it requires an outside actor doing something unusual and deliberate at short range to receive the message. It's interesting mainly because it doesn't require control of anything beyond memory-access pattern, which means that some machine learning algorithms could do it without needing the intermediate step of gaining arbitrary code execution or even acquiring the control necessary to make a signal out of their heating/cooling.
- Eliezer 11y agoIt's a good example of "no, seriously, AIs might figure out how to do weird stuff".
- sgt101 11y agoBut this is not an example about intelligence, it's an example about uncontrolled actuators. A paperclip making machine could be made without any intelligence involved, and many people have been concerned about a grey goo type scenario which is similar. "There is no plausible scenario in which we build a strong AI... and it doesn't take over the world" First off you are using polemic here with the "plausible" card - it seems to me as a reader that you are saying "I won't consider any other argument". I think you should. There are in fact no proofs that a strong AI will take over, I can't imagine any way that anyone could construct such a proof given that it appears that we are dealing with concepts (consciousness, intelligence, freedom of will) which are under defined. Additionally there is no indication that "intelligence" beyond human level is at all possible. What if the strong AI doesn't want to take over?
- JoshTriplett 11y ago> A paperclip making machine could be made without any intelligence involved And very much should be, yes. It's extraordinarily dangerous to build any strong AI (or optimization process) with a value function less general than one that will best satisfy the value functions of sapient beings. (Side note: that doesn't necessarily mean we need to build the AI to itself be sapient; I've seen the term "optimization process" used as a substitute for AI, to imply that it need not have any agency of its own, only a goal to satisfy people's goals.) Any strong AI that doesn't specifically value "people" will hurt people as a side effect of maximizing whatever it does value. >> "There is no plausible scenario in which we build a strong AI... and it doesn't take over the world" > First off you are using polemic here with the "plausible" card I very specifically mentioned "strong AI": AI capable of self-improvement to better satisfy some value function. We could quibble over the word "capable", but I'd argue that a computer with the capability of self-improvement that specifically chooses not to do so seems rather pointless; why build the capability if not to use it? > There are in fact no proofs that a strong AI will take over I used "take over" very loosely there. A strong AI will self-improve to maximize its value function, whatever that value function is. Most interesting value functions tend to benefit from more resources. Even seemingly trivial value functions can potentially blow up if the system decides it could benefit by expanding the amount of computational capacity it has available; more computational capacity requires more hardware, which requires more matter; you're made of matter, and that matter isn't currently being used optimally to maximize the value function. > Additionally there is no indication that "intelligence" beyond human level is at all possible. In which case my statement is vacuously true. :)