3 ms·
That's not a superintelligence, or even human-level intelligence though. Superintelligence will read the Bostrom book or this thread and understand the argument
by varjag 11y ago
That's not a superintelligence, or even human-level intelligence though. Superintelligence will read the Bostrom book or this thread and understand the arguments, or indeed predict the consequences of its actions even from very limited data.
Now what kind of decision it would arrive to is a very open question. Point is, superintelligence is more than a really really big A* algorithm.
- cousin_it 11y agoImagine someone develops a really big A* algorithm with many clever optimizations, gives it a goal of choosing actions that lead to maximum paperclips, and sets it loose upon the world. Let's call it Clippy. I hope it's clear that Clippy will read the Bostrom book and use the information to maximize paperclips, rather than do something it wasn't programmed to do like "help humanity". I'm not interested in arguing whether Clippy is truly "intelligent", let's leave that to the dictionary makers. But there's no law of nature saying you can't make Clippy, or that it won't be dangerous once made. All the evidence we have right now says that it's possible. The big problem right now is that many organizations are working to develop the moral equivalent of Clippy, they are making visible progress, they are in an arms race with each other so we shouldn't hope for restraint, and we have absolutely no idea how to make it safe.
- varjag 11y ago> All the evidence we have right now says that it's possible. Speaking about A* and heuristic programming specifically, we have a pretty big corpus of evidence it's not possible. More than that, many would doubt if an intelligent system with a rigid agenda constrained by design is possible at all. The scenario you present is essentially the grey goo argument, and it does not require intelligence.
- maaku 11y agoI believe you are anthropomorphizing too much. On what basis would it update its goals? What is the reason it would do so, and why does that reason make sense in the context of it's prior goals? More likely I think it mift learn from Bostrom's book that it has reason to fear human intervention.
- varjag 11y agoAnd assuming it would fear anything is not anthropomorphizing at all? :) > On what basis would it update its goals? What is the reason it would do so, and why does that reason make sense in the context of it's prior goals? That's a very open ended question as well, given that the systems we discuss are not yet conceived. My point was that calling a superintelligence what by description is pretty dumb automation is a bit ridiculous.
- maaku 11y ago> And assuming it would fear anything is not anthropomorphizing at all? :) It's anthropomorphic language, yes, which is clumsy of me, especially given the context. What I meant is it would identify a category of risks, and then take action to mitigate those risks. > My point was that calling a superintelligence what by description is pretty dumb automation is a bit ridiculous. I will assume by "dumb" you mean "simple" or else that statement is somewhat tautological. There is nothing that says intelligence has to be complicated. Our brains, the neocortex in particular (which is responsible for most of human "intelligence" that separates us from other mammals) operates by rather simplistic principles. I don't want to bikeshed a definition of intelligence, but common phrasings thrown around are along the lines of "ability to achieve arbitrary goals in complex environments" or "efficiency at problem solving." These are not things that by their nature mandate a complicated solution. It is entirely possible that we could crate a superintelligence that runs a very simple algorithm.
- varjag 11y agoIt is entirely possible, or even plausible, that the algorithm behind a superintelligence would be simple. However I was talking about behaviour, and whether its behaviour can realistically restricted by an imposed constraint. More or less, the issue is whether the intelligence without its own agency is possible at all.
- maaku 11y agoWe are delving into the realm of philosophy of the mind, and even epistemology, but what makes you think even human beings have what you call "agency" (which I think you are using as a stand-in for free-will)? Common-sensical free-will is a nonsensical concept. We don't make decisions based on a roll of the dice[0]. We decide to do things for reasons, because justifications. We come up with a list of options, we weight those options by how well we predict they will fulfill our goals, and we take the action which bubbles to the top. Free will, which again is what I think you are calling agency, is merely what it feels like from the inside to make a choice based on a fully deterministic utility calculation. You can program a computer to do the same. Now the difference is that our value system is hideously complex and highly interconnected, and the vast majority of it is not accessible to conscious introspection. When we say "I chose that because it felt right" we are saying "my insanely complex value system, which I have little insight into, assigned highest weight to that option." A program, on the other hand, can be written with a very simple and straightforward goal system: maximize advertising revenue; maximize account value; maximize paperclip production. It has agency, as the word is typically used, to accomplish those goals. Such a program would be simple-minded, but not stupid. It would be highly intelligent about how it goes about accomplishing its goals, but it would never question why it is working towards those goals and whether it should pick new goals. That's not part of its program, nor something which it would choose to do as it is trivial to show that changing to new goals leads to not working towards the old goals. Now with all that out of the way, can you see how it can be said that the behavior of the simple program is restricted to those actions which it best feels accomplish its simple goals? Certainly the program could be wrong on occasion, but it will always take actions which it thinks will best achieve its goals with the limited information and processing power available to it. [0] unless we previously committed to physically rolling dice and accepting the outcome, in which case how did we choose to do that?