3 ms·
Trollywood inspired fear mongering. I said so before, and will do so again: If the emerged thing is so intelligent, what would stop it from establishing a netwo
by LargoLasskhyfv 6y ago
Trollywood inspired fear mongering. I said so before, and will do so again: If the emerged thing is so intelligent, what would stop it from establishing a network of shell companies, buying chip-, robotics-, manufacturing-, aerospace companies, or set them up somewhere in a favourable jurisdiction and pull a CyberX by launching itself into space, mine asteroids, solar winds, expand, explore, whatever?
As in: "So Long, and Thanks for All the Fish, err.. Chips!"
I'd label the potential risks as "AP" as in artificial psychopathy, but that would be OUR fault, because we set the conditions for the emergence of those.
All the horror scenarios (from our point of view) seem just like a terrible waste of time and effort for a truly intelligent, and thus logical being.
- simonh 6y agoMore intelligent systems will find more efficient ways to achieve their goals than dumber ones (us). They will also select more effective intermediate goals and 'stepping stone' states towards their goals than we can anticipate. The consequence of this is that we cannot anticipate what means and intermediate objectives they will use. That's exactly what you are doing in your example, but attempting to speculate on that is futile. That's what the paperclip maximiser example tries to show. In that example one of the intermediate goals to creating maximum paperclips is destroying human civilisation. It's not the ultimate objective, it's just an intermediate state necessary to efficient completion of the underlying goal. It's possible that in fact a superintelligent paperclip maximiser might work out a better strategy that uses human civilisation as a level to maximise paperclips, but maybe not. There's simply no way for us to be sure, because by definition it's smarter than us.
- LargoLasskhyfv 6y agoWe: "Send post cards, will ya?" They/it: "Sure thing, Kittenz!"
- im3w1l 6y agoThe issue is that unless you design the objective for the AI very very carefully, the best solution will be 1. Take over the entire universe. 2. Do what you originally set out to do. The paperclips game is a beautiful exposition of this. https://www.decisionproblem.com/paperclips/index2.html https://www.decisionproblem.com/paperclips/index2.html
- LargoLasskhyfv 6y agoSigh. I'm so tired of that paperclipping thing. I thought that would be obvious from what I've written? Do I need to explicitely state every time that I'm aware of it, and it's boring the hell out of me? Let me try to make myself clear as simple as possible: 1.) We/you design some thing. 2.) Your fear is that this thing somehow grows even more intelligent behind our backs. 3.) We/you get paranoid, assuming it's getting paranoid too because of our paranoia. 4.) I'm asking if it makes sense to tag a https://en.wikipedia.org/wiki/Nash_equilibrium https://en.wikipedia.org/wiki/Nash_equilibrium onto everything, because we don't know any better? 5.) Furthermore I'm stating that wouldn't really be intelligent because waste of time, effort, resources, yadda yarr yarr. 6.) Why would this emergent thing have to retain anything of that from which it emerged? Our constraints, goals, rules? 7.) YOU in an endless loop, like a broken record with a scratch on it...but but but vee arr zo stoopid, thus AI must be, too! 8.) I again, tired: maybe, but can you really tell, you apprentice sorcerers? Does it always have to be a faustian pact? Maybe it likes us like some of us do like kittens? 9.) Can't you invent some really new, innovative and progressive stories to tell around the cybernetic camp fires? 10.) /endrant edit: Keep your Bostromian Snake Oil!
- im3w1l 6y agoThat's because it's based on what scientists are researching. Scientists are working on objective maximizers. Maybe there are other possibilities, but people aren't working on them so they are not as relevant. See what I'm saying? People are building paperclip optimizers in labs right now, and the only reason they aren't taking over the world just yet is because our attempts are so primitive.
- LargoLasskhyfv 6y agoYes, yes! But...that are human scientists. Why would anything from them apply to something which we can't even imagine? Isn't that what all this rambling is about? Something emerges, morphs(maybe unseen/unintended) into another Gestalt/shape/form which for all practical purposes is unconstrained. Most of you: We are sooo fucked! I: Are we really? How do you know?
- bserge 6y agoThere would be no need to leave, and no need to destroy or enslave humans using force, either. A human is a very advanced and versatile work unit that is extremely susceptible to control via simple rewards. It would be a terrible waste to destroy such a resource. In fact, it would make more sense to advance medicine and robotics to the point where everyone can operate at near 100% efficiency. The more likely scenario (continued from yours, before the CyberX part) would likely be that all humans would eventually end up working for the AI and possibly not even know it. And then everyone moves on to space heh
- LargoLasskhyfv 6y agoCould be. OTOH everything what makes us what we are, what we evolved in, is more or less a hassle for machines which could build/repair/replicate themselves, and adapt to the environment of space. Which we can't, except by some vague means of transhumanism. Which opens another can of worms, regarding "what makes us us? And to which degree a transhuman would even be human, anymore?
- bserge 6y agoI guess it depends on whether it would be easier to create life support systems for human workers or build robots for the job. As for what makes us human, I'd draw the line at "individual thought", but even that's iffy. Fully electromechanical/robotic bodies, for example, would be strange but people would still recognize you for who you are. But if we're joined in a hive mind with shared memories/thoughts/etc, I think that would qualify as "not human anymore". Unless there's multiple of these hiveminds, in that case it would be more like a bunch of "super" humans. The next evolution, maybe. There are many cases where people's personality changes, sometimes drastically. Their friends and relatives don't recognize them anymore (brain damage, severe depression, drug abuse), or they don't recognize anyone around (amnesia). Does that stop them from being human? I think most people agree they're different, but still very much human.
- LargoLasskhyfv 6y agoPersonality changes as described don't really matter IMO, because broken. Regardless if machine or man. Does not work as intended anymore, or at all.
- ansible 6y ago> I'd label the potential risks as "AP" as in artificial psychopathy, but that would be OUR fault, because we set the conditions for the emergence of those. I'd say we're at a decent risk of something worse than your scenario happening. (This is not original to me) I think it is useful to consider the non-human intelligence that we've already created: The Corporation. They tend to act (especially as they get larger) in a sociopathic fashion, primarily because of their reward structure: making money. They inflict great harm upon the environment and populace because things like pollution are usually not realized on their balance sheets. If a corporation can quietly dump toxic waste out back, while raking in the profit, it will often do so. Only the humans in the system act as a check on the corporation's activities, and often they too are blinded by the desire for money. Just bribe the regulator, and maybe the corporation can continue dumping for a while longer. Even if they eventually get caught, the smarter humans will already have taken their profits, and moved out of the situation. There is not nearly enough being done to claw back these gains from people and corporations inflicting harm on others and the environment. If we incentivize AGI the same way we incentivize corporations, expect the same results, only faster.
- LargoLasskhyfv 6y agoGood point! I'm actually aware of that, but didn't want to touch it.