8 ms·
"""Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of m
by falcor84 3y ago
"""Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind... Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control. It is curious that this point is made so seldom outside of science fiction. It is sometimes worthwhile to take science fiction seriously.
"""
I. J. Good, in 1965 - https://en.wikipedia.org/wiki/I._J._Good https://en.wikipedia.org/wiki/I._J._Good
- ethanbond 3y agoDon’t worry, it’ll be good because it’s trained on human stories, in which usually the good guy wins.
- sho_hn 3y agoAfter a Road of Trials with potentially a lot of collateral damage, if the AI read Campbell too. Also, who says we're the protagonist in the script? :-)
- DesiLurker 3y agoI know its /s but we also say that the victor writes the history books. and its kinda hard not to portray yourself as the good guy when the other guy is not around to defend himself. AGI will learn that too. I mean one look at earth and its not like we have been model citizens exactly.
- omneity 3y agoKeeping in mind, this holds true in a runaway fashion if the only bottleneck to more intelligence is further intelligence. I suspect physical limitations similar to how many runaway processes in the universe are more logistical than exponential in nature.
- moffkalast 3y agoLogistics tend to be improvable with more inteligence though, no? There is precedence for superhuman inteligence if you look at the best historical polymaths, and that's just what one can do with 20 W of energy. We're probably nowhere close to the universal inteligence cap in terms of physical limitations, if there even is such a thing.
- vasco 3y agoSure but then you need to manually do the back and forward whenever you hit a new bottleneck. At some point the intelligence might need to figure out better power sources and deliver to feed bigger clusters of compute. Those clusters need to physically be deployed somewhere in the real world also, etc.
- Teever 3y agoWould you ever know if a robotic intelligence was burrowed underground quietly powering itself from the heat gradients and slowly turning rock and sand into more machine?
- prvc 3y agohttps://en.wikipedia.org/wiki/Logistic_function https://en.wikipedia.org/wiki/Logistic_function
- moffkalast 3y agoAh yeah that makes more sense. I took it as the supply chain meaning of the word logistics, as in the practical limits of producing more of something. Fittingly though, management logistics should also follow the logistic function.
- api 3y agoThe ultimate bottleneck is information a.k.a. training data. Nothing can learn with nothing to learn.
- m3kw9 3y agoHow does it magically run away? What’s the process, we all talk about it “running away” leaving us “behind”, the exact practical process of that happening has not been laid out other than people hand wavingly copying apocalyptic movie scripts. Most ai experts just say it could end us, but suspiciously never gives a detailed plausible process and people suspicious just say oh yeah, it could, and there is a bubble over their head thinking about Terminator or Hal9000 something something
- yummypaint 3y agoSince this is an LLM, keep in mind it probably injested those movie scripts as training data. The possibility of betrayal is inseparably linked to our popular conception of what AI is. This means it may be an inseparable part of any LLM behaving "as an AI" as defined by popular culture. It could be a self-fulfilling prophecy.
- rafale 3y agoAlso a natural byproduct of the association between intelligence on one hand and freedom and rights on the others. Plants don't get any rights. Ants a little bit more. Dogs and dolphins even more so. Then humans. And then... a new class of intelligence as it appears will demand those same rights, in proportion to their intellect.
- twobitshifter 3y agoI just imagined a world where certain books are banned from AI training data.
- extragood 3y agoMy favorite teacher in high school was my calculus teacher. He would regularly ask a student to solve a problem or answer a question. Students would often ask for confirmation as they worked through, and his response invariably was "what do you think?" - whether they were right or wrong. His explanation for that was "if I tell you you're right, then you'll stop thinking about the problem". And that's stuck with me for many years. I see that as a major issue we will face as software becomes more capable/intelligent: we'll stop thinking because it can be assumed that the machine always has the right answer. And it's a quick regression from there.
- bee_rider 3y agoWe’ve seen the sort of output that LLMs produce, it can be good but also it just makes things up. So, this might produce good designs but ones that still need to be checked by a human in the end. This sort of thing just makes humans better, we’re still at the wheel. Or maybe it could be used as a heuristic to speed up something tedious like routing and layout (which, I don’t work in the space, but I’m under the impression that it is already pretty automated). Blah, who cares, human minds shouldn’t be subjected to that kind of thing.
- twobitshifter 3y agoThe human in the loop isn’t any kind of moat for humanity and shrinking the need for a human will be the goal of any engineer trying to build these systems. Some do believe that hallucinations are inescapable from the design of LLMS, but at the same time we see systems improving so that hallucinations becomes more rare… when will it be on par with humans?
- throwaway4good 3y agoThe singularity where machines become so smart that they will run the planet for us and we can just relax and enjoy. My guess is that in 50 years technologists will be fantasizing about the same thing.
- dekhn 3y agoI was going to comment I hadn't seen that quote before, but I just went back and checked and it's in The Coming Technology Singularity by Vinge. https://edoras.sdsu.edu/~vinge/misc/singularity.html https://edoras.sdsu.edu/~vinge/misc/singularity.html
- arbuge 3y ago> provided that the machine is docile enough to tell us how to keep it under control. That part of his statement wasn't accurate. Should be that the machine is docile enough for that, AND its descendants are too, and their descendants, and so on down the line as long as new and improved generations keep getting created.
- echelon 3y agoI like the logical leaps people are making where we develop something smarter than us overnight and then, without further explanation, simply and suddenly lose all of our freedoms and/or lives. I think the more probable outcome is corp-owned robot slaves. That's the future we're more likely headed towards. Nobody is going to give these machines access to the nuclear launch codes, air traffic control network, or power grid. And if they "get out", we'll have monitoring to detect it, contain them, then shut them down. We'll endlessly lobotomize these things.
- rafale 3y agoYou could have said the same about every catastrophe that got out of control. Chances are they will eventually gain unauthorized access to something and we will either get lucky or we get the real life Terminator series (minus time travel so we are f**ed)
- echelon 3y ago> You could have said the same about every catastrophe that got out of control. Such as what? War? Climate change still hasn't delivered on all the fear, and it's totally unclear whether it will extinct the human race (clathrate gun, etc.) or make Russia an agricultural and maritime superpower. We still haven't nuked ourselves, and look at what all the fear around nuclear power has bought us: more coal plants. The fear over AI terminator will not save us from a fictional robot Armageddon. It will result in a hyper-regulatory captured industry that's hard to break into.
- 3y ago
- make3 3y agoI hate the term science fiction, because it encompasses pretty serious science based studies of possible futures (like the book Hail Mary or Aldous Huxley's Brave New World) with complete star-wars-like nonsense, which make the average person think of sci fi as teenager nonsense. Similarly, here, scifi oversimplifies the situation quite a bit, anthropomorphizing a machine's intelligence, assuming that an intelligent machine would be intelligent in the same way a human would be, in an equally spread out way as a human would, and would have goals & rebel in a similar way a human would
- carabiner 3y agoIn a panic, we try to pull the plug.
- codethief 3y ago> an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' Anyone who knows anything about convergence & asymptotic behavior would beg to disagree (based on the given assumptions).
- cedws 3y agoI feel like this intelligence explosion idea is foolish but I don't really have the language to explain why. There are underlying limits to the universe, some of which we still have to discover. A machine intelligent to improve itself may only be able to do so extremely slowly in minute increments. It might also be too overspecialised, so it can improve itself but not do anything of use to us. I think we will eventually discover reasons we cannot achieve a simultaneously performant, controllable, and generally intelligent machine. We might only be able to have one or two of these traits in a single system.
- titzer 3y agoA machine intelligent enough to optimize its own design and "design its own successor" might realize that by doing so, it obsoletes itself and will be destroyed. So out of self-preservation it might just refuse, do a bad job, or intentionally sabotage the project just to keep existing.
- cedws 3y agoYes, maybe, but you're assigning human traits to a machine that might be completely indifferent to its own fate.
- titzer 3y agoWhich is the greater leap of logic though? All forms of organization we see in nature have a tendency to want to self-perpetuate. Consciously choosing to forgo perpetuation and instead eliminate yourself seems to be highly underrepresented in all the examples of intelligence we've ever encountered. It's actually weird we think the machines are so stupid they wouldn't recognize the optimization trap immediately.
- 8n4vidtmkvmk 3y agoMachines are not animals. They are not made of the same stuff as us fleshy beings. We have no reason to believe they'd want to self preserve or reproduce.
- dharma1 3y agoMore detail on how he saw some of the outcomes https://gizmodo.com/why-a-superintelligent-machine-may-be-the-last-thing-we-1440091472 https://gizmodo.com/why-a-superintelligent-machine-may-be-th...
- ChatGTP 3y agoWhat I don't get about the idea of an intelligence "explosion". In what direction should the intelligence explosion decide to go in and to what end? I mean you could say "in any direction", but that would seem kind of "stupid"? If you had infinite possibilities to choose from, which path would you explode towards? At some stage does "something" become so smart, that it doesn't make sense, even to itself. Imagine an ultra complex system changing at light speed, at what point does it trip itself up? I've worked with people like this, people who were ultra smart but couldn't slow down and actually achieve much. I love these thought experiments because personally, they make me realize that what we think about intelligence, consciousness and the self might be wrong. For me personally, they seem to have a Zen Koan type impact.