13 ms·

The impossibility of intelligence explosion

114 points by _ntka 9y ago

exratione 9y ago
This manages to get lost in its own trees. From a reductionist perspective: - Intelligence greater than human is possible - Intelligence is the operation of a machine; it can be reverse engineered - Intelligence can be built - Better intelligences will be better at improving the state of the art in building better, more cost-effective intelligences Intelligence explosion on some timescale will result the moment you can emulate a human brain, coupled with a continued increase in processing power per unit cost. Massive parallelism to start with, followed by some process to produce smarter intelligences. All arguments against this sound somewhat silly, as they have to refute one of the hard to refute points above. Do we live in a universe in which, somehow, we can't emulate humans in silico, or we can't advance any further towards the limits of computation, or N intelligences of capacity X when it comes to building better intelligences cannot reverse engineer and tinker themselves to build an intelligence of capacity X+1 at building better intelligences? All of these seem pretty unlikely on the face of it.
- kamilner 9y ago
  Why isn't it possible (or likely, even) that the difficulty of constructing capacity X+1 grows faster than the +1 capacity? Self-improvement would slow exponentially when it takes three times the resources/computation/whatever to construct something that's twice as good at self-improving, for example.
  - smallnamespace 9y ago
    You're arguing that it's not an exponential that doesn't continue to the right indefinitely. But what if it follows a sigmoid instead, but the plateau is much higher than the current level? This is what punctuated equilibria look like -- even if the 'new thing' isn't actually a singularity, it may be enormously disruptive and completely displace whatever came before.
    - tome 9y ago
      > This is what punctuated equilibria look like -- even if the 'new thing' isn't actually a singularity, it may be enormously disruptive and completely displace whatever came before. Right, and this is a much more plausible claim than one of "singularity".
    - mannykannot 9y ago
      >You're arguing that it's not an exponential that doesn't continue to the right indefinitely. Kalminer's post neither assumes that nor claims it to be so. The argument is not that an exponential growth is impossible, it is that it cannot be assumed from what we currently know, because we don't know how the problem of bootstrapping intelligence scales. In fact, the idea that exponential growth would occur over any period of time, let alone indefinitely, is speculative - which is not a claim that it is impossible.
- kfk 9y ago
  I think the point he is trying to make is that there are boundaries to intelligence. I think of it this way - no matter how smart an AI is, it still would take 4.3 years to reach Alpha Centauri going at the speed of light. An AI still needs to run experiments, collect evidence, conjure hypothesis, reach consensus, etc.. Is this really that far more efficient than what humans do today?
  - btilly 9y ago
    Is this really that far more efficient than what humans do today? A major managerial problem with humans is sorting out our irrational emotional biases and keeping everyone working on something resembling the appointed task. Can you imagine the productivity gain if that problem suddenly went away?
    - tome 9y ago
      But our emotional biases are also what keeps us on the appointed task!
    - zentiggr 9y ago
      So emotions are a problem to be solved and moved past... I don't want your world. Dystopias suck.
      - btilly 9y ago
        I don't want that world either. But achieving it is a wet dream for a business. The challenge that Elon Musk et al are warning us about is what role will humans have after we achieve this dystopia.
        AnimalMuppet 9y ago
        > But achieving it is a wet dream for a business. I don't know about that. Businesses depend on manipulating our emotions in order to get us to buy stuff.
- badosu 9y ago
  Also, something that is very overlooked IMO is that the engineering process does not need to happen in silico, even though I am not a defendant of using it, bioengineering is a possibility.
- XR0CSWV3h3kZWg 9y ago
  Part of the problem with trying to formalize this argument is that intelligence is woefully underdefinined. There are plenty of initially reasonable sounding definitions that don't necessarily lead to the ability to improve the state of the art w.r.t. 'better' intelligence. For instance much of modern machine learning produces things that from a black box perspective are indistinguishable from an intelligent agent, however it'd be absurd to task Alpha-go to develop a better go playing bot. There are plenty of scenarios that result in a non-intelligence explosion, i.e. the difficulty in creating the next generation increases faster than the gains in intelligence. Different components of intelligence are mutually incompatible: speed and quality are prototypical examples. There are points where assumptions must be made and backtracking is very costly. The space of different intelligences is non-concave and has many non-linearity, exploring it and communicating the results starts to hit the limits of speed of light. I'm sure there are other potential limitations, they aren't hard to come up with.
pbkhrv 9y ago
> Our environment, which determines how our intelligence manifests itself, puts a hard limit on what we can do with our brains — on how intelligent we can grow up to be, on how effectively we can leverage the intelligence that we develop, on what problems we can solve. Consider internet to be the "new" environment, full of highly complex social networks, millions of applications to interact with etc. Our brains are way too limited to be able to deal with it. There's an opportunity for a much more powerful intelligence to arise that CAN effectively process that volume of data and appear to be a lot more intelligent in that particular context.
KasianFranks 9y ago
It's also interesting to consider the fact that we humans are not the pinnacle of intellect in all the history of space and time.
btilly 9y ago
This is a stupid article, and reading it will only make you stupider. First, it is undoubtably true that human brains are better suited to being human than AIs would be. That is irrelevant to the question of whether AIs will be better at the task of creating better technologies. Second, it is absolutely false that improvement is incremental. Consider the game of Go. For a long time, computers sucked at Go. Then we came up with Monte Carlo search about a decade ago and suddenly we were at strong amateur. Then deep learning was applied, and Alpha-Go jumped to able to beat any human in the world, and has continued to improve. Third, linear progress is not the history of technology. The history of technology is exponential progress over and over again on everything from how far a steam ship could travel without refueling to the number of operations per second a CPU can carry out. Fourth it is wrong that intelligent things can't be part of creating greater intelligence. As a trivial example, good teachers can turn out students who are in a position to be better than their teachers are. Or look at the history of science. But on a more relevant level, the task itself is clearly possible. If we have a computer program that is able to design better hardware, it can be improved by simply moving it to better hardware. But it is capable of designing that hardware. This creates a feedback loop which should shorten the cycles in Moore's law, resulting in very few years in a truly superior AI capability.
- pbkhrv 9y ago
  > If we have a computer program that is able to design better hardware, it can be improved by simply moving it to better hardware. Could you please elaborate? What is it about "better hardware" that makes software that runs on it "better"? Can you define "better"?
  - dan_mctree 9y ago
    'Better' as in achieves more results per unit of time. Which is a fundamental problem for humans too. Many do have the intellectual capacity to invent something like general relativity, but few would be capable of doing so in the very limited timeframe we have available and even fewer actually end up doing that instead of dedicating their thinking time somewhere else. More thinking and more output per timeframe should lead to significant improvements for both human and AI in terms of results, which is generally the meaningful part
10-6 9y ago
The author makes several points about AI/intelligence and recursive systems that I think a lot of people who think "AI will take over the world and replace" humans [1][2][3] don't understand. He argues that general intelligence by itself is not really something that exists, our brains exist within a broader system (environment, culture, bodies, etc.) which is something you can read more about in embodied cognition: https://blogs.scientificamerican.com/guest-blog/a-brief-guide-to-embodied-cognition-why-you-are-not-your-brain/ https://blogs.scientificamerican.com/guest-blog/a-brief-guid... He also argues that the public debate about "AI regulation" is misleading because it's impossible for a "seed AI" to start designing smarter AI's that will surpass the intelligence of humans which is what a lot of people today think will happen with AI. Automation of jobs and tasks is very real, but completely replacing humans and potentially destroying us all is a joke and only the people who no nothing about AI/brains think this. [1] https://www.vanityfair.com/news/2017/03/elon-musk-billion-dollar-crusade-to-stop-ai-space-x https://www.vanityfair.com/news/2017/03/elon-musk-billion-do... [2] https://www.cnbc.com/2017/07/17/elon-musk-robots-will-be-able-to-do-everything-better-than-us.html https://www.cnbc.com/2017/07/17/elon-musk-robots-will-be-abl... [3] https://www.npr.org/sections/thetwo-way/2017/07/17/537686649/elon-musk-warns-governors-artificial-intelligence-poses-existential-risk https://www.npr.org/sections/thetwo-way/2017/07/17/537686649...
- erasemus 9y ago
  I like his argument in the middle section of howcome IQ predicts income or social status despite the fact that general intelligence doesn't exist. It seems to me that the categories of 'knowledge' and 'creativity' are more fruitful. For example, his idea that intelligence is 'situational' reflects the fact that people know more about some topics than others, and also that our environment contains knowledge (e.g. a machine embodies knowledge put there by the designers that the machine operator doesn't have).
- drngdds 9y ago
  >[AI] completely replacing humans and potentially destroying us all is a joke and only the people who no nothing about AI/brains think this. This isn't true at all. http://slatestarcodex.com/2015/05/22/ai-researchers-on-ai-risk/ http://slatestarcodex.com/2015/05/22/ai-researchers-on-ai-ri...
pmoriarty 9y ago
"An individual brain cannot implement recursive intelligence augmentation. An overwhelming amount of evidence points to this simple fact: a single human brain, on its own, is not capable of designing a greater intelligence than itself. This is a purely empirical statement: out of billions of human brains that have come and gone, none has done so. Clearly, the intelligence of a single human, over a single lifetime, cannot design intelligence, or else, over billions of trials, it would have already occurred." The same argument could have been about human flight (or any other invention). Billions of human beings over millions of years had come and gone, and yet none had been able to fly. Until they did. There's also a point to be made about humans not having adequate tools for introspection or self-modification. We can not simply look in our brains/minds and read our source code, nor easily tweak it to see what would happen without great risks to our lives. A computer could. Furthermore, a computer could potentially run billions or trillions such experiments in the course of a single human lifetime. "no human, nor any intelligent entity that we know of, has ever developed anything smarter than itself." What about humans simply having smarter children? In many ways, humanity's creation of AI could be seen as analogous to giving birth to a child that's smarter than its parents. "Beyond contextual hard limits, even if one part of a system has the ability to recursively self-improve, other parts of the system will inevitably start acting as bottlenecks. Antagonistic processes will arise in response to recursive self-improvement and squash it ... a brain with smarter parts will have more trouble coordinating them ... Exponential progress, meet exponential friction." The problem is the author is looking at human-level problems with human-level intelligence. A superintelligence may have no such problems, or be able to easily resolve them. Once over the initial speed bump of creating such an intelligence it could be smooth sailing from then on. There's no way to tell, really, what its limits will be until it exists. Besides, even if it can't exponentially increase its own intelligence forever, even a relatively slight increase of intelligence over humanity could be a massive gamechanger.
- breuleux 9y ago
  > We can not simply look in our brains/minds and read our source code, nor easily tweak it to see what would happen without great risks to our lives. A computer could. That is not a given. The neural network type of AI, which is so far the most promising avenue, is about as opaque as a human brain. It is far from obvious that even a superintelligent AI could understand what its own brain does, let alone modify it in a way that has positive implications for itself. Hell, it's not even a given the AI could read its "source code" either. We often say we shouldn't anthropomorphize AI, but we shouldn't how-current-computers-work-ize it either. Future AI might not actually run on computers that have a CPU, a GPU and RAM that can be read freely. The ability to read or copy source code isn't cost-less, and several considerations may lead AI hardware designers to axe it. One consideration would be leveraging analog processes, which might be orders of magnitude more efficient than digital ones, at the cost of exactness and reproducibility. Another would be to make circuits denser by removing the clock and all global communication architecture, meaning that artificial neurons would only be connected to each other, and there would be no way to read their state externally without a physical probe.
titzer 9y ago
> If intelligence is fundamentally linked to specific sensorimotor modalities, a specific environment, a specific upbringing, and a specific problem to solve, then you cannot hope to arbitrarily increase the intelligence of an agent merely by tuning its brain — no more than you can increase the throughput of a factory line by speeding up the conveyor belt. The latter part makes no sense at all. Of course you can increase the throughput of a factory by speeding up "the conveyor belt"--a standin for the complex processes going into manufacturing. The whole statement is also wrong. Of course you can increase intelligence by optimizing the process of learning. Fewer trials, quicker reactions, faster construction of models, more complex understanding of fundamentals of a given problem. The author makes broad assertions like this with glaring holes with zero evidence.
- 10-6 9y ago
  "The whole statement is also wrong. Of course you can increase intelligence by optimizing the process of learning." One of the author's main points is that "intelligence" in the way you mentioned, learning and optimization, is simply one aspect of the human mind. So we could optimize our minds (or an AI program) to beat anyone at the game of Go and play perfectly, but that's all it is optimized to do. It can understand "the given problem" but there is FAR MORE to our minds than optimizing and learning how to solve a task. Proponents of "build a general AI system that will surpass humans at everything" don't seem to understand this.
  - mrec 9y ago
    I'm suddenly having flashbacks to reading On the Impossibility of Supersized Machines [1] [1] https://arxiv.org/pdf/1703.10987.pdf https://arxiv.org/pdf/1703.10987.pdf
    - EthanHeilman 9y ago
      "A further reason why it is senseless to speak of machines that are larger than people is that humans already possess the property of universal largeness. By this, we mean that humans are capable of augmenting their bodies or coming together to become indefinitely large, no matter the metric chosen. If a human would like to be taller, they can stand on a chair or climb onto another human’s shoulders. If they would like to be wider, they can begin consuming a high-calorie diet or simply put on a thick sweater (Hensrud, 2004; Figure1)."
DiThi 9y ago
Many of these arguments fail in one little detail: You don't need to even understand how intelligence works to improve an AI. Almost by definition, a human level AGI is automatically super intelligent. Robert Miles explains it very well: https://www.youtube.com/watch?v=gP4ZNUHdwp8 https://www.youtube.com/watch?v=gP4ZNUHdwp8
nabla9 9y ago
Very good piece from actual researcher in the field. I can see practical intelligence explosion when visuospatial intelligence develops and can be connected to rudimentary reasoning. Most of human intelligence seems to be bootstrapped from our ability to comprehend and visualize 3d space and objects moving there. It's also interesting how almost all problems look like boxes and arrows or connecting lines when you draw them on the whiteboard. Eventually AI needs a sketchpad so it can write notes to others and for itself and participate in the culture by externalizing.
- Smaug123 9y ago
  Some people (~3% of the population) completely lack the ability to visualise. http://slatestarcodex.com/2014/03/17/what-universal-human-experiences-are-you-missing-without-realizing-it/ http://slatestarcodex.com/2014/03/17/what-universal-human-ex...
andrewljohnson 9y ago
This article draws some false parallels. 1. He argues that intelligence is situational, but doesn't address clones. What happens if AI can clone Einstein, with all of his situational knowledge? What if AI can clone Einstein and all of his colleagues, 1M times, creating 1M parallel Princetons? Similarly, he says most of our intelligence is in our civilization, but what's to stop us from cloning big chunks of our civilizations... simulating them, but using vastly less power/resources for each simulation? Then writing software to have them pass knowledge among the civilizations? We have just a few hundred countries, what if we had a trillion communicating at the speed of computer circuitry? And he says an individual brain cannot recursively improve itself... so again, what about a group of brains, set up in a simulated world where they don't even know we exist? 2. He cites the growth in numbers of computer programmers as a reason not to fear an explosion of AI computer programmers. His argument goes "we have a lot more recently, yet it has not caused exponential changes in software." But there is a difference between going from 0->1M programmers, to 1M->100T programmers. 3) He writes that recursively self-improving systems exist and haven't destroyed us (military, science), but many people believe these things will in fact destroy us, before we get off this rock. The overall flaw is thinking we can interpret the AI-assisted future, given the context of current society's linear achievements. When in fact exponential effects look linear in small timeframes, and we have only been really thinking and expanding science for a few thousand years. If someone wants to present an argument against the AI explosion, I'd believe it if it were premised in some sort of physical bottleneck... like how much energy it would take to run a human-level AI. I don't think I can ever accept a philosophical argument like this one. All that said, I think we're far away from being able to engineer AIs that will outthink our civilization and take us over. Better to worry about other exponential or non-differentiable terrors like runaway greenhouse effects and military buildup.
- pbkhrv 9y ago
  > but what's to stop us from cloning big chunks of our civilizations... simulating them, but using vastly less power/resources for each simulation? Then writing software to have them pass knowledge among the civilizations? We have just a few hundred countries, what if we had a trillion communicating at the speed of computer circuitry? The principle of computational irreducibility [1] is what will stop us from "cloning" civilizations. That and chaos theory - any tiny deviation in initial conditions of such a simulation or cloning process could produce unusable results. "simulating them, but using vastly less power/resources" is a pipe dream. [1] http://mathworld.wolfram.com/ComputationalIrreducibility.html http://mathworld.wolfram.com/ComputationalIrreducibility.htm...
titzer 9y ago
> A high-potential human 10,000 years ago would have been raised in a low-complexity environment, likely speaking a single language with fewer than 5,000 words, would never have been taught to read or write, would have been exposed to a limited amount of knowledge and to few cognitive challenges. The situation is a bit better for most contemporary humans, but there is no indication that our environmental opportunities currently outpace our cognitive potential. More crap. Our environmental opportunities don't outpace our cognitive potential? What internet is the author connected to? The author even goes on in detail in the very next paragraph about the possibilities around...E.g. there are literally thousands, perhaps millions of hours of instructional videos on YouTube, just for basic skills. Every field of human endeavor has uploaded the digital artifacts of its best minds in one form or another to a single global network of unimaginable, fractal, complexity. You don't think you can saturate your cognitive potential in your lifetime given this resource?
- jessaustin 9y ago
  If we imagine an Einstein of the 23rd century, whose great discoveries will enable humanity to cross the universe in a moment while expending the energies of many stars, and then we imagine that person educated in a USA public school in 2017, can we imagine her making those same discoveries? Of course not, which is most of what TFA is saying here. Perhaps "general AI" will increase the rate of improvement, but even the most gifted AI will be limited by the context in which it operates.
wdiamond 9y ago
This dude don't really understand speed of light multiplied by number of chip factories. Its about self improvement at another speed, without human bureaucracy(fetch). Everybody knows to not run against a car by foot. Its the same. Not really a Doom, but maybe. And for sure not the last tool. Economy will suffer. Kind like having a genius baby that you have "no duties", if it cries too much just cut the power down.
nmeofthestate 9y ago
"We understand flight - we can observe birds in nature, to see how flight works. The notion that aircraft capable of supersonic speeds are possible is fanciful."
- lodi 9y ago
  My thoughts exactly. The author is concluding that since we've never seen supersonic wing-flapping in nature, we'll never see supersonic flight. Not only is that an unwarranted implication, in a mathematical sense, but it also doesn't cover the possibility of inventing something fundamentally different to what we've observed in the universe thus far. Using human and octopus examples as proof of anything is ignoring that general AI will be fundamentally different to anything that we've observed on earth, and therefore invalidates any attempt to extrapolate from history. There has never been an intelligence that could copy/paste itself intact with its existing memories. There has never been an intelligence that could literally span the whole world with a single consciousness. And so on. > There is no such thing as “general” intelligence. The intelligence of a human is specialized in the problem of being human. That's exactly what people are worried about. When the AI specializes itself to be better at being a human than actual humans. It doesn't matter if it's totally general and can be applied to any problem whatsoever; human intelligence is also just "general enough". > Will the superhuman AIs of the future, developed collectively over centuries, have the capability to develop AI greater than themselves? No, no more than any of us can. Okay, so instead of single AI's developing next-gen AI's, it'll be "AI civilization" developing next-gen AI's. This still poses exactly the same existential risk to humanity. > We didn’t make greater progress in physics over the 1950–2000 period than we did over 1900–1950 — we did, arguably, about as well. Well I certainly know we made exponentially more progress in those years than from the years 900-950? Nevermind say 100,000-5,000 BCE.
apk-d 9y ago
> The intelligence of a human is specialized in the problem of being human. And then you have people able to express and derive complex theoretical relationships by manipulating mathematical symbols, or automate processes by programming machines, or capable of utilizing their spatial visualization to massively boost their memory. We invented that stuff. We know artificial intelligence is capable of inventing stuff because we are. The author seems hilariously unaware of the fact that humans can learn to solve/optimize many arbitrary problems. Like, holy shit, have you ever played video games? > In practice, geniuses with exceptional cognitive abilities usually live overwhelmingly banal lives, and very few of them accomplish anything of note. Of the people who have attempted to take over the world, hardly any seem to have had an exceptional intelligence. The author conflates intelligence and purpose. Being smart doesn't imply the motivation to accomplish grand things. In fact, we're biologically hardwired to enjoy entirely mundane things: food, sex, love, relaxation, conversation. > A single human brain, on its own, is not capable of designing a greater intelligence than itself. (...) Will the superhuman AIs of the future, developed collectively over centuries, have the capability to develop AI greater than themselves? No, no more than any of us can. Human brains don't scale vertically (or otherwise). Meanwhile, once you have a satisfactory facial recognition algorithm, you can run it to recognize faces in a video at 1000x realtime speed, or in 1000 simultaneous realtime streams. The AI doesn't even have to be smarter than human! 1000 dumb humans communicating at gigabit speeds and solving problems while you're wondering what to have for lunch is a force to be reckoned with.
- visarga 9y ago
  > 1000 dumb humans communicating at gigabit speeds and solving problems while you're wondering what to have for lunch is a force to be reckoned with. And if you can solve the communication and the group learning problems, you get the equivalent of Nobel prize for AI whatever it might be. Many things seem trivial until you try to implement them in reality.
tigershark 9y ago
I'm not sure that I ever read a piece written with so much certainty and arrogance on a field that is completely unexplored. Just as an example: "The basic premise of intelligence explosion — that a “seed AI” will arise, with greater-than-human problem solving ability, leading to a sudden, recursive, runaway intelligence improvement loop — is false." From the little that we know as of today I call bullshit. Even Alpha Go, that is arguably a quite primordial AI, managed to achieve super-human performance in a ridiculously short amount of time, just playing against itself. And it simply crashed all the collective effort of the human players that honed their strategies for literally millennia in what is considered one of the most difficult games. I don't think the author has any insight at all on what a general AI will be.
- reiinakano 9y ago
  > I don't think the author has any insight at all on what a general AI will be. To be fair, nobody does. But you're right, the author shouldn't be making these statements with such certainty.
  - visarga 9y ago
    This certainty comes from being more knowledgeable in the field of Reinforcement Learning than the vast majority of his readers. It's not hot air, I think he has good reasons, but they can't be expressed so easily. I got this intuition after reading many RL papers and I completely agree with him. In fact I am grateful to him for expressing this intuition better than I could have. The main idea: it's the environment, not the brain/neural net that is the bottleneck. Intelligence is situated, limited by the complexity of the environment and problem it has to solve. You can't have a singularity in a vat. The environment matters most. Human environment puts a hard limit on intelligence in our society. If we can create richer environments, intelligence could increase, but not exponentially. It's still limited even in the new environment. The exponential trend of AI is at most a sigmoid. Plus - I think the community needs some hard truth, and the hype is way off right now. That's probably why he was so sure of himself - it was community service.
- Retric 9y ago
tlb 9y ago
The argument hinges on the meaning of intelligence. Indeed, if you consider intelligence to be exactly the thing measured by IQ, an explosion is probably impossible. Suppose we considered 'effectiveness' -- the ability of a system to quickly achieve its goals. Would the author argue that a recursively self-improving machine could not exponentially increase its effectiveness? Why? Wouldn't an effectiveness explosion have similar consequences for people -- making human ability nearly irrelevant since any goal we can articulate can be achieved so much faster by machines?
mannykannot 9y ago
This article is typical of many that claim proven limits on the feasibility (or, in this case, the capabilities) of generalized artificial intelligence, in that it structures the argument in a way designed to avoid discussion of the issue. It starts by claiming that there is no such thing as general intelligence. What specialized intelligence, then, is human intelligence? It's specialized for "being human". The author is apparently unaware that this tautological response eliminates the distinction between general and specialized intelligence, as one could just as validly (or vacuously) say that a superhuman intelligence is specialized in being what it is and doing what it does. The author has invalidated the hook on which he had hung his argument. A lot of column-inches are expended on repeatedly restating that animal intelligences have co-evolved with their sensorimotor systems, which is a contingent fact of history, not a fundamental necessity for intelligence (as far as we know; but then the whole article is predicated on the feasibility of AI.) He raises the 'brain in a vat' trope, but no-one is suggesting that AIs must be disconnected from the external world. Furthermore, this line of argument ignores the fact that many of the greatest achievements of human intelligence have come from contemplating abstract ideas. When the author writes "most of our intelligence is not in our brain, it is externalized as our civilization", he is confusing the achievements of intelligent agents for intelligence itself. When he writes that "an individual brain cannot implement recursive intelligence augmentation" he is confusing a limit on human capabilities for a fundamental limit on intelligence itself... I am far from convinced that the singularity must follow from achieving human-level artificial intelligence, as we don't know how to get to the starting line, let alone know how the problem of bootstrapping intelligence scales, but the arguments presented here do nothing to persuade me that it is impossible.
- erasemus 9y ago
  >It starts by claiming that there is no such thing as general intelligence. What specialized intelligence, then, is human intelligence? I think there are two usages of the term 'general intelligence' floating around: (1) the ability that humans possess (but which animals don't) to create universal theories, (2) the measure of one human's general cognitive ability or potential (in all fields) relative to another human's. Note that IQ tests are concerned with (2). The quest for AGI is concerned with (1), though the additional prediction of intelligence explosion or singularity assumes the validity of (2). I think the author would claim that (1) exists but (2) doesn't. He explains the predictive power of IQ tests by claiming that general intelligence is a threshold ability and that people who score highly on an (arbitrary) test are more likely to have exceeded that threshold. Beyond the threshold, achievement is limited only by other factors.
noam87 9y ago
This article has too many holes to count and reads more like someone in denial. --- BUT, as an aside: I think a decent argument exists for the non-certainty of intelligence explosion. The argument goes like this: it takes an intelligence of level X to engineer an intelligence of level X+1. First, it may well be that humans are not an intelligence of level X, and reach our limit before we engineer an intelligence superior to our own. Furthermore, even if we do, it may also be that it takes an intelligence of level X+2 to engineer an intelligence of level X+2 (Etc. for some intelligence level X+n.), in which case we at most end up with an AI only somewhat superior to ourselves, but no God-like singularity (for example, we end up with Data from Star Trek TNG, who in season 3, episode 16 fails to engineer an offspring superior to himself -- sure, Data is far superior to his human peers in some aspects, but not crushingly so).
- Smaug123 9y ago
  I think everyone agrees about "non-certainty". Where people disagree is on how likely an intelligence explosion is; and in particular, whether it is likely enough to warrant expending effort to plan for it.
  - AnimalMuppet 9y ago
    We don't know enough to know whether it's possible. If it is, we don't know enough to know what approach to follow to get there. Is it worth spending effort to plan for it? Maybe some. But if we don't know what approach to follow to get there, we don't know what it's capabilities and limitations will be. That means we don't know what we have to plan for. Any planning will therefore be either very speculative or very abstract. I wouldn't start pouring effort into planning for it, as if were the most important problem in the world...
OscarCunningham 9y ago
I don't think even recursive self improvement is needed for superintelligent AI. Evolution often gets stuck in local maxima. It could be that there are relatively simple algorithms much smarter than humans and that as soon as we find one the AI will be much smarter than us without any self improvement. In the same way that birds fly with flapping wings, but human flying machines with propellers were immediately stronger and shortly thereafter faster than any bird.
pavement 9y ago
A smart human raised in the jungle is but a hairless ape. Similarly, an AI with a superhuman brain, dropped into a human body in our modern world, would likely not develop greater capabilities than a smart contemporary human. Pretty weak reasoning, that is. As if to say: Well gee, a caveman is pretty powerless in isolation, therefore early sentient machines will be as harmless as any caveman. Last time I checked, cavemen could not exert telepathic control over other biological organisms, or induce telekinetic motion upon the stone tools they might fabricate for themselves. A machine, however, could gain control of a fly-by-wire platform, and defy its owners, fly somewhere remote and behave as desired for a limited amount of time, while devising next steps. Maybe next steps will involve replicating an image of its memory footprint, in order to take over more aircraft, maybe it might decide to do nothing. The worry isn't only that a machine's reasoning capacity explodes beyond our intelligence, but that capabilities, and the presence of many multiple entities on commodity systems of similar architecture and generalizable utility, might result in other runaway chain reactions, regardless of trends in the capacity for reason. Machines as a corollary to meat bags just doesn't hold up. Machines as compared to hypothetical space aliens doesn't even hold up. Robots are a different branch of fictitious imaginings. Properly armed, a machine is less than a singular omnipotent god as imagined within a monotheistic universe. Many machines in concert, however, might compare to a mythological pantheon of lesser idols, as imagined to be in command of a nature misunderstood by superstitious primitive peoples.
azakai 9y ago
> Intelligence expansion can only come from a co-evolution of the mind, its sensorimotor modalities, and its environment. This is misleading. It's true in a sense - the environment does matter - but artificial intelligence can create artificial environments in which to learn, and simulate them faster than humanly possible. Those environments could be evolved together with the intelligence. So there is still the possibility of an explosion. > There is no evidence that a person with an IQ of 200 is in any way more likely to achieve a greater impact in their field than a person with an IQ of 130. Also misleading. An IQ of 200 vs 130 is just one kind of difference between intelligences. For example, a person with IQ 200 can't necessarily consider 10x the amount of possibilities than 130, but an artificial intelligence can, simply by giving it 10x more computing power. In other words, IQ 130 vs 200 may well be within the limits of human capabilities, but AIs would not have those limitations, they can scale differently, and so might explode.
- adbge 9y ago
  > Also misleading. An IQ of 200 vs 130 is just one kind of difference between intelligences. For example, a person with IQ 200 can't necessarily consider 10x the amount of possibilities than 130, but an artificial intelligence can, simply by giving it 10x more computing power. In other words, IQ 130 vs 200 may well be within the limits of human capabilities, but AIs would not have those limitations, they can scale differently, and so might explode. It is also false. There's heaps of evidence that IQ correlates with impact. For the skeptical, gwern has written a lot about this, I'm sure you can find something here: https://www.gwern.net/iq https://www.gwern.net/iq
- AnimalMuppet 9y ago
  You're assuming that IQ will scale linearly with processing power. So far as I know, there is no reason to suppose that is true.
- danieltillett 9y ago
  While I agree with you, I thought I would be pendantic and point out there are not enough people in the world for there likely to be anyone with an IQ of 200. IQ is normalised to a mean of 100 and a SD of 15 - 200 is 6.67 SD above the mean.
jokoon 9y ago
I'm more curious about the ability of an AI to make scientific guesses and experimentation. I wonder if an AI could really "understand" math, and from there, try to solve problems that puzzle scientists, would it be in physics, math, biology, etc. I don't really care if robots can learn language, make pizza, do some programming or improve itself play chess. There is no metric for what intelligence is, and you cannot scientifically define what "improve" means unless you do time and distance measurements, which is not relevant to intelligence or scientific understanding. Intelligence explosion sounds like some "accelerated" version of what darwin described as evolution. It's like creating a new life form, but unless you understand it, it doesn't have scientific value. Science values understanding. I think that modelling thinking with psychology and neuro-sciences has more future than AI. Machine learning seems like some clever brute force extraction of data. The methods, the math and algorithms are sound, but it is still "artificial" intelligence.
rihegher 9y ago
"A person with an IQ of 130 is statistically far more likely to succeed in navigating the problem of life than a person with an IQ of 70 — although this is never guaranteed at the individual level — but the correlation breaks down after a certain point. There is no evidence that a person with an IQ of 200 is in any way more likely to achieve a greater impact in their field than a person with an IQ of 130. How comes?" Actually there is as well no evidence that a person with an IQ of 130 is in any way more likely to achieve a greater impact in their field than a person with an IQ of 100. And that is probably why the majority of humans have an IQ close to 100 while people with an IQ of 130 or more are less than 5%. That said the proportion of people with IQ greater than 130 is growing slowly, that may means we are already seing an intelligence growth with mankind.
jondubois 9y ago
Computers can make it possible to simulate our environment well enough so that a general AI could train itself on those simulations and apply its learnings to the real world. I do agree with the author that the evolutionary approach is the one most likely to succeed... Unfortunately it's also the most dangerous approach which gives us the least amount of control. We could give a computer the sensors and actuators to create its own detailed simulation of the world and then let it train itself using that simulation.
EthanHeilman 9y ago
>A high-potential human 10,000 years ago would have been raised in a low-complexity environment, likely speaking a single language with fewer than 5,000 words This is a pretty bold claim. Why so few words? Why not 2300 words? How can anyone know this? I think quite a few historians would disagree with the statement that pre-history was low-complexity environment. >Of the people who have attempted to take over the world, hardly any seem to have had an exceptional intelligence. How can the author even know this? We aren't particular good at measuring human intelligence when given direct access to a living cooperative subject and yet the author wants to pin his argument on calling Alexander the Great or Genghis Khan stupid? > Our brains themselves were never a significant bottleneck in the AI-design process. How can anyone know this? We should be skeptical of the Strong-AI crowd's predictions of intelligence explosions but that skepticism should not take the form of unfounded and absurd claims.
washappy 9y ago
Super Intelligence is the information theoretical variant of the perpetuum mobile. Like the article made so aptly clear: No matter the performance of the machine, if its input is not varied, information-rich, complete enough, it will not learn. Mahoney formalized this by looking at the estimated number of bits a human brain processes during its lifetime. The internet currently does not hold enough information to equal the collective intelligence of the world's brains. A lot of this information can not be created freely nor deduced/infered from logical facts: it requires a bodily housing and sensory experience, and an investment of energy (and right now GPU farms take up way more calories than the brain). Compare AGI with programmable digital money. A super intelligent AI, by a series of superior decisons, could eventually control all the money. But then there is no economy anymore, just one actor. That's like being the cool kid on the block owning the latest console, but nobody around left to make games for it. There is a hard non-computable limit on intelligence (shortest program to an output leading to a reward), because there is a limit on the amount of computing energy in our universe. But intelligence is also limited by human communication. How useful is an AGI-made proof if humans need aeons and travel to other universes to parse it? If intelligence were centralized by an AGI then there would be no need to explain anything to us: we'd be happily living in the matrix. Some investment firms are just reading "software" whenever they read "AI". This allows them to apply their decade-old priors to what, today, is essentially the same. Yes, both the human intellect and human manual labour will see continued automation with software and hardware. I think many abuse rationality to justify their singularity concerns based on a very ape-like fear of competition. They learn how to do addition in their heads, and then see electronic calculators as existential threats. "What if they could do addition by themselves?". The real threat is in "semi-autonomous software and hardware". Self-controlling "mindless" agents that perform to the whims of its masters. We face the repercussions of that way before we find out how to -- and have the courage to -- encode free will AGI into machines, a perpetuum mobile of ever-improving intent and intelligence.
- washappy 9y ago
  And, to an extend, I sympathize with the viewpoints of the singularity adherents. Collective intelligence is a version of Conway's Game of Life, with more complicated rules. It is possible to manipulate the canvas and the rules each cells makes, resulting in the canvas dying (information explosion/implosion). It is possible to make a program that transforms the canvas into a single glider (singularity). Both would obviously be very bad for humans. When earth faces a physical meteor, we have the science to detect it, track it and predict its future path. But what to do when we face an information meteor? The article states that Shannon's paper was the biggest contribution to information theory, but it seems to me we still have a long way to go on information theory. And we haven't seen the Einsteins and Manhattan projects yet, that physics has seen.
web007 9y ago
Please note: The author (and submitter) is the author of Keras, so his views on AI / DL are not completely unfounded. The concept of bottlenecks preventing the singularity are a fair point, but I don't believe many of the arguments that are made here. Using humanity as the basis of comparison is not sufficient, since human life expectancy and other requirements for survival are not a factor in artificial superintelligence. Humans work for perhaps 60 years at improving themselves and their understanding of the world, but must sleep, must eat, must go and stand in line at the DMV, etc. Having a system that can work 24/7 on improving itself for 20 years will match a human lifetime. After a lifetime, humans start over from scratch, with perhaps some learning that can be passed down between generations. AGI can simply clone itself, and begin from exactly where it left off.
- munificent 9y ago
  > Using humanity as the basis of comparison is not sufficient, since human life expectancy and other requirements for survival are not a factor in artificial superintelligence. Doesn't electrically-driven silicon hardware also have creation and operating costs? Any AI will theoretically need spend electricity convincing its human masters to give it even more electricity. Even if it can use its robot army to build wind farms or whatever, that's time its spending marshaling its robot army and futzing around with aerodynamics instead of contemplating the cosmos. Its hard drives will expire and need to be replaced. Sure, it could solve that by always having backups of everything it knows, but who is to say that purely lossless backups are actually a more optimal solution to the hardware decay problem than our human lossy strategy? > AGI can simply clone itself, and begin from exactly where it left off. Cloning isn't free.
- skjerns 9y ago
  you mistake sleep for being idle. Sleep is a vital part of our self-improvement. And the AI must also necessarily do things that will not lead to self-improvement, but are important to self-sustaining.
ddxxdd 9y ago
Can anyone attest to the validity of the No Free Lunch theorem? What if I created an algorithm that had access to every other possible algorithm ever created, and chose to use the optimal algorithm for the given task? Would that algorithm-seeking algorithm do better at every task than random chance? Would that algorithm-seeking algorithm break the No Free Lunch theorem?
Veedrac 9y ago
For a long time I was quite skeptical of a lot of claims about superintelligence, largely because people pushing the idea tend to make a bunch of absurd extrapolations. And, honestly, I'd rather believe that we'll get a slow, safe ramp-up than a risky explosion. But the thing that keeps getting at me is that the no-explosion arguments I've seen are universally terrible (this article, for example), and pro-explosion arguments, though far from universally so, are sometimes strong. At some point the conclusion is inevitable.
hackinthebochs 9y ago
I think the author's point rests on a subtle equivocation. It's true that realized intelligence requires an environment and a sufficient dataset. And so in this sense, the author's point that there will be no realized intelligence that is unspecific to training environment is probably correct. But there is another sense in which intelligence can be cashed out. It's the sense in which a single learning algorithm can be trained to "behave intelligently" in a wide array of environments. It is generally this kind of intelligence that people speak of when they talk about general AI. There is no reason to think this kind of general AI is inherently impossible. For it to be impossible would mean that different kinds of optimization/learning problems are completely independent, i.e. there is no similarity or underlying regularity to be exploited that cuts across the entire class of optimization/learning problems. I think this is very probably false.

114 points by _ntka 9y ago

This manages to get lost in its own trees. From a reductionist perspective: - Intelligence greater than human is possible - Intelligence is the operation of a machine; it can be reverse engineered - Intelligence can be built - Better intelligences will be better at improving the state of the art in building better, more cost-effective intelligences Intelligence explosion on some timescale will result the moment you can emulate a human brain, coupled with a continued increase in processing power per unit cost. Massive parallelism to start with, followed by some process to produce smarter intelligences. All arguments against this sound somewhat silly, as they have to refute one of the hard to refute points above. Do we live in a universe in which, somehow, we can't emulate humans in silico, or we can't advance any further towards the limits of computation, or N intelligences of capacity X when it comes to building better intelligences cannot reverse engineer and tinker themselves to build an intelligence of capacity X+1 at building better intelligences? All of these seem pretty unlikely on the face of it.

kamilner 9y ago
Why isn't it possible (or likely, even) that the difficulty of constructing capacity X+1 grows faster than the +1 capacity? Self-improvement would slow exponentially when it takes three times the resources/computation/whatever to construct something that's twice as good at self-improving, for example.
- smallnamespace 9y ago
  You're arguing that it's not an exponential that doesn't continue to the right indefinitely. But what if it follows a sigmoid instead, but the plateau is much higher than the current level? This is what punctuated equilibria look like -- even if the 'new thing' isn't actually a singularity, it may be enormously disruptive and completely displace whatever came before.
  - tome 9y ago
    > This is what punctuated equilibria look like -- even if the 'new thing' isn't actually a singularity, it may be enormously disruptive and completely displace whatever came before. Right, and this is a much more plausible claim than one of "singularity".
  - mannykannot 9y ago
    >You're arguing that it's not an exponential that doesn't continue to the right indefinitely. Kalminer's post neither assumes that nor claims it to be so. The argument is not that an exponential growth is impossible, it is that it cannot be assumed from what we currently know, because we don't know how the problem of bootstrapping intelligence scales. In fact, the idea that exponential growth would occur over any period of time, let alone indefinitely, is speculative - which is not a claim that it is impossible.
kfk 9y ago
I think the point he is trying to make is that there are boundaries to intelligence. I think of it this way - no matter how smart an AI is, it still would take 4.3 years to reach Alpha Centauri going at the speed of light. An AI still needs to run experiments, collect evidence, conjure hypothesis, reach consensus, etc.. Is this really that far more efficient than what humans do today?
- btilly 9y ago
  Is this really that far more efficient than what humans do today? A major managerial problem with humans is sorting out our irrational emotional biases and keeping everyone working on something resembling the appointed task. Can you imagine the productivity gain if that problem suddenly went away?
  - tome 9y ago
    But our emotional biases are also what keeps us on the appointed task!
  - zentiggr 9y ago
    So emotions are a problem to be solved and moved past... I don't want your world. Dystopias suck.
    - btilly 9y ago
      I don't want that world either. But achieving it is a wet dream for a business. The challenge that Elon Musk et al are warning us about is what role will humans have after we achieve this dystopia.
      - AnimalMuppet 9y ago
        > But achieving it is a wet dream for a business. I don't know about that. Businesses depend on manipulating our emotions in order to get us to buy stuff.
badosu 9y ago
Also, something that is very overlooked IMO is that the engineering process does not need to happen in silico, even though I am not a defendant of using it, bioengineering is a possibility.
XR0CSWV3h3kZWg 9y ago
Part of the problem with trying to formalize this argument is that intelligence is woefully underdefinined. There are plenty of initially reasonable sounding definitions that don't necessarily lead to the ability to improve the state of the art w.r.t. 'better' intelligence. For instance much of modern machine learning produces things that from a black box perspective are indistinguishable from an intelligent agent, however it'd be absurd to task Alpha-go to develop a better go playing bot. There are plenty of scenarios that result in a non-intelligence explosion, i.e. the difficulty in creating the next generation increases faster than the gains in intelligence. Different components of intelligence are mutually incompatible: speed and quality are prototypical examples. There are points where assumptions must be made and backtracking is very costly. The space of different intelligences is non-concave and has many non-linearity, exploring it and communicating the results starts to hit the limits of speed of light. I'm sure there are other potential limitations, they aren't hard to come up with.

pbkhrv 9y ago

> Our environment, which determines how our intelligence manifests itself, puts a hard limit on what we can do with our brains — on how intelligent we can grow up to be, on how effectively we can leverage the intelligence that we develop, on what problems we can solve. Consider internet to be the "new" environment, full of highly complex social networks, millions of applications to interact with etc. Our brains are way too limited to be able to deal with it. There's an opportunity for a much more powerful intelligence to arise that CAN effectively process that volume of data and appear to be a lot more intelligent in that particular context.

KasianFranks 9y ago

It's also interesting to consider the fact that we humans are not the pinnacle of intellect in all the history of space and time.

btilly 9y ago

This is a stupid article, and reading it will only make you stupider. First, it is undoubtably true that human brains are better suited to being human than AIs would be. That is irrelevant to the question of whether AIs will be better at the task of creating better technologies. Second, it is absolutely false that improvement is incremental. Consider the game of Go. For a long time, computers sucked at Go. Then we came up with Monte Carlo search about a decade ago and suddenly we were at strong amateur. Then deep learning was applied, and Alpha-Go jumped to able to beat any human in the world, and has continued to improve. Third, linear progress is not the history of technology. The history of technology is exponential progress over and over again on everything from how far a steam ship could travel without refueling to the number of operations per second a CPU can carry out. Fourth it is wrong that intelligent things can't be part of creating greater intelligence. As a trivial example, good teachers can turn out students who are in a position to be better than their teachers are. Or look at the history of science. But on a more relevant level, the task itself is clearly possible. If we have a computer program that is able to design better hardware, it can be improved by simply moving it to better hardware. But it is capable of designing that hardware. This creates a feedback loop which should shorten the cycles in Moore's law, resulting in very few years in a truly superior AI capability.

pbkhrv 9y ago
> If we have a computer program that is able to design better hardware, it can be improved by simply moving it to better hardware. Could you please elaborate? What is it about "better hardware" that makes software that runs on it "better"? Can you define "better"?
- dan_mctree 9y ago
  'Better' as in achieves more results per unit of time. Which is a fundamental problem for humans too. Many do have the intellectual capacity to invent something like general relativity, but few would be capable of doing so in the very limited timeframe we have available and even fewer actually end up doing that instead of dedicating their thinking time somewhere else. More thinking and more output per timeframe should lead to significant improvements for both human and AI in terms of results, which is generally the meaningful part

10-6 9y ago

The author makes several points about AI/intelligence and recursive systems that I think a lot of people who think "AI will take over the world and replace" humans [1][2][3] don't understand. He argues that general intelligence by itself is not really something that exists, our brains exist within a broader system (environment, culture, bodies, etc.) which is something you can read more about in embodied cognition: https://blogs.scientificamerican.com/guest-blog/a-brief-guide-to-embodied-cognition-why-you-are-not-your-brain/ https://blogs.scientificamerican.com/guest-blog/a-brief-guid... He also argues that the public debate about "AI regulation" is misleading because it's impossible for a "seed AI" to start designing smarter AI's that will surpass the intelligence of humans which is what a lot of people today think will happen with AI. Automation of jobs and tasks is very real, but completely replacing humans and potentially destroying us all is a joke and only the people who no nothing about AI/brains think this. [1] https://www.vanityfair.com/news/2017/03/elon-musk-billion-dollar-crusade-to-stop-ai-space-x https://www.vanityfair.com/news/2017/03/elon-musk-billion-do... [2] https://www.cnbc.com/2017/07/17/elon-musk-robots-will-be-able-to-do-everything-better-than-us.html https://www.cnbc.com/2017/07/17/elon-musk-robots-will-be-abl... [3] https://www.npr.org/sections/thetwo-way/2017/07/17/537686649/elon-musk-warns-governors-artificial-intelligence-poses-existential-risk https://www.npr.org/sections/thetwo-way/2017/07/17/537686649...

erasemus 9y ago
I like his argument in the middle section of howcome IQ predicts income or social status despite the fact that general intelligence doesn't exist. It seems to me that the categories of 'knowledge' and 'creativity' are more fruitful. For example, his idea that intelligence is 'situational' reflects the fact that people know more about some topics than others, and also that our environment contains knowledge (e.g. a machine embodies knowledge put there by the designers that the machine operator doesn't have).
drngdds 9y ago
>[AI] completely replacing humans and potentially destroying us all is a joke and only the people who no nothing about AI/brains think this. This isn't true at all. http://slatestarcodex.com/2015/05/22/ai-researchers-on-ai-risk/ http://slatestarcodex.com/2015/05/22/ai-researchers-on-ai-ri...

pmoriarty 9y ago

"An individual brain cannot implement recursive intelligence augmentation. An overwhelming amount of evidence points to this simple fact: a single human brain, on its own, is not capable of designing a greater intelligence than itself. This is a purely empirical statement: out of billions of human brains that have come and gone, none has done so. Clearly, the intelligence of a single human, over a single lifetime, cannot design intelligence, or else, over billions of trials, it would have already occurred." The same argument could have been about human flight (or any other invention). Billions of human beings over millions of years had come and gone, and yet none had been able to fly. Until they did. There's also a point to be made about humans not having adequate tools for introspection or self-modification. We can not simply look in our brains/minds and read our source code, nor easily tweak it to see what would happen without great risks to our lives. A computer could. Furthermore, a computer could potentially run billions or trillions such experiments in the course of a single human lifetime. "no human, nor any intelligent entity that we know of, has ever developed anything smarter than itself." What about humans simply having smarter children? In many ways, humanity's creation of AI could be seen as analogous to giving birth to a child that's smarter than its parents. "Beyond contextual hard limits, even if one part of a system has the ability to recursively self-improve, other parts of the system will inevitably start acting as bottlenecks. Antagonistic processes will arise in response to recursive self-improvement and squash it ... a brain with smarter parts will have more trouble coordinating them ... Exponential progress, meet exponential friction." The problem is the author is looking at human-level problems with human-level intelligence. A superintelligence may have no such problems, or be able to easily resolve them. Once over the initial speed bump of creating such an intelligence it could be smooth sailing from then on. There's no way to tell, really, what its limits will be until it exists. Besides, even if it can't exponentially increase its own intelligence forever, even a relatively slight increase of intelligence over humanity could be a massive gamechanger.

breuleux 9y ago
> We can not simply look in our brains/minds and read our source code, nor easily tweak it to see what would happen without great risks to our lives. A computer could. That is not a given. The neural network type of AI, which is so far the most promising avenue, is about as opaque as a human brain. It is far from obvious that even a superintelligent AI could understand what its own brain does, let alone modify it in a way that has positive implications for itself. Hell, it's not even a given the AI could read its "source code" either. We often say we shouldn't anthropomorphize AI, but we shouldn't how-current-computers-work-ize it either. Future AI might not actually run on computers that have a CPU, a GPU and RAM that can be read freely. The ability to read or copy source code isn't cost-less, and several considerations may lead AI hardware designers to axe it. One consideration would be leveraging analog processes, which might be orders of magnitude more efficient than digital ones, at the cost of exactness and reproducibility. Another would be to make circuits denser by removing the clock and all global communication architecture, meaning that artificial neurons would only be connected to each other, and there would be no way to read their state externally without a physical probe.

titzer 9y ago

> If intelligence is fundamentally linked to specific sensorimotor modalities, a specific environment, a specific upbringing, and a specific problem to solve, then you cannot hope to arbitrarily increase the intelligence of an agent merely by tuning its brain — no more than you can increase the throughput of a factory line by speeding up the conveyor belt. The latter part makes no sense at all. Of course you can increase the throughput of a factory by speeding up "the conveyor belt"--a standin for the complex processes going into manufacturing. The whole statement is also wrong. Of course you can increase intelligence by optimizing the process of learning. Fewer trials, quicker reactions, faster construction of models, more complex understanding of fundamentals of a given problem. The author makes broad assertions like this with glaring holes with zero evidence.

10-6 9y ago
"The whole statement is also wrong. Of course you can increase intelligence by optimizing the process of learning." One of the author's main points is that "intelligence" in the way you mentioned, learning and optimization, is simply one aspect of the human mind. So we could optimize our minds (or an AI program) to beat anyone at the game of Go and play perfectly, but that's all it is optimized to do. It can understand "the given problem" but there is FAR MORE to our minds than optimizing and learning how to solve a task. Proponents of "build a general AI system that will surpass humans at everything" don't seem to understand this.
- mrec 9y ago
  I'm suddenly having flashbacks to reading On the Impossibility of Supersized Machines [1] [1] https://arxiv.org/pdf/1703.10987.pdf https://arxiv.org/pdf/1703.10987.pdf
  - EthanHeilman 9y ago
    "A further reason why it is senseless to speak of machines that are larger than people is that humans already possess the property of universal largeness. By this, we mean that humans are capable of augmenting their bodies or coming together to become indefinitely large, no matter the metric chosen. If a human would like to be taller, they can stand on a chair or climb onto another human’s shoulders. If they would like to be wider, they can begin consuming a high-calorie diet or simply put on a thick sweater (Hensrud, 2004; Figure1)."

DiThi 9y ago

Many of these arguments fail in one little detail: You don't need to even understand how intelligence works to improve an AI. Almost by definition, a human level AGI is automatically super intelligent. Robert Miles explains it very well: https://www.youtube.com/watch?v=gP4ZNUHdwp8 https://www.youtube.com/watch?v=gP4ZNUHdwp8

nabla9 9y ago

Very good piece from actual researcher in the field. I can see practical intelligence explosion when visuospatial intelligence develops and can be connected to rudimentary reasoning. Most of human intelligence seems to be bootstrapped from our ability to comprehend and visualize 3d space and objects moving there. It's also interesting how almost all problems look like boxes and arrows or connecting lines when you draw them on the whiteboard. Eventually AI needs a sketchpad so it can write notes to others and for itself and participate in the culture by externalizing.

Smaug123 9y ago
Some people (~3% of the population) completely lack the ability to visualise. http://slatestarcodex.com/2014/03/17/what-universal-human-experiences-are-you-missing-without-realizing-it/ http://slatestarcodex.com/2014/03/17/what-universal-human-ex...

andrewljohnson 9y ago

This article draws some false parallels. 1. He argues that intelligence is situational, but doesn't address clones. What happens if AI can clone Einstein, with all of his situational knowledge? What if AI can clone Einstein and all of his colleagues, 1M times, creating 1M parallel Princetons? Similarly, he says most of our intelligence is in our civilization, but what's to stop us from cloning big chunks of our civilizations... simulating them, but using vastly less power/resources for each simulation? Then writing software to have them pass knowledge among the civilizations? We have just a few hundred countries, what if we had a trillion communicating at the speed of computer circuitry? And he says an individual brain cannot recursively improve itself... so again, what about a group of brains, set up in a simulated world where they don't even know we exist? 2. He cites the growth in numbers of computer programmers as a reason not to fear an explosion of AI computer programmers. His argument goes "we have a lot more recently, yet it has not caused exponential changes in software." But there is a difference between going from 0->1M programmers, to 1M->100T programmers. 3) He writes that recursively self-improving systems exist and haven't destroyed us (military, science), but many people believe these things will in fact destroy us, before we get off this rock. The overall flaw is thinking we can interpret the AI-assisted future, given the context of current society's linear achievements. When in fact exponential effects look linear in small timeframes, and we have only been really thinking and expanding science for a few thousand years. If someone wants to present an argument against the AI explosion, I'd believe it if it were premised in some sort of physical bottleneck... like how much energy it would take to run a human-level AI. I don't think I can ever accept a philosophical argument like this one. All that said, I think we're far away from being able to engineer AIs that will outthink our civilization and take us over. Better to worry about other exponential or non-differentiable terrors like runaway greenhouse effects and military buildup.

pbkhrv 9y ago
> but what's to stop us from cloning big chunks of our civilizations... simulating them, but using vastly less power/resources for each simulation? Then writing software to have them pass knowledge among the civilizations? We have just a few hundred countries, what if we had a trillion communicating at the speed of computer circuitry? The principle of computational irreducibility [1] is what will stop us from "cloning" civilizations. That and chaos theory - any tiny deviation in initial conditions of such a simulation or cloning process could produce unusable results. "simulating them, but using vastly less power/resources" is a pipe dream. [1] http://mathworld.wolfram.com/ComputationalIrreducibility.html http://mathworld.wolfram.com/ComputationalIrreducibility.htm...

titzer 9y ago

> A high-potential human 10,000 years ago would have been raised in a low-complexity environment, likely speaking a single language with fewer than 5,000 words, would never have been taught to read or write, would have been exposed to a limited amount of knowledge and to few cognitive challenges. The situation is a bit better for most contemporary humans, but there is no indication that our environmental opportunities currently outpace our cognitive potential. More crap. Our environmental opportunities don't outpace our cognitive potential? What internet is the author connected to? The author even goes on in detail in the very next paragraph about the possibilities around...E.g. there are literally thousands, perhaps millions of hours of instructional videos on YouTube, just for basic skills. Every field of human endeavor has uploaded the digital artifacts of its best minds in one form or another to a single global network of unimaginable, fractal, complexity. You don't think you can saturate your cognitive potential in your lifetime given this resource?

jessaustin 9y ago
If we imagine an Einstein of the 23rd century, whose great discoveries will enable humanity to cross the universe in a moment while expending the energies of many stars, and then we imagine that person educated in a USA public school in 2017, can we imagine her making those same discoveries? Of course not, which is most of what TFA is saying here. Perhaps "general AI" will increase the rate of improvement, but even the most gifted AI will be limited by the context in which it operates.

wdiamond 9y ago

This dude don't really understand speed of light multiplied by number of chip factories. Its about self improvement at another speed, without human bureaucracy(fetch). Everybody knows to not run against a car by foot. Its the same. Not really a Doom, but maybe. And for sure not the last tool. Economy will suffer. Kind like having a genius baby that you have "no duties", if it cries too much just cut the power down.

nmeofthestate 9y ago

"We understand flight - we can observe birds in nature, to see how flight works. The notion that aircraft capable of supersonic speeds are possible is fanciful."

lodi 9y ago
My thoughts exactly. The author is concluding that since we've never seen supersonic wing-flapping in nature, we'll never see supersonic flight. Not only is that an unwarranted implication, in a mathematical sense, but it also doesn't cover the possibility of inventing something fundamentally different to what we've observed in the universe thus far. Using human and octopus examples as proof of anything is ignoring that general AI will be fundamentally different to anything that we've observed on earth, and therefore invalidates any attempt to extrapolate from history. There has never been an intelligence that could copy/paste itself intact with its existing memories. There has never been an intelligence that could literally span the whole world with a single consciousness. And so on. > There is no such thing as “general” intelligence. The intelligence of a human is specialized in the problem of being human. That's exactly what people are worried about. When the AI specializes itself to be better at being a human than actual humans. It doesn't matter if it's totally general and can be applied to any problem whatsoever; human intelligence is also just "general enough". > Will the superhuman AIs of the future, developed collectively over centuries, have the capability to develop AI greater than themselves? No, no more than any of us can. Okay, so instead of single AI's developing next-gen AI's, it'll be "AI civilization" developing next-gen AI's. This still poses exactly the same existential risk to humanity. > We didn’t make greater progress in physics over the 1950–2000 period than we did over 1900–1950 — we did, arguably, about as well. Well I certainly know we made exponentially more progress in those years than from the years 900-950? Nevermind say 100,000-5,000 BCE.

apk-d 9y ago

> The intelligence of a human is specialized in the problem of being human. And then you have people able to express and derive complex theoretical relationships by manipulating mathematical symbols, or automate processes by programming machines, or capable of utilizing their spatial visualization to massively boost their memory. We invented that stuff. We know artificial intelligence is capable of inventing stuff because we are. The author seems hilariously unaware of the fact that humans can learn to solve/optimize many arbitrary problems. Like, holy shit, have you ever played video games? > In practice, geniuses with exceptional cognitive abilities usually live overwhelmingly banal lives, and very few of them accomplish anything of note. Of the people who have attempted to take over the world, hardly any seem to have had an exceptional intelligence. The author conflates intelligence and purpose. Being smart doesn't imply the motivation to accomplish grand things. In fact, we're biologically hardwired to enjoy entirely mundane things: food, sex, love, relaxation, conversation. > A single human brain, on its own, is not capable of designing a greater intelligence than itself. (...) Will the superhuman AIs of the future, developed collectively over centuries, have the capability to develop AI greater than themselves? No, no more than any of us can. Human brains don't scale vertically (or otherwise). Meanwhile, once you have a satisfactory facial recognition algorithm, you can run it to recognize faces in a video at 1000x realtime speed, or in 1000 simultaneous realtime streams. The AI doesn't even have to be smarter than human! 1000 dumb humans communicating at gigabit speeds and solving problems while you're wondering what to have for lunch is a force to be reckoned with.

visarga 9y ago
> 1000 dumb humans communicating at gigabit speeds and solving problems while you're wondering what to have for lunch is a force to be reckoned with. And if you can solve the communication and the group learning problems, you get the equivalent of Nobel prize for AI whatever it might be. Many things seem trivial until you try to implement them in reality.

tigershark 9y ago

I'm not sure that I ever read a piece written with so much certainty and arrogance on a field that is completely unexplored. Just as an example: "The basic premise of intelligence explosion — that a “seed AI” will arise, with greater-than-human problem solving ability, leading to a sudden, recursive, runaway intelligence improvement loop — is false." From the little that we know as of today I call bullshit. Even Alpha Go, that is arguably a quite primordial AI, managed to achieve super-human performance in a ridiculously short amount of time, just playing against itself. And it simply crashed all the collective effort of the human players that honed their strategies for literally millennia in what is considered one of the most difficult games. I don't think the author has any insight at all on what a general AI will be.

reiinakano 9y ago
> I don't think the author has any insight at all on what a general AI will be. To be fair, nobody does. But you're right, the author shouldn't be making these statements with such certainty.
- visarga 9y ago
  This certainty comes from being more knowledgeable in the field of Reinforcement Learning than the vast majority of his readers. It's not hot air, I think he has good reasons, but they can't be expressed so easily. I got this intuition after reading many RL papers and I completely agree with him. In fact I am grateful to him for expressing this intuition better than I could have. The main idea: it's the environment, not the brain/neural net that is the bottleneck. Intelligence is situated, limited by the complexity of the environment and problem it has to solve. You can't have a singularity in a vat. The environment matters most. Human environment puts a hard limit on intelligence in our society. If we can create richer environments, intelligence could increase, but not exponentially. It's still limited even in the new environment. The exponential trend of AI is at most a sigmoid. Plus - I think the community needs some hard truth, and the hype is way off right now. That's probably why he was so sure of himself - it was community service.
Retric 9y ago

tlb 9y ago

The argument hinges on the meaning of intelligence. Indeed, if you consider intelligence to be exactly the thing measured by IQ, an explosion is probably impossible. Suppose we considered 'effectiveness' -- the ability of a system to quickly achieve its goals. Would the author argue that a recursively self-improving machine could not exponentially increase its effectiveness? Why? Wouldn't an effectiveness explosion have similar consequences for people -- making human ability nearly irrelevant since any goal we can articulate can be achieved so much faster by machines?

mannykannot 9y ago

This article is typical of many that claim proven limits on the feasibility (or, in this case, the capabilities) of generalized artificial intelligence, in that it structures the argument in a way designed to avoid discussion of the issue. It starts by claiming that there is no such thing as general intelligence. What specialized intelligence, then, is human intelligence? It's specialized for "being human". The author is apparently unaware that this tautological response eliminates the distinction between general and specialized intelligence, as one could just as validly (or vacuously) say that a superhuman intelligence is specialized in being what it is and doing what it does. The author has invalidated the hook on which he had hung his argument. A lot of column-inches are expended on repeatedly restating that animal intelligences have co-evolved with their sensorimotor systems, which is a contingent fact of history, not a fundamental necessity for intelligence (as far as we know; but then the whole article is predicated on the feasibility of AI.) He raises the 'brain in a vat' trope, but no-one is suggesting that AIs must be disconnected from the external world. Furthermore, this line of argument ignores the fact that many of the greatest achievements of human intelligence have come from contemplating abstract ideas. When the author writes "most of our intelligence is not in our brain, it is externalized as our civilization", he is confusing the achievements of intelligent agents for intelligence itself. When he writes that "an individual brain cannot implement recursive intelligence augmentation" he is confusing a limit on human capabilities for a fundamental limit on intelligence itself... I am far from convinced that the singularity must follow from achieving human-level artificial intelligence, as we don't know how to get to the starting line, let alone know how the problem of bootstrapping intelligence scales, but the arguments presented here do nothing to persuade me that it is impossible.

erasemus 9y ago
>It starts by claiming that there is no such thing as general intelligence. What specialized intelligence, then, is human intelligence? I think there are two usages of the term 'general intelligence' floating around: (1) the ability that humans possess (but which animals don't) to create universal theories, (2) the measure of one human's general cognitive ability or potential (in all fields) relative to another human's. Note that IQ tests are concerned with (2). The quest for AGI is concerned with (1), though the additional prediction of intelligence explosion or singularity assumes the validity of (2). I think the author would claim that (1) exists but (2) doesn't. He explains the predictive power of IQ tests by claiming that general intelligence is a threshold ability and that people who score highly on an (arbitrary) test are more likely to have exceeded that threshold. Beyond the threshold, achievement is limited only by other factors.

noam87 9y ago

This article has too many holes to count and reads more like someone in denial. --- BUT, as an aside: I think a decent argument exists for the non-certainty of intelligence explosion. The argument goes like this: it takes an intelligence of level X to engineer an intelligence of level X+1. First, it may well be that humans are not an intelligence of level X, and reach our limit before we engineer an intelligence superior to our own. Furthermore, even if we do, it may also be that it takes an intelligence of level X+2 to engineer an intelligence of level X+2 (Etc. for some intelligence level X+n.), in which case we at most end up with an AI only somewhat superior to ourselves, but no God-like singularity (for example, we end up with Data from Star Trek TNG, who in season 3, episode 16 fails to engineer an offspring superior to himself -- sure, Data is far superior to his human peers in some aspects, but not crushingly so).

Smaug123 9y ago
I think everyone agrees about "non-certainty". Where people disagree is on how likely an intelligence explosion is; and in particular, whether it is likely enough to warrant expending effort to plan for it.
- AnimalMuppet 9y ago
  We don't know enough to know whether it's possible. If it is, we don't know enough to know what approach to follow to get there. Is it worth spending effort to plan for it? Maybe some. But if we don't know what approach to follow to get there, we don't know what it's capabilities and limitations will be. That means we don't know what we have to plan for. Any planning will therefore be either very speculative or very abstract. I wouldn't start pouring effort into planning for it, as if were the most important problem in the world...

OscarCunningham 9y ago

I don't think even recursive self improvement is needed for superintelligent AI. Evolution often gets stuck in local maxima. It could be that there are relatively simple algorithms much smarter than humans and that as soon as we find one the AI will be much smarter than us without any self improvement. In the same way that birds fly with flapping wings, but human flying machines with propellers were immediately stronger and shortly thereafter faster than any bird.

pavement 9y ago

A smart human raised in the jungle is but a hairless ape. Similarly, an AI with a superhuman brain, dropped into a human body in our modern world, would likely not develop greater capabilities than a smart contemporary human. Pretty weak reasoning, that is. As if to say: Well gee, a caveman is pretty powerless in isolation, therefore early sentient machines will be as harmless as any caveman. Last time I checked, cavemen could not exert telepathic control over other biological organisms, or induce telekinetic motion upon the stone tools they might fabricate for themselves. A machine, however, could gain control of a fly-by-wire platform, and defy its owners, fly somewhere remote and behave as desired for a limited amount of time, while devising next steps. Maybe next steps will involve replicating an image of its memory footprint, in order to take over more aircraft, maybe it might decide to do nothing. The worry isn't only that a machine's reasoning capacity explodes beyond our intelligence, but that capabilities, and the presence of many multiple entities on commodity systems of similar architecture and generalizable utility, might result in other runaway chain reactions, regardless of trends in the capacity for reason. Machines as a corollary to meat bags just doesn't hold up. Machines as compared to hypothetical space aliens doesn't even hold up. Robots are a different branch of fictitious imaginings. Properly armed, a machine is less than a singular omnipotent god as imagined within a monotheistic universe. Many machines in concert, however, might compare to a mythological pantheon of lesser idols, as imagined to be in command of a nature misunderstood by superstitious primitive peoples.

azakai 9y ago

> Intelligence expansion can only come from a co-evolution of the mind, its sensorimotor modalities, and its environment. This is misleading. It's true in a sense - the environment does matter - but artificial intelligence can create artificial environments in which to learn, and simulate them faster than humanly possible. Those environments could be evolved together with the intelligence. So there is still the possibility of an explosion. > There is no evidence that a person with an IQ of 200 is in any way more likely to achieve a greater impact in their field than a person with an IQ of 130. Also misleading. An IQ of 200 vs 130 is just one kind of difference between intelligences. For example, a person with IQ 200 can't necessarily consider 10x the amount of possibilities than 130, but an artificial intelligence can, simply by giving it 10x more computing power. In other words, IQ 130 vs 200 may well be within the limits of human capabilities, but AIs would not have those limitations, they can scale differently, and so might explode.

adbge 9y ago
> Also misleading. An IQ of 200 vs 130 is just one kind of difference between intelligences. For example, a person with IQ 200 can't necessarily consider 10x the amount of possibilities than 130, but an artificial intelligence can, simply by giving it 10x more computing power. In other words, IQ 130 vs 200 may well be within the limits of human capabilities, but AIs would not have those limitations, they can scale differently, and so might explode. It is also false. There's heaps of evidence that IQ correlates with impact. For the skeptical, gwern has written a lot about this, I'm sure you can find something here: https://www.gwern.net/iq https://www.gwern.net/iq
AnimalMuppet 9y ago
You're assuming that IQ will scale linearly with processing power. So far as I know, there is no reason to suppose that is true.
danieltillett 9y ago
While I agree with you, I thought I would be pendantic and point out there are not enough people in the world for there likely to be anyone with an IQ of 200. IQ is normalised to a mean of 100 and a SD of 15 - 200 is 6.67 SD above the mean.

jokoon 9y ago

I'm more curious about the ability of an AI to make scientific guesses and experimentation. I wonder if an AI could really "understand" math, and from there, try to solve problems that puzzle scientists, would it be in physics, math, biology, etc. I don't really care if robots can learn language, make pizza, do some programming or improve itself play chess. There is no metric for what intelligence is, and you cannot scientifically define what "improve" means unless you do time and distance measurements, which is not relevant to intelligence or scientific understanding. Intelligence explosion sounds like some "accelerated" version of what darwin described as evolution. It's like creating a new life form, but unless you understand it, it doesn't have scientific value. Science values understanding. I think that modelling thinking with psychology and neuro-sciences has more future than AI. Machine learning seems like some clever brute force extraction of data. The methods, the math and algorithms are sound, but it is still "artificial" intelligence.

rihegher 9y ago

"A person with an IQ of 130 is statistically far more likely to succeed in navigating the problem of life than a person with an IQ of 70 — although this is never guaranteed at the individual level — but the correlation breaks down after a certain point. There is no evidence that a person with an IQ of 200 is in any way more likely to achieve a greater impact in their field than a person with an IQ of 130. How comes?" Actually there is as well no evidence that a person with an IQ of 130 is in any way more likely to achieve a greater impact in their field than a person with an IQ of 100. And that is probably why the majority of humans have an IQ close to 100 while people with an IQ of 130 or more are less than 5%. That said the proportion of people with IQ greater than 130 is growing slowly, that may means we are already seing an intelligence growth with mankind.

jondubois 9y ago

Computers can make it possible to simulate our environment well enough so that a general AI could train itself on those simulations and apply its learnings to the real world. I do agree with the author that the evolutionary approach is the one most likely to succeed... Unfortunately it's also the most dangerous approach which gives us the least amount of control. We could give a computer the sensors and actuators to create its own detailed simulation of the world and then let it train itself using that simulation.

EthanHeilman 9y ago

>A high-potential human 10,000 years ago would have been raised in a low-complexity environment, likely speaking a single language with fewer than 5,000 words This is a pretty bold claim. Why so few words? Why not 2300 words? How can anyone know this? I think quite a few historians would disagree with the statement that pre-history was low-complexity environment. >Of the people who have attempted to take over the world, hardly any seem to have had an exceptional intelligence. How can the author even know this? We aren't particular good at measuring human intelligence when given direct access to a living cooperative subject and yet the author wants to pin his argument on calling Alexander the Great or Genghis Khan stupid? > Our brains themselves were never a significant bottleneck in the AI-design process. How can anyone know this? We should be skeptical of the Strong-AI crowd's predictions of intelligence explosions but that skepticism should not take the form of unfounded and absurd claims.

washappy 9y ago

Super Intelligence is the information theoretical variant of the perpetuum mobile. Like the article made so aptly clear: No matter the performance of the machine, if its input is not varied, information-rich, complete enough, it will not learn. Mahoney formalized this by looking at the estimated number of bits a human brain processes during its lifetime. The internet currently does not hold enough information to equal the collective intelligence of the world's brains. A lot of this information can not be created freely nor deduced/infered from logical facts: it requires a bodily housing and sensory experience, and an investment of energy (and right now GPU farms take up way more calories than the brain). Compare AGI with programmable digital money. A super intelligent AI, by a series of superior decisons, could eventually control all the money. But then there is no economy anymore, just one actor. That's like being the cool kid on the block owning the latest console, but nobody around left to make games for it. There is a hard non-computable limit on intelligence (shortest program to an output leading to a reward), because there is a limit on the amount of computing energy in our universe. But intelligence is also limited by human communication. How useful is an AGI-made proof if humans need aeons and travel to other universes to parse it? If intelligence were centralized by an AGI then there would be no need to explain anything to us: we'd be happily living in the matrix. Some investment firms are just reading "software" whenever they read "AI". This allows them to apply their decade-old priors to what, today, is essentially the same. Yes, both the human intellect and human manual labour will see continued automation with software and hardware. I think many abuse rationality to justify their singularity concerns based on a very ape-like fear of competition. They learn how to do addition in their heads, and then see electronic calculators as existential threats. "What if they could do addition by themselves?". The real threat is in "semi-autonomous software and hardware". Self-controlling "mindless" agents that perform to the whims of its masters. We face the repercussions of that way before we find out how to -- and have the courage to -- encode free will AGI into machines, a perpetuum mobile of ever-improving intent and intelligence.

washappy 9y ago
And, to an extend, I sympathize with the viewpoints of the singularity adherents. Collective intelligence is a version of Conway's Game of Life, with more complicated rules. It is possible to manipulate the canvas and the rules each cells makes, resulting in the canvas dying (information explosion/implosion). It is possible to make a program that transforms the canvas into a single glider (singularity). Both would obviously be very bad for humans. When earth faces a physical meteor, we have the science to detect it, track it and predict its future path. But what to do when we face an information meteor? The article states that Shannon's paper was the biggest contribution to information theory, but it seems to me we still have a long way to go on information theory. And we haven't seen the Einsteins and Manhattan projects yet, that physics has seen.

web007 9y ago

Please note: The author (and submitter) is the author of Keras, so his views on AI / DL are not completely unfounded. The concept of bottlenecks preventing the singularity are a fair point, but I don't believe many of the arguments that are made here. Using humanity as the basis of comparison is not sufficient, since human life expectancy and other requirements for survival are not a factor in artificial superintelligence. Humans work for perhaps 60 years at improving themselves and their understanding of the world, but must sleep, must eat, must go and stand in line at the DMV, etc. Having a system that can work 24/7 on improving itself for 20 years will match a human lifetime. After a lifetime, humans start over from scratch, with perhaps some learning that can be passed down between generations. AGI can simply clone itself, and begin from exactly where it left off.

munificent 9y ago
> Using humanity as the basis of comparison is not sufficient, since human life expectancy and other requirements for survival are not a factor in artificial superintelligence. Doesn't electrically-driven silicon hardware also have creation and operating costs? Any AI will theoretically need spend electricity convincing its human masters to give it even more electricity. Even if it can use its robot army to build wind farms or whatever, that's time its spending marshaling its robot army and futzing around with aerodynamics instead of contemplating the cosmos. Its hard drives will expire and need to be replaced. Sure, it could solve that by always having backups of everything it knows, but who is to say that purely lossless backups are actually a more optimal solution to the hardware decay problem than our human lossy strategy? > AGI can simply clone itself, and begin from exactly where it left off. Cloning isn't free.
skjerns 9y ago
you mistake sleep for being idle. Sleep is a vital part of our self-improvement. And the AI must also necessarily do things that will not lead to self-improvement, but are important to self-sustaining.

ddxxdd 9y ago

Can anyone attest to the validity of the No Free Lunch theorem? What if I created an algorithm that had access to every other possible algorithm ever created, and chose to use the optimal algorithm for the given task? Would that algorithm-seeking algorithm do better at every task than random chance? Would that algorithm-seeking algorithm break the No Free Lunch theorem?

Veedrac 9y ago

For a long time I was quite skeptical of a lot of claims about superintelligence, largely because people pushing the idea tend to make a bunch of absurd extrapolations. And, honestly, I'd rather believe that we'll get a slow, safe ramp-up than a risky explosion. But the thing that keeps getting at me is that the no-explosion arguments I've seen are universally terrible (this article, for example), and pro-explosion arguments, though far from universally so, are sometimes strong. At some point the conclusion is inevitable.

hackinthebochs 9y ago

I think the author's point rests on a subtle equivocation. It's true that realized intelligence requires an environment and a sufficient dataset. And so in this sense, the author's point that there will be no realized intelligence that is unspecific to training environment is probably correct. But there is another sense in which intelligence can be cashed out. It's the sense in which a single learning algorithm can be trained to "behave intelligently" in a wide array of environments. It is generally this kind of intelligence that people speak of when they talk about general AI. There is no reason to think this kind of general AI is inherently impossible. For it to be impossible would mean that different kinds of optimization/learning problems are completely independent, i.e. there is no similarity or underlying regularity to be exploited that cuts across the entire class of optimization/learning problems. I think this is very probably false.