20 ms·

The Singularity Is Further Than It Appears (2015)

105 points by _qc3o 10y ago

Moshe_Silnorin 10y ago
Gwern's response to the complexity arguments: https://www.gwern.net/Complexity%20vs%20AI https://www.gwern.net/Complexity%20vs%20AI
- mdale 10y ago
  Nicely captures a strong set of counter arguments.
- dwaltrip 10y ago
  Great counter post. An off-the-top-of-my-head summary of some key points: - Don't dismiss diminishing returns. Even if takes 10^6 times more computations to double intelligence, we may very well build a machine powerful enough. - Units of computing per dollar, in contrast to moore's law, continues to double consistently and shows no sign of slowing down. - Computational complexity is a theoretical model that makes certain assumptions that don't always matter in the real world, such as optimality and worst-case performance. Approximate solutions can be far cheaper and plenty sufficient. Average case performance is often more important. - Impassable barriers can sometimes be entirely avoided by solving the root problem in a different way. Self-driving cars don't need human level vision, they just need a way of sensing the immediate environment, which LIDAR allows, despite being technically inferior to the human eye. Of course none of this says we will achieve strong AI or a singularity. It is primarily a response to the type of arguments found in the linked article above.
samatman 10y ago
Mad respect to Mez, but I was disappointed by something here. He devotes a very thoughtful paragraph to the algorithmic complexity of intelligence augmentation, then cites Intel achieving n^2 improvements in transistor density in linear time as being non-transcendental. He's correct, but the reason is obscured: Moore's Law appears to be following a logistic curve, and is leveling out as we speak. If it wasn't, the compounding interest of quadratic transistor increases over linear time could well lead to a (relatively) hard takeoff at the point where a single chip contains a human brain's worth of calculating ability. Granted that concept (a brain's worth of computation) is hand-wavey and poorly defined: but the point is that if 20n's Intel processor has one brain's worth, than 20n+1 has two, and 20n+2 has four, and so on.
- ThomPete 10y ago
  Moores Law is important but it's not more important than the emerging complexity of connected networks and devices IMO.
- Hondor 10y ago
  And all the CPUs in the world have a 20 billion brains worth. Despite the large number of transistors, they're still not organized in a way which makes them intelligent, no matter how many times you multiply them.
- tim333 10y ago
  >Granted that concept (a brain's worth of computation) is hand-wavey and poorly defined It's not that bad. There are two measures - function equivalence in devices using optimal algorithms - things like Siri, self driving cars and secondly trying to simulate a copy of the neurons in the brain similar to the Blue Brian project. The numbers are roughly 100 teraflops for the first case, about 1000x that for the second. Moores law gets all the press coverage but the more interesting thing is the amount of computing per dollar keeps doubling, did so long before Moore and probably will for a while yet. https://ourworldindata.org/wp-content/uploads/2013/05/Calculations-per-Second-per-1000-Exponential-Growth-of-Computing-for-110-years-Kurzweil.png https://ourworldindata.org/wp-content/uploads/2013/05/Calcul...
legatus 10y ago
An important aspect that the author, in my opinion, doesn't touch is attention. Most of us, humans, tend to be capable of paying attention to a _single_ task for little time, such as some hour. Now, take a computer. A computer, given sufficient electricity is capable of paying attention to such a task until its hardware has problems. Imagine if someone such as Einstein or Schroedinger were capable of paying attention to a single task (such as unifying physical theories) without needing food, water, waste release, sleep, social life. Also the task is poorly defined: increasing an A.I. intelligence isn't a single task, it can be achieved by, for example, creating faster hardware, increasing efficiency, optimizing software, as well as much higher level tasks.
- ianai 10y ago
  That's an underlying motivation for AI research and development. If you have something that can improve simply through uptime then computational hardware will make it happen. Frankly what I'd like is a 1000 foot high overview of all current types of AI technologies. I.e. What exists and what it's useful to solve. I think it's all just training neural nets on large datasets though. I can't help think there's a lot more potential than that.
- snowwrestler 10y ago
  This seems to reflect a fundamental misunderstanding of cognition, which is that only the conscious part matters. In fact we don't really understand how the brain works on problems, but there is plenty of evidence for continuous background processing: ever heard of someone getting an idea in the shower? Ever had the answer to a question pop into your head when you were doing something else?
  - robertk 10y ago
    The point stands that one can probably reduce the necessity for many daily actions when attention and willpower is unlimited. Even if you get an idea in the shower, you're still "wasting" time not being able to write a paper on it immediately, instead having to take time to dry and clothe yourself first. Whether this provides a few percentage points or an order of magnitude advantage is a different question.
fisherjeff 10y ago
> 2) There’s a huge lack of incentive This is THE big one for me. In the classic paper-clip-maximizer-gone-wrong example, there's no fathomable reason why generalized AI is necessary for such a specific, mundane task. But the organizations that have the (enormous) resources to develop such an intelligence are almost entirely focused on small numbers of tightly scoped problems. It's very difficult to see any marginal ROI for any organization whose end goal is not some form of world domination.
- soared 10y ago
  Well if a general AI already exists, wouldn't you rather use that than develop your own AI just to maximize paper clips? If general AI exists it only makes sense to apply it wherever possible (excluding security, etc) to see if it can find any optimizations. It sounds like a viable business plan and good incentive to me. Google builds gAI, sells it as a service to solve any problem.
  - user837387 10y ago
    I really think you are underestimating the impact that GAI will have on the planet. Anybody with it will become the most powerful person on Earth ever. Honestly, I think we are naive in thinking that GAI will just become another service. From our perspective it will be like inventing magic. Ask it what you want and it will give it to you. The future will truly be stranger than fiction once GAI arrives. It will be like nothing we ever imagined.
    - TheOtherHobbes 10y ago
      I'm unconvinced that GAI is possible in practice, and only marginally more convinced it's possible in theory. Consider how buggy most software is. Consider that we have nothing approaching a general theory of computation. Consider that we cannot measure the quality of any given non-trivial piece of software. Consider that we don't even know what "quality" means in this context. Consider there are minor mathematical issues - the halting problem, P != NP, the Incompleteness Theorems - which strongly suggest that all possible symbolic systems must be either incomplete or lacking rigour. Consider that the thinking around AI is an unholy mess of millenarianism, techno-evangelism, philosophical confusion between unrelated concepts (learning, personality, emotional drives, sentience, self-improvement, general intelligence, "magical" post-human physics...) and wishful thinking. I'm completely happy that we can build AIs for specific domains that perform better in those domains than humans do - because we already have. I'm fairly sure we can build learning machines that can analyse a domain and provide a practically useful overview of it to a much deeper and broader extent than we do already. I'm also fairly sure that process has practical and mathematical limits which are analogous to the transistor size limit for Moore's Law, and that the nature of symbolic computation itself makes it the most difficult of all domains to master. My guess is there won't ever be a post-singularity god-AI. But there will be AIs that appear smarter than all humans in specific areas. And that these AIs will continue to be dumb slave machines which do what we tell them to, because domain mastery is completely orthogonal to any requirement for sentience, personality, or motivation.
leepowers 10y ago
> It’s wrong because most real-world problems don’t scale linearly. In the real world, the interesting problems are much much harder than that. And this is what has been nagging me about the Singularity and its associated predictions. Exponential growth in our problem-solving capabilities is explosive when the problem-space is linear. But what happens when the problem-space itself grows exponentially with each iteration? Then we're back to linear progress. Futurism, thinking about the future of AI and technology is still important. Long-term thinking, planning, and prediction are always good. But progress will probably be much slower than anticipated.
- eli_gottlieb 10y ago
  >Exponential growth in our problem-solving capabilities is explosive when the problem-space is linear. But what happens when the problem-space itself grows exponentially with each iteration? Then we're back to linear progress. I've always wondered: why should recursive intelligence improvement be possible? That is, if you're an agent, you're nominally searching for improved versions of your source-code to self-improve. That's obviously going to be a discrete, combinatorially large search space. Why should each search space have constant or falling entropy, conditional on the agent's existing source code and knowledge? I don't really think "intelligence" can drive the entropy (or the compute-time) of any given search-problem to zero just by existing, so it seems like it should need to draw some resource other than memory space and energy from its environment. This resource is probably information: you would need some knowledge of the world to improve yourself. Successive improvements would then require successively better, finer-grained understandings of the world. That makes sense, but implies that self-improvement becomes more difficult with each round you attempt, since you've already conditioned on most of the environmental information you can get. Entropy can't go to zero, precision can't go to infinity.
  - naasking 10y ago
    > I've always wondered: why should recursive intelligence improvement be possible? That it's possible has already been proven [1]. Obviously there are limits to self-improvement, but we are ourselves limited in similar ways, ie. we are very likely simply finite state automatons. [1] https://en.wikipedia.org/wiki/G%C3%B6del_machine https://en.wikipedia.org/wiki/G%C3%B6del_machine
AndrewKemendo 10y ago
The corporation example always kills me because it's a terrible metaphor. A corporation optimizes for shareholder value, market cap or some other X related to business/market goals. They do not optimize for global intelligence capability. They do not benchmark the company based on quantitative capacity to meet or beat human capabilities across the spectrum of activities. Corporations focus on one or a handful of market specific metrics where they meet or beat their competitors. Full stop. You could argue that it would be in the company's interest to focus on general corporate capabilities, in theory giving them a major advantage in the market, but they functionally don't do that, and I would argue can't because that's not what they are designed to do. I think the only major company that might be doing something close is Alphabet, and even they are hamstrung by it. What I do agree with is this part: Lack of incentives means very little strong AI work is happening Which is my primary frustration with the field. Most people don't even want to discuss it, let alone try and specifically work on it (even if it means working on subsets which could help lead to it). I think there needs to be a philosophical MOVEMENT to create AGI. I think it will take that to get there in a short horizon. I think it will happen regardless, but without evangelical AGI proponents it's going to take a lot longer.
- marktangotango 10y ago
  Using a measurable quantity N as proxy for singularity progress and or status is valid, Vinge himself did it in a long now talk (power generation per person iirc). So in some sense asset accumulation per entity or organization could be a similar measure. Not saying this author does a particularly good job of it, or that it's a good measure however. http://longnow.org/seminars/02007/feb/15/what-if-the-singularity-does-not-happen/ http://longnow.org/seminars/02007/feb/15/what-if-the-singula...
  - AndrewKemendo 10y ago
    Yes I'm familiar with the proxy case and I think it's a worthless metaphor. I explained my reasoning in my comment but the whole point is that goal direction is the major difference.
- eli_gottlieb 10y ago
scottm84 10y ago
the industrially complex game that is military dictates that in the end most guns will point at one target. The singularity has already picked up that flag and waved it.
akuma73 10y ago
What will replace Moore's law as the engine of exponential growth? The progress in the last 30 years in computing will not be replicated in the next 30 years for the simple reason that we are hitting hard physical limits of atomic scale and thermodynamics.
- pacificmint 10y ago
  Right now chips are mostly two dimensional. Some of the structures might have three dimensions, but the general layout of the chip is two dimensional. Going into the third dimension would allow you to cram way more transistors into a small volume of space. Once we run out of improvements along the first two dimensions, up might be the only way to go to keep transistor growth going for the future.
  - akuma73 10y ago
    There are thermal issues with die stacking. Where does the heat go? This is a general problem in solid state physics. Power density is already a huge problem.
    - p1esk 10y ago
      Getting the heat out is an engineering problem. If nature could solve it (brain is 3D), we will find a way too.
      - dkarapetyan 10y ago
        But we haven't yet and saying keep doing the thing that increases power density when we know that currently doesn't work doesn't really address the main complaint. No one knows how the brain computes and the current hardware models are very poor approximations as the article outlines. Stacking more silicon is probably not the way to do it.
        p1esk 10y ago
        What does not work? Flash is already 3D. DRAM is rapidly becoming 3D. High end FPGA chips have been 3D for a couple of years. Going vertical is the obvious way to extend Moore's Law, and the main reason why we don't see more of it today is that until recently it was easier to shrink transistors, so that's what people did. Now that's changing, and people will solve engineering problems related to 3D just like they solved engineering problems related to transistor shrinking. We might not know how exactly brain performs some of the tasks, but we are pretty sure neurons are arranged in 3D structures, and heat dissipation is not a problem.
randallsquared 10y ago
People often opine that groups of humans are more intelligent than any human in the group. In terms of raw processing power, sure. But in any analogous way to AGI, it would seem not: for that it would have to be the case that the collection of humans can accomplish something in fewer human-hours than the smartest of them could. That might be the case in carefully designed situations, but generally it doesn't seem to be.
- pbw 10y ago
  The essay talks about Intel. Intel the company could design its next generation chip much much faster than its smartest member could alone.
- TheOtherHobbes 10y ago
  Groups of humans are culturally smarter than any individual human because culture persists. So the smartest human can invent something useful, and other humans can then expand and elaborate on the invention. Specific groups of humans - committees, office groups, political groups, etc - may well be dumber than the human average. For cultural intelligence you need specific processes that generate, share, and try to maximise collective intelligence. Most groups lack those processes, and without them humans reduce to a confused and milling herd which is easily led by charismatic individuals - who are not necessarily the best and brightest. Edit to add: processes to maximise collective human intelligence (like science, and persistent education) are exactly analogous to the processes that are claimed to be needed for GAI. Humanity as a whole is already an evolving GAI which has completely outperformed the limited potential of individual humans.
jumby 10y ago
Kurzweil's point is the following: "who cares". Even if it takes one-thousand years, it's a blink in the history of humanity / life.
- p1esk 10y ago
  Actually, Kurzweil cares. He keeps saying that according to his "laws", Singularity will happen by 2029.
  - leepowers 10y ago
    Even more than that Kurzweil boldly predicts that nanobots will cure virtually all disease within ~15-25 years (by 2030s at least, but no later than the 2040s). I certainly hope this prediction is true. But I think it has more to do with that fact that Kurzweil will be creeping towards the end of his life (gauged by average U.S. life expectancy) in the next 15 years.
    - fapjacks 10y ago
      Yeah, it's the old joke about futurists: When will humans become immortal? And every one of them gives an answer just short of their own expected demise.
      - kobeya 10y ago
        Not all of them, but the others are signed up for cryonics.
      - Koshkin 10y ago
        > humans become immortal The obligatory note: the true immortality is unattainable in principle without indestructibility (which is something that is hard to imagine even theoretically). Otherwise people will continue to die, en masse, from various unnatural causes. My guess as to the half-life of a human in the best of circumstances is just a couple of hundred years...
bryanrasmussen 10y ago
hmm, I figured the Singularity was just small, and it was fusion that was far away.
armitron 10y ago
He writes: "Nothing about neuroscience, computation, or philosophy prevents it. Thinking is an emergent property of activity in networks of matter. Minds are what brains – just matter – do. Mind can be done in other substrates." Yet the rest of his post is obsessed with "building" minds or AGI and tries to extrapolate based on that premise. There is a school of thought that views AGI as an emergent cybernetic process, "a metasystem-transition" [1]. This has roots all the way back to the concept of "Noosphere" that comes from Teilhard/Vernadsky [2]. Even if one does not feel aligned with such ideas, it is intellectually dishonest to posit AGI solely as the product of directed human engineering. It is far more likely in my view, that AGI will be a Black Swan event and therefore all attempts to place it on a time scale, fraught with peril. [1] https://en.wikipedia.org/wiki/Metasystem_transition https://en.wikipedia.org/wiki/Metasystem_transition [2] https://en.wikipedia.org/wiki/Noosphere https://en.wikipedia.org/wiki/Noosphere
- AnimalMuppet 10y ago
  There are also schools of philosophy where minds are not just matter.
  - Filligree 10y ago
    None of which ever explain why computers can't do the same whatever-it-is as flesh can.
    - AnimalMuppet 10y ago
      Because they claim that the mind is not just flesh. I mean, if you want to disagree, feel free. But at least understand what the claim is that you're disagreeing with.
      - Filligree 10y ago
        No, I got that. But why can't computers be not just silicon? If natural selection can blunder into exploiting unusual physics, why can't we do so deliberately?
blueyes 10y ago
He wrote this about a year before AlphaGo beat Lee Sedol... which happened 10 years before anyone expected. The singularity in the mirror could be much closer than it appears, and everything he writes tells me he knows nothing about AI. This piece is full of sloppy thinking as well as obsolete. Calling corporations superhuman AIs doesn't clarify the problem; it introduces oranges to a discussion of apples. And even in this irrelevant tangent, he is wrong. As we so often see in government and the private sector, many of us can be dumber than a few of us. Collective decision making has pernicious emergent properties, which means we should consider many corporations as subhuman AIs. > The most successful and profitable AI in the world is almost certainly Google Search. This, too, is false. Parts of Google ads might qualify as the most lucrative. But other parts of Google outside search, notably DeepMind, are much more successfully pushing AI forward. Autonomous cars and drones are two very successful examples of tech using AI. The fact that he even brings up Jeopardy Watson in a discussion of AI shows that he knows little about the state of the art, which is light years ahead of IBM's question-answer system. Ethical issues will not prevent nation-states and corporations from continuing to pursue the AI arms race. And there are huge incentives to be the one to get this right. Which is why enormous investments in AI are being made by governments and the private sector alike. Google's DeepMind is going to more than double from 400 to 1000 people, half of whom are AI researchers. DeepMind is obviously a research powerhouse, and that investment alone must cost hundreds of millions of dollars beyond the acquisition price of 400M pounds. AI advances hand in hand with hardware capacity. Distributed computing and faster chips will continue to progress, and pull AI along with them. A breakthrough in quantum computing will entail a huge step-wise leap in computing power and therefore AI. So progress will be non-linear, but not in the sense he thinks.
- mehwoot 10y ago
  What year do you think the singularity will happen?
  - blueyes 10y ago
    Nobody knows exactly. So any precise estimate is guaranteed to be wrong. But people smarter than me, and deeply involved with current research, have said they think strong AI could happen in 10 years or so. Others think it's much further away -- and that's probably the consensus view among respected researchers. It's been 20 years away for the last 80 years, right? /s
mSparks 10y ago
that bit hes bolded isnt key is it? the singluarity isnt "ai making new ai" the singularity is ai solving problems that we cant - "greater than human intelligence." which has basically already arrived. albeit bounded such that we still have ultimate control over what problems we direct ai to solve.
vonnik 10y ago
This guy has no idea what he's talking about, and the original post of which the linked article is an update was published in 2014. That's a lifetime in AI research, which is moving very fast. Real advances are happening monthly. The people closest to those advances in AI, at DeepMind for example, are moving the field forward quickly and can see strong AI on the horizon. Compute will determine how quickly we get there, but new chips and hardware are coming onto the market that will speed this along.
- dkarapetyan 10y ago
  You should address actual points instead of appealing to authority and ad-hominem attacks.
  - Chronic2h 10y ago
    Let me help you out by responding as an authority to the parent. I worked at DeepMind as an AI researcher. I can guarantee you will not see "AGI" or "strong AI" in your or your children's lifetimes. It's fun to believe, it gives us something to look forward to, talk about with friends, and discuss on HN. But in reality, we are so far from artificial general intelligence, even with an exponential curve, it will take us 100 more years. The current deep learning era (or more aptly named, pattern recognition) will last another 5-10 years, at best. Then another winter will come.
    - blueyes 10y ago
      I can assure you that some of your former colleagues disagree with you.
    - Eliezer 10y ago
      What on Earth do you think you know, and how on Earth do you think you know it?
      - daveguy 10y ago
        Most people actually doing AI research are much more conservative about their estimates and expectations of AGI. They understand very clearly how much of current AI "breakthroughs" really just barely nudge the ball forward (market speak aside). Do you use an virtual assistant -- OK Google, Alexa, Siri, etc? Has your experience with those assistants consistently improved or do they regress in annoying ways that make them seem obviously ignorant of basic facts, previously known facts, or common sense?
ilaksh 10y ago
Have people had success in efforts to train grounded deep learning systems across a variety of tasks in simulated environments? Have they had success in transfering that learning to new tasks?
tim333 10y ago
It's a good essay. The term singularity in the AI context has always bugged me as being ill defined. I think the interesting point will be when intelligent machines can run things and build other intelligent machines such that if all the humans disappeared they would keep going. That doesn't mean there needs to be a sudden increase in a division by zero way or 'sentience' in a way that keeps the philosophers happy, just that the robots can survive, reproduce and evolve without us. It would be a big change in history though.
joshuak 10y ago
It's curious why people try to build arguments for linear or even worse growth. The arguments are so odd, it seems to me that they are more about the wishes of the author then an attempt to provide a counter argument. Perhaps it's just a misunderstanding of the theory? Why is this argument about AI? The idea of a technological Singularity has nothing to do with AI. In fact the theory is pretty explicit about the fact the details of future technology are unknowable, and generally never predicted correctly. The theory attempts to explain a global evolutionary model in which global evolution continues despite the speed of localized evolutionary systems, like say biological evolution. Local effects of any particular technology's growth are not predicted, and moreover they are said to be unpredictable. Even if the op was successful at arguing that AI does not work like every other technological system that is not salient to the ETA of the Singularity. The point is not that cpus double in transistor density every 18 months, and therefore Skynet. The point is that evolution was exponential, human knowledge was exponential, electrical technology was exponential, digital technology was exponential, biological technology was exponential, and in aggregate exponential growth across all technologies is consistent and predictable. Even as one technology's growth or usefulness tappers off, others supplant it. If you arbitrary pick something say, vacuum tubes or the printing press, and create some sort of argument that it doesn't in and of it self experience exponential growth, you may succeed in your argument, but you haven't said anything.

105 points by _qc3o 10y ago

Gwern's response to the complexity arguments: https://www.gwern.net/Complexity%20vs%20AI https://www.gwern.net/Complexity%20vs%20AI

mdale 10y ago
Nicely captures a strong set of counter arguments.
dwaltrip 10y ago
Great counter post. An off-the-top-of-my-head summary of some key points: - Don't dismiss diminishing returns. Even if takes 10^6 times more computations to double intelligence, we may very well build a machine powerful enough. - Units of computing per dollar, in contrast to moore's law, continues to double consistently and shows no sign of slowing down. - Computational complexity is a theoretical model that makes certain assumptions that don't always matter in the real world, such as optimality and worst-case performance. Approximate solutions can be far cheaper and plenty sufficient. Average case performance is often more important. - Impassable barriers can sometimes be entirely avoided by solving the root problem in a different way. Self-driving cars don't need human level vision, they just need a way of sensing the immediate environment, which LIDAR allows, despite being technically inferior to the human eye. Of course none of this says we will achieve strong AI or a singularity. It is primarily a response to the type of arguments found in the linked article above.

samatman 10y ago

Mad respect to Mez, but I was disappointed by something here. He devotes a very thoughtful paragraph to the algorithmic complexity of intelligence augmentation, then cites Intel achieving n^2 improvements in transistor density in linear time as being non-transcendental. He's correct, but the reason is obscured: Moore's Law appears to be following a logistic curve, and is leveling out as we speak. If it wasn't, the compounding interest of quadratic transistor increases over linear time could well lead to a (relatively) hard takeoff at the point where a single chip contains a human brain's worth of calculating ability. Granted that concept (a brain's worth of computation) is hand-wavey and poorly defined: but the point is that if 20n's Intel processor has one brain's worth, than 20n+1 has two, and 20n+2 has four, and so on.

ThomPete 10y ago
Moores Law is important but it's not more important than the emerging complexity of connected networks and devices IMO.
Hondor 10y ago
And all the CPUs in the world have a 20 billion brains worth. Despite the large number of transistors, they're still not organized in a way which makes them intelligent, no matter how many times you multiply them.
tim333 10y ago
>Granted that concept (a brain's worth of computation) is hand-wavey and poorly defined It's not that bad. There are two measures - function equivalence in devices using optimal algorithms - things like Siri, self driving cars and secondly trying to simulate a copy of the neurons in the brain similar to the Blue Brian project. The numbers are roughly 100 teraflops for the first case, about 1000x that for the second. Moores law gets all the press coverage but the more interesting thing is the amount of computing per dollar keeps doubling, did so long before Moore and probably will for a while yet. https://ourworldindata.org/wp-content/uploads/2013/05/Calculations-per-Second-per-1000-Exponential-Growth-of-Computing-for-110-years-Kurzweil.png https://ourworldindata.org/wp-content/uploads/2013/05/Calcul...

legatus 10y ago

An important aspect that the author, in my opinion, doesn't touch is attention. Most of us, humans, tend to be capable of paying attention to a _single_ task for little time, such as some hour. Now, take a computer. A computer, given sufficient electricity is capable of paying attention to such a task until its hardware has problems. Imagine if someone such as Einstein or Schroedinger were capable of paying attention to a single task (such as unifying physical theories) without needing food, water, waste release, sleep, social life. Also the task is poorly defined: increasing an A.I. intelligence isn't a single task, it can be achieved by, for example, creating faster hardware, increasing efficiency, optimizing software, as well as much higher level tasks.

ianai 10y ago
That's an underlying motivation for AI research and development. If you have something that can improve simply through uptime then computational hardware will make it happen. Frankly what I'd like is a 1000 foot high overview of all current types of AI technologies. I.e. What exists and what it's useful to solve. I think it's all just training neural nets on large datasets though. I can't help think there's a lot more potential than that.
snowwrestler 10y ago
This seems to reflect a fundamental misunderstanding of cognition, which is that only the conscious part matters. In fact we don't really understand how the brain works on problems, but there is plenty of evidence for continuous background processing: ever heard of someone getting an idea in the shower? Ever had the answer to a question pop into your head when you were doing something else?
- robertk 10y ago
  The point stands that one can probably reduce the necessity for many daily actions when attention and willpower is unlimited. Even if you get an idea in the shower, you're still "wasting" time not being able to write a paper on it immediately, instead having to take time to dry and clothe yourself first. Whether this provides a few percentage points or an order of magnitude advantage is a different question.

fisherjeff 10y ago

> 2) There’s a huge lack of incentive This is THE big one for me. In the classic paper-clip-maximizer-gone-wrong example, there's no fathomable reason why generalized AI is necessary for such a specific, mundane task. But the organizations that have the (enormous) resources to develop such an intelligence are almost entirely focused on small numbers of tightly scoped problems. It's very difficult to see any marginal ROI for any organization whose end goal is not some form of world domination.

soared 10y ago
Well if a general AI already exists, wouldn't you rather use that than develop your own AI just to maximize paper clips? If general AI exists it only makes sense to apply it wherever possible (excluding security, etc) to see if it can find any optimizations. It sounds like a viable business plan and good incentive to me. Google builds gAI, sells it as a service to solve any problem.
- user837387 10y ago
  I really think you are underestimating the impact that GAI will have on the planet. Anybody with it will become the most powerful person on Earth ever. Honestly, I think we are naive in thinking that GAI will just become another service. From our perspective it will be like inventing magic. Ask it what you want and it will give it to you. The future will truly be stranger than fiction once GAI arrives. It will be like nothing we ever imagined.
  - TheOtherHobbes 10y ago
    I'm unconvinced that GAI is possible in practice, and only marginally more convinced it's possible in theory. Consider how buggy most software is. Consider that we have nothing approaching a general theory of computation. Consider that we cannot measure the quality of any given non-trivial piece of software. Consider that we don't even know what "quality" means in this context. Consider there are minor mathematical issues - the halting problem, P != NP, the Incompleteness Theorems - which strongly suggest that all possible symbolic systems must be either incomplete or lacking rigour. Consider that the thinking around AI is an unholy mess of millenarianism, techno-evangelism, philosophical confusion between unrelated concepts (learning, personality, emotional drives, sentience, self-improvement, general intelligence, "magical" post-human physics...) and wishful thinking. I'm completely happy that we can build AIs for specific domains that perform better in those domains than humans do - because we already have. I'm fairly sure we can build learning machines that can analyse a domain and provide a practically useful overview of it to a much deeper and broader extent than we do already. I'm also fairly sure that process has practical and mathematical limits which are analogous to the transistor size limit for Moore's Law, and that the nature of symbolic computation itself makes it the most difficult of all domains to master. My guess is there won't ever be a post-singularity god-AI. But there will be AIs that appear smarter than all humans in specific areas. And that these AIs will continue to be dumb slave machines which do what we tell them to, because domain mastery is completely orthogonal to any requirement for sentience, personality, or motivation.

leepowers 10y ago

> It’s wrong because most real-world problems don’t scale linearly. In the real world, the interesting problems are much much harder than that. And this is what has been nagging me about the Singularity and its associated predictions. Exponential growth in our problem-solving capabilities is explosive when the problem-space is linear. But what happens when the problem-space itself grows exponentially with each iteration? Then we're back to linear progress. Futurism, thinking about the future of AI and technology is still important. Long-term thinking, planning, and prediction are always good. But progress will probably be much slower than anticipated.

eli_gottlieb 10y ago
>Exponential growth in our problem-solving capabilities is explosive when the problem-space is linear. But what happens when the problem-space itself grows exponentially with each iteration? Then we're back to linear progress. I've always wondered: why should recursive intelligence improvement be possible? That is, if you're an agent, you're nominally searching for improved versions of your source-code to self-improve. That's obviously going to be a discrete, combinatorially large search space. Why should each search space have constant or falling entropy, conditional on the agent's existing source code and knowledge? I don't really think "intelligence" can drive the entropy (or the compute-time) of any given search-problem to zero just by existing, so it seems like it should need to draw some resource other than memory space and energy from its environment. This resource is probably information: you would need some knowledge of the world to improve yourself. Successive improvements would then require successively better, finer-grained understandings of the world. That makes sense, but implies that self-improvement becomes more difficult with each round you attempt, since you've already conditioned on most of the environmental information you can get. Entropy can't go to zero, precision can't go to infinity.
- naasking 10y ago
  > I've always wondered: why should recursive intelligence improvement be possible? That it's possible has already been proven [1]. Obviously there are limits to self-improvement, but we are ourselves limited in similar ways, ie. we are very likely simply finite state automatons. [1] https://en.wikipedia.org/wiki/G%C3%B6del_machine https://en.wikipedia.org/wiki/G%C3%B6del_machine

AndrewKemendo 10y ago

The corporation example always kills me because it's a terrible metaphor. A corporation optimizes for shareholder value, market cap or some other X related to business/market goals. They do not optimize for global intelligence capability. They do not benchmark the company based on quantitative capacity to meet or beat human capabilities across the spectrum of activities. Corporations focus on one or a handful of market specific metrics where they meet or beat their competitors. Full stop. You could argue that it would be in the company's interest to focus on general corporate capabilities, in theory giving them a major advantage in the market, but they functionally don't do that, and I would argue can't because that's not what they are designed to do. I think the only major company that might be doing something close is Alphabet, and even they are hamstrung by it. What I do agree with is this part: Lack of incentives means very little strong AI work is happening Which is my primary frustration with the field. Most people don't even want to discuss it, let alone try and specifically work on it (even if it means working on subsets which could help lead to it). I think there needs to be a philosophical MOVEMENT to create AGI. I think it will take that to get there in a short horizon. I think it will happen regardless, but without evangelical AGI proponents it's going to take a lot longer.

marktangotango 10y ago
Using a measurable quantity N as proxy for singularity progress and or status is valid, Vinge himself did it in a long now talk (power generation per person iirc). So in some sense asset accumulation per entity or organization could be a similar measure. Not saying this author does a particularly good job of it, or that it's a good measure however. http://longnow.org/seminars/02007/feb/15/what-if-the-singularity-does-not-happen/ http://longnow.org/seminars/02007/feb/15/what-if-the-singula...
- AndrewKemendo 10y ago
  Yes I'm familiar with the proxy case and I think it's a worthless metaphor. I explained my reasoning in my comment but the whole point is that goal direction is the major difference.
eli_gottlieb 10y ago

scottm84 10y ago

the industrially complex game that is military dictates that in the end most guns will point at one target. The singularity has already picked up that flag and waved it.

akuma73 10y ago

What will replace Moore's law as the engine of exponential growth? The progress in the last 30 years in computing will not be replicated in the next 30 years for the simple reason that we are hitting hard physical limits of atomic scale and thermodynamics.

pacificmint 10y ago
Right now chips are mostly two dimensional. Some of the structures might have three dimensions, but the general layout of the chip is two dimensional. Going into the third dimension would allow you to cram way more transistors into a small volume of space. Once we run out of improvements along the first two dimensions, up might be the only way to go to keep transistor growth going for the future.
- akuma73 10y ago
  There are thermal issues with die stacking. Where does the heat go? This is a general problem in solid state physics. Power density is already a huge problem.
  - p1esk 10y ago
    Getting the heat out is an engineering problem. If nature could solve it (brain is 3D), we will find a way too.
    - dkarapetyan 10y ago
      But we haven't yet and saying keep doing the thing that increases power density when we know that currently doesn't work doesn't really address the main complaint. No one knows how the brain computes and the current hardware models are very poor approximations as the article outlines. Stacking more silicon is probably not the way to do it.
      - p1esk 10y ago
        What does not work? Flash is already 3D. DRAM is rapidly becoming 3D. High end FPGA chips have been 3D for a couple of years. Going vertical is the obvious way to extend Moore's Law, and the main reason why we don't see more of it today is that until recently it was easier to shrink transistors, so that's what people did. Now that's changing, and people will solve engineering problems related to 3D just like they solved engineering problems related to transistor shrinking. We might not know how exactly brain performs some of the tasks, but we are pretty sure neurons are arranged in 3D structures, and heat dissipation is not a problem.

randallsquared 10y ago

People often opine that groups of humans are more intelligent than any human in the group. In terms of raw processing power, sure. But in any analogous way to AGI, it would seem not: for that it would have to be the case that the collection of humans can accomplish something in fewer human-hours than the smartest of them could. That might be the case in carefully designed situations, but generally it doesn't seem to be.

pbw 10y ago
The essay talks about Intel. Intel the company could design its next generation chip much much faster than its smartest member could alone.
TheOtherHobbes 10y ago
Groups of humans are culturally smarter than any individual human because culture persists. So the smartest human can invent something useful, and other humans can then expand and elaborate on the invention. Specific groups of humans - committees, office groups, political groups, etc - may well be dumber than the human average. For cultural intelligence you need specific processes that generate, share, and try to maximise collective intelligence. Most groups lack those processes, and without them humans reduce to a confused and milling herd which is easily led by charismatic individuals - who are not necessarily the best and brightest. Edit to add: processes to maximise collective human intelligence (like science, and persistent education) are exactly analogous to the processes that are claimed to be needed for GAI. Humanity as a whole is already an evolving GAI which has completely outperformed the limited potential of individual humans.

jumby 10y ago

Kurzweil's point is the following: "who cares". Even if it takes one-thousand years, it's a blink in the history of humanity / life.

p1esk 10y ago
Actually, Kurzweil cares. He keeps saying that according to his "laws", Singularity will happen by 2029.
- leepowers 10y ago
  Even more than that Kurzweil boldly predicts that nanobots will cure virtually all disease within ~15-25 years (by 2030s at least, but no later than the 2040s). I certainly hope this prediction is true. But I think it has more to do with that fact that Kurzweil will be creeping towards the end of his life (gauged by average U.S. life expectancy) in the next 15 years.
  - fapjacks 10y ago
    Yeah, it's the old joke about futurists: When will humans become immortal? And every one of them gives an answer just short of their own expected demise.
    - kobeya 10y ago
      Not all of them, but the others are signed up for cryonics.
    - Koshkin 10y ago
      > humans become immortal The obligatory note: the true immortality is unattainable in principle without indestructibility (which is something that is hard to imagine even theoretically). Otherwise people will continue to die, en masse, from various unnatural causes. My guess as to the half-life of a human in the best of circumstances is just a couple of hundred years...

bryanrasmussen 10y ago

hmm, I figured the Singularity was just small, and it was fusion that was far away.

armitron 10y ago

He writes: "Nothing about neuroscience, computation, or philosophy prevents it. Thinking is an emergent property of activity in networks of matter. Minds are what brains – just matter – do. Mind can be done in other substrates." Yet the rest of his post is obsessed with "building" minds or AGI and tries to extrapolate based on that premise. There is a school of thought that views AGI as an emergent cybernetic process, "a metasystem-transition" [1]. This has roots all the way back to the concept of "Noosphere" that comes from Teilhard/Vernadsky [2]. Even if one does not feel aligned with such ideas, it is intellectually dishonest to posit AGI solely as the product of directed human engineering. It is far more likely in my view, that AGI will be a Black Swan event and therefore all attempts to place it on a time scale, fraught with peril. [1] https://en.wikipedia.org/wiki/Metasystem_transition https://en.wikipedia.org/wiki/Metasystem_transition [2] https://en.wikipedia.org/wiki/Noosphere https://en.wikipedia.org/wiki/Noosphere

AnimalMuppet 10y ago
There are also schools of philosophy where minds are not just matter.
- Filligree 10y ago
  None of which ever explain why computers can't do the same whatever-it-is as flesh can.
  - AnimalMuppet 10y ago
    Because they claim that the mind is not just flesh. I mean, if you want to disagree, feel free. But at least understand what the claim is that you're disagreeing with.
    - Filligree 10y ago
      No, I got that. But why can't computers be not just silicon? If natural selection can blunder into exploiting unusual physics, why can't we do so deliberately?

blueyes 10y ago

He wrote this about a year before AlphaGo beat Lee Sedol... which happened 10 years before anyone expected. The singularity in the mirror could be much closer than it appears, and everything he writes tells me he knows nothing about AI. This piece is full of sloppy thinking as well as obsolete. Calling corporations superhuman AIs doesn't clarify the problem; it introduces oranges to a discussion of apples. And even in this irrelevant tangent, he is wrong. As we so often see in government and the private sector, many of us can be dumber than a few of us. Collective decision making has pernicious emergent properties, which means we should consider many corporations as subhuman AIs. > The most successful and profitable AI in the world is almost certainly Google Search. This, too, is false. Parts of Google ads might qualify as the most lucrative. But other parts of Google outside search, notably DeepMind, are much more successfully pushing AI forward. Autonomous cars and drones are two very successful examples of tech using AI. The fact that he even brings up Jeopardy Watson in a discussion of AI shows that he knows little about the state of the art, which is light years ahead of IBM's question-answer system. Ethical issues will not prevent nation-states and corporations from continuing to pursue the AI arms race. And there are huge incentives to be the one to get this right. Which is why enormous investments in AI are being made by governments and the private sector alike. Google's DeepMind is going to more than double from 400 to 1000 people, half of whom are AI researchers. DeepMind is obviously a research powerhouse, and that investment alone must cost hundreds of millions of dollars beyond the acquisition price of 400M pounds. AI advances hand in hand with hardware capacity. Distributed computing and faster chips will continue to progress, and pull AI along with them. A breakthrough in quantum computing will entail a huge step-wise leap in computing power and therefore AI. So progress will be non-linear, but not in the sense he thinks.

mehwoot 10y ago
What year do you think the singularity will happen?
- blueyes 10y ago
  Nobody knows exactly. So any precise estimate is guaranteed to be wrong. But people smarter than me, and deeply involved with current research, have said they think strong AI could happen in 10 years or so. Others think it's much further away -- and that's probably the consensus view among respected researchers. It's been 20 years away for the last 80 years, right? /s

mSparks 10y ago

that bit hes bolded isnt key is it? the singluarity isnt "ai making new ai" the singularity is ai solving problems that we cant - "greater than human intelligence." which has basically already arrived. albeit bounded such that we still have ultimate control over what problems we direct ai to solve.

vonnik 10y ago

This guy has no idea what he's talking about, and the original post of which the linked article is an update was published in 2014. That's a lifetime in AI research, which is moving very fast. Real advances are happening monthly. The people closest to those advances in AI, at DeepMind for example, are moving the field forward quickly and can see strong AI on the horizon. Compute will determine how quickly we get there, but new chips and hardware are coming onto the market that will speed this along.

dkarapetyan 10y ago
You should address actual points instead of appealing to authority and ad-hominem attacks.
- Chronic2h 10y ago
  Let me help you out by responding as an authority to the parent. I worked at DeepMind as an AI researcher. I can guarantee you will not see "AGI" or "strong AI" in your or your children's lifetimes. It's fun to believe, it gives us something to look forward to, talk about with friends, and discuss on HN. But in reality, we are so far from artificial general intelligence, even with an exponential curve, it will take us 100 more years. The current deep learning era (or more aptly named, pattern recognition) will last another 5-10 years, at best. Then another winter will come.
  - blueyes 10y ago
    I can assure you that some of your former colleagues disagree with you.
  - Eliezer 10y ago
    What on Earth do you think you know, and how on Earth do you think you know it?
    - daveguy 10y ago
      Most people actually doing AI research are much more conservative about their estimates and expectations of AGI. They understand very clearly how much of current AI "breakthroughs" really just barely nudge the ball forward (market speak aside). Do you use an virtual assistant -- OK Google, Alexa, Siri, etc? Has your experience with those assistants consistently improved or do they regress in annoying ways that make them seem obviously ignorant of basic facts, previously known facts, or common sense?

ilaksh 10y ago

Have people had success in efforts to train grounded deep learning systems across a variety of tasks in simulated environments? Have they had success in transfering that learning to new tasks?

tim333 10y ago

It's a good essay. The term singularity in the AI context has always bugged me as being ill defined. I think the interesting point will be when intelligent machines can run things and build other intelligent machines such that if all the humans disappeared they would keep going. That doesn't mean there needs to be a sudden increase in a division by zero way or 'sentience' in a way that keeps the philosophers happy, just that the robots can survive, reproduce and evolve without us. It would be a big change in history though.

joshuak 10y ago

It's curious why people try to build arguments for linear or even worse growth. The arguments are so odd, it seems to me that they are more about the wishes of the author then an attempt to provide a counter argument. Perhaps it's just a misunderstanding of the theory? Why is this argument about AI? The idea of a technological Singularity has nothing to do with AI. In fact the theory is pretty explicit about the fact the details of future technology are unknowable, and generally never predicted correctly. The theory attempts to explain a global evolutionary model in which global evolution continues despite the speed of localized evolutionary systems, like say biological evolution. Local effects of any particular technology's growth are not predicted, and moreover they are said to be unpredictable. Even if the op was successful at arguing that AI does not work like every other technological system that is not salient to the ETA of the Singularity. The point is not that cpus double in transistor density every 18 months, and therefore Skynet. The point is that evolution was exponential, human knowledge was exponential, electrical technology was exponential, digital technology was exponential, biological technology was exponential, and in aggregate exponential growth across all technologies is consistent and predictable. Even as one technology's growth or usefulness tappers off, others supplant it. If you arbitrary pick something say, vacuum tubes or the printing press, and create some sort of argument that it doesn't in and of it self experience exponential growth, you may succeed in your argument, but you haven't said anything.