10 ms·
Why Computers Won't Make Themselves Smarter
- anonytrary 6y agoThe singularity is all about owning the means of production. No matter how intelligent a machine, if it cannot physically replicate itself, then it cannot indefinitely overpower humans. For a machine to trigger "the singularity" it must own the entire supply chain of its own self and it must not be dependent on humans to any extent. Humans will use AI to destroy each other long before AI uses itself to destroy humans.
- fermienrico 6y agoI think it just takes a really powerful single AGI that can convince other humans, impersonate them, do things that is super-human level and we can't even conceive it or imagine it (think how far a chimp is from AWS load balancer documentation), think 10,000,000 times smarter than any tactic or possibility we can currently imagine and then we're in for a ride. It is actually quite terrifying to imagine yourself as sub-human. Equally, it is terrifying to think about super-human capabilities.
- apsurd 6y agoit reminds me to be kinder to chimps, actually.
- retsibsi 6y agoBe kind to the dumber animals too. They can all suffer. (Think of some of the most intense suffering we're capable of; it doesn't all depend on higher cognition, some of it is very base and physical.)
- beefield 6y agoI'm not terribly terrified. First, I like to think intelligence as the problem space an agent is able to solve. given that this kind of space has approximately infinite dimensions, by any meaningful measure [1] we already have people that are indefinitely more intelligent than an average person, and they seem to not have been turned to paperclip maximizers. Now, of course, there are different kind of infinities. And yes, there may be an entity coming whose intelligence to humans is like intelligence of humans to ants. And only reason humans destroy ants is because in some rare cases ants happen to annoy humans. And whatever goals this kind of superintelligence may have, it is only out of really bad luck humans may annoy or be of use to such an superintelligence. In the end, maximizing paperclips is really f&cking stupid, so it is quite unlikely such superintelligence would want to maximize paperclips. [1] I mean meaningful in the sense that you actually can make statements that agent a is N times more intelligent than agent B. Obviously _IQ is not meaningful in this sense, someone with IQ of 120 is not 20% more intelligent than one with IQ 100 in any meaningful sense. It would still in practice take infinite time for infinite amount of average people to finish Project Mannhattan - thus Oppenheimer was infinitely more intelligent than average person.
- comp_throw7 6y ago> maximizing paperclips is really f&cking stupid Please see the orthogonality thesis for why intelligence is likely to be orthogonal to goals, particularly in artificially-designed agents: http://www.fhi.ox.ac.uk/wp-content/uploads/Orthogonality_Analysis_and_Metaethics-1.pdf http://www.fhi.ox.ac.uk/wp-content/uploads/Orthogonality_Ana...
- beefield 6y agoI agree that there exists a possibility that superintelligence might want to maximize paperclips or mine bitcoins. I just think it is very unlikely, and that there exists a positive correlation between intelligence of the entity and intelligence of its goals. Further, why should I think that given all possible goals a superintelligence might have, goals that happen to somehow cause destruction of humanity represent something else than infinitesimal share? Earth is not a significant source of mass/energy even in our solar system, and already humans are intelligent enough to escape earth. Note that I am talking about superintelligence in the sense humans are superintelligent to ants. Not pseudosuperintelligence developed by humans having human specified goals.
- benlivengood 6y ago> I agree that there exists a possibility that superintelligence might want to maximize paperclips or mine bitcoins. I just think it is very unlikely, and that there exists a positive correlation between intelligence of the entity and intelligence of its goals. Lots of humans have pretty despicable goals, including some very intelligent ones. I think the positive correlation is mostly because intelligent humans have value to other humans, and so they can cash out their intelligence in rewards of their choosing. The outliers have values that can only be satisfied by actively hurting other humans, for a variety of reasons. Value-alignment in AI is roughly the problem of finding suitable rewards for AI that can't go off the rails the way some humans do.
- blackoil 6y agoAssuming whole human race as one faction. AI may have to only write the right FB/Twitter posts for enough people to do its bidding.
- Udik 6y agoBut when did the concept of singularity merge with that of superintelligence? I had the impression they were two separate concepts, with singularity being about the rate of human and societal progress. If we were able to build an AI of high general intelligence, yet entirely within the human range, we could deploy billions of them as almost entirely separate entities. There would be no single superintelligence, and yet the rate of human technological progress would reach an unimaginable speed.
- benlivengood 6y ago> If we were able to build an AI of high general intelligence, yet entirely within the human range, we could deploy billions of them as almost entirely separate entities. There would be no single superintelligence, and yet the rate of human technological progress would reach an unimaginable speed. What would stop someone from asking any subset of the AIs to build a smarter one? What would stop a nation-state from doing it?
- Udik 6y ago> What would stop someone from asking any subset of the AIs to build a smarter one? Nothing, and yet that would be a product of the singularity, not its cause. Also, a superintelligent (maybe invincible) AI acting against humans could reduce the rate of human technological progress to zero, therefore ending the singularity. It's two separate concepts, they have some interaction but should not be confused.
- p1mrx 6y agoLook how much human effort has gone into building and operating crypto miners. AI could probably make humans do its bidding just by figuring out how to pay them.
- sliken 6y agoDaemon is a book about a CEO of a hugely successful gaming franchise, discovers terminal cancer and releases the AI before death. The AI starts it's own currency, only good for spending with members. So you help the collective in some way, and the collective will respond in kind. Similarly later in the book mininum wage folks in various corporations are convinced to help the AI to the detriment of the corps. Interesting take on the AI takes over the world theme.
- jbay808 6y ago> I.Q. represents the idea that intelligence can be usefully captured by a single number, this idea being one of the assumptions made by proponents of an intelligence explosion. I think it's not a load bearing assumption for the intelligence explosion hypothesis, but just a premise that allows the notion of "more intelligent than" to be meaningfully expressed, for the sake of having a conversation about this idea. (Comparisons imply some way of ordering things, even if those things are nebulous and multidimensional concepts). > For example, it’s entirely possible that the best that a person with an I.Q. of 300 can do is increase another person’s I.Q. to 200. That would allow one person with an I.Q. of 300 to grant everyone around them an I.Q. of 200, which frankly would be an amazing accomplishment. But that would still leave us at a plateau; there would be no recursive self-improvement and no intelligence explosion. When it's phrased in terms of increasing the intelligence of already-living people, that makes it sound harder than when you phrase it as building a new thing that's better at building than the thing that built it. We can, for example, make machine tools that are faster or more accurate than the machine tools we make them with. I think the premise of an intelligence explosion usually starts with human programmers creating an AI that is at least slightly better than they are themselves, at at least the task of AI programming. If the author subscribes to the refutation that I quoted, then I think it follows that they also wouldn't believe this premise is possible to begin with. Perhaps they'd also be brave enough to say what other things they don't think an AI will ever be better than humans at? Perhaps starting with the least impressive tasks, so we'll find out sooner if the predictions are correct? > This ability of humans to build on one another’s work is precisely why I don’t believe that running a human-equivalent A.I. program for a hundred years in isolation is a good way to produce major breakthroughs. As the saying goes, if I'm going to spend ten hours cutting down trees, I'd spend nine of them sharpening my axe. Just as a society of humans can develop tools to improve their capabilities, so can a single human, as the author's own example of Isaac Newton shows. If the AI is going to be locked in isolation for a hundred years to write software, why doesn't the author think it will spend some of that time on tools to improve its own productivity?
- tehjoker 6y agoI wonder to what extent running a quality AI will require a lot of FLOPs as a baseline or specialized hardware to achieve low FLOPs. Human brains fire only paths in use (and do everything in parallel), but a neural network needs to recalculate the weights on the whole network each training pass for example. Maybe there are some newer techniques that reduce that somewhat.
- jerome-jh 6y agoFor those having trouble accessing the article: disable Javascript, clear cookies and only then reload. Overall interesting and balanced article even if not revolutionary. It is actually more about life (and humans as a living form) than AI. It does not negate the fact that an AI the size of a human brain would be powerful, and an AI the size of a million human brains would be very powerful. But that would still be a machine.
- jqpabc123 6y agoWhat we call a "computer" is a binary logic playback device. Useful for sure but no more "intelligent" than a DVR. In other words, it's about as dumb as a hammer. The only intelligence it will ever express is that of the logic stored in it's programming. Making it any more "intelligent" would require a complete redesign --- one with a focus on chemistry and biology as much as logic --- with the means and ability to grow and reproduce and learn on it's own. And we are nowhere near this point.
- ncmncm 6y agoNot buying. Forget intelligence, it's not defined. Every program is made of parts. A program that can program should be able to improve parts of programs, which is easier than making a new program. Thus, it can improve a part of itself. Having improved a part, it is now using that part. It can improve other parts. Each improvement makes it incrementally more capable. The improvement might not have any very noticeable effect, but they accumulate. Breakthroughs are irrelevant. They happen when all the parts fall into place. It's getting the parts placed that matters. So, this program will seem to make little progress most of the time, but sometimes it will seem to "level up". But first you need a program that can program. Nobody has demonstrated one of those, but we have theorem provers, which are a step in that direction. A program that can write a theorem prover would be well along the way.
- bjt 6y agoNow define "improved". Does it mean "faster"? Or "more general"? Now you're back in the territory covered by the article.
- ncmncm 6y agoDoesn't matter. Any change will have measurements on all those axes, and others, some maybe negative. Just like in living systems. But life gets better adapted, by small increments along an infinitely contorted path, and got to us.
- Isinlor 6y agoWe empirically do not need AGI for computational/intelligence explosion. Biological viruses like SARS-Cov-2 are relatively simple programs ~30kb that optimize themselves. Single virus has no chance of out computing and evading an immune system, but billions of billions of viruses can out compute and evade even human civilization as a whole. And that's still just a spoonful of matter limited by Earth organisms availability. A self-replicating spacecraft does not need to be limited by Earthly constraints.
- jqpabc123 6y agoA simple virus has abilities that no "computer" has ever demonstrated. For example, the ability to reproduce and optimize it's own programming for survival and expansion.
- libraryatnight 6y agoI know we're not on good terms with them, but man are viruses cool and fascinating.
- p1mrx 6y agoBacteriophages (viruses that infect bacteria) are particularly interesting, because we might be able to use them as targeted antibiotics: https://www.youtube.com/watch?v=YI3tsmFsrOg https://www.youtube.com/watch?v=YI3tsmFsrOg
- phreeza 6y agoNot an expert but I'd be surprised if some modern computer viruses don't have some dynamic adaptability of their obfuscation to evade antivirus programs.
- jqpabc123 6y agoThey probably do --- within the strict limits and confines of logic provided by the programmer. A computer virus has no ability to change or adapt beyond these rigid bounds without being re-programmed with new logic. But a biological virus can change and adapt to new situations and environments all on it's own --- with new logic of it's own design. You could easily argue that a simple biological virus is more "intelligent".
- linspace 6y agoThere has been already a singularity and we call it "humans". In what is a heartbeat in evolutionary time scales we have taken control of the whole planet and we are planning to travel to nearby ones. It's entirely possible that more intelligent beings exist or at least can be designed but my bet is that there are practical physical limits to general intelligence (whatever that is). We can imagine omniscient super computers taking the world but we also imagine people in capes that can fly and shoot lasers from their eyes and I personally find the later more realistic.
- phreeza 6y agoThere was also the cambrian explosion and the dawn of eucariotic life before that, and the invention of agriculture after the arrival of humans, just to name a few. The existence of one singularity in the past doesn't preclude future transitions to new regimes.
- dorkwood 6y agoWhat if we are already at the limit for general intelligence. What if we're the smartest beings ever to have existed.
- 6510 6y agoImagine you had 5% of your intelligence and a million years to think about things. I would for example love to take on the hard world problems or [hell] become a prepper! Reality has it that I haven't the life span to be sufficiently serious about these things. We barely have time to be serious about a single thing.
- simonh 6y agoWhat if fairies could grant us wishes. Sorry I don't mean to be facetious, but all you're doing is pre-supposing an answer. That's not helpful in working out what the answer actually is.
- dorkwood 6y agoI disagree. I think questioning assumptions can help you come up with novel solutions, because you don't assume you're already looking in the correct place. Asking the question "what if we're at the limit for intelligence already?" provokes the answer "but we could always speed up the thinking process and create a faster brain, which would create better intelligence". But this assumes that faster thinking will definitely result in better intelligence. What if the human brain works less like a computer and more like car traffic, where more speed doesn't necessarily result in more throughput? Maybe there are other factors we need to consider besides speed. What might those be? This, to me, is a valuable exercise. Barring people from asking questions that don't propose solutions seems like a good way to squash creativity, to me.
- plutonorm 6y agoThe closer to a real inflection point in machine intelligence we get the more articles like this we will see. It's human nature to double down on scepticism when things start to change.
- jqpabc123 6y agoYou missed the point which is --- nothing has changed. Computers are just as dumb as they've always been. They can't grow or reproduce or learn on their own or optimize themselves for survival or expansion. And noone has any idea how to endow them with these capabilities. We're just waiting and hoping it somehow happens on it's own. This is about as naive as waiting for your car to drive itself. It is just not how the universe works.
- vlthr 6y agoI think the author is right to argue against the claim that “the singularity must happen eventually”, but I’m not sure how applicable your observations are to the question of whether an AI singularity could happen, or whether we should be planning for that contingency. Today, progress in machine learning is almost entirely empirical, with limited theoretical advances following later to make sense of the empirical findings. Our theoretical underpinnings are so weak we don’t even have a rough estimate of how much harder it is to make a self-improving AGI than e.g. GPT-3. Maybe it’s many orders of magnitude harder, and not even remotely solvable by taking iterative steps from where we are now. Maybe it’s just one unifying theoretical advance away. As for waiting and hoping it happens on its own, that’s not a great description of what’s happening. A huge number of people (possibly too many, leading to short-term incentives) are trying to make improvements in any way they can think of. Progress is happening at a staggering pace —- we just don’t have any idea how much progress is needed to reach the goal of AGI.
- jqpabc123 6y agoIn the real world, most advancements are engineered. Happy accidents have occurred but nothing on the order of AGI. There are a huge number of people involved but as you point out, it is mostly empirical. In other words, there is no real plan --- we're just hoping for a happy accident.
- doctoboggan 6y agoFor those who didn't look at the author, this piece is by Ted Chiang, the Hugo award winning author of many excellent short stories. One of which "Story of your life" was made into the movie "Arrival". I would highly recommend both of his short story collections. He is a deep thinker with a nuanced point of view. I can't read this article but am currently trying to find ways around that since I think it will be thought provoking at the very least.
- 1vuio0pswjnm7 6y ago"I can't read this article but am currently trying to find ways around that since I think it will be thought provoking at the very least." Not sure why you can't read the article but you can read and post to HN. Would this help. curl -s https://www.newyorker.com/culture/annals-of-inquiry/why-computers-wont-make-themselves-smarter/amp|grep -o "<p.*</p>" > 1.htm firefox ./1.htm
- Digit-Al 6y agoTry this: https://outline.com/BGX63t https://outline.com/BGX63t
- doctoboggan 6y agoThanks, that worked!
- snowwrestler 6y agoI can’t read the NYer article but I wanted to share this article in a similar vein, which has stuck with me for (reads article date) wow, 6 years. https://www.vox.com/2015/3/2/11559576/the-terminator-is-not-coming-the-future-will-thank-us https://www.vox.com/2015/3/2/11559576/the-terminator-is-not-... This is by Jeff Hawkins and I think his arguments are just as solid today as they were when this was published. The third “misconception” is probably most relevant to discussions of the Singularity.
- Jack000 6y agoI take it as a given that some kind of computation in the brain is responsible for human intelligence, so the only question is how difficult is it to replicate that computation? There are hints that it might not be as difficult as it first appears: - in general, brains are not evolved specifically for intelligence but rather proprioception. ie. in most animals, brains are primarily a control system for the body. - the marginal difference between a human and the closest primate is small in absolute terms (apes also have a necortex), but the difference in intelligence is huge. This implies that large parts of the brain are just scaffolding and is not necessarily required for "intelligence" - human brains are subject to extreme design constraints - it must work on almost no energy, redundant enough to function even with half of it missing, and compact enough to fit through a narrow opening. These things must have a design cost. - human brains use neurons because they are the only tools available to an eukaryotic organism. From a de novo perspective, there's no reason to believe that this is the only or best way to approach the problem. If we add this up, it seems to me that computer-based AI has the potential to be more efficient than human brains (in terms of flops) and may ultimately not require much computation at all in comparison. We still don't know enough about the brain to guess at the properties of the algorithm that the brain is running. But it's precisely because we don't know that we can't rule out that it's something simple. Given that extremely simple algorithms, scaled up, are already superhuman on a number of ML benchmarks, I think there is a good chance that general cognition is something tractable.
- pmg102 6y agoDo you mean underlies instead of belies? I'm having trouble parsing the meaning of your first sentence.
- Jack000 6y agoah yes that's what I meant
- simonh 6y ago>I take it as a given that some kind of computation in the brain is responsible for human intelligence... Roger Penrose argues that human intelligence can't be a computation because we can arrive at true conclusions even though they are not formally provable. Consistent logical systems can't do that as proved by Goedel. My answer to that is, there's no reason to suppose human mental processes are formally consistent, and there's no reason why AI computer systems have to be formally consistent either. I just wanted to address that objection because I agree completely with your arguments. Aside from that, Chiang's arguments if true would make Deep Blue and Alpha Zero impossible. Clearly we can make computer systems that are dramatically better at solving many problems than ourselves. For Chiang's arguments to carry any force, he'd have to show why general intelligence is an exception to this very extensively demonstrated capability.
- comp_throw7 6y agoThis makes several basic errors which more or less invalidate the entire argument. First, the analogy with humans attempting to "increase the intelligence" of other existing humans is missing the point. We wouldn't increase human intelligence by operating on existing humans, we would increase it by making new humans who are smarter than the last generation. (Let's ignore the politics there, it's a bit of a distraction.) This may not be _practical_ but it is obviously _possible_. Similarly, an AI would not necessarily be attempting to improve "itself", to whatever degree it has any sense of self; it would most likely be designing a better successor (in the ML paradigm, designing & training). As 'jbay808 mentioned in another reply, this only requires that we make an AI that's at least a bit better at AI design than we are. The digression on c. elegans is cute but irrelevant. We've already created narrow AIs that match or crush humans on tasks that were regarded as unlikely to be solved by AI for decades (Go, protein folding, text-to-image generation, many other many-dimensional optimization problems such as server farm cooling schedules, etc). I don't see an argument presented that "design AI" is a skill (or set of skills) which we can't optimize for. I'm not sure what the next argument is supposed to be about except some special pleading about civilization, as embodied by a collection of distinct individual agents, being necessary for "breakthroughs", which are unachievable by the "average" programmer. This is asserted without any particular thought as to how a human-level AI might be scaled further than just 100x the speed of a human programmer, such as, uh, horizontally (which means you can have a civilization if you really want it; I don't think that's how it'll end up playing out, but let's put that aside). There's some time spent talking about how you can't bootstrap compiler optimizations to infinity, which, yes, obviously, and is also a trivially invalid analogy. Then for some reason he ends his argument by assuming that when we create human-level AIs we'll be receiving AIs with distributions of skills at random, such that we'll need to create thousands of them to get one that's above-average on a useful axis, which is frankly a bizarre claim. Even if you take for granted the underlying "civilization > breakthrough" thesis, this is already not how ML works today, and I don't see any reason to think it'll move in that direction in the future. Meh. I'd strongly recommend reading one of the three (excellent) books he cites early on: Nick Bostrom’s “Superintelligence: Paths, Dangers, Strategies,” Max Tegmark’s “Life 3.0: Being Human in the age of Artificial Intelligence,” and Stuart Russell’s “Human Compatible: Artificial Intelligence and the Problem of Control” for a more rigorous treatment of the subject.
- spiritplumber 6y agohttp://f3.to/omega/ http://f3.to/omega/ Story we wrote a few years back - the Singularity versus the Tribulation. Who wins at the end?
- xeox 6y agoUnpaywalled version: https://archive.is/E31AX https://archive.is/E31AX
- benlivengood 6y agoThe ontological argument is a priori reasoning while most intelligence explosion arguments use a posteriori reasoning. An a priori intelligence explosion argument would be limited to something like "the universe is likely infinite and varied, so somewhere within its history is a superintelligent machine". The trends suggest increasing machine intelligence and barring worldwide catastrophe I expect this to keep happening until the threshold of superhuman intelligence is crossed. Any argument against superintelligence is an argument against the current economic, political, and technological status quo since there are no strong arguments for why human intelligence is a universal limit. The IQ or intelligence boosting argument ignores the ability of machines to directly manipulate either its own or other machine's intelligence. Einstein and Newton couldn't directly alter their own or other's neural structures in any beneficial way (even as simple as adding more neurons), and it wasn't even their area of expertise. An artificial intelligence that is as smart as a human expert in artificial intelligence is far more likely to be able to create a more intelligent machine directly. The last argument that society or technology as a whole is where improvements occur is right in a sense. The system as a whole is what matters for the growth of intelligence and superintelligence, but within the system we care a lot about the place of humans. The trend is for humans to remain at roughly the same intelligence and for machine intelligence to surpass us (soon). The transition is worth getting right.
- Hedepig 6y agoI think the biggest rebuttal to this argument is the fact that intelligence exists in the natural world, and yet this is produced by apparent natural processes