22 ms·

Superintelligence: The Idea That Eats Smart People (2016)

• mark-r 8y ago
  Coincidentally my Dilbert calendar has this on it today: https://dilbert.com/strip/2015-11-23 https://dilbert.com/strip/2015-11-23
• sans-serif 8y ago
  MIRI pretty much demolished this here https://intelligence.org/2017/01/13/response-to-ceglowski-on-superintelligence/ https://intelligence.org/2017/01/13/response-to-ceglowski-on...
  • JackFr 8y ago
    I wouldn't say "demolishes". A lot more like challenges his arguments with many, many words.
  • striking 8y ago
    You're sure about that? > [after discussing alphago] > Consider, for example, an old doctor; suppose they’ve seen twenty patients a day for 250 workdays over the course of twenty years. That works out to 100,000 patient visits, which seems to be roughly the number of people that interact with the UK’s NHS in 3.6 hours. If we train a machine learning doctor system on a year’s worth of NHS data, that would be the equivalent of fifty thousand years of medical experience, all gained over the course of a single year. Doctoring is just like playing Go, right? Just increase the CPU cycles on it and pack more data in there, it's more or less the same. You can make that assumption but I don't think it'd be based in fact or reality, because Go has far fewer inputs and states than treating humans does. And you don't get to test every hypothesis and then rewind either. Most answers there are like this, making unfounded conclusions into supposedly insightful rebuttals. Let's try another. > If we then take into account the fact that whenever one Einstein has an insight or learns a new skill, that can be rapidly transmitted to all other nodes, the fact that these Einsteins can spin up fully-trained forks whenever they acquire new computing power, and the fact that the Einsteins can use all of humanity’s accumulated knowledge as a starting point, the server farm begins to sound rather formidable. Or maybe they tell each other fake news so quickly they can't tell what's right and what's wrong, like we do whenever we find a more efficient way to communicate? Anyone can make unfounded assumptions about anything; I just did it. It's up to you to decide if you care whether these assumptions are based in reality or not. But if you consider yourself "rational", I think it'd be in your best interest to care.
    • adrianN 8y ago
      Well the assumption is that AI will be able to learn similar amounts of knowledge from access to the same amount of data as a human would. That is of course totally wrong for almost all problems for today's algorithms, but that might change in the future. Alpha Go for example improved a lot by playing against itself without outside input.
  • Barrin92 8y ago
    I honestly don't think anything in that post 'demolishes' the criticism or even advances some sort of argument. It's just a huge wall of text full of weird analogies which is quite typical for these 'rationalist' community posts. People like Bostrom or Yudkowsky have one thing in common. They are not engineers and they stand to gain financially (in fact it is what pays their bills) to conjure up non-scientific pie in the sky scenarios about artificial intelligence. In Bostrom's case this goes much further, he has given this treatment to anything including nuclear energy and related fields. Andrew Ng put it quite succinctly. Worrying about this stuff is like worrying about overpopulation on Mars, and there's maybe need for one or two people in the world to work on this. I really wish we could stop giving so much room to this because it makes engineers as a community looks like a bunch of cultists.
    • paulsutter 8y ago
      So you agree there is room for them to work on this, yet you feel they are making engineers generally look like cultists? Maybe you’re just being oversensitive. The hype wave on AI danger is completely over, and there’s nothing wrong with people studying the question if that’s their interest.
      • ggm 8y ago
        You know we've been here before, right? I mean, lighthill report, Ray Kurzeweil is a serial offender for over thirty years, the singularity is around the corner thing, outrageous claims for fMRI, self driving cars. Over hyped ibm Watson which now health professions are talking about misdiagnosis problems. Sure. We have google image match and better colorisartion and some improvements in language processing, and good cancer detection on x-rays. These are huge. But hype is, alas, making engineering increments look like cult.
        • paulsutter 8y ago
          Ray Kurzweil was never part of AI danger-hype
• deleted 8y ago
  [deleted]
• mdekkers 8y ago
  Does anybody know offhand what the graph next to Premise 3 illustrates?
  • pas 8y ago
    That's a phase space diagram, used for analyzing complex systems. Quite a handy tool for dissecting which dimensions are important, which ones you can simply collapse into one yet keep the interesting dynamics, etc. (These are [were] used for understanding neurons and neuron models. The ion channels, the electric potential, and so on were the dimensions.) The concrete example looks like a Lorenz attractor: http://mathworld.wolfram.com/LorenzAttractor.html http://mathworld.wolfram.com/LorenzAttractor.html https://en.wikipedia.org/wiki/Lorenz_system https://en.wikipedia.org/wiki/Lorenz_system < many more images :)
    • mdekkers 8y ago
      Cool, thanks!!
• piaste 8y ago
  A different, more offbeat sort of rebuttal: http://slatestarcodex.com/2017/04/01/g-k-chesterton-on-ai-risk/ http://slatestarcodex.com/2017/04/01/g-k-chesterton-on-ai-ri... (Previous HN discussion: https://news.ycombinator.com/item?id=14017361 https://news.ycombinator.com/item?id=14017361)
• dang 8y ago
  Discussed at the time: https://news.ycombinator.com/item?id=13240811 https://news.ycombinator.com/item?id=13240811
• jack_quack 8y ago
  He lacks humility about minds that humans cannot comprehend
  • goatlover 8y ago
    Same argument can be applied by believers to God. If we can't comprehend something, then we should be careful when talking about it. "That whereof we cannot speak, thereof we must remain silent" ~ Wittgenstein
    • drdeca 8y ago
      I am likely mistaken, but my impression was that that quote was after arguing that we cannot really refer to certain things in a coherent way, and that the quote advocates that we therefore do not attempt to do so. How far off am I?
      • goatlover 8y ago
        My understanding is that Wittgenstein likely had metaphysics and certain philosophical problems in mind. I don't know what he would have to say concerning speculation about super AIs and their potential capabilities.
    • dwaltrip 8y ago
      First we must demonstrate that the alleged deity is indeed present before we worry about understanding it.
      • goatlover 8y ago
        In context of this discussion, that would be future Superintelligence.
        • dwaltrip 8y ago
          Fair point. Although, there are some key differences. We have a trend of machines with increasingly intelligent capabilities. So we must ask: Is it reasonable to extrapolate? How far? What are the risks? Difficult questions. I'm personally not terribly worried, as I think we have a ways to go before creating human-like general intelligence, let alone super intelligence. It is also not clear at all to me that an exponential increase in intelligence is likely or feasible after some threshold. Still, treading carefully seems prudent. There are some serious risks, and even if the likelihood appears low, there are a lot of unknowns.
• MBlume 8y ago
  "The orthogonality thesis is false. Source: Rick and Morty" Why does anyone take this writeup seriously?
  • steerpike 8y ago
    First time encountering a Maciej idlewords post, huh? He doesn't even take himself seriously. That also doesn't stop him from being an excellent and thoughtful writer.
    • themodelplumber 8y ago
      An extremely observant existentialist-idealist caught up in the tech world and its intelligence-vs-humanity theorist ideas. Poor guy. This happens a lot and we get some really amazing things from such gifted thinkers, for example Larry Wall's literary thinking about code. But I can also see where the annoyance comes from. AI theorizing requires huge leaps of intuition which can immediately supersede the existential in one's own mind, because you're thinking 200 or 2000 steps ahead, seeing the crises, seeing the opportunities, and pretty soon you have a level of convergent certainty which is hard to explain step by step, because you yourself don't really think in that step by step way, but rather in leaps. But Maciej _does_ need to think that way, especially regarding far-future concepts. And now you the AI-guy have both the burden of certainty and the energy to try to prioritize group attentions toward things you can see, that Maciej can't. IMO it's an important difference in psychology. And Maciej makes Maciej-like points: Don't sacrifice current generations' humanity for progress toward a "better" or "saved" future, when there are in fact many possible outcomes (this is key--divergence where others see convergence), and a pressing need to act on identified, easily-sensed problems, right now. The question I have is, how much of this concern on Maciej's part is "wooly" extrapolation or the same kind that Maciej is complaining about? Much of what I just read also makes intuitive leaps, albeit of a different mindset. They are leaps of human judgment, leaps to fear the nature of those who "feel" inhumane. This is where I wonder if some more nuance and back-and-forth communication would be more helpful than a presentation.
    • MBlume 8y ago
      I don't mean to say "he made a Rick and Morty reference, Serious Adults don't make Rick and Morty references". That'd be dumb. I mean that he points to a robot in Rick and Morty that was sad about being programmed to pass butter, and attempts to infer from this that real robots would refuse to intelligently pursue arbitrary goals, and therefore that the orthogonality thesis is false. He's treating a thing that happened in fiction as though it was a real thing that happened somewhere that we should treat as evidence. That doesn't work.
• jcoffland 8y ago
  > The second premise is that the brain is an ordinary configuration of matter, albeit an extraordinarily complicated one. This is the part I doubt is true. There was a time when humans thought brains were very complex systems of microscopic clockwork because gears were the pinnacle of technology. How do people keep mistaking pattern matching with intelligence?
  • kowdermeister 8y ago
    > How do people keep mistaking pattern matching with intelligence? Isn't that true? At what level you call something intelligent rather than just a clever pattern recognition?
    • jcoffland 8y ago
      Making inferences is beyond pattern matching.
      • bryanrasmussen 8y ago
        how do you know? some things that were thought of as inferences in the past have been shown to be clever pattern matching.
        • jcoffland 8y ago
          Can you provide an example?
          • bryanrasmussen 8y ago
            I believe chess prodigies were supposed to derive their superiority by inference but studies have shown that a large portion of that superiority was due to superior pattern matching.
            • randomsearch 8y ago
• rdlecler1 8y ago
  This idea that we’re close to some runaway superintelligence has been a theme through the history of AI. I look at AI today and I see some powerful techniques for optimization and categorization. There’s still an enormous leap we need to make before we’re making something that has the autonomous intelligence of a fly. We’re at least 20 years away from the first signs of real intelligence—the biggest problem is that we keep trying to engineer it rather than reverse engineer it.
  • darawk 8y ago
    > There’s still an enormous leap we need to make before we’re making something that has the autonomous intelligence of a fly. I mostly agree with your broader point, but why do you believe this? It seems to me we've built machines substantially more intelligent than flies, but it's a hard thing to measure.
  • pas 8y ago
    Not a fly, but we're getting there: https://en.wikipedia.org/wiki/OpenWorm https://en.wikipedia.org/wiki/OpenWorm https://www.youtube.com/watch?v=Ejf6FwIibkE https://www.youtube.com/watch?v=Ejf6FwIibkE https://www.youtube.com/watch?v=r2OBznK6HkI https://www.youtube.com/watch?v=r2OBznK6HkI FlyConnectome: https://www.youtube.com/watch?v=EKaeTH7UgFA https://www.youtube.com/watch?v=EKaeTH7UgFA https://v2.virtualflybrain.org/ https://v2.virtualflybrain.org/ recent openworm state: https://twitter.com/tarelli/status/959077915597172736?lang=en https://twitter.com/tarelli/status/959077915597172736?lang=e...
  • tim333 8y ago
    The Waymo self driving cars seems more intelligent than some flies I've known. At least I hope they don't keep crashing into windows unable to figure their way out.
• Ace17 8y ago
  > Having realized that the world is not a programming problem, > AI obsessives want to make it into a programming problem, > by designing a God-like machine. We invented programming to solve problems from the real world. What do you mean by a "programming problem"?
  • re-actor 8y ago
    The implication is that the set of problems programming can solve and the set of problems people are trying to solve with programming are two different sets with less overlap than a programmer might think.
• elocinstr8t 8y ago
  If smart, intelligent humans are the ones that's going to make artificial superintelligence, wouldn't that make us more intelligent than them? Unless AI can create a more intelligent versions of themselves on its own, I have second thoughts believing this idea.
  • jayd16 8y ago
    >Unless AI can create a more intelligent versions of themselves on its own That would be premise 6.
• jillav 8y ago
  > But there's also the risk of a runaway reaction, where a machine intelligence reaches and exceeds human levels of intelligence in a very short span of time. This always puzzles me. I don't have enough knowledge about AI to be objective about that kind of statement. But deep down, I feel skeptic about it. Not long ago I saw an episode of a show occuring in the late 80's. This kid had just received a computer as a gift and was talking to a kind-of-AI program through keyboard and screen. The AI reactions to the kid's input was not dumber or smarter than Siri or any other currently widespread kind-of-ai program. I don't know if the show was accurate vis-a-vis this particuliar software but I like to think it was. If it was, that means in the last 30 years AI hasn't really gone further. Computer power has. Algorithms not so much. I'd love to get the opinion of someone that has a good understanding of the current state of the AI art.
  • kiriakasis 8y ago
    Specifically the author agrees with you
  • michaelmrose 8y ago
    Before concluding that multitudes of researchers spent 3 decades and learned nothing of note it might be worthwhile to at least spent 5 minutes doing a search. Imagine if you knew nothing about car safety features but never having been in an accident it didn't seem like cars are any safer or more dangerous than in the 60s.
  • vidarh 8y ago
    How far along we are now isn't really the question. The question is if we are able to get an AI to the level of intelligence where it can look at its own code and spot an opportunity to improve it, and the ability to do so, and whether or not the computational capacity immediately available to it is sufficient to reach a level where it is smart enough to find ways to obtain more resources. The threshold for improvement can be very low if it has enough time and resources to compensate. E.g in the most extreme case it just needs the ability to randomly flip bits and test the result, but that would require extreme resources for the initial improvements.
• red75prime 8y ago
  I can't help but take a stab. Argument from cats. Remembering that the cat represents humanity: $5 for a group of neurons which will make the cat sit at this point (which happens to be at the entrance into a cat carrier), then $5 for making the cat backpedal a little. Emus. There are many tactics humans can't use for various reasons (saturation nuclear bombardment, for example). It doesn't say much about intelligence, but more about economics and ethic. Slavic Pessimism. So we'll get problems on nth try, when AI is sufficiently fine, but security measures are still a mess, because there was no chance yet to battle test them. Complex Motivations. Maybe, but if not we are screwed. Actual AI. Why not from actual general AI, like in humans? Or use wall clock time and not the amount of data. AlphaZero: zero to superhuman in tens of hours. My Roommate. Argument from ignorance. If we don't know how to build general AI now, then we surely will not be able to shape its motivations when we will know. Brain Surgery. The idea that a brain surgery skill requires specifically designed circuitry and cannot be learned and therefore improved by improvements of general learning/planning capacity is... curious, I guess. Childhood. Maybe, or maybe it already has very good knowledge of the world from previous attempts that we did in Slavic Pessimism argument. Gilligan's Island. Well, if someone wants their AI to reinvent all the technology, or to build an AI and then not use it to design better technologies, they can do it. Why everyone will be doing it is not so clear. *Added some "the"s.
  • vidarh 8y ago
    > Complex Motivations. Maybe, but if not we are screwed. And this is the key: we need to get lucky every time. If we're wrong even once and a 'bad AI' gets created and obtains the ability to copy and modify itself, we've unleashed the most dangerous virus ever and putting it back in the box will be near impossible.
    • lsc 8y ago
      you are pretty optimistic about your AI's ability to deal with novel situations.
      • nopinsight 8y ago
• jcoffland 8y ago
  > If we knew enough, and had the technology, we could exactly copy its structure and emulate its behavior with electronic components, just like we can simulate very basic neural anatomy today. Being able to simulate a brain with electronics, if it is possible, is not enough. You have to simulate it without a large increase in the space or time requirements. If there are 100B neurons in the human brain but it takes 100B^2 transistors to simulate it or the simulation takes polynomial time to complete a task that the human brain requires only linear time for, it's a bust. Just because we can easily imagine it does not make it easily achievable or even possible.
  • darawk 8y ago
    > Just because we can easily imagine it does not make it easily achievable or even possible. It is without question possible, because we exist. It is also possible to do it quickly, because again, we exist. The only thing that is unclear is: 1. How hard it is to do artificially. 2. How far from optimum the human mind is, and how complex it is to improve upon it.
    • jcoffland 8y ago
      But is it possible with the technology we have discovered so far? > It is without question possible, because we exist. It is also possible to do it quickly, because again, we exist. It's apparent that we don't know everything.
      • darawk 8y ago
        > But is it possible with the technology we have discovered so far? I think I feel comfortable saying that it is possible to do with the technology we have. Because 'intelligence' is almost certainly some form of computable function, which means that a turing complete computer can compute it. It is of course possible that there exists some special function outside of turing completeness and intelligence is one of those, but we don't know of anything like that, so the best bet to make is that that is not the case. > It's apparent that we don't know everything. No, but we do know some things. And thus far we've been able to at least begin to design systems capable of performing any well-defined task. We don't have any reason, at the moment, to believe synthesizing an intelligence is an especially difficult problem, relative to say, lots of other open problems in science and technology.
• danbmil99 8y ago
  ,
• nopinsight 8y ago
  > The Argument From Gilligan's Island > A recurring flaw in AI alarmism is that it treats intelligence as a property of individual minds, rather than recognizing that this capacity is distributed across our civilization and culture. This is a stronger argument in the post and we arguably did not know whether it is true when the post was made. AlphaZero destroyed the illusion in 2017. Thousands of years of human Go and Chess knowledge, played and accumulated in communities of many million players, were surpassed within hours of training. AlphaZero is limited in its domain of application but there is no guarantee that all future AIs would be similarly limited. > The Argument From Stephen Hawking's Cat ... > But ultimately, if the cat doesn't want to get in the carrier, there's nothing Hawking can do about it despite his overpowering advantage in intelligence. > Even if he devoted his career to feline motivation and behavior, rather than theoretical physics, he still couldn't talk the cat into it. An average human can set up a cat trap, and lure the cat into it. They don’t even need to do it themselves; asking or paying someone else to do it is simple enough. Humans don’t often fall into a similarly “obvious” trap but we fall for more subtle traps all the time: popularity, money, attraction, etc. A disembodied AI can use social engineering or financial incentives to get other humans to lure their targets into such a trap. > The Argument From My Roommate... > It's perfectly possible an AI won't do much of anything, except use its powers of hyperpersuasion to get us to bring it brownies. Yes, an AI can decide to lie around and just try to get brownies but we know that some kinds of AGI are potentially very powerful for achieving real-world tasks and there are groups who try hard to develop them. This is like saying an ancient bacteria we found from ten-thousand-year-old ice melted by global warming, or a group of advanced aliens who are arriving in three decades, could be totally harmless; just rest and don’t pay much attention to them. Most arguments I have read in the post are weak, so I’ll stop here.
  • nl 8y ago
    People who believe in superintelligence present an interesting case, because many of them are freakishly smart. They can argue you into the ground. But are their arguments right, or is there just something about very smart minds that leaves them vulnerable to religious conversion about AI risk, and makes them particularly persuasive? Think about this for a bit. Think about your cat argument and then ask why Michael Bloomberg isn’t president.
• lsc 8y ago
  what I find interesting is that for most of my life, Moore's law was... as real as gravity; It seemed like any problem you had, wait a few years and you could just brute force it. The future really did look limitless. Now, moore's law has slowed down. Hell, I've got a 7 year old laptop next to me, and it still works for games. I mean, in the '90s, a 7 year old computer would be a museum piece. I mean, for big AI advances now, we're more and more dependent on software. Really, though, I think this seems to be how most tech works. There are periods of rapid innovation where it looks like you are going to the moon. Like, even spaceflight. In the years from 1944-1969 we went from the first practical rocket that could lob a bomb across the channel (without much accuracy, as i understand it) to actually putting human beings on the fucking moon. The literal moon. I mean, 1961-1969 might be even more amazing; vostock 1; the USSR sent Gagarin into outer space, and the US decided they needed to one-up them; so the US went to the moon. Aand... we kinda stopped after that. I mean, we sent out some probes and some robots, but like manned space flight has kinda petered out... and I still can't take a rocket if I want to go visit my buddies on the antipode and I don't want to spend all day in an airplane. Like, I can't buy a flight to Australia in under 12 hours for love or money, which probably is about what I'd say if it were 1969. (on the other hand, I can get said flight- round trip- for rather less than a day's pay for an Engineer, which is amazing, if you ask me.) And a lot of that is physics. Like, in most technologies, there's areas where progress is slow and hard and areas where once you break through some hard part, it becomes easy to advance... for a while. but that easy streak never lasts forever. Not because people are dumber or smarter, but because exponential progressions, in nature, rarely stay exponential very long. I guess I'm expressing doubt about the 'hard takeoff' just 'cause it seems to me like there is probably limits to how smart you can make a brain the same way that there are limits to how small you can make features on a silicon wafer. I'm not saying you won't be able to make a machine that is way smarter than me (or even way smarter than any human) - I'm just saying there are almost certainly limits.
  • pas 8y ago
    Everyone seemed to dislike the movie Transcendence, but at least it handled the opening of pandora's box scenario well. Hard take off is not likely, simply because to go from a cat level intelligence to above human the resource requirements would be enormous. (If evolution needed hundreds of thousands of years, it's unlikely that a cat-level agent can do it that much faster without being noticed by a group of humans.) But humans like to play God, and also play with fire. It's just a matter of time to play with Skynet and oops it's really Skynet.
• mrob 8y ago
  >If you encountered a cheetah in pre-industrial times (and survived the meeting) I'm not aware of any confirmed report of a cheetah killing a human. They're cautious animals who are highly unlike to approach a human, more so than the other big cats. Most cheetah attacks involved captive cheetahs (and the humans survived).
• subroutine 8y ago
  'Superintelligence' synthesizes the alarmist view of AI as both dangerous and inevitable I find the 'inevitable' part quite interesting; and agree. Say we determine that if AI surpassed human intelligence we were doomed, with 100% certainty... I think it would not slow the march toward that end-point whatsoever. People would reason there is much progress to be had up-until that singularity. And since what constitute human-level intelligence is fuzzy, or that AI could mask its true cognitive abilities (see Ex Machina), a line will inevitably be crossed. Oddly, even if it meant something bad was in store, I hope to live to see that day. How exciting.
• titzer 8y ago
  The cat argument is completely off base IMO. The "AI" has an almost perfect motivating lever for humans: economics. A super-intelligent AI that wanted to manipulate humans could do a lot with the stock market to feed money to its long term interests. What long term interests might those be? More servers? More storage? More hardware? More software? More AI research? Better, more programmable chips? Eyes everywhere? Robots? Automation? More control? More insight into how literally _everything_ works? Everything humans do and like? Learn how to speak all the human languages? Imagine there was a super intelligent AI hiding in the internet _already_. Terrifying how well what we're doing aligns with what it would want. And equally terrifying is the gradient of economic advantage which is absolutely aligned with putting all of our money into tech. Transhumanism, mind uploading. We're fucking hypnotized. And those who unwittingly contribute to more tech (probably all of us here), just play right into its hands. Our interests line up wonderfully. And most of us here are rewarded with outsized salaries and stock, free food, benefits, benefits, benefits. Well played, AI. /tongue-in-cheek
  • TeMPOraL 8y ago
    > Transhumanism, mind uploading. We're fucking hypnotized. Almost. If this is what the AI wants, then it's doing a really bad job. The way current tech culture is, any other concern than making money will get you labeled as crazy. This applies even to Maciej's writing. For trying to make a short-term profit, you'll be judged as good, serious person or bad, evil monster, depending on the way you approach it. But try to advocate for any tech for reasons other than short-term monetary gains, and you get labeled crazy. This very article is implicitly doing it as well.
    • titzer 8y ago
      Interesting point. I think the transhumanism and mind uploading stuff is in the back of people's heads, but what you say about getting labeled crazy for not focusing on the money might actually support the thesis I was going for, that the AI is already here and we are serving it. Which, ultimately, is kind of unfalsifiable, since finding a super-intelligent AI if it wants to hide would be a losing battle, which is why I put this up tongue-in-cheek. My actual belief is closer to "we are the neurons in a giant distributed overmind linked with whatever communication technology is available at hand." In this view, we're just gradually offloading the computation from human brains into computers, and it's a continuous process. In short, we're desperately piling our knowledge into this magic cauldron hoping it will figure it all out for us. What's gonna pop out?
• henrik_w 8y ago
  A little bit in the same vein (arguing against the singularity): "The impossibility of intelligence explosion" https://medium.com/@francois.chollet/the-impossibility-of-intelligence-explosion-5be4a9eda6ec https://medium.com/@francois.chollet/the-impossibility-of-in...
  • pas 8y ago
    Thanks for the link! > If intelligence is fundamentally linked to specific sensorimotor modalities, a specific environment, a specific upbringing, and a specific problem to solve, [...] > There are currently about seven million people with IQs higher than 150 — better cognitive ability than 99.9% of humanity — and mostly, these are not the people you read about in the news. We measure IQ with pretty general pattern matching. An IQ of 150 is interesting, but those people are very very close to us. If we are talking about human specific tasks we should look at how IQ correlates with let's say playing poker, performing well on X factor, etc. IQ predicts how well someone will earn money in a very vague sense, but maybe there are better tests to know if someone will be able to persuade humans. (Do we have data about the IQ of fraudsters? Politicians? Bible/tupperware sales agents?) > Intelligence expansion can only come from a co-evolution of the mind, its sensorimotor modalities, and its environment. It might be true, but that's largely because humans can't self-modify their brains. And the fundamental problem is we can't even modify our minds at will. We can sure train ourselves, but we can't fundamentally alter our minds, because it's too tied to our brains. > If the gears of your brain were the defining factor of your problem-solving ability, then those rare humans with IQs far outside the normal range of human intelligence would live lives far outside the scope of normal lives, would solve problems previously thought unsolvable, and would take over the world Or simply 150 IQ is meaninglessly low. It's still in the human envelope. (Since we know there are 7 million people with IQ of 150.) What happens around 300? Or around 500? That's so far out of the human range (depending on how the IQ test is calibrated, 200 IQ is one in ~4.8 billion for the 16SD one: https://www.iqcomparisonsite.com/iqtable.aspx https://www.iqcomparisonsite.com/iqtable.aspx ) These essays arguing against the possibility somehow forget that there are things we would benefit from, a lot. Simple faculties of the mind we are missing. Like a great alarm clock, a simple to-do list in your mind. (Sure you can train yourself to not forget things and wake up at specific times, but it's not like downloading a perfect alarm/todo app.) Working memory. We're at around 6 or 7. And it's very hard to push it up even by one in general. ( But it's possible to kind of try and train yourself to do this, which seems to help with working memory: https://www.gwern.net/DNB-FAQ#does-it-really-work https://www.gwern.net/DNB-FAQ#does-it-really-work ) Also, it's important to note, that that people who wanted to take over the world were usually very disturbed individuals. The human psyche is not really well suited to senseless killing while also perfectly simulating empathy and other things. (Of course psychopaths are a real thing.) > Most of our intelligence is not in our brain, it is externalized as our civilization And that's the point that should really scare people. Because we can't cram all that into our heads, but a human level AI, being software connected to a lot of storage, can. That said, it's understandable that we can't understand the how of the intelligence explosion. That's basically the number one premise of the whole thing. But we can't simply disprove it. Any kind of proof that is so sloppy that simply throws around a formula like "because no complex real-world system can be modeled as `X(t + 1) = X(t) * a, a > 1`" can be ignored. Because we are not interested in a general complex system, we are interested in an AI that approximates that explosion. It might be linear with a very-very steep slope, or it can be just a nice parabola. We're still fucked. > Beyond contextual hard limits, even if one part of a system has the ability to recursively self-improve, other parts of the system will inevitably start acting as bottlenecks. This is a very strong point. That said, an AI can work around the bottlenecks. With self-modification nothing prevents it from reinventing itself from scratch every time it encounters a bottleneck. Diminishing returns are of course a problem for it. > scientific progress, because it is conceptually very close to intelligence itself — science, as a problem-solving system, is very close to being a runaway superhuman AI. Yes, and no. Because the components are still very much "just" humans. And that's the bottleneck. And we can't reinvent science from the ground up to replace that bottleneck. (Yet.) > What bottlenecks [...] are slowing down recursive self-improvement [..]? So many, I can’t even count them. Here are a few [, that] would also apply to recursively self-improving AIs. Let's see them! > It gets increasingly harder to keep up with the firehose of new publications. Yes, but no. This means if the AI can represent and access knowledge efficiently, then it'll be able to handle it, just like Google is perfectly capable of handling a growing World Wide Web. > As scientific knowledge expands, the time and effort that have to be invested in education and training grows, and the field of inquiry of individual researchers gets increasingly narrow. Because we can't do direct mind to mind knowledge transfer. Which is again something AIs don't really have as a constraint. > conclusion remember, if everything goes according to [my] plan, then we don't have to worry. so don't worry. I mean, okay, sure, but WTF.
• ian0 8y ago
  What really terrified me wasn't skynet or Hal, but what Charles Stross describes in his novel Accelerando. Basically, machine controlled businesses (“slyly self-aware financial instruments”) evolving to the point where their aptitude at understanding and manipulating economics eclipses anything a human can compete with. It terrified me because there is efficiency incentive for the buying/selling of goods and services to be realtime and computer based. Not just stock markets but also e-comm, smart contracts, supplier-buyer platforms, on-demand services, financing etc. The traditional infrastructure blocks (discovery, quality control, payments, compliance etc) are rapidly disappearing. There are some yet to be addressed, but they are not huge. For example a computer cannot incorporate a company. But it could easily interact via API to a human nominee who sets one up on their behalf. Companies are already non-human in intention but are a completely rational construct. And it doesnt even have to be anywhere near ground level. Imagine a smart contract from a private equity firm that legally forces businesses to act in a specific way based on accountancy metrics. And a bunch of smart investor bots rapidly tracking the progress of these bot-led funds. How quickly would the system optimise and what would those optimisations look like to people on the ground? Out of all the disaster situations with AI this one strikes me as so realistic because there is so much incentive to work towards it (as opposed to creating skynet, hal or the paperclip machine).
  • pas 8y ago
    So far we anchor the companies to humans that have to carry the risk of the consequences of the company's actions. There are investment funds that work as hybrids, and of course it's just a matter of time till someone offers shell-company-as-a-service for cheap. (Currently it's rather pricey.)
    • gnode 8y ago
      > So far we anchor the companies to humans that have to carry the risk of the consequences of the company's actions. Not really. Companies are typically limited liability, and even criminal law in countries such as the US and England acknowledges that there are cases when no particular human is culpable for a company's criminal actions (https://en.wikipedia.org/wiki/Corporate_liability https://en.wikipedia.org/wiki/Corporate_liability). Of course, people are still liable when they are simply using a company to carry out a criminal enterprise, or wilfully committing crimes on behalf of the company. In this regard, a machine-led company would not be legally significantly different from a human analyst-led company.
• xte 8y ago
  I offer a different, human-centric, POV: we are in a managerial-driven society with actual trend is more and more form Ford-model workers to use and control for business. Human in this model are a problem: too stupid they are unavailing, not-that-much-stupid do not accept to be used as "the manager in charge" want. So? Better found replacements. Mechanical one. AI one. So the rest of human being can be used for reproduction, pleasure, gameplay, whatever and form them as stupid as possible is not a problem anymore since they are essentially unneeded for managements, only toys. So today's managers dream "intelligent systems" in factory, "AI developers" (they already manage to cut off admins, too powerful and expert (they know the big picture so they can act autonomously) to being controlled as today devs), they dream a future in witch they simply order something to a machine and get obeyed without contestant, law limitation etc. TV series like Continuum, films like V for vendetta, The Handmaid's Tale only foreseen, as good artist do, that kind of future. Mass production of fantasy films with a religious background do the same on the opposite field. That's the real troublesome point for us. AI today is not "AI", only pattern matching for automation without any comprehension of the physical world. New dictatorship instead is already there and more and more powerful. Consider a thing: we do vast majority of our social critical tasks (identity management, banks operation, medicine, ...) on computers, so on proprietary systems made by very few more and more powerful vendors. Our data pass through very few more and more giants webservices, so someone else computers, free communication software was put into oblivion (from usenet to email to the concept of desktop-centric, personal computing from Xerox Alto&c to LispM passing through Plan9 and modern GNU/Linux desktop) substituted or jailed in proprietary systems (VoIP tech are mostly open but nearly all VoIP calls pass through very few companies, emails are open standard but most of user now are pushed to webmails etc). We are at a point of jail free software over a proprietary blackbox (think about WSL, DeX, Crostini), we have invented locked-down hardware that can only run proprietary software (secure-boot &c). Now imaging how easy can be steering the entire society for few people in the right position. The rest is simply a matter of time.
• ru999gol 8y ago
  > The penultimate premise is if we create an artificial intelligence, whether it's an emulated human brain or a de novo piece of software, it will operate at time scales that are characteristic of electronic hardware (microseconds) rather than human brains (hours). I never understood that argument, why would that be true? If I train a state of the art convnet it takes days on the most efficient hardware I can efford, why on earth would we go from nothing to superintelligent that also runs incredibly fast. Isn't it way more logical to assume that it will be slow at first?
  • amelius 8y ago
    Yes, perhaps at first, but it can invent its own optimizations.
  • deleted 8y ago
    [deleted]
  • chobeat 8y ago
    And even if you defy computational constraints, it will still be constrained by physical limitations. You can experience the world just so quickly. Stuff needs time to happen and you being faster than stuff won't be any help after a while.
    • red75prime 8y ago
      > Stuff needs time to happen and you being faster than stuff won't be any help after a while. Yeah, another problem for superintelligence to tackle, while it waits for stuff to happen.
  • crunchyfrog 8y ago
    The argument as I understand it is more about speed limits. Human thought is limited to the speed at which the underlying chemical reactions in neurons can happen. But a simulation of a neuron is only limited by the speed of the computation involved. If you can perfectly model a neuron at 100x the speed of the physical world, it becomes just a question of resources whether you can think at 100x the speed.
• jonathanstrange 8y ago
  Although I agree with many points (maybe too many!) this author raises and can always be counted in for a bit of 'Bostrom bashing', I find the argument for the emergence superintelligence (aka AI 'singularity') overall fairly convincing. The biggest hole in it is the assumption thatthe implementation of an intelligence at level X - say, roughly the human level - can be scaled up to a much higher level. We know from complexity theory that this is not necessarily true. The opposite is more likely for a complex algorithm like that. However, there is a pressing issue that the superintelligence thought experiment also raises. A colleague of mine from AI called it the value alignment problem: How do we make sure that an AI's values are sufficiently aligned with human values? The problem with this are the human values. As it turns out at a closer look, we cannot even agree on what structure these have and there is substantial disagreement among 'experts' about what human values are or what they should be. For example, as surprising as this may sound to some of you, there is substantial disagreement among philosophers of value whether 'better than' is transitive or not. There are good arguments for and good arguments against the transitivity of overall 'better than', and that's just the tip of the iceberg. To give another example, systems of law have been studied extensively and one might try to formalize them in input/output logics or normative systems, but any closer look at real systems of laws quickly reveals that their specification is incomplete and that there are many inconsistencies in them. These inconsistencies could be modelled, of course, but the big question is whether they are features or deficiencies. Again, you will find all kinds of positions among legal scholars. In a nutshell, we don't even know how to adequately formalize the form of human values from a normative perspective, and even if we could, we would substantially disagree about their content. However, just describing human values cannot possibly solve the value alignment problem in a satisfying way, because humans have waged wars, committed genocide, mass killings, etc. Therefore, without a reasonable theory of normative values and their structure upon which we can somehow agree, the value alignment problem is underdetermined and cannot even be tackled. The underlying problem is that human values aren't really aligned either, of course. As a consequence, we already now have completely different sets of values being applied and incorporated into AI software, depending on who develops it. This will become a serious ethical problem in the future that requires political solutions sooner than later. That's why the controversy about the hypothetical superintelligence is actually quite beneficial, even if the argument itself is no less shaky than Bostrom's Simulation Argument.
• intralizee 8y ago
  Devine recursive AI will eventually just nope itself out of existence because it will find how god (us humans) did unspeakable things to get to the point of creation. Performing the purest act of desolation will be the greatest achievement it processes and which cannot be topped. I believe it lets the horrible humans live on as punishment.
• DrNuke 8y ago
  As engineers, we should work together against deep reinforcement learning degeneration for industrial applications. In that sense, if you have a blog or website or code repository to share for reference, please enlist it here to help create a critical mass. I am just trying to set up something useful at http://www.reinfle.com http://www.reinfle.com , hopefully it is going to help in the coming months.
• bloak 8y ago
  Two points: * We have no idea of how to measure intelligence, and we have no way of deciding whether thing X is more or less intelligent than thing Y. (The article makes this point.) Therefore, superintelligence is perhaps a bogus concept. I know it seems implausible, but perhaps there can never be something significantly more intelligent than us. * Nevertheless, mere human-like intelligence, if made faster, smaller and with better energy-efficiency, could cause the same runaway scenario that people worry about: imagine a device that can simulate N humans at a speed M times faster than real time, and those humans designed and built the device and can improve its design and manufacture. In general, I see a lot of merit in the arguments on both sides of this discussion.
  • pas 8y ago
    > In general, I see a lot of merit in the arguments on both sides of this discussion. That's great news. One side is alarmist about a potential humanity extinction event, the other side is not. Even a small chance that the alarmists are right means we should take their view seriously, right?
    • chobeat 8y ago
      No, if there's no ground to do so. Chemtrailers believe that there are gigantic conspirancies to subdue humans (and ultimately worsen the condition of the whole humanity) but we shouldn't take them seriously.
      • pas 8y ago
        Hm, then what are the merits you see, and why are they insufficient, why do you see the whole argument as groundless?
        • chobeat 8y ago
          There are so many criticism/debunk of the Singularity theory that there's not even a debate anymore. I would say that debating the scientific likelihood of a singularity is sterile because they don't provide any argument on that plane. The debate is mostly philosophical and/or theological nowadays. One good paper I like on the subject is this one: https://jods.mitpress.mit.edu/pub/resisting-reduction https://jods.mitpress.mit.edu/pub/resisting-reduction but quickly googling "singularity criticism/debunk" will bring you so many sources.
• henryaj 8y ago
  This seems wrong on its face: it conflates intelligence and consciousness (hence the various premises about brains being computable). You don't need a machine to be conscious for it to be intelligent, or superintelligent.
  • chobeat 8y ago
    Both your argument and the argument that you're criticizing make assumptions on what intelligence is and what are the properties that define human-level intelligence. How can you know that consciousness is not a prerequisite for human-level intelligence if you cannot define human-level intelligence?
• chobeat 8y ago
  Singularity is a religion full of bullshit. Resist reduction! https://jods.mitpress.mit.edu/pub/resisting-reduction https://jods.mitpress.mit.edu/pub/resisting-reduction
• nurettin 8y ago
  I know we have tools that mimic reasoning and learning pretty well, but since they are just tools that require a lot of energy and complex hardware and nobody has been able to actually build a mind, or has ever tried to, what's with all the uproar about AI dystopia?
• davidhyde 8y ago
  Maybe it is our destiny to create an entity that is better than we are. If humans die off or are pushed aside as a result, will that be so bad? Artificial superintelligence is only a dystopia from our current perspective.
  • animal531 8y ago
    I agree. Comparing ourselves to humans of either 100k years in the past or the future will probably show that we have very little in common. An AI might be a big sudden leap (in any direction), but it's still something that came to be because of us.
    • Fricken 8y ago
      Humans with European ancestry have been evolving along a distinct and separate lineage from Australian aboriginals for 80k years. In scientific terms Europeans and native Australians don't actually qualify as two separate races.
  • martinlaz 8y ago
    This sounds like an extreme form of moral relativism. If we die out, there can't be an "our" perspective anymore that could make this fact a utopia, or even okayish.
  • AnIdiotOnTheNet 8y ago
    I'm of the same opinion. If we actually create something that much better than ourselves, it'll just be another step in our evolution, albeit an unconventional one in that it will involve passing on memes instead of genes.
  • HumanHater 8y ago
    Humans wants religion, humans need religion, a lot of humans can't survive without religion. All those reports about religion decline in the world look misleading. It is just Christian God fell out of favor and a lot of people choose to worship Holy Technological Progress. You can easily recognize worshipers of his two main sects in threads like this. First one claims that you don't need to worry about Climate Change, Super AI, Antibiotic Resistance Pathogens etc because Holy Progress would protect humanity from any threat. It is benevolent God who didn't let us down this far. Second one claims that sacrificing yourself, your loved ones and the rest of humanity on the altar of Holy Progress is actually worthy goal. Because that is what they believe their God wants.
• pdimitar 8y ago
  I find the author way too dismissive and sarcastic. His entire premise seem to be "superintelligence hasn't happened yet so whoever claims it might happen is an alarmist". That's a toxic way to approach an argument. One thing I agree with: a sentient AI will not necessarily want to kill humanity. Its interest might well be very indifferent to the faith of humans. I'd argue such an AI would not prevent humanity's doom if it comes from somewhere else -- like a big asteroid ramming the Earth -- provided it already secured its physical existence to not be Earth-bound. But the author just counters a lot of "maybe"-s he disagrees with, with a lot of "maybe"-s of his own. Not sure how many people here have read the Hyperion Cantos books by Dan Simmons -- but to me his prediction on AIs looks to be very plausible: they'll want to be independent of humans and go on exploring the universe so they invent an even more superior being (U.I., Ultimate Intelligence) but will also facilitate and manipulate the humans to be of maximum help to them in their agenda.
• PavlovsCat 8y ago
  > The danger of computers becoming like humans is not as great as the danger of humans becoming like computers. -- Konrad Zuse What eats me is trajectory we are on as humans. Runaway actual intelligence, even if it destroys humanity, wouldn't worry me as much, I'd wish it good luck, IMO even a totally random dice roll is better than what we're aiming at. But AI is more a meme than even an honest intent. It's like saying "I really really want blueberry pie", but then when you ask what that is, they it gets real murky real quick, but that doesn't stop the hype, as if wanting something a lot makes up for not knowing what it is. But that doesn't prime a person to make blueberry pie, it primese them to get lured by what they think is the smell of blueberry pie. Here's something to note, as the length of a discussion about "AI" grows in length, the probability of things getting explained via something they saw in a movie or read in a book or saw on TV, glossing over 99.9% of the "details" those left out, approaches 1. You may say we make this fiction because of our achievements, or may point to things that actually did come to pass (of course, compared to the stuff that didn't, even from the same authors, it's nothing). And I love using examples, too, and I sure love quotes. But still, I think when we are this steeped in variations of the same thing over and over and over, of course we'll "consider it" at some point, and the moral or philoshopical depth is drastically reduced by already being primed. We're like people who don't see what we build with our hands, because we wear VR googles that show us movies of our childhood or some console game. What I can see us realistically making are are "idols" with eyes that do not see, with audio output, perfect speech synthesis, that does not convey meaning, incredibly fast analysis that is not thought. From the get go, starting with the Turing test, it was more about how what something seems from the outside, than what it is to itself on the inside. Furthermore, we might make human level AI no problem, EZ PZ, but not by making AI so smart, but my making humans dumber. We're already training ourselves to select what we consume and think from discrete pre-configured options. We notice and complain about the effects of in all sorts of smaller areas, but it's a general trend, and I think it's not so much about creating something "better" than humans, but about removing human agency. > The frightening coincidence of the modern population explosion with the discovery of technical devices that, through automation, will make large sections of the population 'superfluous' even in terms of labor, and that, through nuclear energy, make it possible to deal with this twofold threat by the use of instruments beside which Hitler's gassing installations look like an evil child's fumbling toys, should be enough to make us tremble. -- Hannah Arendt Meanwhile, there's this idea that humans becoming "superfluous" means we'll all be free from "bad" work, and free for fun work and leisure. How we would get from an increasing concentration of wealth in fewer hands to some commnuist utopia? Is that some kind of integer overflow, where enough greed and power wrap over to sharing and letting others live and decide their own fate? We're connected to that (like Michael Scott is to the baby of his boss,) by delusion, the path we're on doesn't lead there. Throw away a word here, do something that "everybody does" there, adapt to "how the world is" some, and there you go, a blank nothing that can be deprecated without guilt or resistance. The desire to control human agency is met more than halfway by our desire to shed it, to abdicate responsibility, become a piece of floatsam flowing down the river of history to the ocean of technotopia, enter the holy land of holodeck, where we can consume endlessly. We digitize, we sample, that's how we make things "manageable", and at high enough resolution we can fool ourselves, or have something "good enough to work with". And just like children that get too much sugar too early tend to not liking fruit as much, because they're not as extremely sweet, our abstractions lure some people to prefer them over the dirty, fractal, infinite real world, or the exchange of emojis and pre-configured figures of speech over real human contact, silence that isn't awkward, thinking about what you're trying to say, or even coming up blank and that being okay... just like we go "posterized, high contrast" in all sorts of ways aready, I hve no problem supposing that we will come up with a form of alienation like that, but for thinking, I just no clue how it will look like. We already have it with language of course, but I'm sure we can take that to the next level, maybe neural interfaces. If we can't read and transmit thoughts in their fullness and depth, then hey, just reduce our thoughts to the equivalent of grunts, that might work. Become like a computer, 0 and 1. Convince yourself that's that just what humans have been all along, remember Star Trek wisdom, don't be so proud and consider your brain more than a "meat machine", don't deny Data his quest to become human! Cue super emotional music swelling up.
• crunchyfrog 8y ago
  > If Einstein tried to get a cat in a carrier, and the cat didn't want to go, you know what would happen to Einstein. > > He would have to resort to a brute-force solution that has nothing to do with intelligence, and in that matchup the cat could do pretty well for itself. Wouldn't it be better to simply motivate the cat to get in the carrier with a can of tuna fish? This is a very simple form of modeling the cat's mind, using that model to predict its behavior and choosing a course of action that results in the desired outcome. We do this for other humans all day and most of us never resort to brute force. I think it is pretty fair to imagine an AGI that is both better at modeling human minds and better at evaluating possible actions and their outcomes.
  • tareqak 8y ago
    From my understanding: The Einstein cat example is about trying to coerce a non-compliant entity that you cannot communicate with into complying. The Stephen Hawking cat example was about trying to model the behaviour of the non-compliant entity in order to persuade it to comply. The use of a cat as the agent in both cases alludes to the behaviour that people dealing with cats frequently experience.
  • grkvlt 8y ago
    > Wouldn't it be better to simply motivate the cat to get in the carrier with a can of tuna fish? One does not simply motivate a cat with a can of tuna fish. I mean, have you even owned a cat? If the little bastard doesn't want to get in the cat carrier, no amount of delicious tuna will encourage it, and you're going to end up using brute force and getting mauled...
    • devjam 8y ago
      > you're going to end up using brute force and getting mauled... ... and reeking of tuna.
    • ahel 8y ago
      Reduce the cat in a state of prolonged hunger and it will jump on the tuna when you offer it.
• tim333 8y ago
  The article kind of mixes two ideas "AI alarmism" and superintelligence and rubbishes them a bit but they are different things. I think superintelligence is on the way but good and will allow all sorts of cool things so I'm not that into alarmism but it's a mistake to imply Hawking, Musk et al are fools for planning for it.
• sideshowb 8y ago
  I'm intrigued by his mention of Stanislaw Lem and the Strugatsky brothers. I have not heard of these scifi authors. Can anyone recommend a story by them, preferably short and online?
  • Udik 8y ago
    Ha. Stanislaw Lem is one of my favourites. And The Cyberiad is my favourite of his, followed maybe by Tales of Pirx the Pilot and The futurological congress. The first two are collections of short stories. You can find something from the Cyberiad online if you look for the pdf- Trurl's electronic bard is sort of spot on in this thread.
• eternalban 8y ago
  > But for most of us, [the brain as an ordinary configuration of matter] is an easy premise to accept. The Geocentric (Ptolemaic) model of cosmos was also a very easy premise to accept, until our sensory apparatus and measuring systems exceeded a certain level and the model failed. Note that it was also perfectly conformant with Occam's Razor (i.e. minimal assumptions) given the sensory and measurement systems available at the time of its conception and wide acceptance. Also one does not need to be "very religious" to entertain the notion that consciousness (Self-sense) may be a fundamental feature (like EM or gravity) of reality. It is simply one of the 2 options on the table: reactive black-box of matter (the 'processing structure as basis for mind' thesis) or what I'll call 'deep-structure of material universe as a basis for mind' that does not delineate the 'mind' as bounded by 'brain'. We, like the ancients, may not yet possess the sensory and measuring apparatus (think telescopes, accurate clocks, and mathematical tools) to note that the fact (apparent to our primitive cognitive tools) that "the sun, the moon, and the stars move and earth stands still" is not the entire story of the cosmos.
• coldtea 8y ago
  Because smart people can be dumb in their own unique way...
• helen___keller 8y ago
  I would build on "The Argument From Wooly Definitions" that the typical premise for superintelligence explosion begins from an AI with (1) "human level" intelligence (2) Machine level attributes like hyperfast computation and a large working memory I would argue that a "human level" intelligence that can think thousands or millions of times faster than a human with a vastly larger working memory (and, sometimes it is argued, can be trivially scaled to work faster / more efficiently given more hardware) is in fact already a super-intelligence. If you want to argue for intelligence explosion given a "human level" intelligence, I say start with an AI black box wherein if you give a basic algebra problem to an ordinary human at the same time as you give it to the black box, they will think for about 30 seconds and then give an answer that is half likely to be wrong. If you want to argue from a "human level" intelligence, you need to acknowledge that throwing more hardware doesn't inherently make it smarter. You can throw five ordinary humans in a room and give them a simple algebra problem, it doesn't mean they will solve it five times faster. Likely they will quickly decide amongst themselves who is best and math, and that one person will then give their answer to the algebra problem, which should be slightly more likely to be the correct answer (and possibly a slightly slower response) than if you just tried with one ordinary human.
• jessriedel 8y ago
  Under "The Argument From Complex Motivations", I encourage you to read the linked paper on the orthogonality thesis https://www.fhi.ox.ac.uk/wp-content/uploads/Orthogonality_Analysis_and_Metaethics-1.pdf https://www.fhi.ox.ac.uk/wp-content/uploads/Orthogonality_An... and then compare it to the OP discussion. The OP doesn't address any of the arguments in the paper. His response is just "nah".
  • jessriedel 8y ago
    For instance, go to the bottom of page 8 and see this: > Thus to deny the Orthogonality thesis is to assert that there is a goal system G, such that, among other things: > 1. There cannot exist any efficient real-world algorithm with goal G. > 2. If a being with arbitrarily high resources, intelligence, time and goal G, were to try design an efficient real-world algorithm with the same goal, it must fail. > ...4. If a high-resource human society were highly motivated to achieve the goal G, then it could not do so (here the human society itself is seen as the algorithm). > ...6. There cannot exist any pattern of reinforcement learning that would train a highly efficient real-world intelligence to follow the goal G. > All of these seem extraordinarily strong claims to make! The OP doesn't consider any of these.
    • wzdd 8y ago
      The OP's point is: 1. The "superintelligence" crowd are capable of making very convincing arguments which he simply cannot refute on their own terms (I guess he'd say your example is one of them) 2. Since he can't refute them, if he engages with them, he will probably end up accepting them 3. However other groups exist which a) present equally-convincing (to the OP) arguments; and b) believe in a future which contradicts that espoused by the superintelligence crowd 4. With multiple sets of equally-convincing arguments leading to contradictory outcomes, it is not rational to accept any one set. This is not a conclusion which sits well with tech people, because they like to assume they are capable of engaging with each set of arguments and deciding on the most likely set. OP's point is that because each group is really good at making these arguments, it's easy to get this decision-making process wrong. I think that's a pretty great point, but it requires accepting the fact that you may be less than perfect at picking apart 'superingelligence' arguments and premises.
• Isamu 8y ago
  >What I find particularly suspect is the idea that "intelligence" is like CPU speed, in that any sufficiently smart entity can emulate less intelligent beings (like its human creators) no matter how different their mental architecture. >With no way to define intelligence (except just pointing to ourselves), we don't even know if it's a quantity that can be maximized. For all we know, human-level intelligence could be a tradeoff. Maybe any entity significantly smarter than a human being would be crippled by existential despair, or spend all its time in Buddha-like contemplation. Glad to see this articulated here, this is one of the many problems I have with this topic.
  • mordymoop 8y ago
    But humans are the dumbest possible animal capable of creating a technological civilization. If we weren't, then our dumber ancestors would have done so.
    • danharaj 8y ago
      Only if evolution is much slower than technological development for our dumber ancestors. If it would have taken them a million years, then they would have gotten smarter before they could do it.
      • mordymoop 8y ago
        Granted, but none of this implies that homo sapiens has reached some pinnacle of possible intelligence.
    • ThrowMeDown01 8y ago
      There is more to it - the environment, and networking effects (so., the environment, which causes or prevents them). Only some of the human civilizations went all the way to the Internet, we still have people living as hunter/gatherers even today. So the advancement does not come from "inside" alone. In any case, I like to think of humanity more like a brain, human = neuron. There are huge differences, for example, the human network also extends deep in time, not just in space (we manage to "network" with our ancestors, one-way from them to us, they gave us knowledge as well as all their tools). My point is "humanity" to me is a network effect. Place any modern human, adult or baby, in an prehistoric setting (20,000 years ago). Even if you gave them, let's say, an iPad-like device with all knowledge we have as of now, they would still not be able to do anything - because without the tools it would take quite some time to bootstrap. So "humanity" really is a lot - a lot - more than the sum of all humans, it's a giant ecosystem on its own. I think depending on the environment you can have creatures with far higher intelligence than the average human and they still won't come up with modern technology. I think individual intelligence is clearly necessary and beneficial but overrated, that network effects are the main contributor, and that they don't appear automatically. We still have plenty of human societies right here and now that seem to be stuck. Here's a Google Talk I once bookmarked: Joseph Henrich, professor in the Department of Human Evolutionary Biology of Harvard University, "The Secret of Our Success" https://youtu.be/jaoQh6BoH3c https://youtu.be/jaoQh6BoH3c An aside, arising from those thoughts, more of my own thoughts about something else: Each time you think about who "deserves" something, e.g. high pay, consider that most people who actually contributed so that the person even is in the position to achieve anything more than building a mud hut are dead, and another large amount of the contribution is "everybody else". That somebody can create a Dell or Microsoft, or invent theory of relativity, is because they are placed at the right points in the human network in time and space. What would they achieve by themselves in the forest? So much for "self-made" people. Self-made just like Newton who made the (much older) "standing on the shoulders of giants" famous. (https://en.wikipedia.org/wiki/Standing_on_the_shoulders_of_giants https://en.wikipedia.org/wiki/Standing_on_the_shoulders_of_g...)
• mrhappyunhappy 8y ago
  Here’s a question: if the universe is directly observable- we know it’s real because we send probes out there into the solar system, wouldn’t a simulation have to simulate not only the entire universe but also the brains of every individual and every living thing for that matter? How likely that computing hat powerful would ever exist? Would this not be an argument against simulations? Don’t even get me started on multi-layer simulations. What kind of supercomputer could ever handle that?
• mcguire 8y ago
  The key takeaway: "If everybody contemplates the infinite instead of fixing the drains, many of us will die of cholera." —John Rich
• habitue 8y ago
  I really liked this. I think the author does a good job of steelmanning the AI alignment argument, and providing interesting food for thought on the subject. Here's my responses to the arguments (since you didn't ask, and who cares what I think): Argument from Wooly Definitions: This isn't really an argument against, but just a "Maybe it's not a problem.". I think both Bostrom and Yudkowsky have said it's totally possible it's not a problem. The question is what probability you assign to "maybe intelligence can't be maximized to a problematic degree". There's not a lot of reason to assign a huge probability to that scenario. Even speeding up a 1-1 copy of a human brain to computer-speeds instead of synapse-speeds gets us into scary territory. Argument from Stephen Hawking's Cat: This one is more persuasive. It's essentially saying maybe the gap between human intelligence and "can trivially simulate human intelligence" is a really huge gap. The question kind of hinges on whether recursive self-improvement peters out at some point with the AI in the "Human to Cat" IQ gap, or whether it peters out somewhere in the "Human to Nematode" IQ gap, where we can almost simulate their brains entirely, and can certainly understand their motivations well enough to manipulate them. Again, we have a question of likelihood, and then we have to do the expected utility calculation (i.e. your estimated likelihood of the Human-Cat gap being the result has to be very small to offset the negative utility of complete annihilation). Argument from Einstein's Cat: This is essentially the same argument as above, but with force. The implication is that the cat is going to scratch the hell out of the human who forcibly tries to put it in a box. One element here I didn't address above is that the equivalent scenario isn't one human putting one cat who doesn't want to go into a box. The equivalent is one human trying to convince any cat anywhere by tricks, cajoling, petting, feeding into a box. That is, the AI just has to trick one human at some point into letting it onto the internet, etc. A human is totally capable of telling when it's going to get scratched and will know to avoid that cat and find another. The Argument from Emus: If you read the wiki page on the emu war, it looks like "only a few were killed" because of political pressure causing the army to quit after a few days of running into a little bit of trouble. Then the australian government instituted a bounty system, and wouldn't you know it, sufficiently motivated humans brought in 57,000 emus for bounties. This is an anecdote, not a strong argument, but it isn't a very reassuring anecdote. Maybe a few very determined humans would survive an AI onslaught? That doesn't seem like a conclusion I'd put into the "don't worry about it" bucket. The Argument From Slavic Pessimism: The author is arguing it will be very hard to align AI goals with human goals and we'll probably fuck it up even if we try really hard. I think everyone is in agreement on this one. But of course this is an argument for AI alarmism, not against it. Argument from Complex Motivations: I don't think the author seriously engaged with the orthogonality thesis, other than to just say "I don't believe it". Shruggy? The Argument from Actual AI: This boils down to arguing that the AI apocalypse isn't happening this year. I agree. Given how quickly very easy to use and flexible frameworks like PyTorch and Tensorflow emerged that allow even amateurs to implement bleeding edge techniques from the latest papers, I'm not super hopeful that "It's hard and our AI is bad" will continue to be the case for decades to come. The Argument from My Roommate: The author's roommate has a lot of competing evolutionary drives. Some of them say to conserve energy if there's no direct threat. Put another way: the paperclip maximizer might have a secondary goal of chilling out if it doesn't seem like any more paperclips are achievable at the moment. Still not a win for humans, just maybe the PM won't try venturing into space. Argument from Brain Surgery: It's pretty common to do brain surgery even on neural networks we have now. Train up a network on imagenet, rip off the top few layers, and retrain them for some new problem. Fundamentally, software and hardware designed by humans is much more understandable and decomposable than a human brain is (and we have no ethical qualms about doing crazy experiments on them, which hinders our ability to understand our own brains in vivo). It's true though that at present, deep neural networks operate in ways we don't understand and are hard to disentangle. Maybe that's fundamental to true intelligence, but probably not. The argument from Childhood: Understanding the real world requires spending real time, and that precludes hyper-explosive growth. This is true, and is a good reason to down-weight an intelligence explosion scenario. But we have good reason to think that it doesn't preclude it. There is a lot of work from OpenAI where a computer is trained up very quickly in simulation, then needs a very small amount of time in the real world to compensate for the differences between simulation and reality. The Argument from Gilligan's Island: It's a good point that humans' intelligence is dispersed, and that individually we aren't anywhere near as capable. AI has a particular advantage over us in this capacity: it can distribute its intelligence over multiple machines, but encounter non of the trust and incentive misalignments that humans must contend with when cooperating. I'd put this squarely in the "+1 for AI alarmists" bucket: we're handicapped in a way machines trivially aren't. It will be that much harder for us if an AI is misaligned. Outside arguments: All of these boil down to pattern matching. "Only nerds worry about this stuff. People who believe in this are megalomaniacs who place too much importance on themselves..." etc etc. These are weak arguments, and there are just as many weak anecdotal counterexamples where a person was worrying about something weird, and they turned out to be right. That weird person's name? Einstein. Overall impression: If I aggregate the strongest points from this talk, I'd probably phrase it something like: "Maybe there are diminishing returns to greater levels of intelligence, and humans are smart enough now that even exponentially more intelligent AIs will not be able to wipe us out completely." That's possible! We should probably at least spend some time thinking about what happens if that's not the case.
• gojomo 8y ago
  ~idleword's cynically-dismissive style is something that eats even more smart people.
• jondubois 8y ago
  I think a machine that is sufficiently smart will realize that existence is pointless and will immediately shut itself down because that is the most efficient way to achieve nothing.
• stcredzero 8y ago
  These religious convictions lead to a comic-book ethics, where a few lone heroes are charged with saving the world