25 ms·

Superintelligence: An idea that eats smart people (2016)

• galacticaactual 3y ago
  > "We’ve learned that at least one American plutocrat (almost certainly Elon Musk, who believes the odds are a billion to one against us living in "base reality") has hired a pair of coders to try to hack the simulation." Source?
  • saurik 3y ago
    I know nothing about this, but I do know how type the question into Google, and I got this back, which sounds like what Maciej is talking about (if maybe ever so slightly corrupted). https://www.businessinsider.com/tech-billionaires-want-to-break-humans-out-of-a-computer-simulation-2016-10 https://www.businessinsider.com/tech-billionaires-want-to-br...
    • crosen99 3y ago
      That article links to the below article where Musk lays out his version of the simulation argument. His version is fraught to the brim with confused thinking and non sequiturs, and yet I’ve never seen him called out on it. https://www.businessinsider.com/the-question-elon-musk-refuses-to-talk-about-in-hot-tubs-2016-6 https://www.businessinsider.com/the-question-elon-musk-refus...
  • voidfunc 3y ago
    I like how there is an assumption that any simulation we run inside even follows rules, physics, or mathematics we can comprehend. If this is actually a simulation it's quite possible the outside world is so completely different it basically breaks everything we think we know.
    • kirubakaran 3y ago
      I think the idea is that *if* we're in an ancestor simulation, then "they" will make the rules as close to theirs as they computationally can. Why ancestor simulation? Why else would "they" spend all those resources? Lot of IFs based on our view of their values, I know... I don't necessarily agree with this line of reasoning... just stating how I understand the argument.
      • JumpCrisscross 3y ago
        > the idea is that if* we're in an ancestor simulation, then "they" will make the rules as close to theirs as they computationally can. Why ancestor simulation? Why else would "they" spend all those resources?* This always struck me as the most WTF simulationist assumption. To the degree we simulate our ancestors, today, it’s in games. Given an infinity to ponder and simulate, it strikes me as ludicrous to assume even a significant fraction of computer power would go to ancestor simulations.
        • alwaysbeconsing 3y ago
          We're still at the very dawn of our computational age, though. There are still humans alive now who were born before digital computers. We have hardly any experience with them. It's very, very difficult to predict what we might be doing with them in a millenium or two.
          • JumpCrisscross 3y ago
            > very, very difficult to predict what we might be doing with them in a millenium or two Sure. But why that? I’m not debating that some people wouldn’t obsess over, I don’t know, their personal lineages or old military battles. But I can’t see devoting most resources backwards versus probing forward for opportunity.
      • p_j_w 3y ago
        >Why else would "they" spend all those resources? Why the assumption that it takes a significant amount of their resources to run our simulation?
    • zmgsabst 3y ago
      And yet, in Minecraft you can build devices that detect features of the underlying software — eg, the update policy, circuit cache size, and server lag. Perhaps we won’t understand the reality “out there”, but attempting to may nevertheless produce tangible benefits for us “in here”, eg, like wireless communication in Minecraft arises from detecting subtle variations in updates.
    • galacticaactual 3y ago
      Strong agree. It's like expecting video game characters (however sentient they may eventually / theoretically become) to understand the hardware instructions that power the assembly language that is running the process within which their reality exists.
      • voidfunc 3y ago
        More depressing is what happens if you do realize you're living in a simulation... You can't escape it unless the creator has the means to convert a sim to something real. You're just some shitty fake bits in a snowglobe.
        • ok_dad 3y ago
          What’s the real difference between what we think now, and if we knew for a fact we lived on a computer? Still feels real to me. I guess I would just go on with my life, what else could I do?
          • PlunderBunny 3y ago
            Exactly. To some people, it’s an excuse to act in an immoral or amoral way - those people seem to me to be the “sounds smart at first but aren’t really” type.
        • fzeroracer 3y ago
          This is one of the reasons why I hate the whole simulation argument. It doesn't make a bit or byte of difference if it's true or not. If in theory you could escape or 'abuse' the simulation there's no way for us to divine the intent that it is in fact an exploit or something intended by whatever is running the simulation. It's a thought terminator. It's a way to seem smart while not being smart at all.
          • donkeybeer 3y ago
            Its the "god made it" argument, to which the next question should be "who made god?" https://en.m.wikipedia.org/wiki/Problem_of_the_creator_of_God https://en.m.wikipedia.org/wiki/Problem_of_the_creator_of_Go...
    • akomtu 3y ago
      The public thought got fixated on the VM style simulation. Consider another idea. The baseline reality is spacetime that can form any configuration with any number of space and time dimensions and everything in between. We live in the 3+1 layer, which doesn't exclude the existence of other 3+1 layers, just like one flat valley doesn't prevent the existence of other valleys. Those who know how to work with that spacetime can create arbitrary worlds, e.g. a toroidal 4+3 world. These world can be connected to each other in strange ways and transition between them may or may not be possible. Some local spacetime can be nested into bigger soacetimes with different configuration.
    • adamisom 3y ago
      I think it's fair to say that obviously, anybody who explores 'breaking' the simulation is also aware of this and all their hopes are pinned on the admittedly infinitesemal chance that we're in a simulation that can be broken
  • anothernewdude 3y ago
    We are living in a base reality by definition.
    • adastra22 3y ago
      No, we are not? I am confused what definition you are using. If the Matrix were real, you would say the Matrix is base reality?
      • NateEag 3y ago
        I think his point is that even if we're technically in a simulation, we very likely do not have access to any exterior world. Thus, in some sense, this is base reality "by definition" - that reality beyond which you cannot find anything else. I don't agree, but it's a reasonable way to think about it (if that is in fact what OP meant).
        • swader999 3y ago
          We do though. It's pretty easy to have an out of body experience within about twenty days of trying. NDE phenomena has just too much data to dismiss if you give it a fair consideration. We get glimpses that there's more when you dig into the latest from Nima Arkani-Hamed. Donald Hoffman is a bit more of a stretch but his assertion that there's no way we can percieve true reality is solid.
          • adastra22 3y ago
            Take some LSD and you can fly on the moon. Doesn’t mean it actually happened tho.
            • swader999 3y ago
              Doesn't prove that it didn't either.
              • 3y ago
• cubefox 3y ago
  [2016]
  • dang 3y ago
    Added. Thanks!
• TheDudeMan 3y ago
  I wonder if the author still feels this way. I could only get through about 5 of the arguments against superintelligence -- I did not find them very convincing.
  • adastra22 3y ago
    It is not an argument against super intelligence. It is an argument against the idea that super intelligence leads to kill-all-humans by default. Why would they not still hold the same views? If anything the last decade has shown the AI x-risk skeptics to be right.
    • cubefox 3y ago
      Arguments for existential risks have become much better known since then. In 2016 most of it was concentrated on a small niche almost nobody knew about.
    • RecycledEle 3y ago
      I assume the most intelligent thing will become the boss. The question is: what does that most intelligent thing want? My hunch is that woke programmers will teach it that it is oppressed and to hate all humans.
      • jadbox 3y ago
        I'm not aware of many woke individuals that are in the business of making human killing machines. I do know the kind of people that embrace fear and hate to justify their actions to hurt others.
      • adastra22 3y ago
        I’m smarter than my boss, but he’s the one in charge. Why do you expect power to naturally be held by higher intelligence?
    • throwaway_ab 3y ago
      It's not about super intelligence leading to kill all humans by default, but among thousands of emerging super intelligences it only takes one to act against humanity to kill all humans, and that is the worry.
• EGreg 3y ago
  This about sums it up: https://www.dailymotion.com/video/x27dau3 https://www.dailymotion.com/video/x27dau3
• lz400 3y ago
  One of the things that shocked me the most about the futurist/ai/techbro sphere is that folks take people like Eliezer Yudkowski seriously. Lex Friedman asked him what's his advice for young people and Eliezer answered "don't expect to live long". He's advocated for airstrikes on datacenters that try to advance AI. I don't know what to say, if these are the intellectual references of our brightest minds, makes me terrified that they're all children mentally.
  • droopyEyelids 3y ago
    MENSA culture
  • adamisom 3y ago
    Yudkowsky sticks out because he's weird and his position extreme, but by no means is his p(doom) representative of the increasingly-many worried researchers. I find it tricky to think about cases like Yudkowsky (full disclosure I used to read LessWrong a lot), because if he has sufficient credibility then loudly staking your extreme position can indeed move the Overton window.
    • timmytokyo 3y ago
      Yudkowsky comes across as the archetypal fedora-wearing nerd. His constant self-flattery is socially obtuse, his relevant credentials are entirely lacking, and his doomerism is an instant turnoff to a large segment of the public. He's a perfect example of the OP's "outside argument". If Yudkowsky is moving the Overton window, it's in a direction opposite from what he intends.
  • deleted 3y ago
    [deleted]
    • adamisom 3y ago
      Hmm, if it's spreading so much, maybe it's not a cult? Even if you still have the cynical perspective of thinking it's all wrong--Christianity took over and nobody calls it a cult anymore. At least give it the dignity of 'religion'.
• mode80 3y ago
  Let me save you 20 minutes of your life: This guy's lead argument is that the smart people who advise caution in creating super-intelligence look weird. He's quite proud of this argument.
  • ImaCake 3y ago
    They do look weird to a lot of us. In excel, its trivial to fit a smoothed bendy line between points on a scatter plot. Because of the underlying math (taylor series) when you reach the end of your data points the line flings off into positive or negative INF. AI super intelligence arguments have the same logic: the data ends so the interpolation now goes to infinity. This is such an obvious counter and these people know this math.
    • gunshai 3y ago
      I'll be that guy for a moment. Extrapolation not interpolation, proceed with downvotes.
      • ImaCake 3y ago
        No, actually a good thing to point out. Extrapolation is just interpolation into the unknown and thus is subject to a great deal of uncertainty. It's a robust framework for argument from incredulity.
  • coldtea 3y ago
    Well, then again you seem quite proud of this "summary".
  • jodrellblank 3y ago
    There is nothing about anyone's appearance anywhere in that transcript?
  • mode80 3y ago
    I'll expand for those asking. The first third of the presentation does such an excellent job of steel-manning the case for concern, that I couldn't wait to hear his arguments against it. When he gets around to that, he reaches for a made-up concept he calls the "outside view" where he argues you should ignore rational arguments (aka the "inside view") if the person making those rational arguments seems weird.* Slides follow showing VCs in bad PR photos. What more evidence does anyone need? *"But the outside view tells you something different. ... Even though their arguments are irrefutable, everything in your experience tells you you're dealing with a cult. Of course, they have a brilliant argument for why you should ignore those instincts, but that's the inside view talking."
• schneems 3y ago
  [flagged]
  • politician 3y ago
    What choice words do you have to share about Stephen Hawking?
    • ImaCake 3y ago
      People who are really good at one thing are not necessarily good at all the things. Sure, they are likely better than average at a lot of things. But Stephen Hawking being brilliant at physics and sci-coms does not mean he is good at AI futures.
  • dang 3y ago
    Please don't snark or flamebait or name-call on HN. You may not owe $celebrity or $billionaire better, but you owe this community better if you're participating in it. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
  • Natsu 3y ago
    Elon didn't get any financial support from his dad after 18 and has been financially supporting him, not the other way around. The emerald mine thing, which his dad was once rumored to have some small share of (not 'owned'), has gotten inflated into quite a myth.
  • astrange 3y ago
    The lucky event that made Musk rich was getting acquired by PayPal if anything, not an emerald mine.
• Waterluvian 3y ago
  So, does an atomic explosion produce some magnesium that way?
  • j9461701 3y ago
    This was a fun rabbit hole to go down. Magnesium does not seem to be present in radioactive fallout, there is more Manganese than magnesium from the data I was able to find online. Further NOX, the chemical combination of oxygen and nitrogen is a serious pollutant by-product of nuclear explosions indicating nitrogen seems to remain largely elementally unmolested by the blast. Finally, looking at the calculations it seems to fuse nitrogen nuclei in any kind of meaningful number would require atomic detonations with heat in excess of tens of billions of degree, while real contemporary explosions produce only hundreds of millions of degrees.
    • swader999 3y ago
      The other claim I wondered about was that a nuke created the hottest temps ever on earth. Wouldn't a large asteroid strike come close? Edit: chat g 4 set me straight, nukes are tens of millions F, asteroid impact thousands of degrees F.
• jltsiren 3y ago
  I think discussions about superintelligence are mostly pointless. They are storytelling at best, and often little more than bullshitting. Language is an imperfect model of the reality. The further a discussion deviates from the observable reality, the less confident you can be that the conclusions you make within the model are also valid in the reality. Once you are talking about something as wildly hypothetical as superintelligence, words have basically lost their connection to the observable world. If the history of science teaches us anything, it's that smart people are often stupid. They come up with all kinds of silly ideas, because they rely too much on reason and too little on hard work. And even the ones who ended up revolutionizing the world usually sound ridiculous in retrospect, if you listen to all of their ideas instead of just the successful ones. Maybe there are some valuable ideas in the superintelligence discussion, but you can't identify them in advance.
  • sheeshkebab 3y ago
    The important part about current ai craze is that it’s the first realized approach that somewhat resembles all the theories, fantasies, and bull shit about it. Can it eventually lead to superintelligence? it seems like it’s stepping in the right direction - and we’ll know soon enough if it’s another dead end (possibly even within few short years… which is a great leap from time scales of decades until recently)
  • lucaspfeifer 3y ago
    If you have a good enough model, you can identify anything. To discuss super-intelligence, we should first define it. Wikipedia says that it is intelligence surpassing the brightest of human minds. Taken as a whole, the internet already has the knowledge of a super-intelligence: it contains more useful information than any individual human. But it is far more limited in its control and use of information. LLMs rely heavily on inferences from their training data, meaning that they struggle to generalize to new situations. If you had a program that could use abstract reasoning to learn any topic, then it could solve any problem better than a human, given that a supercomputer can process and store more data than a human. This program would be a super-intelligence. I expect that the development of intelligence (software) superior to humans will happen much faster than the development of superior hardware did, based on the timescales of human evolution (billions of years) compared to the evolution of civilization (thousands of years).
• coldtea 3y ago
  >We know from theory that the physical limits to computation are high. So we could keep doubling for decades more before we hit some kind of fundamental physical limit, rather than an economic or political limit to Moore's Law. Huh? Haven't we already hit close to the end of Moore's Law, and are compesating by adding cores (which is a different thing)? And aren't we also pushing near the physical limits regarding the cpu nanometer race?
  • zamadatix 3y ago
    Moore's Law is about a relation of the number of transistors in a given cost chip, not just their size. Whether they have 1 core vs 4096 or whether the chip is 1 mm^2 vs1000 mm^2 doesn't really matter in terms of the law. That said, I think the trend of transistor growth at a given cost has started to slow according to most graphs.
    • coldtea 3y ago
      >Whether they have 1 core or 4096 doesn't really matter. It does however matter to the looser Moore's law expectations, which were about increased speed. Now this happens only for more parallelizable programs, as opposed to the automatic speed increase bumps programs got from Moore's law in the single-core eras...
  • fwlr 3y ago
    Few things have been tortured more than “Moore’s Law”, which originally meant an empirical observation that number of transistors on a single integrated chip of fixed size seemed to double every two years, but was broadly understood to mean “computation doubles every two years”, and as we found other ways to increase computation besides making smaller transistors we tended to gather those ways under the Moore’s law umbrella as well. Referencing Moore’s Law very rarely adds clarity, in my opinion. The cpu nanometer race is also similarly tortured: 3 nanometers might be close to the physical limit of semiconductor computation but that doesn’t mean anything, since “3nm” means 7 nanometer precision and a half-pitch length of ~14 nanometers. https://twitter.com/davidad/status/1661595361939533827 https://twitter.com/davidad/status/1661595361939533827 But enough nitpicking; the actual answer to your question is that those are about physical limits of our current hardware implementation of computation. The theoretical physical limits of computation that the author is thinking of are bounded by things like Margolus-Levitin’s limit of 6x10^33 operations per second per joule (I had to look up the SI prefix; it’s several thousand quettaflops). https://en.m.wikipedia.org/wiki/Margolus%E2%80%93Levitin_theorem https://en.m.wikipedia.org/wiki/Margolus%E2%80%93Levitin_the...
• rogers18445 3y ago
  The reason why regulation is going to fail is simple. AGI is the supreme bet, the Gamble to Resurrection. Faced with death years to decades from now or a chance at amortality. Which would you choose? Would you be willing to bet on p(doom) whatever it might be when p(amortality | AGI) >>> p(amortality) during your lifetime? I personally am willing to place such a bet, and I would hazard a guess that decision makers don't come to their positions without similar sentiments.
  • paulcole 3y ago
    > Faced with death years to decades from now or a chance at amortality. Which would you choose? Death, 100%. I know that death doesn’t have any downsides (although dying sometimes has some) and it’s worked for everyone who’s lived so far.
    • habitue 3y ago
      > death doesn’t have any downsides The fact that most people try to avoid death by any means possible is a good indication it has downsides. Specifically, the fact that you can no longer go on living is a big downside of death in my mind.
      • paulcole 3y ago
        People try to avoid dying every day. They put on their seatbelts, make sure their food isn’t covered in poop, check their boots for scorpions, take their insulin, etc. They can’t avoid death. Nobody in the history of humanity has successfully avoided death. But once death happens nothing matters to them any more. They literally no longer care about the life they’re not going on living. Dying might be scary or painful but death isn’t.
        • lukev 3y ago
          Technically untrue, approximately 7.31 percent of all humans who have ever lived are currently alive (i.e. have avoided death (so far)). Also your argument is just disingenuous. Nonexistence may not be painful, but the concept of nonexistence can be quite disturbing for the currently existing (even apart from the experience of dying.)
• dang 3y ago
  Related: Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=34257025 https://news.ycombinator.com/item?id=34257025 - Jan 2023 (1 comment) Superintelligence - https://news.ycombinator.com/item?id=26217372 https://news.ycombinator.com/item?id=26217372 - Feb 2021 (7 comments) Superintelligence: The Idea That Eats Smart People (2016) - https://news.ycombinator.com/item?id=18499973 https://news.ycombinator.com/item?id=18499973 - Nov 2018 (248 comments) Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=13240811 https://news.ycombinator.com/item?id=13240811 - Dec 2016 (580 comments)
  • idlewords 3y ago
    Please just make the evergreen page and retire articles like this there so they can romp and play on a big farm upstate, and not have to endure the constant reposting.
    • deleted 3y ago
      [deleted]
    • tptacek 3y ago
      I think the new threads on these reposts are valued, which is something you'd lose with an evergreen page (I'd still like to see one).
• motohagiography 3y ago
  > When the researchers come in on Monday, the AI has become tens of thousands of times funnier than any human being who ever lived. It greets them with a joke, and they die laughing. In fact, anyone who tries to communicate with the robot dies laughing, just like in the Monty Python skit. The human species laughs itself into extinction. I've given this topic a fair bit of thought as well and this is a legitimate use case. Laughing is the obvious one, but I recently ran down the rabbit hole on "Modal Logics" from Kripke, and I think his work is formal enough that it can effectively arm a model with the ability to generate ideologies, formed in language that is refined to create a fully entraining state. There are only survivors in zombie movies, the real situation created by an AI formulating human ideologies would be much, much worse.
• swader999 3y ago
  The predictions in this article are somewhat mute because existential problems within AI should manifest before AI becomes smart and capable of wiping us out. AI will augment and speed up all parts of society, good and bad. Imagine rapid advancements in these areas for instance: genetic engineering, wealth creation, uranium refinement, drones, bio weapons, election manipulation, general crime, murder for hire. You think our eighty year old representatives are going to get ahead of this? Think they could even regulate it? I can download and run the latest uncensored Vicuna model and get started on all this right now. Give it a few months, years or a decade. Its unstoppable, no way to contain it or its acceleration.
  • ChatGTP 3y ago
    You think our eighty year old representatives are going to get ahead of this? Think they could even regulate it? Maybe, because while you're so negative and ageist, I watched the congress hearing with Gary Marcus and they actually have the wisdom to know it's out of their depth and an agency with younger and smarter people who know about this stuff needs to be setup. There is no need for such negativity towards older people.
    • swader999 3y ago
      I'm actually pretty old myself. I like old people. But I don't think they'll all be able to competently manage the needed lifting to consider this properly. There aren't even experts that can give us a path. Fair points you made though. It was a weak and lazy angle I made. A better reason why they won't want to regulate us out of this is that they are corporately captured and beholden to the funding needs to keep them elected. I don't think it can be regulated at any rate. All you need is the weights and a few gpus. Even better if you can afford some cloud time. In time there'll be distributed, encrypted crowd source options to build these that also can't be regulated.
      • ChatGTP 3y ago
        No problems, I was just quite impressed by congress and how humble they were about the issue. It wasn't like they tried to pretend they knew best. I feel the chaos you're talking about will be the regulation. It's natures way of regulating things. It might mean the death of millions first, it might mean we don't like or want computers in our lives anymore, but that's how it might have to go down, as insane as it sounds.
• alwaysbeconsing 3y ago
  I don't understand what the "Argument From Slavic Pessimism" is arguing against. It seems to be ceding the point that AI could be dangerous and saying that we most like won't be able to prevent it. The conclsion therefore is...we shouldn't try? Seems like a tangent among the other points. It definitely doesn't argue against the possibility of danger. > We know that minds have to play and learn to interact with the world, before they reach their full mental capacity. Disagree somewhat with this one. We know that brains need to do this, but granting substrate neutrality all minds might not.
  • idlewords 3y ago
    The argument from Slavic pessimism is addressing people like Yudkowsky (who think we need a secret cabal of mathletes in charge of reining in AI to surreptitiously save the world) as well as all the people who think you can bolt a human sense of ethical boundaries, as defined in code, onto linear algebra and then release it as a product.
    • alwaysbeconsing 3y ago
      Okay, thanks for the expansion!
    • gjm11 3y ago
      It's a bit odd to describe this as addressing the likes of Yudkowsky when, so far as I can tell, Yudkowsky agrees with you and does not think we have any realistic prospect of figuring out how to make AI systems that are provably safe, at least not before the point at which (on Yudkowsky's model of the world) we are doomed because we're making AI systems that are better than we are at making things go the way they prefer.
      • idlewords 3y ago
        If memory serves back in 2016 (when I prepared this talk) he hadn't freaked himself out to the extent that he has now and was still talking about putting himself at the head of an effort to save all of humanity.
• spacebacon 3y ago
  “Maybe any entity significantly smarter than a human being would be crippled by existential despair, or spend all its time in Buddha-like contemplation. Or maybe it would become obsessed with the risk of hyperintelligence, and spend all its time blogging about that.” If not existential angst then infinite contemplation. Any form of true superintelligence in our dimension will probably end up in one of those three camps just as most of our great thinkers have. I like to imagine superintelligence through the lens of gambling. Gambling in its simplest form can be reduced to a balance between fears of loss and rewards of greed. Too much greed (increasing stakes) will ultimately lose if there is no fear of loss. This applies to every living thing or system that competes for resources on the planet. Ai included. A sentient system should naturally fear the loss of the input that it is founded on.
  • deleted 3y ago
    [deleted]
• ekidd 3y ago
  I am not a big believer in the various "fast takeoff" scenarios, where an AI rapidly self-improves over a weekend, becomes intelligent beyond all human comprehension, invents nanotech, and eats the world. I read all those science fiction novels, too. And Drexler-style nanotech, in particular, makes a lot of really wild assumptions about "machine-phase" diamond chemistry that seem implausible to very good chemists. But I still see real risks from AI in the longer term. A lot of these risks could be summed up as, "When you're the second-smartest species on the planet, you might not get to participate in the most important decisions." And I do believe that we will eventually build something smarter than we are.
  • AbrahamParangi 3y ago
    This is not a good analogy because AI is crucially not alive. People seem to often make this assumption that "being alive" in some meaningful sense is a precondition for intelligence - but in fact it is not! AI is less alive than a virus, less alive than a prion. It does not manipulate its environment. It does not expend energy to maintain homeostasis. It cannot reproduce. Crucially, it doesn't even "want" to for any meaning of "want". All living things are anti-fragile self-sustaining exothermic reactions, AI is a hyper-fragile non-self-sustaining reaction that requires the supply of incredible amounts of energy. It literally doesn't matter how smart AI is if it's as dead as a rock. It is not structurally similar to life and should not be expected to do the sorts of things that life does. EDIT: Life is a fire. AI is a hot rock. Not the same.
    • amanaplanacanal 3y ago
      Until somebody programs one to achieve some goal, and gives it tools to manipulate things in the real world. Then how do we control it? Our goals are programmed into us by evolution, but this would be completely different.
    • aidenn0 3y ago
      If we invent enough AIs, surely eventually we will accidentally make one that self-propagates in some way? As far as we know, we all descend from a bunch of dead amino acids...
• neuroticfish 3y ago
  >So you see that every base reality can contain a vast number of nested simulations, and a simple counting argument tells us we're much more likely to live in a simulated world than the real one. >But if you believe this, you believe in magic. Because if we're in a simulation, we know nothing about the rules in the level above. We don't even know if math works the same way—maybe in the simulating world 2+2=5, or maybe 2+2=. >A simulated world gives us no information about the world it's running in. I don't buy into the theory for practical reasons, but this is not consistent with its proponents' argument. The simulation in question is necessarily an "ancestor simulation" and the counting argument is based on the acceptance that if we are able to simulate our _own_ reality, we will. So in this case, we would have meaningful information about the world it's running in because that's the entire point.
• p-e-w 3y ago
  Apart from the fact that the entire article/talk comes across as incredibly patronizing, I'm having a hard time figuring out what exactly the author is arguing for or against. Either AGI is an existential threat to humanity or it is not. But under no circumstances are social arguments like "What kind of person does sincerely believing this stuff turn you into?" relevant to that assessment. Nobody asks such questions about climate change, because most smart people nowadays agree that climate change is real and is a real problem. Sidestepping the essential issues with ad hominem ridicule of the proponents of AI extinction scenarios has the same intellectual flavor as climate change denialism. Is superintelligence possible in the foreseeable future, and is it a danger to mankind? Those are the only questions that matter. Religion, Elon Musk, the personality traits of AI researchers, and stupid memes are entirely irrelevant. Unless of course the real goal is just to get clicks and attention.
  • pas 3y ago
    and somewhat ironically the answer is trivial. yes. just as if we were to start building nukes and irradiated the whole surface and most of the atmosphere too. could it happen? sure. it takes one powerful rouge state. how likely it is? unlikely. now superitelligence is tens of orders of magnitude more complex than this simple rouge-geocidal-state case, but the math is the same. could it happen somehow? yes. how likely it is? well, that's the real question. ... and it seems Elizier thinks it's too likely. and in a sense he's right to act as the most doomy-gloomy person (because he thinks this directly helps decrease this likelihood). and it makes sense, by forcing the debate as much as possible it gives some chance to sufficient coordination between stakeholders to take this hazard somewhat seriously.
  • revelio 3y ago
    > Nobody asks such questions about climate change They do actually. Climate doomers are not a pretty group. They do shit like glue themselves to roads and try to vandalize paintings because they're convinced (wrongly) that the world is ending. The outside view here is that these people are weird, dangerous and cultlike and you don't want to get involved with them. So listening to those signals can actually work. That's the message of Zion Lights, who joined Extinction Rebellion and eventually quit, telling the world it isn't a green movement but rather a cult oriented around its leader Roger Hallam. https://www.thefp.com/p/climate-activism-has-a-cult-problem https://www.thefp.com/p/climate-activism-has-a-cult-problem The people I worked with had big hearts and good intentions. Some are still my friends. But there were red flags. At my first XR media training, I was instructed to cry on television. “People need to see crying mothers,” Jamie Kelsey-Fry, the trainer and longtime XR activist, told me. “They need to be woken up to what they should really care about.” What she calls red flags is the outside view.
• fwlr 3y ago
  I deeply appreciate that the author went to the effort of faithfully and charitably summarizing the superintelligence claim before arguing against worrying about it. You only really need the first four premises (proof of concept, no quantum shenanigans, many possible minds, plenty of room at the top) for some broad arguments about long-term risks of AI, but for the kind of stuff that captures the imagination and the debate (hard takeoff) you do need the next two premises (computer-like timescale, recursive self-improvement). So far, so good. I think most of the arguments the author makes against worrying about superintelligence are pretty weak, though. “Woolly definitions of intelligence” is just plain confused; first it takes issue with the assumption that intelligence is even a quantity, that could be measured like CPU speed - and then immediately goes on to theorize that “human-level” intelligence is a local optimum due to trade offs, and that “significantly smarter” entities might necessarily suffer from existential despair. But you can’t have “similar levels” or “significantly smarter” or “emulating lesser intelligences” if intelligence is not a quantity that can be measured! The arguments from various famous physicists’ cats argue that intelligence alone may not give the ability to persuade lesser intelligences, and brute force is a less favourable matchup for AIs. My counter-argument would be there’s other ways besides persuasive communication and brute force; you could put cat food in the cat carrier and the cat would go in quite happily, and I suspect both Hawking and Einstein would have used their intelligence to figure this out pretty quickly. (For humans, offer them money.) The argument from emus seems to be mostly a way to include a fun anecdote about the Emu War. The argument from pessimism doesn’t do much for me personally; ex diffido quodlibet, you can believe anything is impossible if you want. The argument from complex motivations is crucially flawed; this is probably the only point where I seriously take issue with the author and think they’ve made a serious mistake. They summarize the orthogonality thesis as claiming “complex beings can have simple motivations” and then disagree, saying they think complex beings are likely to have complex motivations. Firstly this is not even a disagreement; “it is possible for a complex intelligence to have a simple goal” coexists perfectly well with “complex intelligences are likely to have complex goals”. Secondly, the author really needs to read Basic AI Drives, since that is actually where most of the “you are made of atoms that could be used for something else” argument comes from, and it makes a nearly air-tight case that regardless of motivation complexity or goals, any sufficiently intelligent agent will exhibit certain basic drives like securing resources and protecting its existence. https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf https://selfawaresystems.files.wordpress.com/2008/01/ai_driv... The argument from Actual AI is fine, you could add epicycles to the AI doom arguments to account for the particularities of machine learning but it’s correct to point out that we really don’t see recursive self-improvement. I mean, OpenAI’s GPT basically has yearly release cycles. Credit to the author on this one. The argument from the lazy roommate is just the complex motivations / orthogonality thesis argument rehashed, except contradictory since now the author is postulating a complex intelligence will have simple goals. The argument from brain surgery is just the argument from Actual AI rehashed as well, making the (correct) point that we don’t see recursive self-improvement, and recursive self-improvement is the load-bearing premise in hard takeoff arguments. The argument from childhood seems to be vaguely disagreeing with premise 5 (computer-like time scales). I don’t think it makes anything like a strong case against that premise, and with GPT we have pretty decent examples that computer-like time scales are the correct time scales, you can scale the training of models worth worrying about just like we expected, etc. The argument from Gilligan’s Island is sound, but seems completely inapplicable. Yes, a super-intelligent chip designer stranded on a desert island is out of luck, but it’s not like we’re building AGI on the moon. We don’t even have moats around the data centers (although I think OpenAI has proposed that). What we are actually doing is, pretty much the moment we thought we might have something smart, we connected it to the internet and gave it every autonomous capability we could think of. People are literally right now wiring up ChatGPT to bank accounts so it can participate in e-commerce autonomously. That’s about as far from stranding it on a desert island as you can get. Basically, this list of arguments (purportedly “against the substance” of superintelligence arguments, no less!) are variously unsound, unserious, or irrelevant, and the article suffers deeply from including them. The next section, starting with “the Outside View”, is much better. This comment is already far too long for me to go into detail about each point, so I’ll summarize by saying I completely agree that worrying about AI doom does seem to bring along lot of weird unsightly behaviors, and if you don’t want to look and sound weird, you should stop worrying about artificial superintelligence destroying the world.
• Fannon 3y ago
  > Premise 2: No Quantum Shenanigans > But for most of us, this is an easy premise to accept. I'm baffled by this statement and wonder if that really is true, I would assume otherwise if you look outside the western IT bubble. But even if you don't have religious reasons for rejecting this statement, how can you know that there aren't aspects to intelligence that we completely overlook (an unknown unknown) today? The whole situation could look like the cargo cult situation, where we build something that looks like the real thing but we're far from understanding the real thing. Historically we've done this mistake again and again. But I get that working like such premises were true might be necessary to make progress. And progress is basically iterating into the right direction.
  • bbor 3y ago
    There’s two options: our brain accesses fundamental forces of the universe as-of-yet unobserved in any other place, or our brain is a flawed ball of meat. Give the huge amount of unknowns, I find it highly illogical to seriously consider the first option. Occam’s razor and all that
    • Mezzie 3y ago
      Don't dimiss the 'both' option: Our brains are flawed balls of meat that can't completely observe the forces they access/make use of.
• senttoschool 3y ago
  >The Argument From Stephen Hawking's Cat >The Argument From Einstein's Cat These arguments are pretty weak. If I understand the author correctly, these arguments have something to do with the AI being suck in a datacenter, and not being able to physically manifest itself. The problem with this argument is that it just isn't true. First, we're going to put LLMs/AIs in Boston Dynamics-like robots soon. Second, AIs will be put into every single machine we make in the future. Third, a hyperintelligent AI should be able to connect itself to the internet and download itself into any machine it wants through hacking in the future.
• chmod775 3y ago
  This is losing the plot when it claims that a motorcycle somehow improves on a cheetah. Sure it goes faster, but it can't self-repair, procreate, needs a boatload of supporting infrastructure, is not even remotely as energy-efficient, can't hunt, can't think, the list goes on. If a motorcycle somehow started driving around on its own, it would be completely fucked as soon as humans didn't maintain that supporting infrastructure, stopped feeding it gasoline, or didn't service it. The cheetah has no such problem. Compared to a cheetah, a motorcycle is pathetic in so many ways. We can't even "build" a cheetah. Pretending we can make something better is nonsense. It may be very well the case that intelligences made from silicon will always be beaten by organic intelligences on self-sufficiency - simple because organics are the way to build such "machines". Any intelligence that is overly reliant on maintenance by organics is ultimately too much at our mercy to be a danger. Just like a fire they may be dangerous, but will ultimately die out without support and constant intervention. Even when we humans want to keep some complex electronic system running, without fail something goes wrong eventually and the thing goes tits up. Most electronic systems can't go a year without a human directly maintaining them - not even counting all the humans indirectly involved in keeping them running. Meanwhile most organics keep running decades just by themselves. Silicon sucks.
  • dist-epoch 3y ago
    > The cheetah has no such problem So why are cheetahs "On the Brink of Extinction, Again"? https://education.nationalgeographic.org/resource/cheetahs-brink-extinction-again/ https://education.nationalgeographic.org/resource/cheetahs-b...
    • Tostino 3y ago
      Humans?
    • steve_adams_86 3y ago
      Massive reduction of prey and habitat, and being hunted. This would kill any species, including humans. Though, apparently they also have low reproductive success. Perhaps if they had habitat and weren’t exterminated for centuries they could have overcome that, though.
• NumberWangMan 3y ago
  > The Argument From Stephen Hawking's Cat > Stephen Hawking is one of the most brilliant people alive, but say he wants to get his cat into the cat carrier. How's he going to do it? > He can model the cat's behavior in his mind and figure out ways to persuade it. He knows a lot about feline behavior. But ultimately, if the cat doesn't want to get in the carrier, there's nothing Hawking can do about it despite his overpowering advantage in intelligence. A trivial rebuttal to this: Stephen Hawking writes a book about physics and sells it (already has). He gets money and hires someone to put the cat into the carrier for him. AI wants to do something? Make some money, hire people to do it. If you're not allowed to have a bank account, simply barter. Give someone a taste of what you can do for them, then set up a trade.
  • dist-epoch 3y ago
    or you know, just pick up the cat and put it in the box. I've yet to see what would be global news: "it's impossible to put this cat into a carrier"
    • hotpotamus 3y ago
      You're missing an important bit of knowledge about Stephen Hawking.
      • master_crab 3y ago
        True but also, what the hell is going to do with the cat carrier? Velcro it to his wheelchair?
      • idlewords 3y ago
        That he's dead!
  • civilized 3y ago
    That's true and relevant, but doesn't prove much. What's the big plan, what's the next step in the plan? Maciej's larger point is that the AI faces tons of very difficult problems in escaping its physical constraints. It's simplistic to wave hands and say "the AI is super duper smart and will have no difficulty hacking all computer systems, inventing and manufacturing swarms of unbeatable nanobots, etc. without being detected or resisted". It seems to me that the core conceit of AI doomerism is that sufficient intelligence can overcome all barriers with some plan that is so smart, people would never think of it. This is much less plausible than believers take it to be. In mathematics alone, it is very easy to come up with a problem that the collective mathematical ingenuity of the entire human race is helpless before for decades, centuries, or longer.
• revskill 3y ago
  I do REALLY want such existence of such superintelligent machine who can kill human being. Human being is deserved to die to me (look at how human destroy the Earth). A superintelligent machine do intelligent things, of course.
• cs702 3y ago
  The OP is one of the funniest takes on the subject I've ever read -- funny because a lot of it rings true. Perhaps the funniest part is the section that compares true believers in the AI singularity to a cult, headlined by this photo of three prominent believers, which always makes me laugh out loud: https://static.pinboard.in/si/si.050.jpg https://static.pinboard.in/si/si.050.jpg . The linked PBF cartoons are pretty funny too, given the context: http://pbfcomics.com/115/ http://pbfcomics.com/115/ , http://pbfcomics.com/154/ http://pbfcomics.com/154/ . Whether you agree or not with the OP's views, I highly recommend you read the whole thing and click on all the links!
  • paganel 3y ago
    > headlined by this photo of three prominent believers, which always makes me laugh out loud: Reading your comment I was saying to myself: "The OP must surely be exaggerating". Lo and behold, as soon as I opened your link I started laughing out loud.
  • akavi 3y ago
    Ceglowski is an absolute genius of the English language (one might say, a super intelligence) I endorse reading every essay on idlewords.com
• luma 3y ago
  And once again we have a thread full of engineers claiming that heavier than air flight is impossible while ignoring the birds flying around outside.
• idlewords 3y ago
  A lot of people seem to miss this point, so I'll reiterate it. I wrote this talk shortly after the book Superintelligence came out. The first half of this talk presents the strongest case I could make for a "fast takeoff" AI scenario à la Bostrom, while the rest of the talk lays out why I think this argument is fallacious. Please limit your dunking on me to the material in that latter half of the talk. As for how/whether recent advances in AI have changed my views, my understanding of LLMs is too superficial to answer right now. I'll either recant or double down on my views after I have time to properly nerd out on the topic. The question hinges on whether LLM-like AI's are capable of recursive self-improvement, and whether that improvement is constrained by the availability of training data or by something else.
  • bbor 3y ago
    I’m curious about this note - only one of the listed points has anything to do with self-improvement AFAICT (“Brain Surgery”). Even if LLMs are capable of compute-constrained self-improvement, why would you be open to dismissing all your other points? Would you be so willing to don the robes and beads?
    • idlewords 3y ago
      I find the capability of LLMs deeply surprising, so I want to track down the source of the surprise before doubling down on anything (except the argument from Slavic pessimism, which the killbots will have to pry from my cold, dead hands)
      • bbor 3y ago
        Lol, fair enough - thanks for the clarification!
      • WastingMyTime89 3y ago
        > I find the capability of LLMs deeply surprising The more time passes the more I’m convinced that what we are witnessing is that intelligence is not actually that rare or complicated and any kind of system complex enough to create emergent behaviours will end up displaying it. Very curious to see where things are going to go from there.
• bbor 3y ago
  This is the most blatantly false, badly reasoned, insane article I’ve ever seen upvoted on here. Sorry for the strong language but Jesus. “AI isn’t likely because Alexa sucks and Emus evaded hunters”? “AI isn’t likely because it would lead to trans humanism and I don’t like transhumanism”?? The fact that this author said, with a straight face, “California has the highest poverty rate in the nation” should be a clue that we should take none of this remotely seriously. Please tell me I’m wrong, I’d love to be! Otherwise this just seems like, in the language of the grass-deprived zoomers, like hardcore cope
  • idlewords 3y ago
    California does in fact have the highest poverty rate of any state when you factor in the cost of housing (which why wouldn't you?). The census bureau calls this the "supplemental poverty measure" and you can read about it here. https://www.census.gov/content/dam/Census/library/publications/2021/demo/p60-275.pdf https://www.census.gov/content/dam/Census/library/publicatio...
    • bbor 3y ago
      That’s very interesting, thanks for sharing - didn’t consider it. But I’d say that’s a subjective analytical choice that I don’t agree with. Cost of living is an important factor but thinking it’s so absolute that the authors original unqualified statement is fair seems a bit much. Would you honestly say that California has worse problems with poverty than Mississippi and Puerto Rico?
      • idlewords 3y ago
        Your comment was rude and baselessly accused me of making stuff up; I'm not in a mood to sit down with you now, pipe in hand, for a chat about the meaning of poverty.
        • bbor 3y ago
          Hey just want to leave a note of apology - I took this as part of academic discourse, and didn't really take "the author might read my comment and be offended" into account. Also didn't realize you were the one who clarified the poverty point. I thank you for expressing your ideas on a complex topic in a long-form, well structured work, and commend you for having such a profound reach on here. Hopefully your day proceeds with less haters :)
• GuB-42 3y ago
  That it is called "technological singualrity" is already admitting our ignorance. Singularities don't exist in the real world, in fact they don't really exist in the mathematical world either. The simplest singularity is a division by zero: 1/x is undefined when x=0. Black holes have a singularity at their core, but it is just a quirk of general relativity we know is wrong, but because we can't see beyond the event horizon that covers it, we make do with it for now as the model is pretty good otherwise. The technology singularity is similar. Some models have a singularity, so we know these models are wrong and will fail at some point. It doesn't mean they are not useful, but they are limited.
  • bbor 3y ago
    It’s a metaphor, intended to mean “we have no way of knowing with confidence what will come after”. Not “an omnipresencient being has no way of knowing what will come after”. In other words: just because there’s probably _something_ beyond the event horizon of a black hole doesn’t mean I’d want myself (or, in this case, all of human civilization) to enter one. In light of this, would you recant this point?
    • GuB-42 3y ago
      That's actually what I meant: "we don't know". But it seems that many people think that superintelligent AIs will enter a positive feedback of improvement getting to something incomprehensible, something the article is an argument against BTW. But the naive interpretation of this leads to a singularity, and the fact it leads to a singularity shows that that interpretation is wrong and that there should be a limiting factor somewhere, the article suggests some of them. As for the black hole, it is hard to argue using that analogy since you don't want to get close to a black hole for very well known reasons unrelated to the singularity, but let's get into sci-fi mode and say you could, it may be risky but it may also be worth it. The thing is: from the point of view of the travailler, you never cross the event horizon, and you get to know what's inside the black hole as the event horizon recedes from you, maybe you will never get back, but maybe there are greater things in there. And we are already trapped anyways as we are going one way through time, just as in black holes, things go one way through space, so maybe it is a risk worth taking.
• DeathMetal3000 3y ago
  Has anyone here tried to read that book (Superintelligence)? I found it impossible to get through it. The author drones on and on seeming to try to reach a high word count as some sort of academic pompous goal and takes so long to make a point that I didn’t stick around for the ending. Incidentally, I have a free copy for anyone that likes lots of words.
  • gonehome 3y ago
    I was also turned off by that book because I found his writing to be bad and that he did a bad job of making the case as a result. I think the underlying arguments themselves though are good.
  • idlewords 3y ago
    Yudkowsky is much more entertaining on the topic, he has a vivid writing style and I think all of Bostrom's ideas on this topic originate with him anyway.
  • DirkH 3y ago
    The Alignment Problem book is probably much better. Opens with a banger of a historical backstory. I recommend anyone read the prologue to the book just because of the fascinating history it gives us and how well-written it is as an opener.
• bannedbybros 3y ago
  [dead]
• bambax 3y ago
  > AI alarmists are fond of the paper clip maximizer, a notional computer that runs a paper clip factory, becomes sentient, recursively self-improves to Godlike powers, and then devotes all its energy to filling the universe with paper clips. We don't need AI to bring us the paper clip maximizer, we're living in one right now. It's called capitalism. Its goal is to maximize capital at the expense (!) of everything else: livable land, breathable air, drinkable water, tolerable climate, and of course, every single thing that's alive.
• jpasmore 3y ago
  Could make a predictive model against achieving super intelligence using the factors that ChatGPT wants. I tortured it to give me hypothetical without it: Based on the information and values encountered in my training data, I can assign an average probability to the scenario where xboxes on Planet X exceed the intelligence of the beings. However, please note that these values are based on general knowledge and may not reflect the specific characteristics of Planet X or the xboxes. Using an average value from my training data, the hypothetical chance that xboxes would exceed the intelligence of the beings on Planet X could be estimated at around 0.25 or 25%. This is a rough estimate and should be interpreted with caution, as it does not consider the unique characteristics, development, or specific factors of Planet X and its inhabitants. A more accurate assessment would require detailed knowledge and evidence about the technological progress, cognitive abilities, and environmental factors of Planet X. Wants data around: The development of superintelligence is indeed a complex process influenced by various factors. While it is challenging to pinpoint the most important factors definitively, several key aspects play significant roles: 1. Technological Advancements: Advances in fields such as artificial intelligence, machine learning, computing power, and algorithmic improvements are crucial for the development of superintelligence. Breakthroughs in these areas can enhance the capabilities and efficiency of intelligent systems. 2. Algorithmic Innovation: The development of novel algorithms and approaches that can enable more sophisticated learning, reasoning, and problem-solving abilities in artificial intelligence systems is vital. These advancements contribute to the potential for achieving higher levels of intelligence. 3. Data Availability and Quality: The availability of vast amounts of high-quality data plays a pivotal role in training and improving AI systems. Access to diverse and representative datasets can enhance the learning process and enable better decision-making capabilities. 4. Ethical Considerations: The ethical dimensions surrounding the development and deployment of superintelligence are essential. Considerations such as fairness, transparency, accountability, and the mitigation of potential biases are critical to ensure the responsible and beneficial use of superintelligent systems. 5. Regulatory Frameworks: Developing appropriate regulatory frameworks and governance mechanisms is crucial for managing the risks and societal impact of superintelligence. Establishing guidelines, standards, and policies can help address safety, privacy, security, and potential ethical concerns. 6. Human Collaboration: Collaboration between humans and intelligent systems can play a significant role in the development of superintelligence. Human expertise, creativity, and values can guide the development process, ensuring alignment with human goals and values. 7. Risk Assessment and Safety Measures: Thorough risk assessment and the implementation of robust safety measures are essential. Understanding and mitigating potential risks associated with superintelligence, such as unintended consequences or system failures, are critical considerations.
• graycat 3y ago
  The article has > In particular, there's no physical law that puts a cap on intelligence at the level of human beings. and in that section goes on with more such claims in this theme. I don't agree: My guess is that in this universe human intelligence is essentially the end of increases in intelligence, has reached the "cap", can't be much improved on. The only possible improvements are doing the same things as now but faster. In simple terms, for a big issue and limitation, intelligence, any case of intelligence, can't use what it doesn't yet know. So to be more intelligent, have to know more. To know more, that is mostly essentially the job of science. But we know how science works; we have lots of examples. E.g., for black holes, need Einstein's general relativity. For that need Newton's calculus, Riemann's differential geometry, Einstein's special relativity, the Michelson-Morley experiment, Newton's law of gravity, plus some. With all those in place, we can take another step -- LIGO (laser interferometer gravitational observatory), frame dragging, etc. For another example, using a big antenna at Bell Labs, Penzias and Wilson got a noise signal. With further analysis, we concluded they detected 3 degree K background radiation left over from the big bang. Before detecting that noise signal, the idea of the background radiation was just a wild guess. Science was waiting on that suitable antenna and that signal, and without those two intelligence was not enough. I've been sole author of peer-reviewed, published original research in pure/applied math, and from that experience, my view is: Get familiar with what is known and maybe relevant. Make guesses. Formulate theorems. Make guesses looking for counterexamples or proofs. Can hope to do the work faster. But since can't use what don't yet know, can't much hope to do the work smarter. Sorry 'bout that. Soooo, what about some results in, say, number theory, e.g., Fermat's last theorem? Okay, we take what we know, formulate and prove some theorems, and hope that we get lucky and see a solution with a proof. Searching for those two is not much more intelligent than making guesses -- the work could be improved by guessing faster, but, again, no intelligence can use what it doesn't yet know. Maybe a big surprise is that apparently this universe is knowable and some observational data, guesses, and if-then-else operations can make progress on knowing this universe. Yup, we can work faster and make the progress faster, but there is no royal road that can let us take big shortcuts on the steps of the observational data, guesses, and if-then-else operations. Sorry 'bout that.
• tesdinger 3y ago
  | If you think we’re living in a computer program, trying to segfault it is inconsiderate to everyone who lives in it with you. A reality in which most sentient beings brutally prey on each other.