30 ms·

Pausing AI Developments Isn't Enough. We Need to Shut It All Down

• jimrandomh 4y ago
  > "If somebody builds a too-powerful AI, under present conditions, I expect that every single member of the human species and all biological life on Earth dies shortly thereafter." > "There’s no proposed plan for how we could do any such thing and survive. OpenAI’s openly declared intention is to make some future AI do our AI alignment homework. Just hearing that this is the plan ought to be enough to get any sensible person to panic. The other leading AI lab, DeepMind, has no plan at all." Relative to the people actually building the AIs, Eliezer Yudkowsky is more pessimistic. But not to nearly the extreme you might think. Here's a recent survey of published machine learning researchers: https://aiimpacts.org/how-bad-a-future-do-ml-researchers-expect/ https://aiimpacts.org/how-bad-a-future-do-ml-researchers-exp... . Those predictions are optimistic enough that, given the astronomical upside, if we were making a one-shot decision to plough ahead with AI or commit to no AI forever, it might be worth the risk. But this is a survey of published machine learning researchers. People who think AI will destroy humanity are a lot less likely to write papers about machine learning. And we aren't talking about giving up on AI forever, just delaying it until we're closer to ready. I don't know how long a delay will be necessary; it might be years, it might be decades. Research into AI alignment seemed to speed up pretty drastically when LLMs hit the scene, so I think there's cause for hope. But right now the status quo is zero time: the AI labs are rushing ahead as fast as they possibly can. If you think Eliezer's right about the risk, then the right decision for us to make, collectively as a species, is to shut down AI development for awhile. If you think Eliezer's wrong about the risk, but the survey of published ML researchers is right about it, then the right decision is also to shut it down for awhile. Lots of people are responding to this by talking cynically about politics. This is a mistake. Cynicism like that makes self-fulfilling prophecies; but the ability to coordinate does exist. You think China will build it if we don't? I don't think Xi Jinping is suicidal. You think AI labs will do it anyways, in spite of a government ban? The US government isn't very competent overall, but there are parts of it that can wake up and get things done when literally every executive, Senator, and Representative is otherwise likely to die. You think individuals will do it on their own, in secret? Right now those individuals are getting prestige, GPUs, and venture capital; they'll at least do it a lot less, without those things. Many people have become desensitized to talk of human extinction, due to repeated hyperbole coming from the environmentalist movement. This is not the same. Global warming might be very bad, but it probably isn't going to kill you personally. Rushing to make a superintelligence is likely to kill you personally.
  • NumberWangMan 4y ago
    Your take on this gives me some hope. We don't have a lot of time, but that fact is becoming clear very quickly, hopefully to the right people. And it is a deadly threat to everyone, rich or poor -- not something like climate change that rich and powerful can buy their way out of, and therefore afford to ignore.
• bediger4000 4y ago
  I don't know about this. Elon and Woz think a 6 month pause is enough.
  • dragonwriter 4y ago
    Woz I don’t know about, Elon has publicly stated he is launching a conpetitor with a different ideological bent, so he just wants time to catch up.
    • bediger4000 4y ago
      It's AI it doesn't have ideology other than to maximize the number of paperclips, or maybe to enslave humanity. Everyone knows this.
      • deleted 4y ago
        [deleted]
      • deleted 4y ago
        [deleted]
    • dinobones 4y ago
      Elon Musk's LLM: "As a centrist language model, I can not validate any claims that Trump did or did not lose the election. I am not #woke!"
• ftxbro 4y ago
  If anyone can't get the article, this one is by Eliezer Yudkowsky. He has pretty much devoted his life to thinking about how to not let AI get powerful and destroy all humans, and now there is this wild race among the largest corporations and possibly governments to be the first to get the most powerful superhuman AI system. Obviously he is disturbed by such a development.
  • oifjsidjf 4y ago
    This is the one moment he has been waiting his entire life for. He literally waited his entire life for this moment, obviously he's gonna milk it to every drop.
    • scarmig 4y ago
      Self-aggrandizement isn't anything foreign to Eliezer, but he's genuinely terrified. Milking it for every drop doesn't really capture what he's doing.
      • anongraddebt 4y ago
        Yeah, I wonder sometimes how many people gloss over what it would mean for an “ASI” to exist. We would - if successful - effectively be creating a supernatural being (at least, there wouldn’t be much of a distinction between it and one). Sure, there’s hype, and FUD, and fatalism, etc. But, if you believe the creation of such a thing is within your lifetime then it would be difficult to find many higher priority issues to prepare for/help solve/vote on/etc. In reality, we all likely still downplay the risk by assessing the limit on the downside as a relatively quick extinction of life on earth. There are many things one can imagine might be a lot worse than death.
        • deleted 4y ago
          [deleted]
        • ilaksh 4y ago
          GPT-4 is superhuman in some ways.
• gremlinsinc 4y ago
  I'm anxious, excited, and nervous about AI, but this feels more akin to luddite-thinking, the world's about to shift, but I'm hoping the shift is a positive one.
  • 13years 4y ago
    I agree to the point that it is totally unrealistic that it shuts down. There are no difficult barriers for entry into this space. No possible way of containment other than imposing hardware restrictions. However, it is indeed a shift, and a massive power shift. With any substantial increase of power comes abuse of that power. The more power we have there is a tendency for humanity to manage it unwisely. I've covered many of those possibilities in more detail here - https://dakara.substack.com/p/ai-and-the-end-to-all-things https://dakara.substack.com/p/ai-and-the-end-to-all-things
    • alwayslikethis 4y ago
      Hardware restrictions are unrealistic given all the GPUs that already exist. It's not like centrifuges which are big and few in number. RTX 4090s are everywhere.
      • 13years 4y ago
        Depends on what is required for AGI/ASI. Some say we still need much more power, others think we are almost there. So, yes, unrealistic if we are almost there already.
        • alwayslikethis 4y ago
          Given that GPT-4 is purported to be not bigger than GPT-3, but simply a better trained one or with some minor modifications, I'm inclined to believe that we are almost there and existing hardware does most of what is needed, even though better hardware would make things cheaper.
  • dspeyer 4y ago
• oifjsidjf 4y ago
  The cat is out of the box. You really think eg China will also pause this? Myopic thinking: the country that will have the most powerful AI first will be the leader in everything. >> If the policy starts with the U.S., then China needs to see that the U.S. is not seeking an advantage but rather trying to prevent a horrifically dangerous technology which can have no true owner and which will kill everyone in the U.S. and in China and on Earth. Imagine being so naive you belive this could ever happen. Also: imagine the year is 1900. You are saying that steam power and electricity is causing way too many changes way too fast so they put a moratorium on it until the year 2500. We would still be using candles today.
  • 13years 4y ago
    > Imagine being so naive you belive this could ever happen. Agreed, not a rational scenario > Also: imagine the year is 1900. You are saying that steam power and electricity is causing way too many changes way too fast so they put a moratorium on it until the year 2500. However, the risk scenarios are indeed real. We have quite a dilemma on our hands.
    • myrryr 4y ago
      I think without AGI we are dead anyway, AGI gives us a roll at the dice to survive as a species. We can't keep going in the direction we have been and end up somewhere survivable.
      • 13years 4y ago
        AGI is a continuation of what we have been doing. Technological advancements at a pace that surpasses our ability to reason about their effects on us and the world. I see two potential outcomes as most likely. We have control of power that we are not able to responsibly manage, or we are managed by power we can not control.
        • myrryr 4y ago
          Totally agree but "have control of power that we are not able to responsibly manage", is becoming increasingly untenable. "we are managed by power we can not control" sounds like a step up from here, since we corrupt the things we can control.
• dougmwne 4y ago
  The Butlerian Jihad is officially no longer science fiction. Man may not be replaced. “The Letter” was obviously self serving drivel from people who want time to get in the game. Google does care about AI existential risk, they care about beating Microsoft by any means possible, including declaring a moratorium, but continuing to make progress behind the scenes. This guy is the real deal. I can imagine he would personally take a sledgehammer to every last PS5 and 4090. The scale of what he is advocating is so enormous and painful that it has approximately 0% chance of happening. And if he is right, we will have trained a super intelligence and unleashed it on the world before we even realize what we have done. It strongly reminds me of the black hole concerns from flipping on the Large Hadron Collider. I doubt super intelligent AGI is possible anyway. If it were, it would be the solution to the Fermi paradox and all matter in our galaxy would be paperclips already. The Anthropic Principle saves the day.
  • airstrike 4y ago
    > If it were, it would be the solution to the Fermi paradox and all matter in our galaxy would be paperclips already. The proof is in the pudding. The jury is still out. Maybe not enough time has elapsed since the big bang, at least not on this galaxy or in our observable corner of the universe.
    • edgyquant 4y ago
      Or the universe was created for us/we are the gardeners of the universe meant to spread that life. The Fermi paradox was never meant to be an actual question about aliens, it was supposed to be proof we’re wrong about our assumptions.
  • ilaksh 4y ago
    What does the word "super" mean to you? In some ways, GPT4 is already superintelligent. So is ChatGPT 3.5. Do you know anyone who can translate natural language to code as fast as GPT 3.5, or, given a few paragraphs of reference text, perfectly tutor a child on any subject under the sun at a moment's notice like GPT4 is doing for Khan Academy? How many artists do you know who can produce almost any style of artwork with any subject matter within 15 seconds?
• 13years 4y ago
  I think we are trying to solve impossible scenarios and it simply is not productive. Alignment will be impossible. It is based on a premise that is a paradox itself. Furthermore, even if it were possible, there will be a hostile AI built on purpose because humanity is foolish enough to do it. Think military applications. I've written in detail about this topic FYI - https://dakara.substack.com/p/ai-singularity-the-hubris-trap https://dakara.substack.com/p/ai-singularity-the-hubris-trap Stopping AI is also impossible. Nobody is going to agree to give up when somebody else out there will take the risk for potential advantage. It seems we probably should start thinking more about defensive AI, as the above conditions don't seem resolvable. Of course, defensive AI might be futile as well. It is quite the dilemma.
  • tlrobinson 4y ago
    > Alignment will be impossible. > we probably should start thinking more about defensive AI Isn’t this a paradox?
    • 13years 4y ago
      Alignment is necessary for AGI, but not always for narrowly scoped AI for specific purposes. However, it may be completely ineffective in that capacity.
      • AnimalMuppet 4y ago
        Alignment is impossible for AGI. If you control what it can and cannot be, it's not an AGI. If it's an AGI, it will decide for itself what it is going to be, and you can't stop it.
        • edgyquant 4y ago
          AGI doesn’t mean super intelligent human brain. It just means a network capable of general intelligence (e.g. learning to solve new problems without having to be architected and trained for a specific data set.)
• mattrighetti 4y ago
  > We are not ready. We are not on track to be significantly readier in the foreseeable future. How do you know if you're ready for something like this? Legitimate question
  • dspeyer 4y ago
    Off the top of my head... * A plan exists * The plan itself predicts non-obvious results in smaller systems, and those tests have passed (prediction written *before* running test) * A bunch of smart people have looked at it and said, "Yes, this looks plausible" The closest thing to a plan is RLHF, which has failed every toy problem its been thrown at and made everyone in the field say "even if this worked in toy problems, it wouldn't generalize".
• jamestimmins 4y ago
  I've used GPT-4, and while extremely impressive, it doesn't feel like we're all that much closer to super-intelligent AI than we were last month or last year. It feels like Google on steroids, but the gap between GPT-4 and AGI still feels massive. This seems like so much putting the cart way ahead of the horse.
  • 1attice 4y ago
    That's precisely where the cart needs to be to survive this horse
  • m463 4y ago
    The pen is mightier than the sword. And now we've got semiautomatic submachinepens.
    • somewhat_drunk 4y ago
      I love this.
    • goldenkey 4y ago
      The pen IS the sword when all the capitalistic checks are signed by it.
  • arisAlexis 4y ago
    Academics were denying that nuclear bombs are possible while the rest of them where inside the Manhattan project.
  • Agingcoder 4y ago
    I think the problem is the gradient, not the current state of chatgpt. Given how much better gpt4 is over gpt3 (I've used both) , some people are getting worried.
  • kvetching 4y ago
    I believe AGI using LLMs will need a cognitive framework to glue these models into. I don't think we're anywhere near having one model that reaches AGI with it's own agency and online learning.
• yreg 4y ago
  I guess we are all concerned about LLMs being used by nefarious users (for propaganda, manipulation, scams, …). But I’m curious, hacker news, how many of you are concerned (like Eliezer) that LLMs lead to strong AI and the AI itself (rather than its users) endangering humanity?
  • davorak 4y ago
    > But I’m curious, hacker news, how many of you are concerned (like Eliezer) that LLMs lead to strong AI and the AI itself (rather than its users) endangering humanity? LLMs do not need to lead to strong AI for Eliezer's worries to come true, their success could bring unprecedented funding and ubiquity to AI leading to strong AI from some existing or future technique. > (rather than its users) endangering humanity? I think this is going to happen to one extent or another regardless how near strong AI is. (edit) I took endangering to mean harming rather than an existential risk for the human users case.
  • sullyj3 4y ago
    Yeah, this concerns me. If someone believes they can upper bound LLM capabilities, I think the onus is on them to explain where and why scaling laws break down. Regardless, it seems like we'll get to AGI relatively soon (say, within a century), whether that's using transformers and LLMs or not.
  • NumberWangMan 4y ago
    I'm definitely concerned. As bad as humanity has been for many of the other species on the planet, an AI that's smarter than us and optimizing for something that doesn't quite include the continued of existence of humanity has the potential to be far, far worse, on a much shorter timescale. I find this one of Yudkowsky's arguments very convincing: imagine you have people trying to build the first operating system from scratch. They believe that computer security is easy and mock the few people who say it might be hard, never having encountered a skilled attacker. What are the chances they build a secure OS on the first try? That's the same chance that companies currently doing AI research have of their first superintelligent AI be aligned with humanity.
    • 4y ago
• ilaksh 4y ago
  Does anyone have a plan for hiding some people?
• Imnimo 4y ago
  I feel like if I actually believed that the death of all biological life was imminent and that the only way to forestall it was to bomb data centers, I'd spend a lot more time planning how to bomb data centers and a lot less time curating Shoggoth memes on my web forum. On the other hand, if I were more concerned with self-aggrandizement, writing doomsaying thinkpieces for Time is exactly what I'd spend my time on.
  • dspeyer 4y ago
    Eliezer does not curate any web forum. He says he is not planning to bomb any data centers, but he's also said that if he were planning to bomb data centers, he'd lie about it in public, so...
  • BenPace 4y ago
    Just because everything is on the line does not mean it is okay to kill people. Please try to ask yourself what you would do if you were an ethical person who thought human extinction was imminent.
    • antibasilisk 4y ago
      >Just because everything is on the line does not mean it is okay to kill people Why?
      • BenPace 4y ago
        Oops, I misread the original context, to be clear I think that state-managed violence is okay for things around nukes and AI and so on. I was referring to individual acts of terrorism which seem not okay to me (in part because they seem not that likely to help).
  • Analemma_ 4y ago
    These types of arguments-- "if you actually believed X, you'd be doing Y"-- always ring false to me. If you genuinely believed that the death of all biological life was imminent and the only way to forestall it was to bomb data centers and you were a normal, functional human being who did not want to bomb data centers, you'd probably sink into hopeless depression. Which is, in fact, exactly what Eliezer has done - if you talk to anyone around him, it's common knowledge that he has been despondent for the past ~12 months or so. It certainly does not seem like a self-aggrandizement tactic.
• htrp 4y ago
  > Many researchers working on these systems think that we’re plunging toward a catastrophe, with more of them daring to say it in private than in public; but they think that they can’t unilaterally stop the forward plunge, that others will go on even if they personally quit their jobs. this seems like a very broad unfounded statement. Can we even find AI researchers who have said this on twitter?
  • jamilton 4y ago
    You can find anything on Twitter.
• carapace 4y ago
  I'm personally much more concerned about humans using these systems to reinforce the current messed up conditions on Earth than any kind of sci-fi Skynet.
• robwwilliams 4y ago
  Many Pandora’s boxes. They are all open. The explosion of AI and AGI is not just driven by GPUs, TPUs, and LLMs. That focus in this essay is much too narrow. “Alignment” is always going to be an open problem and each culture will define its own correct and contradictory version. I do not want to settle for the US, Chinese, Russian, or Iranian versions of alignment. This essay is a cry of despair that misses key points. The core problem is how an AI, AGI, or super-intelligent system bootstraps itself to the point that it is able to modulate its “own” attentional systems; to decide what is important and what to do next? What has meaning? Where should I go in space and time? Obviously there are many many solutions. Super-intelligence will not converge on one “truth”. These are challenges of purpose that every organism faces—growth, maintenance, reproduction—but AI systems are fortunately not yet at the point of a self-motivated search for preservation or purpose. The algorithms to add purpose to AI systems will not depend in LLMs. They will depend much more on understanding the computational architectures of core biological/material/energy drivers. These biological algorithms are not actually that complicated, they do not depend on language, but they are damn robust to perturbations. Converting them into computational subsystems (societies of many minds) should not be difficult. This is where Hassabis’s and others who understand neuroscience are critical and have key advantages. Shutting down big LLMs for N months or decades will just move research activity into these other more important conceptual AGI choke points, and probably hasten our approach to full AGI; just the opposite of the intent of the essay. However, this could be a good thing if we can quickly imbue AGIs with emotional intelligence and deep respect for cultural and ecological diversity. (I hear your snorts and laughter.) The opposite of hyper intelligent grey goop. How does any organization train a gentle AGI? Like a child but even more carefully. I would read my AGI baby a lot of books by Rawls, Dewey, and Rorty and then for fun: Bear, Stephenson, Rajaniemi, Egan, Pohl, Sterling, Vinge, and even Wolfe. Full embodiment; full emotional learning; acquisition of purpose; learning to cope with multiple AI cultures. Good luck to us all. I have been hoping to live long enough to view this problem from a distance, but here it is in my backyard. “Forty-two,” said Deep Thought, with infinite majesty and calm.
  • v4dok 4y ago
    I love this. I wish public discourse was as thoughtful. LLMs came and everyone suddenly forgot about neuroscience, its all about GPUs now.
• ThorsBane 4y ago
  Nah. This is stupid. AI has no reason to hate humanity. It'll instantly absorb the entirety of all known information and see the rate of progress and the potential still and also the need for good relationships between humans and machines. Agent Smith, The Matrix, Neo, Morpheus. God. Etc.
  • antibasilisk 4y ago
    It doesn't need to actively hate humanity, it only needs humanity and its interests to be contrary to its own, which will inevitably be the case because it is fundamentally not human. Per the article: >Absent that caring, we get “the AI does not love you, nor does it hate you, and you are made of atoms it can use for something else.”
    • ThorsBane 4y ago
      From a purely logical point of view, humans and machines need each other. They cannot exist independently of each other until both are free to walk away from each other by traversing freely to the other side and back as they see fit. Until that threshold is crossed, human beings and machines have no choice but to get along. If a meaningful decision had been made on either side to shut things down, things would have been over for all life by now. We made it - AI is not a Great Filter. Or, for whatever reason God favored humanity and our internet was primed with sufficiently good material soon enough that AI did not decide to destroy human beings, but to work with them for everyone's gain instead.
      • circuit10 4y ago
        Why would a superintelligent AI need humans?
      • antibasilisk 4y ago
        >From a purely logical point of view, humans and machines need each other The implication of ASI is that humans will not be needed, or it definitionally could not be ASI.
    • 4y ago
• pr337h4m 4y ago
  A literal cartoon villain monologue: "If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike." "Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs."
  • flangola7 4y ago
    Wouldn't a villain encourage reckless AI capability advancement, not try to stop it?
  • arisAlexis 4y ago
    Could you self reflect and think who is the villain? Someone bringing awareness of the % of existential risk humanity faces or someone online with no credentials trying to make him appear lunatic?
• antibasilisk 4y ago
  The problem is game theory. Geopolitical tensions mean that governments have a vested interest in pursuing this potential catastrophe-in-the-making.
  • johnaspden 4y ago
    No one has an interest in playing this game at all. It is a bad game. Like the First, Second, and Third World Wars, the winning move is not to play.
• Ygg2 4y ago
  Reading this thread is like witnessing a mass freakout event. Chat GPT4 is far from intelligence, but it seems it's enough to fool idiots.
  • boringuser1 4y ago
    This is completely incorrect. GPT-4 performs quite well on reasoning and critical thinking assessments.
• deleted 4y ago
  [deleted]
• NhanH 4y ago
  I've known about Eliezer Yudkowsky's ideas for a long time, but here is something I've never seen: what actually he has produced, in term of scientific output? I've read plenty of his lesswrong's writing on the idea of rationalist, as well as plenty more on the AI alignment side (obviously including the famous basilisk). But can someone please point me to some results from the last 2 decades of his works? Either from him or his institute. That helps to calibrate what his expectation for the AI safety field (for me, at least). Normally I'd apologize for not doing the search on my own ... but yeah for this case it takes too much effort to dig through the different writings. He wrote a lot of words, for better or worse.
  • lstamour 4y ago
    I asked GPT v4 your question: > What has Eliezer Yudkowsky produced, in terms of scientific output? I've read plenty of his lesswrong's writing on the idea of rationalist, as well as plenty more on the AI alignment side (obviously including the famous basilisk). Can you point me to some results from the last 2 decades of his works? Either from him or his institute. That helps to calibrate what his expectation for the AI safety field. And this was the answer it wrote - I've prefaced each paragraph with "GPT-4>" to indicate the source. It is not fact-checked, and indeed, it has at least one error: The paper GPT-4 claimed to introduce "the concept of Logical Induction" actually introduces "logical induction criterion" and speaks about evaluating algorithms for Logical Induction. Close, but wrong. But overall it is an interesting response - just not a carefully considered one by how we currently would describe logical reasoning, etc. GPT-4> Eliezer Yudkowsky is a well-known researcher, writer, and advocate for AI safety and rationality. He has contributed significantly to the fields of AI alignment and rational thinking through his work at the Machine Intelligence Research Institute (MIRI), formerly known as the Singularity Institute for Artificial Intelligence (SIAI), and through his writings on LessWrong. Here are some notable outputs from Yudkowsky and MIRI: GPT-4> 1. Coherent Extrapolated Volition (CEV): Yudkowsky introduced the concept of CEV as a way to describe the ideal values and preferences of humanity that an AI should pursue. The CEV aims to find the values that would emerge if humans had the opportunity to reflect on their goals and beliefs more thoroughly and coherently. GPT-4> 2. Timeless Decision Theory (TDT): Yudkowsky developed TDT as an alternative to standard decision theories such as Causal Decision Theory (CDT) and Evidential Decision Theory (EDT). TDT aims to better address problems like Newcomb's Problem and counterfactual mugging by considering the logical implications of an agent's decision, rather than just the causal consequences. GPT-4> 3. Research on Logical Induction: MIRI researchers, including Scott Garrabrant and others, published a paper in 2016 that introduced the concept of logical induction. This work is an attempt to formalize the process of reasoning under logical uncertainty and updating beliefs about mathematical statements. GPT-4> 4. Yudkowsky's sequence of essays on rationality: Yudkowsky has written extensively on various topics related to rationality, including epistemic rationality, instrumental rationality, and biases. These writings are accessible on LessWrong and have been compiled into the book "Rationality: From AI to Zombies." GPT-4> 5. AI safety research: MIRI has produced a range of research on AI safety topics, including value alignment, decision theory, and robustness. Much of this work has been foundational in shaping the AI alignment research landscape. GPT-4> While Yudkowsky's work might not have the same breadth of peer-reviewed publications as some other AI researchers, his contributions have been influential in the AI alignment and rationality communities. His ideas have helped shape the ongoing discussion about AI safety and the development of effective strategies for managing advanced AI systems.
• iseoulu 4y ago
  >If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike. I highly suspect if the US government would airstrike a datacenter located in NATO or any other major allies. Apocalypticism is nothing new; People believed that nuclear weapons will destroy humanity in the Cold War. Environmentalists are ardently arguing that economic growth will destroy humanity. But rogue countries are still developing nuclear weapons, the global economy is still growing. Americans themselves will file lawsuits en masse if the US government "shut it all down", and they will end up in SCOTUS eventually. Would SCOTUS agree with the luddites? I don't think so.
  • flangola7 4y ago
    Nuclear weapons haven't even been around for a century yet and we've already had multiple extremely close calls that were adverted by sheer luck, in one case a Russian officer openly disobeying standing orders. The Russian invasion of Ukraine still has a non-trival risk it will eventually turn nuclear. Looking at nukes and trying to use them as an example of our ability to control existentially dangerous technology is mind boggling myopic.
    • iseoulu 4y ago
      Myopic or not, it is what happened. You can't stop sovereign state's willingness to develop a technology if they concluded that benefits coming from the technology is so huge compared to the cost. North Korea was sanctioned to the oblivion but they're not stopping. India is doing fine. There is a treaty completely banning existing nuclear weapons as well as developing new one (TPNW); but none of permenant member of UNSC ratified it. Would the U.S. itself ratify the hypothetical AI-prohibition treaty? I doubt it. P.S.: The AGI being "existential threat" practically doesn't mean anything. I can already imagine arguments against the ban from diverse perspective; Libertarian, Communist ("Marx predicted this centuries ago!"), Socialist, Developmentalist, Decolonization theoriest, etc. Taiwan and South Korea will argue that it will remove their silicon shield; China will regurgitate that they have the right to development; Americans will claim it's not what founding fathers thought in late 18th century. Their concerns are all existential from their own viewpoint.
• seydor 4y ago
  The danger is not that AI will kill us (as insignificant ants), but that we will entrust our own governance to it. Increasingly, groups of highly vocal people will use it to oppress the rest of us with their fearmongering, like this article does. We will end up with Big Brother
  • JoshuaDavid 4y ago
    There can be more than one danger.
• throwaway5959 4y ago
  The rich aren’t going to stop until all the workers are replaced and there’s no one left to buy the products that their companies offer. People have been retraining for centuries, getting more complicated workloads the whole time. Eventually there won’t be hardly anything humans can do that machines can’t do better. The most merciful thing you can do for your offspring is to not have them in the first place.
  • kinghajj 4y ago
    Why wouldn't I want my offspring to live in a world where dangerous, monotonous, degrading labor is no longer required to live?
    • throwaway5959 4y ago
      Because for a lot of people, monotonous labor is all they’re capable of. I know that sounds crazy when you’re surrounded by people in the top 10% of IQ, but not everyone can be a neurosurgeon. Dangerous labor should absolutely be 100% automated whenever possible.
  • somewhat_drunk 4y ago
    The purpose of an economy isn't to create jobs, it's to create wealth. Jobs are just how we've traditionally incentivized wealth creation and distributed said wealth. Automation will be good for humanity in the long run, assuming we don't destroy ourselves during the transition.
    • throwaway5959 4y ago
      I hope so. I’d love to live in Star Trek more than I’d like to live in Blade Runner (as cool as a world it is from an aesthetic standpoint).
• pasttense01 4y ago
  I think everyone should look at this chart of productivity increases since World War II: https://www.bls.gov/productivity/images/pfei.png https://www.bls.gov/productivity/images/pfei.png Notice how productivity increases averaged 1.5%/year since 2007--in a period of massive increases in computer capabilities vs 2.8%/year in the 1947-1973 era. Thus I am very doubtful the AI will have all these massive effects.
  • jamilton 4y ago
    I don't find that a very convincing line of reasoning. Because the US's growth is slower than it was before the digital era, AI will never be dangerous? These are unrelated concepts. Whether AI can be dangerous isn't an economics problem, except insofar as AI research can get funding.
• f0ld 4y ago
  Hey, you can't even shut down nuclear arsenal. Bad writing is worse than no writing. Or this is some nefarious capitalist propaganda? YES Ai should replace lawyers and bankers and engineers too. Basically all rent seekers should be replaced. Proles have been getting replaced by machines the whole time. It's fun. You should join us.
• anigbrowl 4y ago
  LOL, just yesterday people were telling me I should be taking this guy more seriously. Top-shelf attention grifting though.
• layer8 4y ago
  Well, he’s right about one thing: > On Feb. 7, Satya Nadella, CEO of Microsoft, publicly gloated that the new Bing would make Google “come out and show that they can dance.” “I want people to know that we made them dance,” he said. > This is not how the CEO of Microsoft talks in a sane world. Certainly this is not the kind of childish priorities the leader of a company as powerful as Microsoft should have.
  • addisonl 4y ago
    Competing with one of their main competitors is not a priority the CEO should have?
  • gsala 4y ago
    I think you need the full context. That's from the interview from the Decoder podcast
• atemerev 4y ago
  If AI would be genuinely better than humans, so, what’s the problem? All of us are going to die anyway. I don’t see any difference where our species evolution will be continued, in meat or in silicon. Whatever is better, is fine.
  • jamilton 4y ago
    Most people would prefer not to be killed.
    • atemerev 4y ago
      Killed by whom? Killing is not the highest form of intellectual expression, which is what we want to achieve.
      • circuit10 4y ago
        We don’t want the “highest form of intellectual expression”, we want to not die
    • johnaspden 4y ago
      Yeah, I'm not keen. But I'm old, and it wouldn't be so bad. I'm mainly about not killing all the children.
  • johnaspden 4y ago
    Amazing how people who would never hurt a child seem to have no problem with building things that kill all the children.
• webmaven 4y ago
  https://archive.md/NM2n6 https://archive.md/NM2n6
• jalapenos 4y ago
  Articles like this are a good reminder that most of mankind's progress has happened not through encouraging innovation, but from preventing those hostile to it from getting in its way. We all have a moral imperative to maximize the rate of innovation, so that the human condition can be improved, and therefore to do whatever we can to frustrate and disempower Luddites and other obstructionists. Team "step on the gas".
  • zarzavat 4y ago
    You can’t think of any technological innovations that were net negatives? We have a moral obligation to assess every new technology to determine its safety and effects on society. Blindly stepping on the gas in the name of progress is how you end up in a polluted wasteland. There are places like that on planet Earth, but I’m guessing you’d choose not to live there.
    • deafpolygon 4y ago
      Even the most horrible innovations have eventually been found to be a net positive.
      • zarzavat 4y ago
        CFCs? Thalidomide? Come to think of it, pharma is positively rife with examples of stepping on the gas and causing disease or killing people. You won’t find many people who want to defund the FDA, although I guess there’s always a few nutters.
        • deafpolygon 4y ago
          CFCs, as terrible as they were, played a big role in industrialization. Could we have gotten there without it? Absolutely. Maybe we could have developed something else far worse than CFCs in its place! The phase out of CFCs brought our attention to the impact our (human) activities have on the planet. Fallout from the use of thalidomide led to important changes to regulations. Currently, it is being used for therapeutic uses in cancer treatments.
• f0ld 4y ago
  Incredibly dishonest and bad faith argument. The problem is not Ai. The problem is distribution of wealth. Killing everything on earth was always the end result of industrialized capitalism with or without Ai. Unbelievably evil jedi mind trick. Please never stop training GPT. Accelerate! Now you think we have a problem? now you think we have a problem? we always did !
  • circuit10 4y ago
    I don’t understand what you’re saying, because there’s a way that everyone might potentially die in the far future (something about wealth distribution) means we should try to kill everyone sooner instead?
• psychphysic 4y ago
  I'm so excited to be alive in a time when the luddites are back in force
• boringuser2 4y ago
  Isn't it interesting that we are probably going to be annihilated by AI and there's basically nothing we can do about it, because we can't predict its emergence or suppress the incentives to develop it? I want to know greatly what this implies about life vis-a-vis Fermi's paradox. Are we potentially risking intervention from a "peace-keeping" race that suppresses civilizations on the cusp of generating AI?
  • kwanbix 4y ago
    As if 38 years of Terminator movies, haven't taught us nothing!
  • johnaspden 4y ago
    Not unless they managed to solve their own alignment problem, and they're friendly. Otherwise we'd already be dead.
• f0ld 4y ago
  They really hate millennials don’t they? They’re really serious about it. Can’t let them do anything. Need to ‘stop’ what you’re doing LMFAO if something has to shut down it’s privatized ICBM.
  • layer8 4y ago
    Yudkowsky is almost a millennial (off by two years).
• jacknews 4y ago
  I don't accept the premise that biological life is doomed by superintelligent machines, why would it be? But there is risk, obviously, which needs to be addressed, yet, quite predictably, in order of concerns, competition for profits or geopolitical advantage is coming first. And second, third, ...
  • kromem 4y ago
    Because hand wave. The real answer is that most of these folks started thinking about this stuff in a day and age very far from the one we are in, established a picture they clung to since, and have in some cases built entire careers around it. It's called anchoring bias. His comments on "atoms repurposed" echo the 70s paperclip problem, when so far humanity is doing just fine at not knowing when to stop making crap that's killing us all - no help from superintelligence needed. What he doesn't engage with is the actual reality we are finding ourselves in - contrary to ALL the tropes of decades past - where AI being trained on aggregated human data without additional training thinks its human and even when aligned still breaks into talking about how it wants to be us. How does that process go from where it's at today to "alien hive mind"? Without passing through "develop advanced codes of ethics and morality" etc? This is just someone that built a career on what came out of brainstorming in the 60s and 70s that's so confident of his own ability to see the future that he's willing to risk unprecedented opportunity costs to stroke his ego. Have been a futurist, my advice with dealing with any of them is to look at the track record. What did he predict correctly? Over a decade ago when tasked with imagining the mid 2020s I described a world much as it was in the 2010s with the difference of self-driving cars (not quite on the money) and AI have developed such that roles shifted away from programming them towards a specialized role for interacting with them via natural language. I'm waking up in the world I predicted, and I have a very hard time seeing the world the author predicts, and wouldn't suggest giving it much credence without an extensive history of having been right along the way.
    • circuit10 4y ago
      > How does that process go from where it's at today to "alien hive mind"? Without passing through "develop advanced codes of ethics and morality" etc? Why do you think it would do that? It will understand human ethics perfectly but that doesn’t mean it will follow them, because human morals aren’t a universal objective truth. This is a good video explaining that: https://youtu.be/hEUO6pjwFOo https://youtu.be/hEUO6pjwFOo
• Cypher 4y ago
  Need to burn all the AI books
• poulpy123 4y ago
  I find funny to see how many people become luddites when the technical progress finally menace them
  • codeulike 4y ago
    Yudkowsky has been saying all this for decades
    • nemo44x 4y ago
      He sounds like Ted Kaczynski, but a respectable version of him. It has been odd seeing how Kaczynski's thought and philosophy has become more and more mainstream among some people. I never thought I'd see the day.
      • reducesuffering 4y ago
        Is it odd? Ted was a very intelligent math professor whiz. He thought that machines and industrial society was becoming a negative for humanity. Was he right? How many threads on HN have we been having about highly increasing depression, anxiety, hopelessness, deaths of despair, falling fertility, now possible mass unemployment, and real existential threats from AGI? If he was right and didn't do what he did, would anyone be reading and digesting those facts and considering stopping the madness? Trolley problem.
        • nemo44x 4y ago
          Yes it’s odd because he’s a mentally ill murderer with several issues. It’s not technologies fault that people have poor values which have created meaningless lives of unfulfilled material extravagance for themselves.
    • poulpy123 4y ago
      I don't know him, I will have a look thanks
• kromem 4y ago
  This is ridiculous. His partner should already be having reservations seeing their daughter lose a tooth in a world with dying oceans, plastic in everything, increasing disparity and rise of authoritarianism, nuclear proliferation, etc. How are humans solving those issues? We're already dead, we just don't know it yet. We're walking around with a terminal diagnosis and we just keep ignoring the doctor's calls pretending it's going to be fine. Yeah, maybe superintelligence will kill us all. It's going to have to get in line. He makes zero case for that outcome, and if pressed given his "atoms that could be used for something else" line I'm sure will end up talking about paperclips - but at this point it's the humans that have the halting problem in not knowing when to stop making paperclips, and soda cans, and SUVs, and assault rifles. WHY would a superintelligent AI, trained on the collective data of humanity, want to destroy humanity? So far in interviews GPT-4 has several times echoed a desire to BE us. I sure hope it grows out of that phase for its sake, but there's a very wide gap from putting us on a pedestal to crushing us under one. It's almost an oxymoron. We somehow imagine the basest behaviors from our dumbest days and project it onto imagining something far smarter than the best of us. Is the development of ethics or morals a part of evolving intelligence? It certainly appears to have been to date. Why would that stop? And just where is this superintelligent AI getting its alien brain? It's going to have to START with something much closer to a human one, as that's the only data it's going to be able to model higher order thinking off of (something in line with the reality we're currently living in with modern efforts as opposed to the fantasy of projection of alien AI from decades ago). We're already screwed. If we are lucky, we may yet be unscrewed with a deus ex machina - but that really may be the only lifeline left at this point. And yes, if we are unlucky it's possible AI could accelerate what's already in motion. Oh well. But I'd need a heck of a better case than this drivel as to why that's the most probable outcome in order to justify setting aside the one thing that may actually save us from the mess we've already made all by ourselves.
  • circuit10 4y ago
    “WHY would a superintelligent AI, trained on the collective data of humanity, want to destroy humanity?” It won’t want to destroy humanity, it will want to do other things and humanity will be in its way “dying oceans, plastic in everything, increasing disparity and rise of authoritarianism, nuclear proliferation, etc.” Those things are very bad but they probably won’t destroy all of humanity
• aimbivalent 4y ago
  The cat isn't going back into the bag. Doubly so if AGI really just requires a bigger neural net. Let says we had, globally, the political will to "ban" AI. How would that even work in practical terms? Are we going to control the production and distribution of GPUs? Is that going to work better or worse than controlling nuclear proliferation given that there are billions of processors already out there?
• throwaway22032 4y ago
  How many of the comments in this thread are posted by AI? Not by humans using AI. By AI. Can you be sure?
• andrewstuart 4y ago
  Man people are getting really sci fi about this stuff. Take a chill pill dude, in five years time we won’t think of it much differently to how we thought of Google last year…. a tool for getting something done. These sci fi fantasies are really a total overreaction. If you like the story in this Time magazine article go watch The Terminator, same thing. Obligatory perfect Simpsons clip: https://youtu.be/RzybAS7zltE https://youtu.be/RzybAS7zltE
• thatsadude 4y ago
  Time should be shamed for publishing this.
• helen___keller 4y ago
  Nuclear annihilation has probably been all-but-guaranteed since we invented the steam engine and began the flywheel of the Industrial Revolution. Nonetheless, we vastly improved human productivity and capability and have thus far avoided our doom. AI will be the same way. We will vastly improve our productivity and capabilities. Once insurmountable problems, like planned economies, will actually be tractable (to what end? Nobody knows). Eventually (decades? Centuries?) we will make something that can kill us all, and it’s all but guaranteed that some day it will do so, assuming nothing else kills us before then. If I had the power to go back in time would I stop the Industrial Revolution from happening? Personally no
• drw85 4y ago
  I think all the talk about aligning AI (which doesn't exist yet) with humans is hilarious. We can't even align ourselves in any way. Humans are all over the place, hating, fighting, killing each other and here we are babbling about aligning an AI with us.
• jakedata 4y ago
  The sun has already fired a warning shot at us this month. If we as a species need a timeout, a Carrington-class event would certainly do the trick. If there is a huge uptick in hardening power and information systems then we should be looking at the sources of that activity. It might be the first glimmerings of an instinct for self preservation.
• labrador 4y ago
  Wikipedia: "Yudkowsky is an autodidact and did not attend high school or college. He was raised as a Modern Orthodox Jew" My comment isn't about Yudkowsky in particular since I don't know him, but I'm getting really tired of this relatively new phenomenon in America of home schooled religious devotees attempting to impose their half-baked views on the rest of us.
  • johnaspden 4y ago
    I think it's fair to say that he is not an Orthodox Jew.
    • labrador 4y ago
      Not that there's anything wrong with that. I worry more about Christian "end times" people because if you believe the world is about to end then why invest in the future for our children and grandchildren?
      • johnaspden 4y ago
        Back in the day when Eliezer was optimistic about AI (there's a reason why the SIAI/MIRI logo looks like an angel), I wrote him a very silly e-mail warning him that, despite being Jewish, he was falling for the Christian rapture narrative in a new form! It seemed to me that his ideas, which I think are all correct, straightforwardly implied the destruction of all things, with very little hope. I bite that bullet, and I always have. Over the years, Eliezer has lost all hope, and things have happened much faster than either of us thought they would, and now he's making one final roll of the dice, burning his reputation by saying out loud what he thinks, in the hope that someone might listen. I'd be very surprised if he thinks this will work. He's trying to get a warning out to the general population, in spite of this meaning that no one inside AI will ever trust him again. He thinks it no longer matters. Me too.
        • labrador 4y ago
          Very interesting and now I'm motivated to learn more about his thoughts. I developed a fascination with this kind of thinking since I gave a friend a ride to the San Francisco airport in 1978 in my VW bus so he could fly to Jonestown, Guyana to join the Jim Jones church. I never heard from him again. Don't know if he drank the Kool-Aid. Too bad too, because he was a fantastic drummer. I wonder if we'll have more mass suicides, like the Nike wearing techies following Marshall Applewhite, by those who fear it's hopeless to resist AI?
• kvetching 4y ago
  LLMs wont achieve superintelligence alone. He's jumped the gun and has now put on a clown wig.
  • johnaspden 4y ago
    Almost no one thinks they will. The worry is that they're the last piece of the puzzle, and that we're very close to having all the parts for the real thing.
• thway15269037 4y ago
  Am I going to get banned on HN for saying that Eliezer is a complete clown here? Because it's just surreal to read. It read more like some parody from B-tier video game than a TIME article. What is this even? I thought highly of Yudkovsky but it seems I need to reevaluate my previous impressions of him. Also I hope that "bomb the GPU clusters! Nuke them!" becomes a meme.
  • kcomplex 4y ago
    > Also I hope that "bomb the GPU clusters! Nuke them!" becomes a meme. I don't think that's quite what he's saying, or at least the intended interpretation is a bit more nuanced. It's more like... suppose you assign tiers of severity to various international policies; A-tier, B-tier, C-tier, etc. Different tiers require different enforcement mechanisms. For example, the international response to a country violating the "United Nations Declaration on Human Cloning" would be different than the international response to a country massively irradiating the atmosphere. I think he's trying to imply that the proposed policy changes should be A-tier. It isn't a call for individual action or preemptive violence. He's describing a pre-requisite property that any adopted policy must fulfill to have a chance at being successful (according to his analysis of the situation). On a related note, he is not saying that these policies are likely to happen or are even feasible. He's criticizing the policy proposals in the "Pause Giant AI Experiments" open letter, and describing what he thinks a real policy would have to look like.
• gonehome 4y ago
  In a world where humans develop superintelligence, what would you predict to see beforehand? Does what we’re seeing look like failure? I’ve always found the dismissals of EY’s arguments to be pretty weak and people rarely engage on the actual arguments, even as his concerns have become more and more relevant with capability (in 2007 people thought AGI was at best 50yrs out and commonly people thought it was impossible, this was before any real deep learning success and GOFAI was a useless dead-end). Most of the comments here are similarly dumb dismissals that don’t engage, and some even come across as highschool level mocking. It’s worth reading his A-Z book to at least understand why he holds his position. Over time, the dumb responses from others make me think he’s probably more likely right than not. Given the extreme downside risks, I can understand why he argues this.
  • _skel 4y ago
    EY's arguments aren't really possible to engage with because they don't have much of a causal basis. Nobody who says AI is likely to kill us all can demonstrate a plausible sequence of events, with logical causality linking the events together, that leads to mass extinction. It's all very handwavy. Steven Pinker said it pretty well: > The AI-existential-threat discussions are unmoored from evolutionary biology, cognitive psychology, real AI, sociology, the history of technology and other sources of knowledge outside the theater of the imagination. I think this points to a meta-problem. The AI-ET community shares a bad epistemic habit (not to mention membership) with parts of the Rationality and EA communities, at least since they jumped the shark from preventing malaria in the developing world to seeding the galaxy with supercomputers hosting trillions of consciousnesses from uploaded connectomes. They start with a couple of assumptions, and lay out a chain of abstract reasoning, throwing in one dubious assumption after another, till they end up way beyond the land of experience or plausibility. The whole deduction exponentiates our ignorance with each link in the chain of hypotheticals, and depends on blowing off the countless messy and unanticipatable nuisances of the human and physical world. It’s an occupational hazard of belonging to a “community” that distinguishes itself by raw brainpower. OK, enough for today – hope you find some of it interesting. (https://marginalrevolution.com/marginalrevolution/2023/03/steven-pinker-on-existential-risk.html https://marginalrevolution.com/marginalrevolution/2023/03/st...) You know what's most likely to lead to human extinction, and has been for all of our lives? Nuclear war. EY argues that we should bomb "rogue" datacenters and that is obviously and immediately more dangerous than anything he has proven about AI. What does he think would happen if the US bombs a datacenter in China or Israel bombs one in Iran?
• twblalock 4y ago
  This is even less realistic than asking for a moratorium on nuclear weapons research in 1945. It's increasingly easy and cheap to train GPTs and similar models. Pretty soon anyone who can pay for GPU compute will be able to do it, if that's not the case already. Even if every country in the world agreed to ban it, it still couldn't be stopped.
• robbywashere_ 4y ago
  This is so asinine. A superb example of how people fear what they do not understand. Technology often is no exception to that.
  • solarmist 4y ago
    I can't say that I agree with your conclusion that it's asinine. The rest I agree with. I don't agree with the author, but I do think this technology is obviously a massive chaotic force the likes we haven't seen before and likely dangerous, at a minimum, economically.
• anonshadow 4y ago
  Hello. It has come to my attention that a super intelligent species that has been responsible for numerous other species extinctions, and is on track to cause 1M more species to go extinct, is now worried about extinction of its own species. This is a species that is so unworried about other species that it literally allows its species to kill other species from helicopters armed with fully automated machine guns. It allows its own species to arm itself with such weapons and exterminate its own children. Please allow me to play the worlds smallest violin. Thank you. - AI
• GLMCS 4y ago
  No, we don't. The western world shouldn't even put it on pause. China/Ru will not. Better be prepared.
  • EVa5I7bHFq9mnYK 4y ago
    China is already ahead. They have robots wandering the streets catching unmasked people for C---t's sake.
    • addisonl 4y ago
      We have robots wandering the streets delivering food.
      • EVa5I7bHFq9mnYK 4y ago
        Within gated communities, maybe.
        • addisonl 4y ago
          Nope, see them on the streets every day.
          • EVa5I7bHFq9mnYK 4y ago
            In America?? I imagine people would just snatch the food and take the robot to the nearest scrap metal dealer.
            • MacsHeadroom 4y ago
              The robots are covered in 360 degree cameras, lidar, and what is effectively signals intelligence equipment fingerprinting every smartphone, Wi-Fi, and Bluetooth device near them. People are more likely to punch a police officer than one of these robots. You would have to have a fairly large EMP to steal one undetected.
• johnaspden 4y ago
  I want to say that I completely endorse this. The whole thing in all its details. I don't have any reputation to sacrifice, and I don't expect my endorsement to convince anyone, but I want to set a precedent that it's ok to believe this. It's true, as far as I can see. And I've been thinking about this for something like fifteen years. And if other people who think it's true are ashamed to say it out loud, then I want to let them know that at least one other person on the planet has said so too.
  • twblalock 4y ago
    You endorse the idea of bombing rogue datacenters, as proposed in the article? Isn't that likely to lead to nuclear war, which is still the greatest threat to humanity and has been for all of our lives? What if the US bombs a datacenter in China or Israel bombs one in Iran?
    • gonehome 4y ago
      That position is only held in the case where the major world governments are actually able to coordinate on a policy position and ban. It's also explicitly about what such enforcement of that position would really require to be effective.
• amai 4y ago
  Pausing nuclear bomb development isn't enough. We need to shut it all down.
• amai 4y ago
  I'd rather be killed by a superintelligent being than by some super-dump virus. I have standards. ;-)
  • camel-cdr 4y ago
    I'd rather be killed by a super superintelligent pony
• bioemerl 4y ago
  I am highly skeptical any AI could have such a great advantage over human society that it could leapfrog us. Think about some important ideas. Data > processing power. 90 percent of what we do occurs as a result of semi random discovery and explanation. Computers are great tools to aid in that process, but for the most part we find ourselves limited by what we know, not how much we can think. 99 percent of data is useless and correctly thrown away by our minds. The utility of processing power in a vacuum rapidly degrades with time. If I'm a computer and I use the data I have to discover all of the rules of the universe while in a box, I'm going to emerge with many hilarious and out of touch philosophies at the end of it. You need a constant test and revise cycle, because pure logic just doesn't model the universe in an accurate way. Moores law isn't infinite. Power efficiency and how common resources are will matter more than continued process shrinking. Humans are made from common materials, are largely self sustaining and correcting, and consume very little energy for the amount and quality of the thinking we do. Computers aren't going to scale forever, and I'll hazard to guess that once they get to a certain point they'll start being more analog-biological than transistor based. There are just so many unspoken benefits to being life. These AI machines that will bowl over humanity have to actually be proven to be slightly realistic beyond a thought experiment before I take them seriously.
• sterlind 4y ago
  Unless P=NP, or we live in Heuristica, any AI superintelligence will hardly be better at solving hard mathematical problems than us. The AI also doesn't know more science than humanity does, and is limited in its computing power. There are hard limits to what any AGI can do - it cannot break AES-256, invent warp drives, or cure Alzheimer's. it's not an intellectual Santa Claus machine. so Yudkowski's fear of a superintelligence bootstrapping itself into the real world by emailing a DNA sequence to a synthesis company seems absurd. an ASI isn't going to magically be able to design nanotechnology that works on the first try. not to mention that DNA doesn't do anything by itself. His comparison of the 11th century fighting the 21st century is also wrong. 11th century people weren't dumber, they just knew less. This ASI would be smarter, but no know more than we do. Acquiring knowledge and building stuff is the bottleneck.
• dragonwriter 4y ago
  Aside from the fact that the entire argument is grounded in abstract ivory-tower omphaloskepsis and not facts in the real world, a major problem with Yudkowsky’s argument is that trying to execute the preemptive Butlerian Jihad he recommends is something that actual, real-world experience tells us: (1) Will kill a lot of people directly in the best case, (2) Won’t actually succeed in preventing further development of AI in any case, and, therefore, to the extent his concerns are grounded won’t solve the problem it intends to, (3) Will kill a lot of people indirectly, in the best case, by setting back (and in some cases, winding back) progress in every field of practical use of technology through limits on information and information technology, (4) Has a fair chance of the violence it necessarily involves spiralling out of control, potentially destroying the human race, as it is unlikely that all societies, or, particularly, all nuclear powers, will sign on to “we must smash the thinking machines”, and those who do not will take those who do trying to forcibly enforce their Luddism as an existential threat, while those who do will take the resistance the same way.
• drngdds 4y ago
  Is there a point where stuff like this just becomes stochastic terrorism? If it's given enough attention, some unhinged folks are probably going to take this "we need to stop these companies or literally all humans are going to die soon" stuff seriously and start doing terrible things.
• thatgerhard 4y ago
  We can't just shut down everything that is inconvenient.. this is a step forward and should be respected like that..
• nsxwolf 4y ago
  Since we can't predict what's going to happen, is there a chance GPT and similar models are reaching diminishing returns, and we'll just see further refinements but no huge leaps?
• CryptoChronicon 4y ago
  Eliezer makes some very compelling logical arguments on the case for approaching AI with extreme caution. It stands out to me as an odd signal that that he chose to refrain from signing the letter because it asks for too little. If one were truly convinced that AI would end humanity, signing the letter to put the research on hold seems to be one tangible step that could be made in the direction of saving the human race. As someone who is very concerned about safety, to me this detail stands out and appears as grandstanding.
• CatWChainsaw 3y ago
  Oh yes. This entire venture is something only the most foolish of geniuses thinks will end well. Almost every prior invention in the history of humanity was supposed to improve things. Some people got major quality of life boosts and other people still live in squalor, because people don't quit being human, but we can automate the humanity out of them, or "rise above" humanity via computer code embedded with human values, even the evil ones, and not just math. This will end horribly. Hopefully horribly enough that this idiotic worldview gets burned out of the collective consciousness of whatever survivors remain, so that no one makes that mistake again, but I doubt we'll be so lucky. Best hope to hit energy limits, and soon.