20 ms·

The fact that anyone may be taking Eliezer Yudkowsky seriously on this topic is mind-blowing to me. Simply mind-blowing. To imagine that anybody, including the

• jeffparsons 3y ago
  My estimation of Eliezer Yudkowsky just dropped a notch — not because I don't take the existential threat posed by AI seriously — but because he seems to think that global coordination and control is an even remotely possible strategy.
  • concordDance 3y ago
    He actually doesn't think it's viable, but thinks its the only card left to play. He's pretty sure that human civilization will be extinct this century.
    • jeffparsons 3y ago
      It can't be the only card left to play. There are so many others. They may be weak, but they still have to be better than one that is guaranteed to be absolutely useless. Example of a weak, but at least _better_ option: convince _one_ government (e.g. USA) that it is in their interest to massively fund an effort to (1) develop an AI that is compliant to our wishes and can dominate all other AIs, and (2) actively sabotage competing efforts. Governments have done much the same in the past with, e.g. conventional weapons, nuclear weapons, and cryptography, with varying levels of success. If we're all dead anyway otherwise, then I don't see how that can possibly be a worse card. Edit: or even try to convince _all_ governments to do this so that they come out on top. At least then there's a greater chance that the bleeding edge of this tech will be under the stewardship of a deliberate attempt for a country to dominate the world, rather than some bored kid who happens to stumble upon the recipe for global paperclips.
      • concordDance 3y ago
        That depends on making powerful AIs and being sure they are well aligned being as easy as making powerful AIs which look aligned to cursory inspection. The former seems much harder, though I'd say talk to Robert E. Miles for a more detailed look.
        • jeffparsons 3y ago
          It depends on there being _some chance_ that it can be achieved. The benchmark strategy we are comparing it against is tantamount to licking a wall repeatedly and expecting that to achieve something. So even a strategy that may well fail but has _some_ chance of success is a lot better by default.
          • TeMPOraL 3y ago
            Not with this. Failure to align a powerful AI is a mistake we only get to make once. This is like playing Russian roulette, where the amount of bullets missing from the magazine is the degree to which we can properly align the AI. Right now, we're at "the magazine is full, and we're betting our survival on the gun getting jammed".
            • varjag 3y ago
              Assuming you mean drum here; it's pointless to play russian roulette with a magazine fed weapon (which in a way could be an own metaphor for this predicament).
              • TeMPOraL 3y ago
                I did, and you're right in both of your points.
      • johnthewise 3y ago
        He is not really convinced you can align an AI that is intelligent past some threshold with aims of a single entity. His risk calculations derive primarily from the AI itself, not weaponization of AI by others. Rich Sutton seems to agree with this take and embraces our extinction: https://www.youtube.com/watch?v=NgHFMolXs3U https://www.youtube.com/watch?v=NgHFMolXs3U
    • zimpenfish 3y ago
      > He's pretty sure that human civilization will be extinct this century. If they are, it'll almost certainly[1] be climate change or nuclear war, it won't be AI. [1] leaving some wiggle room for pandemics, asteroids, etc.
      • toth 3y ago
        There is no scenario where climate change leads to human extinction. Worst case scenario is global population goes down by a single digit percentage. Horrible, horrible stuff, but not extinction. And most likely scenarios are not nearly that bad. Nuclear war, if the worst nuclear winter predictions are correct, could be a lot worse but there would still be some survivors. Unaligned ASI though, could actually make us extinct.
        • bryanrasmussen 3y ago
          it said end of civilization? >Horrible, horrible stuff, but not extinction. anyway climate change drop single percentage directly caused, but that kind of thing seems like it would have LOTS of side effects.
        • kylebyte 3y ago
          I'm pretty sure the worst case of your nuclear war scenario is actually the worst case of the climate change one. Chunks of the world suddenly becoming unlivable and resources getting more scarce sounds like a recipe for escalation into war to me.
          • ethanbond 3y ago
            The nuclear war scenario, according to all known information about nuclear war policies, is not “chunks of the world becoming unlivable.” The principals in a nuclear conflict do not appear to even have a method to launch just a few nukes in response to a nuclear attacks: they will launch thousands of warheads at hundreds of cities.
            • 3y ago
    • meowface 3y ago
      He's actually suggested it may happen within 10 years. Which makes it hard to take him seriously in general. (I do personally take his overall concerns seriously, but I don't trust his timeline predictions at all, and many of his other statements.)
      • kylebenzle 3y ago
        Can anyone enumerate what exactly your concerns are cause I don't get it. AI I'd a great spell checker, a REALLY good spell checker, and people are losing their minds over it, I don't get it.
        • ahamm 3y ago
          I used to think that this fear was driven by rational minds until I read Michael Lewis' "Going Infinite" and learned more about effective altruism [EA]. Michael Lewis writes that what is remarkable about these guys [i.e. EAs] is that they're willing to follow their beliefs to their logical conclusion (paraphrasing) without regard to social cost/consequences or just inconvenience of it all. In fact, in my estimation this is just the definition of religious fundamentalism, and gives us a new lens with which to understand EA and the well funded brain children of the movement. Every effective religion needs a doomsday scenario, or some 'second coming' apocalyptic like scenario (not sure why, just seems to be a pattern). I think all this fear mongering around AI is just that - it's based on irrational belief at the end of the day - its humanism rewashed into tech 'rationalism' (which was originally just washed from Christianity et. al.)
          • hm-nah 3y ago
            My comment above, which I think you’re replying to, is not based in fear. It’s based on first-hand experience experimenting with the likes of AutoExpert, AutoGen and ChatDev. The tools underlying these projects are quite close to doing, in a very short amount of time and cost, what it takes human knowledge workers a long time (and hence cost) to do. I think it’s as close as Summer 24’ that we get simplified GenAI grounding. Once hallucinations are grounded and there are some cleaner ways to implement GenAI workflows and pipelines…it won’t be long until you see droves of knowledge workers looking for jobs. Or if not, they’ll be creating the workflows that replace the bulk of their work, hence we’ll get that “I only want to work 20hrs a week” reality.
    • tensor 3y ago
      If humans go extinct it sure as hell won't be AI that does it, but rather good old human ignorance and aggression.
      • sokoloff 3y ago
        I think the probability of humans eventually going extinct is 1. Avoiding it requires almost unimaginable levels of cooperation and technical advances.
        • boringg 3y ago
          Well thats not a difficult position to take. The difficult take is when the timing would be. The sun is eventually going to explode rendering our solar system uninhabitable though that is a long time away from now and we have many other massive risks to humanity in the meantime.
        • babyshake 3y ago
          The end of human civilization as we know it doesn't mean extinction. Why is extinction unavoidable?
          • TeMPOraL 3y ago
            Why does extinction matter? If we lose civilization now, we'll be stuck at pre-industrial levels possibly for geological timescales, as we've mined and burned all the high-density energy sources you can get at without already being a high-tech industrial society.
            • ben_w 3y ago
              I don't think that would necessarily limit us. Windmills and water mills can be made with wood, stone, and the small quantities of iron that pre-industrial societies could produce.
              • TeMPOraL 3y ago
    • hodgesrm 3y ago
      > He's pretty sure that human civilization will be extinct this century. This is certainly possible, but if so extinction seems far more likely to come from nuclear warfare and/or disasters resulting from climate change. For example, it's become apparent that our food supply more brittle than many people. Losing access to fertilizers, running out of water, or disruptions in distribution netorks can lead to mass starvation by cutting the productivity we need to feed everyone.
      • civilitty 3y ago
        Is it really possible? Perhaps it's a failure of imagination but the only things I can think of that would guarantee human extinction are impacts from massive asteroids or nearby gamma ray burst or supernova. There are two billion poor subsistence farmers largely disconnected from market economies who aren't dependent on modern technology so the likelihood of them getting completely wiped out is very remote.
      • cjbprime 3y ago
        (Neither nuclear war nor climate change are considered as this-century extinction risks by the singularity-minded side of the existential risk community, just because there would necessarily be pockets of unaffected people.)
        • hodgesrm 3y ago
          I'm always a bit surprised at how people seem to wish away the destructiveness of nuclear weapons or the likelihood of their use. Another commenter alluded to a "failure of imagination" which seems very applicable here. You would think just watching videos of above-ground tests would cure that but apparently not. [0] [0] https://www.youtube.com/watch?v=gZuyPn0_KRE https://www.youtube.com/watch?v=gZuyPn0_KRE
          • cjbprime 3y ago
            I don't think I was doing either? Assume they are widely used. Does every human die?
    • edgyquant 3y ago
      Why exactly should I care what this person thinks?
      • TremendousJudge 3y ago
        He wrote a really engaging Harry Potter fanfiction.
  • YetAnotherNick 3y ago
    Global coordination using coercion has worked for nukes. It's very hard but not impossible to ban NVidia or any competitor and create spy network in all large companies and if any engineer is found to create AI, mark them as terrorist and give them POW treatment. And monitor chip fab like nuclear enrichment facility and launch war with country found to create chip fab, which I belive is hard to do it in complete secrecy. I believe if top 5 nation agrees for this, it could be acheived. Even with US and China enforcing this, it could likely be acheived.
    • jeffparsons 3y ago
      Sure, I think that kind of "coordination" is a little more credible than what I thought Eliezer was proposing. Maybe I just didn't read carefully enough. One big difference here, however, is that the barrier to entry is lower. You can start doing meaningful AI R&D on existing hardware with _one_ clever person. The same was never true for nuclear weapons, so the pool of even potential players was pretty shallow for a long time, and therefore easier to control.
      • YetAnotherNick 3y ago
        > You can start doing meaningful AI R&D on existing hardware with _one_ clever person. With the current hardware, yes. But then again countries could highly restrict all decently powerful hardware and backdoor them, and you need permission from government to run matrix multiplication above few teraflops. Trying to mess with backdoor is a war crime. A weak form of hardware restriction even happens now that Nvidia couldn't export to many countries.
        • upupupandaway 3y ago
          And yet: https://www.scmp.com/news/china/science/article/3239998/chinese-scientists-create-chip-can-perform-ai-task-3000-times-faster-nvidias-a100-study https://www.scmp.com/news/china/science/article/3239998/chin...
      • johnthewise 3y ago
  • reducesuffering 3y ago
    How are you anti-doomers arguments still this naive years later? No one is saying global coordination is in anyway remotely easy at all. They’re saying that it doesn’t matter what the difficulty is, the other option is fucking catastrophic
    • jeffparsons 3y ago
      I don't think that's a fair representation of my position. I'm not suggesting that it's "hard" — I'm suggesting that it's absolutely futile to attempt it, and at least as bad as doing nothing. I'm also suggesting that there are lots of other options remaining that are not necessarily catastrophic. I'm not particularly optimistic about any of them, but I think I've come to the opposite conclusion to you: I think that wasting effort on global coordination is getting is closer to that catastrophe. Whereas even the other options that involve unthinkably extreme violence might at least have some non-zero chance of working. I guess the irony of this reply is that I'm implying that I think _your_ position is naive. (Nuh-uh, you're the baby!) I suspect our main difference is that we have somehow formed very different beliefs about the likelihood of good/bad outcomes from different strategies. But I want to be very clear on one thing: I am not an "anti-doomer". I think there is a very strong probability that we are completely boned, no matter what mitigation strategies we might attempt.
    • ImPostingOnHN 3y ago
      > They’re saying that it doesn’t matter what the difficulty is, the other option is fucking catastrophic That's precisely what people have been saying about climate change. Some progress been made there, but if we can't solve that problem collectively (and we haven't been able to), we aren't going to be able to solve the "gain central control over AI" problem
  • varelse 3y ago
    [dead]
  • SilasX 3y ago
    But it will work fine for limiting GHG or CFC emissions?
    • hanspeter 3y ago
      It will not. The difference is that reducing emissions just 10% is still better than 0% while stopping 90% from doing AI advancements is not better than 0% (it might actually be worse).
      • SilasX 3y ago
        That's a different argument than the parent, who claimed that global restriction isn't even possible.
        • hanspeter 3y ago
          I don't see that being different. I would also claim that global restriction is not possible and even in the unlikely scenario that we're able to restrict 90% of world, we're still not better off than if we did nothing.
          • SilasX 3y ago
            Still a different argument than one that offhand dismissed the very idea of global restriction at any level. Nothing you’re saying is found anywhere in the original comment. You have better uses of your time than retroactively rewriting others’ words to be less stupid so you can “refute” valid critiques of their original versions.
• concordDance 3y ago
  Sure, it's not likely to work, but Yud doesn't know of an idea with a better chance of working. If you agree with him on the likelihood of superintelligence in the next few decades and the difficulty of value alignment, what course of action would you suggest?
  • RandomLensman 3y ago
    I think the core is not to agree with the existential risk scenario in the first place. Once you believe some malevolent god will appear and doom us all, anything can be justified (and people have done so in the past - no malevolent god appeared then, btw.).
    • concordDance 3y ago
      What's your view on where the X risk idea breaks down? Is it that it's impossible to make something smarter than a human? Or that such a thing wouldn't have goals/plans that require resources us humans want/need? Or that such a thing wouldn't be particularly dangerously powerful?
      • RandomLensman 3y ago
        It could break down on any of those, for example (with the first one not forever, but could take much longer than what is mainstream thinking now). A lot of existential risk involves bold hypotheses about machines being able to "solve" a lot of things humans cannot, but we don't know how much is actually just solvable by superior intelligence vs. what just doesn't work in that way. Human collective intelligence failed on a lot of things so far. Even the basic idea of exponential scaling intelligence is a hypothesis which might not hold true. Also, some existential risk ideas involve hypotheses around evolution and on how species dominate - again, might not be right.
        • concordDance 3y ago
          "It could" implies to me that there's a chance they don't. What would you say the odds are on each of those three components? And are those odds independent?
• epups 3y ago
  We do not have AGI yet. In order to get there, very specific chips are needed that can only be built by a handful of companies in a handful of factories in the world. On top of that, the available stock of existing cards is predominantly in the hands of large tech companies. If China and the US agreed - perhaps even if the US alone wanted it -, development of AI would be severely constrained if not stopped tomorrow.
  • csomar 3y ago
    Government will rally to make this internally as part of their secrets projects. This will be worse. The US sanction mechanism has shown that it's highly flawed with the Russian sanctions. This will not work but I can see some bureaucrat out there going all in on this.
• toth 3y ago
  It will be very difficult, but if you think about nuclear weapons it doesn't sound so hopeless. First nuclear weapons were used in 1945, almost 80 years ago. Today, out of ~ 200 countries in the world only 9 have nukes. Of those, only 4 did it after Non-proliferation of Nuclear Weapons Treaty. South Africa had them and gave them up under international pressure. Ukraine and other former soviet republics voluntarily gave them up after the breakup of the USSR. So, yes. Unfortunately we still have nukes and there are way too many of them, but it's not like non-proliferation efforts achieved nothing. Iran, a large country with a lot of resources, has been trying to get them a for a while with no luck for instance. And for frontier AI research you don't just need matrix multiplication, you need a lot of it. There's less than a handful of semiconductors fabs globally that can make the necessary hardware. And if you can't get to TSMC for some reason, you can get to ASML, NVDIA, etc.
  • nazka 3y ago
    Getting uranium, the hardware to have nukes, and all the research and production around it is hard without being detected. Starting to code AI on a Mac and a few good GPUs or even a cluster without being detected is crazy easy. Even if they start to treat the most powerful GPUs as small nukes and only a few can have them it’s super easy to steal, transport, store, use… Nukes not so much.
    • toth 3y ago
      For frontier AI models you don't need a few GPUs or a small cluster. You need thousands of them, you need them all connected with high bandwidth and you need a lot of power. It's not something you can do in your garage or undetected. There is some risk that over time people will be able to do more with less, and eventually you can train an AGI with a few dozen A100s. If that happens I agree there's nothing you can do, but until then there is a chance.
      • pixl97 3y ago
        What is the difference between a weather model, a nuclear bomb simulation, and training a LLM? At the end of the day all these are just calculations, of which you need thousands of processors to do with high accuracy. Pretty much what you're suggesting is a global police force to ensure we're performing the 'correct calculations'. Having to form an authoritarian world computer police isn't much of a solution to the problem either. This just happens to work well in the nuclear world because it's hard to shield and the average person doesn't have a need for uranium.
• amalcon 3y ago
  I like to think of this same argument in another (slightly tongue-in-cheek) way: doing this would require solving the human alignment problem. Since this is certainly harder than the AI alignment problem, if we could do it, we wouldn't need to. edit: s/easier/harder/
• hollerith 3y ago
  Yudkowsky does not expect a stop to the development of AI globally. He, and thousands of others (including me) are merely saying that if there is not a stop, then we are all dead. Yudkowsky thinks we are all dead with probability .999. I don't know the subject nearly as well as he does, which makes it impossible for me to have much certainty, so my probability is only .9. Also, it is only the development of ever-more-capable AIs that is the danger. If our society wants to integrate language models approximately as capable as GPT-4 thoroughly into the economy, there is very little extinction danger in that (because if GPT-4 were capable enough to be dangerous, we'd already all be dead by now). Also, similar to how even the creators of GPT-4 failed to predict that GPT-4 would turn out capable enough to score in the 92th percentile of the Bar Exam, almost none of the people who think we are all dead claim to know when exactly we are all dead except to say that it will probably be some time in the next 30 years or so.
  • kylebenzle 3y ago
    I've been in data science only a few years but I fail to understand what people's issue with AI is. Best 8 can tell is it's ignorance coupled with fear mongering.
    • hollerith 3y ago
      >I fail to understand what people's issue with AI is But you haven't put much effort into trying to understand, have you?
• melagonster 3y ago
  we had seen the result of endeavor to stop global warming. but how can we do...
• shoubidouwah 3y ago
  One good way of staying to a prudent course / halting haphazard - winner takes all progress is to use social tech for it. In particular, there is an interesting possibility: the top researchers who drive the AI scenario all kind of know each other, and can self-organize in an international union, with pooled resources for strikes. AI revolution is dependent on simultaneous progress and cross capitalization of good ideas: a well timed strike of a good portion of research worldwide can stop things from moving on pretty effectively.
  • edgyquant 3y ago
    Laughably naive. International groups of autoworkers won’t work together but you think a group of the highest paid people on the planet, with incentive to compete against each other for national interest, will.
  • nyssos 3y ago
    > the top researchers who drive the AI scenario all kind of know each other, and can self-organize in an international union, with pooled resources for strikes. This is skipping to the end in the same way that "governments should just agree to work together" is. The hard part is building an effective coordination mechanism in the first place, not using it.
• bnralt 3y ago
  Every time I see Yudkowsky’s writing I wonder why people take him seriously. Just a couple of months back he was duped by a fake story about an airforce AI deciding to kill it’s operators in a simulation. It turned out the whole thing was thought experiment. Asking Yudkowsky what we should do about AI is like asking Green Peace what we should do about nuclear power.
  • dragonwriter 3y ago
    > Every time I see Yudkowsky’s writing I wonder why people take him seriously. Its a perfect thing for rich, extremely privileged people to grab onto as a cause to present thenselves as (and maybe even feel like they are, depending on their capacity for self-delusion) “doing something for humanity” while continuing to ignore the substantive material condition of, as a first order approximation, everyone in the world, and their role in perpetuating it.
    • 0x00_NULL 3y ago
      I lost total interest in every “Stop” argument when Elon Musk started a company to develop AGI during the pause that he signed on to and promoted. Yann LeCunn said that this was just an attempt to seize market share among the big companies and he couldn’t have been more right.
      • pclmulqdq 3y ago
        That is transparently what this letter is for most of its signatories. They lost the race (in one way or another) and they want time to catch back up. This is as true for the academics who made big contributions in the past but missed out on the latest stuff as it is for Elon Musk.
        • nineteen999 3y ago
          The joke will be on them when our open-source powered mechwarriors made of bulldozer parts come for them.
• i_am_a_squirrel 3y ago
  > imagine thinking we can stop matrix multiplication globally. It's clear what's happening here. Some "lin-alg" freshman has found a magical lamp.
• bondarchuk 3y ago
  If your main critique of EY is the unfeasibility of his proposed solutions then you might still take him seriously for his identification of the problems.
• _cjse 3y ago
  Well, you can police the entire world quite cheaply on this or any other scientific research program by using fine-insured bounties to do it. Whether it's a good idea is a different question. https://web.archive.org/web/20220709091749/https://virtual-instinct.xyz/ https://web.archive.org/web/20220709091749/https://virtual-i...
• degrews 3y ago
  Obviously no one is talking about "stopping matrix multiplication globally". But setting limits on the supply of and access to hardware, in a way that would meaningfully impede AI development, does seem feasible if we were serious about it. Also, Eliezer is not claiming this will definitely work. He thinks it ultimately probably won't. The claim is just that this is the best option we have available.
• wseqyrku 3y ago
  Eliezer Yudkowsky sounds a lot like Neil deGrasse Tyson of computer science.