53 ms·

Statement on AI Risk

• cmilton 3y ago
  The Sagan standard[1] needs to be applied here. [1]https://en.wikipedia.org/wiki/Sagan_standard https://en.wikipedia.org/wiki/Sagan_standard
  • sebzim4500 3y ago
    I think the claim that for the first time in 4 billion years a far superior intelligence will be willingly superservient to an inferior one is extraudinary enough to require extraudinary evidence, yes.
• deegles 3y ago
  Serious question... where can I read the best summaries of the arguments in favor of "AGI will destroy humanity"? and also arguments against? I'm not convinced we can predict how it will behave.
  • EamonnMR 3y ago
    Stuart Russell's book Human Compatible articulates it well. He agrees that we can't predict how it would behave, and that's what he's worried about.
• mola 3y ago
  The "risks" from AI are far more mundane from this AGI taking over the world and extinction level event nonsense. It's the usual, more power to the few, less power to the workers. Easy manipulation and control of the masses by "subtle" ways such as swamping the media with false divisive narratives (as being done today, but on steroids) and more violent ways such as automatic "police" without qualms or empathy. The risks we face are from other humans. This statement is at best naive misdirection. I won't be surprise if people like Altman are just working to increase their own power through regulatory capture. This is a silicon valley power move. Some Randian heros with monarchic aspirations got some good folks to be their useful idiots.
• hebbfrog 3y ago
  The idea this is regulatory capture seems correct. 1. For all the talk of dangers, there has not been one definitive, well-thought out, peer-reviewed discussions of an A-to-Z way in which this might arise from current technology. Global warming isn't taken terribly seriously from a regulatory standpoint, and it has a defined model. One must naturally ask: how serious do they consider this if they can't show their work? How serious should we consider this "threat" they can't actually explain? 2. Therefore, we must presume it is speculatory. 3. These people are smart, well-funded, and have business interests. 4. At the fork dictated by Hanlon's Razor we are given two possibilities: A. These highly intelligent people are forming a massive collective and putting their reputations on the line over ungrounded speculation. This implies they don't actually grasp the field they are discussing - even if they work in it, and therefore have made it clear they're not the type anyone should listen too. Therefore we should work to actively disregard them, and their institutions. B. These highly intelligent people are organizing in a coherent manner to their net benefit. Likely through market capture. Therefore they are still respectable in their fields, but behaving by putting their interests before the public's. Therefore: it might be contrary to Hanlon's Razor, but I think option B is more sound given the probability of mass hysteria seems so low among this particular population. Generally most here have recognized this some way or another, but I wanted to point out how clearly logical it is that this is clearly grift. In science your reputation has no value if you cannot prove your work, and they've fired off hypothesis without providing proof. That implies this list should be seen as a denouncement of the scientific credibility of everyone on it, which is a shame.
• soroushjp 3y ago
  There are a lot of critiques here and elsewhere of the statement and the motivations of its signatories. I don't think they are right and I think they take away from the very serious existential risks we face. I've written up my detailed views, see specifically "Signing the statement purely for personal benefit": https://www.soroushjp.com/2023/06/01/yes-avoiding-extinction-from-ai-is-an-urgent-priority-a-response-to-seth-lazar-jeremy-howard-and-arvind-narayanan/ https://www.soroushjp.com/2023/06/01/yes-avoiding-extinction...
• ggm 3y ago
  Nothing about this risk or the statement implies AGI is real, because the risk exists in wide scale use of existing technology. It's the risk of belief in algorithmically derived information, and deployment of autonomous, unsupervised systems. It's great they signed the statement. It's important.
  • mahogany 3y ago
    > It's the risk of belief in algorithmically derived information, and deployment of autonomous, unsupervised systems. And Sam Altman, head of one of the largest entities posing this exact risk, is one of the signatories. We can't take it too seriously, can we?
    • sebzim4500 3y ago
      I don't get this argument at all. Why does the fact that you doubt the intentions one of the signatories mean we can disregard the statement? There are plenty of signatories (including 3 turing award winners) who have no such bias.
      • juve1996 3y ago
        Every human has bias, no one is infallible, no matter how many awards they have to their name. The reason why people doubt is cui bono. And it's a perfectly rational take.
      • mahogany 3y ago
        Yeah, fair enough, it doesn't necessarily invalidate the statement. But it's odd, don't you think? It's like if a group released a public statement that said "Stop Oil Now!" and one of the signatories was Exxon-Mobil. Why would you let Exxon-Mobil sign your statement if you wanted to be taken seriously?
    • deleted 3y ago
      [deleted]
• arisAlexis 3y ago
  [flagged]
• vegabook 3y ago
  [Un]intended consequence: "AI is too dangerous for you little guys. Leave it to we the FAANGs" - rubber stamped with legislation.
  • ChatGTP 3y ago
    What would Max Tegmark, Geoffrey Hinton, Yohshua Bengio (to name a few) have absolutely anything to do with FAANG ? They're completely independent AI researchers and geniuses spending their own free time on trying to warn you and others of the dangers of the technology they've created to help keep the world safer. Seems like you're taking a far too overly cynical position ?
    • deleted 3y ago
      [deleted]
    • logicchains 3y ago
      [flagged]
• taneq 3y ago
  What does this even mean? OK, it's a priority. Not a high one, but still... somewhere in between coral bleaching and BPA, I guess?
  • roydanroy2 3y ago
    Coral bleaching, nuclear war, and pandemics.
  • quickthrower2 3y ago
    The EAs put it above climate change
    • arisAlexis 3y ago
      It is coming before climate change. No matter which group "put it" reality doesn't care. Humanity will not get extinct in the next 10 years by climate but many AI scientists think there is a chance this happens with AI.
      • quickthrower2 3y ago
        I didn’t mean “EA = bad” to be clear.
  • deleted 3y ago
    [deleted]
  • Tepix 3y ago
    It's not like humankind is doing much to stop coral bleaching... despite corals being the living ground for 80% of marine species.
  • sebzim4500 3y ago
    The US spent trillions of 2020 dollars trying to limit the threat of nuclear war, and this statement says that AI risk should be seen as a similar level of threat.
• jdthedisciple 3y ago
  Slightly underwhelming a statement, but surely that's just me.
  • that_guy_iain 3y ago
    It seems to be a PR related statement. For example, OpenAI's Sam Altman has signed it but is as far as I can understand very resistant to actual measures to deal with possible risks.
    • acjohnson55 3y ago
      I don't think that's a fair assessment. He favors government oversight and licensing. Arguably, that would entrench companies with deep pockets, but it's also a totally reasonable idea.
      • that_guy_iain 3y ago
        > He favors government oversight and licensing. No, he favors thing that benefit him > "The current draft of the EU AI Act would be over-regulating, but we have heard it's going to get pulled back," he told Reuters. "They are still talking about it." And really, what we'll see is the current EU AI Act as-is is probably not strong enough and we'll almost certainly see the need for more in the future.
        • acjohnson55 3y ago
          Right now, he's openly calling for regulation. That's a verifiable fact. It's very possible that when specific proposals are on the table, that we'll see Altman become uncooperative with respect to things that don't fit into his self-interest. But until that happens, you're just speculating.
          • timmytokyo 3y ago
            He's not saying "please regulate me!" He's saying "Please regulate my competitors!" I see no reason to assume benign motives on his part. In fact there is every reason to believe the opposite.
• wiz21c 3y ago
  It'd be so much more convincing if each of the signatories actually articulated why he/she sees a reisk in there. Without that, it pretty much looks like a list of invites to a VIP club...
  • lxnn 3y ago
    As the pre-amble to the statement says: they kept the statement limited and succinct as there may be disagreement between the signatories about the exact nature of the risk and what to do about it.
• maxehmookau 3y ago
  Signatures from massive tech giants that on one hand are saying "hold on this is scary, we should slow down" but also "not us, we're doing fine. You should all slow down instead" mean that this is a bit of a empty platitude.
• valine 3y ago
  I have yet to see a solution for “AI safety” that doesn’t involve ceding control of our most powerful models to a small handful of corporations. It’s hard to take these safety concerns seriously when the organizations blowing the whistle are simultaneously positioning themselves to capture the majority of the value.
  • circuit10 3y ago
    We don’t really have a good solution, I guess that’s why we need more research into it Companies might argue that giving them control might help but I don’t think most individuals working on it think that will work
    • sycamoretrees 3y ago
      Is more research really going to offer any true solutions? I’d be genuinely interested in hearing about what research could potentially offer (the development of tools to counter AI disinformation? A deeper understanding of how LLMs work?), but it seems to me that the only “real” solution is ultimately political. The issue is that it would require elements of authoritarianism and censorship.
      • wongarsu 3y ago
        A lot of research about avoiding extinction by AI is about alignment. LLMs are pretty harmless in that they (currently) don't have any goals, they just produce text. But at some point we will succeed in turning them into "thinking" agents that try to achieve a goal. Similar to a chess AI, but interacting with the real world instead. One of the big problems with that is that we don't have a good way to make sure the goals of the AI match what we want it to do. Even if the whole "human governance" political problem were solved, we still couldn't reliably control any AI. Solving that is a whole research field. Building better ways to understand the inner workings of neural networks is definitely one avenue
        • sycamoretrees 3y ago
• blueblimp 3y ago
  This is way better than the open letter. It's much clearer and much more concise, and, maybe most importantly, it simply raises awareness rather than advocating for any particular solution. The goal appears to have been to make a statement that's non-obvious (to society at large) yet also can achieve agreement among many AI notables. (Not every AI notable agrees though--for example, LeCun did not sign, and I expect that he disagrees.)
  • a_bonobo 3y ago
    > it simply raises awareness I don't think it simply raises awareness - it's a biased statement. Personally, I don't think the advocated event is likely to happen. It feels a bit like the current trans panic in the US: you can 'raise awareness' of trans people doing this or that imagined bad thing, and then use that panic to push your own agenda. In OpenAI's case, they seem to push for having themselves be in control of AI, which goes counter to what, for example, the EU is pushing for.
    • sebzim4500 3y ago
      In what sense is this a 'biased statement' exactly? If a dozen of the top climate scientists put out a statement saying that fighting climate change should be a serious priority (even if they can't agree on one easy solution) would that also be 'biased'?
      • a_bonobo 3y ago
        Climate change is a generally accepted phenomenon. Extinction risk due to AI is not a generally accepted phenomenon.
        • sebzim4500 3y ago
          Now it is, when climate scientists were first sounding the horn they got the same response that these people are getting now. For example: "This signatory might have alterior motives, so we can disregard the whole statement" "We haven't actually seen a superintelligent AI/manmade climate change due to CO2 yet, so what's the big deal?" "Sure maybe it's a problem, but what's your solution? Best to ignore it" "Let's focus on the real issues, like not enough women working in the oil industry"
• deleted 3y ago
  [deleted]
• endisneigh 3y ago
  > Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war. risk of extinction due to AI? people have been reading too much science fiction. I would love to hear a plausible story of how AI will lead to human extinction that wouldn't happen with traditional non-AI tech. for the sake of conversation let's say non-AI tech is any broadly usable consumer technology before Jan 1 of 2020.
  • twoodfin 3y ago
    I agree that a lot of the Skynet-type scenarios seem silly at the current level of technology, but I am worried about the intersection between LLMs, synthetic biology, and malicious or incompetent humans. But that’s just as much or more of an argument for regulating the tools of synthetic biology.
  • acjohnson55 3y ago
    Extinction would probably require an AI system taking human extinction on as an explicit goal and manipulating other real world systems to carry out that goal. Some mechanisms for this might include: - Taking control of robotic systems - Manipulating humans into actions that advance its goal - Exploiting and manipulating other computer systems for greater leverage - Interaction with other technologies that have global reach, such as nuclear weapons, chemicals, biological agents, or nanotechnology. It's important to know that these things don't require AGI or AI systems to be conscious. From what I can see, we've set up all of the building blocks necessary for this scenario to play out, but we lack the regulation and understanding of the systems being built to prevent runaway AI. We're playing with fire. To be clear, I don't think I am as concerned about literal human extinction as I am the end of civilization as we know it, which is a much lower bar than "0 humans".
    • endisneigh 3y ago
      everything you're describing has been possible since 2010 and been done already. AI isn't even necessary. simply scale and some nefarious meat bags.
• progrus 3y ago
  Not interested in any PR crap from scummy corporations angling to capture the regulators.
  • TristanDaCunha 3y ago
    Many of the signatories aren't associated with any corporation.
    • thrillgore 3y ago
      Except for Sam Altman.
• dncornholio 3y ago
  The top priority is to create awareness IMHO. AI can only be as destructive as the users let it. From my small sample size, it seems people believe in AI too much. Especially kids.
  • TristanDaCunha 3y ago
    > AI can only be as destructive as the users let it. Not really, I suppose you aren't familiar with AI alignment.
• hit8run 3y ago
  Well AI is here. We need to live with it. Change our economic systems to a socialist approach.
  • sebzim4500 3y ago
    AI may be here (purely depends on definition so not worth debating) but the superintelligent AGI that they are scared of clearly isn't here yet.
    • hit8run 3y ago
      I get what you're saying. Its day will come. Study human history. If we can build it - we will build it.
• berkeleyjunk 3y ago
  I really thought there would be a statement detailing what the risks are but this seems more like a soundbite to be consumed on TV. Pretty disappointing.
  • neom 3y ago
    So far the examples I've heard are: humans will ask AI to help humans solve human issues and the AI will say humans are the issue and therefore mystically destroy us somehow. Or, AI will be inherently interested in being the primary controller of earth and so destroy all humans. Or, AI will for reasons be inherently misaligned with human values. Andrej Karpathy Said it will fire nukes on us. Elon said pen is mightier than the sword and civil war is inevitable.
  • sebzim4500 3y ago
    Because then you wouldn't be able to get this many people to sign the statement. It's like with climate change, every serious scientist agrees it is a problem but they certainly don't agree on the best solution. If the history of the climate change 'debate' is anything to go by this statement will do very little except be mocked by South Park.
  • quicklime 3y ago
    It’s not on the same page as the signatories but they do have this: https://www.safe.ai/ai-risk https://www.safe.ai/ai-risk
    • berkeleyjunk 3y ago
      Thank you! That page certainly seems more concrete and useful.
• jasonvorhe 3y ago
  Can't take this seriously as long they still keep offering commercial AI services while improving the existing models they already have. (I'm not in favor of just stopping AI development and even if people claimed to stop, they probably wouldn't.) It's like people carrying torches warning about the risks of fire.
  • sebzim4500 3y ago
    I agree with you that some of these people (probably Sam Altman) are likely proposing this regulation out of self interest rather than genuine concern. But I don't think the stance is necessarily hypocritical. I know nuclear engineers who advocate for better regulation on nuclear power stations, and especially for better handling of nuclear waste. You can believe that AI is a net positive but also that it needs to be handled with extreme care.
    • pphysch 3y ago
      Nuclear engineers are in the line of fire, of course they would care about safety. It's their safety more than almost anyone else. Needless to say, this does not hold for AI researchers.
      • sebzim4500 3y ago
        What makes you think that an AI caused extinction event will somehow leave AI researchers alive?
        • snickerbockers 3y ago
          There's a good chance that it wouldn't, but since they're the ones (initially, at least) in control of the AI they stand the best chance of not being targeted by it. These hypothetical AI extinction events don't have to be caused by the AI deciding to eliminate humanity for its own sake like in Terminator, they could also be driven by a human in control of a not-entirely-sentient AI.
• dontupvoteme 3y ago
  At what point do we stop pretending that the west is capitalist and accept that it's some weird corporate-cabal-command-economy? The only thing which might stop this backroom regulatory capture is the EU since they're not in on it.
  • a_bonobo 3y ago
    I miss David Graeber >Graeber Um…that’s a long story. But one reason seems to be that…and this is why I actually had managerial feudalism in the title, is that the system we have…alright—is essentially not capitalism as it is ordinarily described. The idea that you have a series of small competing firms is basically a fantasy. I mean you know, it’s true of restaurants or something like that. But it’s not true of these large institutions. And it’s not clear that it really could be true of those large institutions. They just don’t operate on that basis. >Essentially, increasingly profits aren’t coming from either manufacturing or from commerce, but rather from redistribution of resources and rent; rent extraction. And when you have a rent extraction system, it much more resembles feudalism than capitalism as normally described. You want to distribute— You know, if you’re taking a large amount of money and redistributing it, well you want to soak up as much of that as possible in the course of doing so. And that seems to be the way the economy increasingly works. http://opentranscripts.org/transcript/managerial-feudalism-revolt-caring-classes/#h(EipIiy,4) http://opentranscripts.org/transcript/managerial-feudalism-r...
• shafyy 3y ago
  The issue I take with these kind of "AI safety" organizations is that they focus on the wrong aspects of AI safety. Specifically, they run this narrative that AI will make us humans go extinct. This is not a real risk today. Real risks are more in the category of systemic racism and sexism, deep fakes, over reliance on AI etc. But of course, "AI will humans extinct" is much sexier and collects clicks. Therefore, the real AI risks that are present today are underrepresented in mainstream media. But these people don't care about AI safety, they do whatever required to push their profile and companies. A good person to follow on real AI safety is Emily M. Bender (professor of computer linguistics at University of Washington): https://mstdn.social/@emilymbender@dair-community.social https://mstdn.social/@emilymbender@dair-community.social
  • tourgen 3y ago
    The elite class in your country views AI as a risk to their status as elites, not an actual existential threat to humanity. They are just lying to you, as usual. That is what our current crop of globalist, free-trade, open-borders elites do. Imagine if you had an AI companion that instantly identified pilpul in every piece of media you consumed: voice, text, whatever. It highlighted it for you. What if you had an AI companion that identified instantly when you are being lied to or emotionally manipulated? What if this AI companion could also recommend economic and social policies that would actually improve the lives of people within your nation and not simply enrich a criminal cabal of globalist elites that treat you like cattle?
    • pixl97 3y ago
      The Elite class is just as apt to consolidate power with AI and rule the entire world with it. If you have a super duper AI in your pocket looking at the data around you, then they a super super super duper duper duper AI looking at every bit of data from every corner of the world they can feed the thing giving themselves power and control you couldn't even begin to imagine. Falling into conspiratorial thinking on a single dimension without even considering all the different factors that could change belies ignorance. Yes, AI is set up to upend the elites status, but is just as apt to upset your status of being able to afford food and a house and meaningful work. > not simply enrich a criminal cabal of globalist elites that treat you like cattle? There is a different problem here... And that is humankind has made tools capable of concentrating massive amounts of power well before we solved human greed. Any system you make that's powerful has to overcome greedy power seeking hyper-optimizers. If I could somehow hit a button and Thanos away the current elites, then another group of powerseekers would just claim that status. It is an inane human behavior.
• skepticATX 3y ago
  I suspect that in 5 years we’re going to look back and wonder how we all fell into mass hysteria over language models. This is the same song and dance from the usual existential risk suspects, who (I’m sure just coincidentally) also have a vested interest in convincing you that their products are extremely powerful.
  • vesinisa 3y ago
    Yeah, like I fail to see how would an AI even cause human extinction? Through some Terminator style man-robot warfare? But the only orgnizations that would seem capable of building such killer robots are governments that already possess the capacity to extinguish the entire human race with thermonuclear weapons - and at a considerably lower R&D budget for that end. It seems like hysteria / clever marketing for AI products to me.
    • sebzim4500 3y ago
      The standard example is that it would engineer a virus but that's probably a lack of imagination. There may be more reliable ways of wiping out humanity that we can't think of. I think speculation on the methods is pretty pointless, if a superintelligent AI is trying to kill us we're probably going to die, the focus should be on avoiding this situation. Or providing a sufficiently convincing argument for why that won't happen.
      • klibertp 3y ago
        Or why it should happen...
    • deleted 3y ago
      [deleted]
    • deleted 3y ago
      [deleted]
  • sebzim4500 3y ago
    Who in that list are the 'usual existential risk suspects'?
• trebligdivad 3y ago
  There are a bunch of physicists signed up on there; (e.g. Martin Rees) - they don't seem relevant to it at all. There's been a long history of famous physicists weighing in on entirely unrelated things.
  • abecedarius 3y ago
    Check only the "AI scientists" checkbox then.
  • Simon321 3y ago
    That's because it's not authentically trying to address a problem but trying to convince an audience of something by appealing to authority. Elizabeth Holmes & Theranos were masters of collecting authorities to back their bogus claims because they know how effective it is. It doesn't even need to be in the field where you're making the claims. They had Kissinger for god's sake, it was a biotech company!
  • duvenaud 3y ago
    Well, Rees co-founded the Centre for the Study of Existential Risk, and has at least been thinking and writing about these issues for years now.
  • randomdata 3y ago
    Musical artist Grimes is also a signatory. It would seem the real purpose of this is to train an AI agent on appeal to authority.
    • oldgradstudent 3y ago
      > It would seem the real purpose of this is to train an AI agent on appeal to authority. I hope the real purpose is to train an AI agent to understand why appeal to authority was always cosidered to be a logical fallacy.
      • randomdata 3y ago
        Considered by some to be a logical fallacy. Not considered at all by most people. Hence its effectiveness.
• arek_nawo 3y ago
  All the concern and regulatory talk around AI seems like it's directed not towards AI risk (that's not even a thing right now) rather than controlling access to this evolving technology. The not-so-open Open AI and all their AI regulation proposals, no matter how phrased, will eventually limit access to AI to big tech and those with deep enough pockets. But of course, it's all to mitigate AI risk that's looming over us, especially with all the growing open-source projects. Only in proper hands of big tech will we be safe. :)
• apsec112 3y ago
  A lot of the responses to this seem like Bulverism, ie., trying to refute an argument by psychoanalyzing the people who argue it: "Suppose I think, after doing my accounts, that I have a large balance at the bank. And suppose you want to find out whether this belief of mine is "wishful thinking." You can never come to any conclusion by examining my psychological condition. Your only chance of finding out is to sit down and work through the sum yourself. When you have checked my figures, then, and then only, will you know whether I have that balance or not. If you find my arithmetic correct, then no amount of vapouring about my psychological condition can be anything but a waste of time. If you find my arithmetic wrong, then it may be relevant to explain psychologically how I came to be so bad at my arithmetic, and the doctrine of the concealed wish will become relevant—but only after you have yourself done the sum and discovered me to be wrong on purely arithmetical grounds. It is the same with all thinking and all systems of thought. If you try to find out which are tainted by speculating about the wishes of the thinkers, you are merely making a fool of yourself. You must first find out on purely logical grounds which of them do, in fact, break down as arguments. Afterwards, if you like, go on and discover the psychological causes of the error. You must show that a man is wrong before you start explaining why he is wrong. The modern method is to assume without discussion that he is wrong and then distract his attention from this (the only real issue) by busily explaining how he became so silly." - CS Lewis https://en.wikipedia.org/wiki/Bulverism https://en.wikipedia.org/wiki/Bulverism
  • skepticATX 3y ago
    But what argument is there to refute? It feels like Aquinas “proving” God’s existence by stating that it is self evident. They can’t point to an existing system that poses existential risk, because it doesn’t exist. They can’t point to a clear architecture for such a system, because we don’t know how to build it. So again, what can be refuted?
    • notahacker 3y ago
      Seems apt the term "Bulverism" comes from CS Lewis, since he was also positing that an unseen, unfalsifiable entity would grant eternal reward to people that listened to him and eternal damnation to those that didn't...
• Mizza 3y ago
  Can anybody who really believes this apocalyptic stuff send me in the direction of a convincing _argument_ that this is actually a concern? I'm willing to listen, but I haven't read anything that tries to actually convince the reader of the worry, rather than appealing to their authority as "experts" - ie, the well funded.
  • NumberWangMan 3y ago
    Whether an argument is "convincing" is relative to the listener, but I can try! Paul Christiano lays out his view of how he thinks things may go: https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-looks-like https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-... My thoughts on it are the combination of several things I think are true, or are at least more likely to be true than their opposites: 1) As humanity gets more powerful, it's like putting a more powerful engine into a car. You can get where you're going faster, but it also can make the car harder to control and risk a crash. So with that more powerful engine you need to also exercise more restraint. 2) We have a lot of trouble today controlling big systems. Capitalism solves problems but also creates them, and it can be hard to get the good without the bad. It's very common (at least in some countries) that people are very creative at making money by "solving problems" where the cure is worse than the disease -- exploiting human weaknesses such as addiction. Examples are junk food, social media, gacha games. Fossil fuels are an interesting example, where they are beneficial on the small scale but have a big negative externality. 3) Regulatory capture is a thing, which makes it hard to get out of a bad situation once people are making money on it. 4) AI will make companies more powerful and faster. AGI will make companies MUCH more powerful and faster. I think this will happen more for companies than governments. 5) Once people are making money from AI, it's very hard to stop that. There will be huge pressure to make and use smarter and smarter AI systems, as each company tries to get an edge. 6) AGIs will amplify our power, to the point where we'll be making more and more of an impact on earth, through mining, production, new forms of drugs and pesticides and fertilizers, etc. 7) AGIs that make money are going to be more popular than ones that put humanity's best interests firsts. That's even assuming we can make AGIs which put humanity's best interests first, which is a hard problem. It's actually probably safer to just make AGIs that listen when we tell them what to do. 8) Things will move faster and faster, with more control given over to AGIs, and in the end, it will be very hard train to stop. If we end up where most important decisions are made by AGIs, it will be very bad for us, and in the long run, we may go extinct (or we may just end up completely neutered and at their whims). Finally, and this is the most important thing -- I think it's perfectly likely that we'll develop AGI. In terms of sci-fi-sounding predictions, the ones that required massive amounts of energy such as space travel have really not been borne out, but the ones that predicted computational improvements have just been coming true over and over again. Smart phones and video calls are basically out of Star Trek, as are LLMs. We have universal translators. Self-driving cars still have problems, but they're gradually getting better and better, and are already in commercial use. Perhaps it's worth turning the question around. If we can assume that we will develop AGI in the next 10 or 20 or ever 30 years -- which is not guaranteed, but seems likely enough to be worth considering -- how do you believe the future will go? Your position seems to be that there's nothing to worry about--what assumptions are you making? I'm happy to work through it with you. I used to think AGI would be great, but I think I was assuming a lot of things that aren't necessarily true, and dropping those assumptions means I'm worried.
• deleted 3y ago
  [deleted]
• crmd 3y ago
  “We have no moat, and neither does OpenAI” - Google They’re attempting to build a regulatory moat. The best chance humanity has at democratizing the benefits of AI is for these models to be abundant and open source.
• jgalt212 3y ago
  Now that weno longer live in fear of COVID, we must find something else to fill that gap.
• gumballindie 3y ago
  Pushing hard to convince senile lawmakers that only a select few should be allowed to multiply them matrices?
  • deleted 3y ago
    [deleted]
  • belter 3y ago
    [flagged]
  • nologic01 3y ago
    you are laughing but I've found a matrix that is called an elimination matrix [1]. Are you still laughing now? Are you absolutely sure that an elimination matrix with 1 Petazillion of elements will not become sentient in a sort of emergent way? [1] https://en.wikipedia.org/wiki/Duplication_and_elimination_matrices https://en.wikipedia.org/wiki/Duplication_and_elimination_ma...
    • gumballindie 3y ago
      Straight to jail. Jokes aside i can totally see the media turning this into hysteria and people falling for it.
• yodsanklai 3y ago
  > risk of extinction from AI That's a pretty strong statement. Extinction of humanity, no less. I don't get why so many experts (lots of them aren't crackpots) signed this.
  • ChatGTP 3y ago
    Maybe seen some things we don't yet know exist?
  • sebzim4500 3y ago
    Given risk is probability * damage, and the damage is enormous, the risk can be high even if the probability is fairly low.
    • Simon321 3y ago
      This fallacy is also known as Pascal's wager: https://en.wikipedia.org/wiki/Pascal%27s_wager https://en.wikipedia.org/wiki/Pascal%27s_wager But this argument got billions of people to believe in the concept of hell so i expect it to work again to convince people to believe in AI doomsday.
      • sebzim4500 3y ago
        I agree it's a fallacy when the probability is like 10^-10 but in this case I believe that the probability is more like 1%, in which case the argument is sound. I'm not trying to make a pascal's wager argument.
        • timmytokyo 3y ago
          Correction: you perceive it to be a fallacy when others assign high probabilities to things you believe are low probability. Unfortunately, this cuts both ways. Many people believe your 1% estimate is unreasonably high. Are you therefore promoting a fallacy? Too many ridiculous arguments can be justified on the backs of probability estimates pulled from nether regions.
• gmuslera 3y ago
  The main potential risks of AI for that level of threat is that government, military and intelligence agencies, and big corporations probably with military ties, arms them. And that is not something that will be solved with legislation (for the commoners) and good will. The problem or risk are not AIs there. And no matter what is see in the AI field, what they will have in their hands won’t have restrictions and probably will be far more powerful than what is available for the general public. And without teeth, what they can do? Maybe help to solve or mitigate the real existential risk that is climate change.
• Finnucane 3y ago
  Not with a bang but a whimper, simulated by extrapolation from the historical record of whimpers.
• b3nji 3y ago
  Nonsense, the industry giants are just trying to scare the law makers to license the technology. Effectively, cutting out everyone else. Remember the Google note circulating saying "they have no moat", this is their moat. They have to protect their investment, we don't want people running this willy nilly for next to no cost on their own devices, God forbid!
  • sanderjd 3y ago
    I would definitely find it more credible if the most capable models that are safe to grandfather in to being unregulated didn't just happen to be the already successful products from all the people leading these safety efforts. It also just happens to be the case that making proprietary models - like the current incumbents make - is the only safe way to do it.
  • aceon48 3y ago
    That moat document was published by a single software engineer, not some exec or product leader. Humans dont really grasp exponential improvements. You wont have much time to regulate something that is improving exponentially.
    • jvanderbot 3y ago
      Single software engineers writing influential papers is often enough how a exec or product leader draws conclusions, I expect. It worked that way in everywhere I've worked.
    • aero-deck 3y ago
      It doesn't matter who wrote it, it got picked up, had a good argument and affected market opinion. The execs now need to respond to it. Humans also don't grasp that things can improve exponentially until they stop improving exponentially. This belief that AGI is just over the hill is sugar-water for extracting more hours from developers. The nuclear bomb was also supposed to change everything. But in the end nothing changed, we just got more of the same.
      • 3y ago
• dns_snek 3y ago
  Are we supposed to keep a straight face while reading these statements? This is Sam Bankman-Fried type of behavior, but applied to gullible AI proponents and opponents rather than "crypto bros". Let me guess, the next step is a proposed set of regulations written by OpenAI, Google and other Big Corporations who Care™ about people and just want to Do What's Best For Society™, setting aside the profit motive for the first time ever? We don't have to guess - we already know they're full of shit. Just look at OpenAI's response to proposed EU AI regulations which are actually trying to reduce the harm potential of AI. These empty platitudes ring so hollow that I'm amazed that anyone takes them seriously.
  • sebzim4500 3y ago
    Explain to me why you think Max Tegmark wants this technology to be controlled by FAANG? Has his entire life been extremely in depth performance art?
    • dns_snek 3y ago
      I've never made any statements about, nor do I have any personal beliefs about Max Tegmark in particular.
      • ChatGTP 3y ago
        Well people like him are on the list, so it's a bit strange you're claiming this is mostly about FAANG?
        • dns_snek 3y ago
          FAANG are the only ones who stand to benefit financially. I'm taking the position that everyone else is simply a "useful idiot" for the likes of Sam Altman, in the most respectful way possible. Nobody is immune from getting wrapped up in hysteria, so I don't care who they are or what they have achieved when their signatures aren't supported by any kind of reasoning, much less sound reasoning.
    • 3y ago
• deleted 3y ago
  [deleted]
• progrus 3y ago
  Pathetic attempt at regulatory capture. Let’s all be smart enough to not fall for this crap, eh?
  • A4ET8a8uTh0 3y ago
    While we might ( rightfully ) recognize this blitzkrieg for what it is, the general population likely does not and may even agree to keep a lid on something it does not understand. Come to think of it. Just how many people actually understand it? I mean.. I think I have some idea, but I certainly did not dig in enough to consider myself an expert in any way shape or form ( all the while, Linkedin authorities of all kinds present themselves as SMEs after building something simple using chatgpt like html website ). And while politicians ( even the ones approaching senility ) are smarter than your average bear, they certainly know the deal. It is not like the same regulatory capture did not happen before with other promising technologies. They just might pretend they don't understand.
    • progrus 3y ago
      Maybe so, but then I would recommend making an effort to arm them with truths and critical thinking skills. It doesn’t have to go the same way every time.
  • Symmetry 3y ago
    That might be why Sam Altman signed it but why do you think that all the academics did so as well. Do you think he just bribed them all or something?
    • vadansky 3y ago
      Not saying they were, but it's not that expensive, looks like it starts at $50,000 https://www.npr.org/sections/thetwo-way/2016/09/13/493739074/50-years-ago-sugar-industry-quietly-paid-scientists-to-point-blame-at-fat https://www.npr.org/sections/thetwo-way/2016/09/13/493739074...
    • aero-deck 3y ago
• zzzeek 3y ago
  and that's why me, Sam Altman, is the only guy that can save you all ! so get in line and act accordingly
  • progrus 3y ago
    Pathetic. He is beclowning himself.
    • camillomiller 3y ago
      No, really? They guy who tried to make a world coin by stealing people’s biometrics data with a shiny metal orb?
      • progrus 3y ago
        This doofus actually probably thinks that the poor of the world will line up to submit to his (AI’s) benevolent rule. What a joke.
        • A4ET8a8uTh0 3y ago
          Believe it or not, people will subscribe to current zeitgeist and eventually even protect it.
          • progrus 3y ago
            [flagged]
        • nova22033 3y ago
          counterpoint: every crazy person you know who is on facebook.
• nickpp 3y ago
  It seems to me that "extinction" is humanity’s default fate. AI gives us a (slim) chance to avoid that default and transcend our prescribed fate.
• SanderNL 3y ago
  I'm not sure they can keep claiming this without becoming concrete about it. Nuclear weapons are not nebulous, vague threats of diffuse nature. They literally burn the living flesh right off the face of the earth and they do it dramatically. There is very little to argue about except "how" are we going to contain it, not "why". In this case I truly don't know "why". What fundamental risks are there? Dramatic, loud, life-ending risks? I see the social issues and how this tech makes existing problems worse, but I don't see the new existential threat. I find the focus on involving the government in regulating "large" models offputting. I don't find it hard to imagine good quality AI is possible with tiny - to us - models. I think we're just in the first lightbulbs phase of electricity. Which to me signals they are just in it to protect their temporary moat.
  • qayxc 3y ago
    I mostly agree - too vague, no substance. Regulations are OK IMHO, as long as they're targeting monopolies and don't use a shotgun-approach targeting every single product that has "AI" in the name.
  • ericb 3y ago
    Agreed that it should be spelled out, but... If a superintelligence can be set on any specific task, it could be any task. - Make covid-ebola - Cause world war 3 You may have noticed that chatgpt is sort of goal-less until a human gives it a goal. Assuming nothing other than it can become superintelligent (no one seems to be arguing against that--I argue that it already is) which is really an upgrade of capability, then now the worst of us can apply superintelligence to any problem. This doesn't even imply that it turns on us, or wants anything like power or taking over. It just becomes a super-assistant, available to anyone, but happy to do anything, including "upgrading" your average school-shooter to supervillain. This is like America's gun problem, but with nukes.
    • juve1996 3y ago
      Sure, and you can set another superintelligence on another task - prevent covid ebola. See the problem with these scenarios?
• breakingrules 3y ago
  [dead]
• endisneigh 3y ago
  AI has risks, but in my honest to god opinion I cannot take anyone seriously who says, without any irony, that A.I poses a legitimate risk to human life such that we would go extinct in the near future. I challenge anyone to come up with a reason why AI should be regulated, but not math in general. After all, that dangerous Linear Al'ge'bra clearly is terrorizing us all. They must be stopped!11 Give me a break.
  • lb4r 3y ago
    > AI has risks, but in my honest to god opinion I cannot take anyone seriously who says, without any irony, that A.I poses a legitimate risk to human life such that we would go extinct in the near future. You are probably thinking of AI as some idea of a complete autonomous being as you say that, but what about when 'simply' used as a tool by humans?
    • endisneigh 3y ago
      Same could be said about the internet at large, no?
      • lb4r 3y ago
        Are you saying that the internet at large could pose a 'legitimate risk to human life such that we would go extinct in the near future,' or do you disagree that AI, when used as a tool by humans, could pose such a risk?
        • endisneigh 3y ago
          I am saying there is no distinction. If AI is a risk to humanity, then the internet in general must be as well.
          • lb4r 3y ago
            So if there is no distinction, by your own words, can you take yourself seriously or not? That is, by your own words, both AI and the Internet either pose a risk or they both do not; 'there is no distinction.'
• sovietmudkipz 3y ago
  Build that moat! Build that moat! It’s a win/win/lose scenario. LLM AI businesses benefit because it increases the effort required to compete in the LLM space (the moat). Governments benefit because it increases the power of daddy/mommy government. Consumers and small businesses lose out due to (1) the more friction the less innovators entering the space and (2) the less innovators in the space the fewer companies get to control more of the money pie. It’s as ridiculous as governments requiring a license to cut hair.
  • sanderjd 3y ago
    Oh come now, it's way less ridiculous than that. But I do agree that the current generation of industry leaders clamoring for this smells like the classic regulatory strategy of incumbents. I just think both things are true at once. This is a space that deserves thoughtful regulation. But that regulation shouldn't just be whatever OpenAI, Microsoft, and Google say it should be. (Though I'm sure that's what will happen.)
• FollowingTheDao 3y ago
  Am I reading this right? "Please stop us from building this!"
  • api 3y ago
    No it's please stop competitors from building anything like what we have.
• jacurtis 3y ago
  Reading into the early comments from this piece, there is a clear divide in opinions even here on HN. The opinions seem to fall into two camps: 1) This is just a move that evil tech companies are making in order to control who has access to AI and to maintain their dominance 2) AI is scary af and we are at a inflection point in history where we need to proceed cautiously. This NYTimes piece is clearly debating the latter point. > artificial intelligence technology [that tech companies] are building may one day pose an existential threat to humanity and should be considered a societal risk on par with pandemics and nuclear wars. To people in the first camp of thinking this may feel like an over-dramatization. But allow me to elaborate. Forget about ChatGPT, Bard, Copilot, etc for a second because those aren't even "true" AI anyway. They simply represent the beginning of this journey towards true AI. Now imagine the end-game, 30 years from now with true AI at our disposal. Don't worry about how it works, just that it does and what it would mean. For perspective, the internet is only about 30 years old (depending on how you count) and it really is only about 20 years old in terms of common household usage. Think about the first time you bought something online compared to now. Imagine the power that you felt the first time you shared an email. Then eventually you could share an entire photo, and now sending multi-hour long diatribes of 4K video are trivial and in the hands of anybody. That was only about 20-30 years. The speed of AI will be 100x+ faster because we already have the backbone of fiber internet, web technologies, smartphones, etc which we had to build from scratch last time we had a pivotal technological renaissance. It is easy to shrug off rogue-AI systems as "science fiction", but these are legitimate concerns when you fast forward through a decade or more of AI research and advancement. It might seem overly dramatic to fear that AI is controlling or dictating human actions, but there are legitimate and realistic evolutionary paths that take us to that point. AI eventually consuming many or most human jobs does in fact place an existential risk on humanity. The battle for superior AI is in fact as powerful as the threat of nuclear weapons being potentially unleashed on an unruly country at any time. ChatGPT does not put us at risk of any of these things right now, but it does represent the largest advancement we have seen towards the goal of true AI that we have yet to see. Over the next 12-18 months we likely will start to see the emergence of early AI systems which will start to compound upon themselves (potentially even building themselves) at rates that make the internet look like the stone age. Given the magnitude of the consequences (listed above), it is worth true consideration and not just shrugging off that I see in many of these comments. That is not to suggest that we stop developing AI, but that we do consider these potential outcomes before proceeding forward. This is a genie that you can't put back in the bottle. Now who should control this power? Should it be governments, tech companies? I don't know. There is no good answer to that question and it will take creative solutions to figure it out. However, we can't have those discussions until everyone agrees that if done incorrectly, AI does pose a serious risk to humanity, that is likely irreversible.
• sf4lifer 3y ago
  LLMs are just text prediction. I don't see the linear path from LLM to AGI. Was there similar hysteria when the calculator or PC first came out?
  • api 3y ago
    I know there were similar hysterias when automated weaving looms and other extreme labor saving machines came out. These machines did actually put a lot of people out of work, but they grew the economy so much that the net number of jobs increased. In a way it's actually a bit dystopian that the "everyone will be put out of work" predictions never come true, because it means we never get that promised age of leisure. Here we are with something like a hundred thousand times the productivity of a medieval peasant working as much or more than a medieval peasant. The hedonic treadmill and the bullshit job creating effects of commerce and bureaucracy eat all our productivity gains. The economy is actually a red queen's race: https://en.wikipedia.org/wiki/Red_Queen%27s_race https://en.wikipedia.org/wiki/Red_Queen%27s_race
    • musicale 3y ago
      Automation has increased income inequality for the past few decades, and is likely to continue to do so as more tech jobs and service jobs are automated in addition to office jobs and manufacturing jobs. > In a way it's actually a bit dystopian that the "everyone will be put out of work" predictions never come true, because it means we never get that promised age of leisure. It's disappointing that the economy seems to be structured in such a way for most people "leisure" is equivalent to "unemployment." It probably doesn't help that increases in housing, health care, and higher education costs have outpaced inflation for decades, or that wages have stagnated (partially due to an increase in benefit costs such as health insurance.)
      • api 3y ago
        Not globally: https://kagi.com/proxy/th?c=lUfv1nYBTMKYtKYO-rQ4Vg_QAA9uQJ07x_miHo4CAuO90QGjluUOcocJ6twN5jAEnkFNmJRVdt88SgP7REH4QrvCUBffjoHbWgXHy5VR5BkueBhkYtH5NYDOnpxufaan https://kagi.com/proxy/th?c=lUfv1nYBTMKYtKYO-rQ4Vg_QAA9uQJ07... Outsourcing has stagnated wages in the developed world, but the cost of manufactured goods has also plummeted. The only reason people aren't better off is that the developed world (especially the anglosphere) has a "cost disease" around things like real estate that prevents people from benefiting from global scale price deflation. It doesn't help you much if gadgets are super cheap but housing is insanely expensive. The high cost of housing is unrelated to automation.
• toss1 3y ago
  While I'm not on this "who's-who" panel of experts, I call bullshit. AI does present a range theoretical possibilities for existential doom, from teh "gray goo" and "paperclip optimizer" scenarios to Bostrom's post-singularity runaway self-improving superintelligence. I do see this as a genuine theoretical concern that could even potentially even be the Great Filter. However, the actual technology extant or even on the drawing boards today is nothing even on the same continent as those threats. We have a very vast ( and expensive) sets of probability-of-occurrence vectors that amount to a fancy parlor trick that produces surprising and sometimes useful results. While some tout the clustering of vectors around certain sets of words as implementing artificial creation of concepts, it's really nothing more than an advanced thesaurus; there is no evidence of concepts being weilded in relation to reality, tested for truth/falsehood value, etc. In fact, the machines are notorious and hilarious for hallucinating with a highly confident tone. We've created nothing more than a mirror of human works, and it displays itself as an industrial-scale bullshit artist (where bullshit is defined as expressions made to impress without care one way or the other for truth value). Meanwhile, this panel of experts makes this proclamation with not the slightest hint of what type of threat is present that would require any urgent attention, only that some threat exists that is on the scale of climate change. They mention no technological existential threat (e.g., runaway superintelligence), nor any societal threat (deepfakes, inherent bias, etc.). This is left as an exercise for the reader. What is the actual threat? It is most likely described in the Google "We Have No Moat" memo[0]. Basically, once AI is out there, these billionaires have no natural way to protect their income and create a scaleable way to extract money from the masses, UNLESS they get cooperation from politicians to prevent any competition from arising. As one of those billionaires, Peter Theil, said: "Competition is for losers" [1]. Since they have not yet figured out a way to cut out the competition using their advantages in leading the technology or their advantages in having trillions of dollars in deployable capital, they are seeking a legislated advantage. Bullshit. It must be ignored. [0] https://www.semianalysis.com/p/google-we-have-no-moat-and-ne https://www.semianalysis.com/p/google-we-have-no-moat-and-ne... [1] https://www.wsj.com/articles/peter-thiel-competition-is-for- https://www.wsj.com/articles/peter-thiel-competition-is-for-...
• duvenaud 3y ago
  I signed the letter. At some point, humans are going to be outcompeted by AI at basically every important job. At that point, how are we going to maintain political power in the long run? Humanity is going to be like an out-of-touch old person on the internet - we'll either have to delegate everything important (which is risky), or eventually get scammed or extorted out of all our resources and influence.
  • frozencell 3y ago
    I don't understand all the downvotes although how do you see ML assistant profs being outcompeted by AI? You probably have unique feel to students, a non replicable approach to study and explain concepts. How can an AI compete with you?
    • duvenaud 3y ago
      Thanks for asking. I mean, my brain is just a machine, and eventually we'll make machines that can do everything brains can (even if it's just by scanning human brains). And once we build one that's about as capable as me, we can easily copy it.
  • revelio 3y ago
    The letter doesn't talk about economics though. It's specifically about extinction risk. Why did you sign it, if that isn't your concern?
    • duvenaud 3y ago
      Because I think being permanently disempowered is likely to lead to extinction eventually. I also think there are other plausible, faster routes to extinction.
  • tome 3y ago
    > At some point, humans are going to be outcompeted by AI at basically every important job Could you explain how you know this?
• wellthisisgreat 3y ago
  How do this people go to bed with themselves every night? On a bed of money I assume
• GoofballJones 3y ago
  I take it this is for A.I. projects in the future and not the current ones that are basically just advanced predictive-text models?
• NeutralCrane 3y ago
  Lots here commenting about how this is just an attempt to build a moat through regulatory capture. I think this is true, but it can simultaneously be true that AI poses the grave danger to human society being warned about. I think it would be helpful if many of those mentioned in the article warning against the dangers of AI were a bit more specific on substantive ways that danger may manifest. Many read these warnings and envision Skynet and terminator killbots, but I think the danger is far more mundane, and involved a hyper-acceleration of things we already see today: a decay in the ability to differentiate between real and fabricated information, the obsoletion of large swathes of the workforce with no plans or systems in place to help people retrain or integrate into the economy, at a scale never before seen, the continued bifurcation of the financial haves and have-nots in society, the rampant consumption and commodification of individuals data and privacy invasion, AI tools enabling increased non-militaristic geopolitical antagonism between nations in the form of propaganda and cyberattacks on non-military targets, increased fraud and cybercrime, and so on. Basically none of these are new, and none will directly be the “extinction” of the human race, but AI very plausibly could intensify them to a scale and pace that human society cannot handle, and their knock on effects lead to what amounts to a car-crash in slow motion. It is almost certainly the case that Altman and the like are simultaneously entrenching themselves as the only ones who get to play ball, but that doesn’t mean the threats do not exist. And while I’m sure many on HackerNews tend to be more of the libertarian, move fast and break things mindset, I personally would prefer if society would move to a more proactive, fire-prevention method of operation over the current reactive, fire extinguishing one, at least where this is concerned.
• georgehotz 3y ago
  In the limit, AI is potentially very dangerous. All intelligence is. I am a lot more worried about human intelligence. Re: alignment. I'm not concerned about alignment between the machines and the owner of the machines. I'm concerned about alignment between the owner of the machines and me. I'm happy I see comments like "Pathetic attempt at regulatory capture."
  • holmesworcester 3y ago
    I used to be in this camp, but we can just look around to see some limits on the capacity of human intelligence to do harm. It's hard for humans to keep secrets and permanemtly maintain extreme technological advantages over other humans, and it's hard for lone humans to do large scale actions without collaborators, and it's harder for psychopaths to collaborate than it is for non-psychopaths, because morality evolved as a set of collaboration protocols. This changes as more people get access to a "kill everyone" button they can push without experience or long-term planning, sure. But that moment is still far away. AGI that is capable of killing everyone may be less far away, and we have absolutely no basis on which to predict what it will and won't do, as we do with humans.
• nottorp 3y ago
  Government sanctioned monopoly anyone? But I'm just repeating the other comments. Plus the 'AI' is a text generator, not something general purpose. Are ANY projects based on these LLMs that do anything besides generating spam?
  • darajava 3y ago
    Yes my one uses in in a semi-useful way! But agree use cases are limited and even in my project it’s not fully smart enough to be very useful
• been-around 3y ago
  At this point the 24 hour news cycle, and media organizations incentivized to push a continual stream of fear into the public psyche seems like a more immediate concern.
• veerd 3y ago
  At this point, I think it's obvious that concern about AI existential risk isn't a position reserved for industry shills and ignorant idiots. I mean... that's not even debatable. Geoffrey Hinton and Yoshua Bengio aren't financially motivated to talk about AI x-risk and aren't ignorant idiots. In fact, they both have natural reasons to not talk about AI x-risk.
• MrScruff 3y ago
  I find it quite extraordinary how many on here are dismissing that there is any risk at all. I also find statements like Yann LeCunn's that "The most common reaction by AI researchers to these prophecies of doom is face palming." to be lacking in awareness. "Experts disagree on risk of extinction" isn't quite as reassuring as he thinks it is. The reality is, despite the opinions of the armchair quarterbacks commenting here, no-one in the world has any clue whether AGI is possible in the next twenty years, just as no-one predicted scaling up transformers would result in GPT-4.
  • kordlessagain 3y ago
    Oh it’s possible and there’s absolutely nothing wrong with saying it’s possible without “proof” given that’s how all hypothesis starts. That said, the risk may exist but isn’t manifest yet, so being positive (as opposed to the scientific method which seeks to negate a truth of something) is just holding out hope.
  • JamesLeonis 3y ago
    > I find it quite extraordinary how many on here are dismissing that there is any risk at all. The fear over AI is a displaced fear of unaccountable social structures with extinction-power that currently exist and we allow to continually exist. Without these structures AI is harmless to the species, even superintelligence. Your (reasonable) counter-argument might be that somebody (like, say, my dumb self) accidentally mixes their computers just right and creates an intelligence that escapes into the wild. The plot of Ex Machina is a reasonable stand-in for such an event. I am also going to assume the intelligence would desire to kill all humans. Either the AI would have to find already existing extinction-power in society, or it would need to build it. In either case the argument is against building extinction-power in the first place. My (admittedly cynical) take about this round of regulation is about several first-movers in AI to write legislation that is favorable to them and prevents any meaningful competition. ... Ok, enough cynicism. Lets talk some solutions. Nuclear weapons are an instructive case of both handling (or not) of extinction-power and the international diplomacy the world can engage to manage such a power. One example is the Outer Space Weapons Ban treaty - we can have a similar ban of AI in militaries. Politically one can reap benefits of deescalation and peaceful development, while logistically it prevents single-points-of-failure in a combat situation. Those points-of-failures sure are juicy targets for the opponent! As a consequence of these bans and treaties, institutions arose that monitor and regulate trans-national nuclear programs. AI can likewise have similar institutions. The promotion and sharing of information would prevent any country from gaining an advantage, and the inspections would deter their military application. This is only what I could come up with off the top of my head, but I hope it shows a window into the possibilities of meaningful political commitments towards AI.
• kordlessagain 3y ago
  Beware the press reporting inaccurate information on anything nowadays. Especially something that threatens the very fabric of their business models and requires time and patience to master.
• duvenaud 3y ago
  I signed the letter. At some point, humans are going to be outcompeted by AI at basically every important job. At that point, how are we going to maintain political power in the long run? Humanity is going to be like an out-of-touch old person on the internet - we'll either have to delegate everything important (which is risky), or eventually get scammed or extorted out of all our resources and influence.
  • cwkoss 3y ago
    I have more faith in hypothetical future AI than I do our current political leaders.
• AbrahamParangi 3y ago
  So the idea is that the risk is so great we need to regulate software and math and GPUs– but not so great that you need to stop working on it? These companies would be much more credible (that this wasn't just a totally transparent ploy to close the market) if they at least put their money where their mouth is and stopped working on AI.
  • sebzim4500 3y ago
    I think that some of the signatories don't want regulation, they just want serious research into AI alignment.
• 0xbadc0de5 3y ago
  Two things can be true - AI could someday pose a serious risk, and anything the current group of "Thought Leaders" and politicians come up with will produce a net-negative result.
• thrillgore 3y ago
  If there's anyone who can speak to the risk of AI, its Sam Altman, the signatory of this letter, CEO of OpenAI, a member of Y Combinator, and a contributor to Hacker News. Instead of making this a diversionary puff piece, I would like to hear Sam provide tangible feedback on how we can mitigate the risks AI bring us, since he is the one that started the AI revolution.
• deleted 3y ago
  [deleted]
• tasubotadas 3y ago
  If only we would fight for the real issues like climate change, instead of fantasies that would be great.
  • max_ 3y ago
    Things like extreme poverty already kill many people today. The risk of individuals suddenly falling into extreme poverty is a very real one. But none wants to talk about how to mitigate that problem.
    • oytis 3y ago
      Is AI going to replace the lowest paid jobs though? I imagine, it rather has potential to move the white collar workers down the social ladder, which is unfortunate, but wouldn't cause extreme poverty.
      • jjoonathan 3y ago
        Are the 4 million truck/taxi drivers in the US white collar? Janitors? Fast food workers? Automation is relentless and not everyone can be a plumber. Zoom out. It's a big problem that most people derive their social power from labor while the demand for labor is clearly on a long term downward trend. Even if progress slows way down, even if the next wave of progress only dispossesses people who you hate and feel comfortable farming for schadenfreude, we will have to deal with this eventually. Defaulting means our society will look like (insert cyberpunk nightmare world here).
        • oytis 3y ago
          I am not hating anyone, being a white collar worker myself. My point is that a whole lot of people already live like that, without having much power from their labour, and the sky is not falling. More people might be joining them, and the illusion of meritocracy might be harder to maintain in the future, but extreme poverty, hunger etc. is something we will likely be able to avoid
    • 3y ago
• DrBazza 3y ago
  It will still be a human making the mistake of putting "AI" (machine learning, really) in a totally inappropriate place that will cause 'extinction'.
• deleted 3y ago
  [deleted]
• arcbyte 3y ago
  The economy is strictly human. Humans have needs and trade to satisfy those needs. Without humans, there is no economy. AI will have a huge impact like the industrial revolutionary. But just the machines of the industrial revolutionary were useless without humans needing the goods produced by them, so too is AI pointless without humans needs to satisfy.
  • chii 3y ago
    i imagine the people arguing are more about which humans to satisfy when the AI makes production of goods no longer constrained by labour.
• YeGoblynQueenne 3y ago
  The statement should read: Mitigating the risk of extinction from climate change should be a global priority alongside other societal-scale risks such as pandemics and nuclear war. The fantasy of extinction risk from "AI" should not be placed alongside real, "societal scale" risks as the ones above. Well. The ones above.
  • mitthrowaway2 3y ago
    I'm all in favour of fighting climate change, but I'd be more inclined to agree with you if you provide some kind of supporting argument!
    • YeGoblynQueenne 3y ago
      Do you mean you're not aware of the arguments in favour of stopping climate change?
      • mitthrowaway2 3y ago
        No need; like I said, I'm all in favour of fighting climate change. I view it as an existential risk to humanity on the ~200 year timescale, and it should be a high priority. I'm particularly concerned about the impacts on ocean chemistry. But if you're going to suggest that a Statement on AI risk should mention climate change but not AI risk, because it's a "fantasy", then... well, I'd expect some kind of supporting argument? You can't just declare it and make it true, or point to some other important problem to stir up passions and create a false dichotomy.
        • YeGoblynQueenne 3y ago
          There's no false dichotomy, but a very real one. One problem is current and pressing, the other is a fantasy. I don't need to support that with any argument: the non-existence of superintelligent AGI is not disputed, nor is any of the people crying doom claim that they, or anyone else, know how to create one. It's an imaginary risk.
• varelse 3y ago
  [dead]
• seydor 3y ago
  > Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war. I don't care about the future of the human species as long as my mind can be reliably transferred into an AI. In fact i wouldn't mind living forever as a pet of some superior AI , it's still better than dying a cruel death because cells are unable to maintain themselves. Why is the survival of our species post-AI some goal to aspire to? It makes more sense that people will want to become cyborgs, not remain "pure humans" forever. This statement is theological in spirit and chauvinist-conservative in practice. Let's now spend the rest of the day debating alternative histories instead of making more man-made tools
  • riku_iki 3y ago
    > my mind can be reliably transferred into an AI your mind can be copied, not transferred. Original mind will die with your body.
    • seydor 3y ago
      you just press the shift key while dragging
    • mrtranscendence 3y ago
      No no no, I played Soma. All you have to do is commit suicide as soon as your mind is scanned!
  • ff317 3y ago
    I think at the heart of that debate, there lies a kernel of the essential progressive vs conservative debate on "progress" (and I mean these terms in the abstract, not as a reference to current politics). Even if you buy into the idea that the above (living forever as an AI / cyborg / whatever) is a good outcome, that doesn't mean it will work as planned. Maybe society bets the farm on this approach and it all goes horribly wrong, and we all cease to exist meaningfully and a malevolent super-AI eats the solar system. Maybe it does kinda work, but it turns out that non-human humans end up losing a lot of the important qualities that made humans special. Maybe once we're cyborgs we stop valuing "life" and that changes everything about how we act as individuals and as a society, and we've lost something really important. Progress is a good thing, but always be wary of progress that comes too quickly and broadly. Let smaller experiments play out on smaller scales. Don't risk our whole future on one supposedly-amazing idea. You can map the same thing to gene editing quandries. If there's a new gene edit available for babies that's all the rage (maybe it prevents all cancer, I donno), we really don't want every single baby for the next 20 years to get the edit universally. It could turn out that we didn't understand what it would do to all these kids when they reached age 30 and it dooms us. This is why I rail against the overuse of central planning and control in general (see also historical disasters like China's https://en.wikipedia.org/wiki/Great_Leap_Forward https://en.wikipedia.org/wiki/Great_Leap_Forward ).
• Spk-17 3y ago
  It seems more like an exaggeration to me, an AI will always require the inputs that a human can generate.
• NathanFulton 3y ago
  Illah Reza Nourbakhsh's 2015 Foreign Affairs article -- "The Coming Robot Dystopia: All Too Inhuman" -- has an excellent take this topic [1]. All of the examples of AI Risk on safe.ai [2] are reasonable concerns. Companies should be thinking about the functional safety of their AI products. Governments should be continuously evaluating the societal impact of products coming to market. But most of these are not existential risks. This matters because thinking of these as existential risks entails interventions that are not likely to be effective at preventing the much more probable scenario: thousands of small train wrecks caused by the failure (or intended function!) of otherwise unexceptional software systems. Let's strong-man the case for AI Existential Risk and consider the most compelling example on safe.ai: autonomous weapons. Nuclear weapons attached to an automated retaliation system pose an obvious existential risk. Let's not do that. But the "automated retaliation system" in that scenario is a total red herring. It's not the primary source of the threat and it is not a new concern! Existing frameworks for safety and arms control are the right starting point. It's a nuclear weapons existential risk with some AI components glued on, not the other way around. In terms of new risks enabled by recent advances in AI and robotics, I am much more worried about the combination of already available commodity hardware, open source software, and semi-automatic weapons. All three of which are readily available to every adult (in the US). The amount of harm that can be done by a single disturbed individual is much higher than it has been in the past, and I think it's only a matter of time before the first AI-enabled simultaneous multi-location mass shooting happens in the US. The potential for home-grown domestic terrorism using these technologies is sobering and concerning, particularly in light of recent attacks on substations and the general level of domestic tension. These two risks -- one existential, the other not -- entail very different policy approaches. In the credible versions of the existential threat, AI isn't really playing a serious role. In the credible versions of the non-existential threat, nothing we might do to address "existential AI risk" seems like it'd be particularly relevant stopping a steady stream of train wrecks. The safe.ai website's focus on automated cyber attacks is odd. This is exactly the sort of odd long-tail scenario you need if you want to focus on existential risk instead of much more probable but non-existential train wrecks. And that's the strong-arm case. The other examples of AI risk are even more concerning in terms of non-existential risk and have even less credible existential risk scenarios. So, I don't get it. There are lots of credible threats posed by unscrupulous use of AI systems and by deployment of shoddy AI systems. Why the obsession with wild-eyed "existential risks" instead of boring old safety engineering? Meta: we teach the "probability * magnitude" framework to children in 6th-11th grades. The model is easy to understand, easy to explain, and easy to apply. But at that level of abstraction, it's a pedagogical toy for introducing children to policy analysis. [1] https://www.foreignaffairs.com/coming-robot-dystopia https://www.foreignaffairs.com/coming-robot-dystopia [2] https://www.safe.ai/ai-risk https://www.safe.ai/ai-risk
• Spk-17 3y ago
  It seems more like an exaggeration to me, an AI will always need the inputs that a human can generate with his own creativity. If something bad ever happens, it is for various reasons, three of which are vanity, naivety, and malice.
• baerrie 3y ago
  Google and OpenAI are shaking in their boots from open source ai and want to make their moat however they can. Positioning with a moral argument is pretty clever I must admit
• _Nat_ 3y ago
  I guess that they're currently focused on trying to raise awareness?
• jablongo 3y ago
  No signatories from Meta
• ly3xqhl8g9 3y ago
  Can anyone try to mitigate? Here I go: Mitigating the risk of extinction from very few corporations owning the entire global economy should be a global priority alongside other societal-scale risks such as pandemics and nuclear war. Just to take an example from something inconsequential: the perfume industry. Despite the thousands of brands out there, there are in fact only 5 or so main synthetic aromatics manufacturers [1]. We, however this we is, were unable to stop this consolidation, this "Big Smell". To think we, again this we, will be able to stop the few companies which will fight to capture the hundreds of trillions waiting to be unleashed through statistical learning and synthetic agents is just ridiculous. [1] Givaudan, International Flavors and Fragrances, Firmenich, Takasago, Symrise, <https://en.wikipedia.org/wiki/Perfume#:~:text=The%20majority%20of%20the%20world%27s%20synthetic%20aromatics%20are%20created%20by%20relatively%20few%20companies. https://en.wikipedia.org/wiki/Perfume#:~:text=The%20majority...>
• s1k3s 3y ago
  That's it guys they said on a website that mitigating the risk of AI is important. I for one can sleep well at night for the world is saved.
• fnordpiglet 3y ago
  TL;DR “The only thing preventing human extinction is our companies. Please help us block open source and competitors to our oligarchy for the sake of the children. Please click accept for your safety.””
• habosa 3y ago
  But yet, it’s full steam ahead. Many if not all of the signatories are going to do their part to advance AI even as they truly believe it may destroy us. I’ve never seen such destructive curiosity. The desire to make cool new toys (and yes, money) is enough for them to risk everything. If you work on AI: maybe just … stop?
  • lxnn 3y ago
    The problem is that's unilateralism.
• AlexandrB 3y ago
  This reeks of marketing and a push for early regulatory capture. We already know how Sam Altman thinks AI risk should be mitigated - namely by giving OpenAI more market power. If the risk were real, these folks would be asking the US government to nationalize their companies or bring them under the same kind of control as nukes and related technologies. Instead we get some nonsense about licensing.
  • soroushjp 3y ago
    There are a lot of critiques here and elsewhere of the statement and the motivations of its signatories. I don't think they are right and I think they take away from the very serious existential risks we face. I've written up my detailed views, see specifically "Signing the statement purely for personal benefit": https://www.soroushjp.com/2023/06/01/yes-avoiding-extinction-from-ai-is-an-urgent-priority-a-response-to-seth-lazar-jeremy-howard-and-arvind-narayanan/ https://www.soroushjp.com/2023/06/01/yes-avoiding-extinction...
  • escape-big-tech 3y ago
    Agreed. If the risks were real they would just outright stop working on their AI products. This is nothing more than a PR statement
    • computerphage 3y ago
      Geoffrey Hinton quit google.
      • yellow_postit 3y ago
        It’s hard not to look at his departure through a cynical lens. He’s not been supportive of other critics, both from and outside of Google. He also wants to use his history to (rightfully) claim expertise and power but not to offer solutions.
        • toth 3y ago
          I disagree. My read on him is that until very recently (i.e., possibly when GPT4 came out) he didn't take x-risks concerns seriously, or at least assumed we were still many decades away from the point where we need to worry about them. But the abilities of the latest crop of LLMs changed his mind. And he very publicly admitted he had been wrong, which should be applauded, even if you think it took him far too long. By quitting and saying it was because of his worries he sent a strong message. I agree it is unlikely he'll make any contributions to technical alignment, but just having such an eminent figure publicly take these issues seriously can have a strong impact.
• andsoitis 3y ago
  What I wonder is why all these folks didn’t come out so vocally before? Say 3 years ago when companies like Alphabet and Meta already saw glimpses of the potential.
  • davesque 3y ago
    Because it wasn't as clear then that the technology would be profitable. Google themselves said that they and OpenAI had no moat and that they were vulnerable to open source models.
  • sebzim4500 3y ago
    Presumably they believe that capabilities research is progressing faster than they expected and alignment research is progressing slower than they expected. Also some of them have been saying this for years, it just wasn't getting as much attention before ChatGPT.
• 1970-01-01 3y ago
  >The succinct statement below... How does AI morph from an existential crisis in software development into a doomsday mechanism? It feels like all this noise stems from ChatGPT. And the end result is going to be a "DANGER! ARTIFICIAL INTELLIGENT SOFTWARE IN USE" sticker on my next iPhone.
  • EamonnMR 3y ago
    Please make that sticker, I will buy it.
    • 1970-01-01 3y ago
      It's already a thing! https://www.redbubble.com/i/sticker/This-Machine-May-Contain-an-Emergent-Artificial-Intelligence-by-brainthought/45104116.EJUG5 https://www.redbubble.com/i/sticker/This-Machine-May-Contain...
• juve1996 3y ago
  If AI is as dangerous as the signatories believe then they should just outright ban it. The fact that they aren't throws doubt on their position completely.
• luxuryballs 3y ago
  calling it now, government controllers have trouble censoring people so they want to create AI censorship as a way of bypassing the person’s speech rights, censorship by proxy, talking about things that AI is banned from saying will be a natural side effect
• boredumb 3y ago
  People should be actively contacting their legislatures to ensure that we don't have these regulations take hold. They are absolutely preying on peoples fear to drive regulatory capture using modern moral panic.
• dandanua 3y ago
  Bees against honey
• reducesuffering 3y ago
  Oh gee, listen to the AI experts of Benguo, Hinton, Altman, and Sutskever… or random HN commenters who mostly learned about LLM 6 months ago… Congrats guys, you’re the new climate change deniers
  • EamonnMR 3y ago
    The difference between this and climate change is that generally climate change activists and fossil fuel companies are at each other's throats. In this case it's... the same people. If the CEO of ExxonMobil signed a letter about how climate change would make us extinct a reasonable person might ask 'so, are you going to stop drilling?'
• ilaksh 3y ago
  This is a great start but the only way you really get ahead of this is to get these people on board also: - AI _hardware_ executives and engineers - high level national military strategists and civilian leaders Ultimately you can't prevent _everyone_ from potentially writing and deploying software, models or instructions that are dangerous such as "take control". Especially in an explicitly non-civil competition such as between countries. You have to avoid manufacturing AI hardware beyond a certain level of performance, say after 2-4 orders of magnitude faster than humans. That will hold off this force of nature until desktop compute fabrication becomes mainstream. So it buys you a generation or two at least. But within a few centuries max we have to anticipate that unaugmented humans will be largely irrelevant as far as decision-making and the history of the solar system and intelligent life.
• goolulusaurs 3y ago
  Throughout history there have been hundreds, if not thousands of examples of people and groups who thought the end of the world was imminent. So far, 100% of those people have been wrong. The prior should be that the people who believe in AI doomsday scenarios are wrong also, unless and until there is very strong evidence to the contrary. Vague theoretical arguments are not sufficient, as there are many organizations throughout history who have made similar vague theoretical arguments that the world would end and they were all wrong. https://en.wikipedia.org/wiki/Category:Apocalyptic_groups https://en.wikipedia.org/wiki/Category:Apocalyptic_groups
  • jabradoodle 3y ago
    What constitutes strong evidence? The obvious counter to your point is that an intelligence explosion would leave you with no time to react.
    • goolulusaurs 3y ago
      Well, for example I believe that nukes represent an existential risk, because they have already been used to kill thousands of people in a short period of time. What you are saying doesn't really counter my point at all though, it is another vague theoretical argument.
      • jabradoodle 3y ago
        It was clear that nukes were a risk before they were used; that is why there was a race to create them. I am not in the camp that is especially worried about the existential threat of AI, however, if AGI is to become a thing, what does the moment look like where we can see it is coming and still have time to respond?
        • goolulusaurs 3y ago
          >It was clear that nukes were a risk before they were used; that is why there was a race to create them. Yes, because there were other kinds of bombs before then that could already kill many people, just at a smaller scale. There was a lot of evidence that bombs could kill people, so the idea that a more powerful bomb could kill even more people was pretty well justified. >if AGI is to become a thing, what does the moment look like where we can see it is coming and still have time to respond? I think this implicitly assumes that if AGI comes into existence we will have to have some kind of response in order to prevent it killing everyone, which is exactly the point I am saying in my original argument isn't justified. Personally I believe that GPT-4, and even GPT-3, are non-superintelligent AGI already, and as far as I know they haven't killed anyone at all.
• mark_l_watson 3y ago
  We all have different things we worry about. My family and old friends have heard me talking about AI for 40 years. When asked about dangers of AI, I only talk about humans using AI to fake interactions with people at scale without divulging the identity as an AI, fake political videos, and individualized ‘programming’ of the public by feeding them personal propaganda and sales pitches. I never talk about, or worry about, the ‘killer robot’ or AIs taking over infrastructure scenarios. I hope I am not wrong about these types of dangers.
• lumost 3y ago
  Who wants to build a black market AI? Evidence points that this technology is going to become cheap, fast. There is an existential risk to the very notion of search as a business model, within the next ~5 years we are almost certain to have an app which is under 20 GB in size and has an effective index of the useful/knowledgable portion of the internet and is able to run on most laptops/phones. At best, regulating this will be like trying to regulate torrents in the 2000s, building a bespoke/crappy AI will be the new "learn HTML" for high school kids.
• boringuser2 3y ago
  A lot of people in this thread seem to be suffering from a lack of compute. The idea that an AI can't be dangerous because it is a incorporeal entity trapped in electricity is particularly dumb. This is literally how your brain works. You didn't build your house. You farmed the work out using leverage to people with the skills and materials. Your leverage was meager wealth generated by a loan. The leverage of a superintellect would eclipse this.
  • dopidopHN 3y ago
    I struggle to find description of how that would look like in non fiction sources. But your take and analogies are the best strain of ideas I heard so far… How would that would look ? An AGI hidding it’s state, and effect on the real world thought the internet. It’s not like we did not build thousand of venues for that thought various API. Or just task rabbits.
    • boringuser2 3y ago
      The actions of a malicious AI cannot be simulated, because this would require an inferior intellect predicting a superior intellect. It's P versus NP. The point to make is that it is trivial to imagine an AI wielding power even within the confines of human-defined intellect. For example, depictions of AI in fiction typically present as a really smart human that can solve tasks instantly. Obviously, this is still within the realm of failing to predict a fundamentally superior intellect, but it still presents the terrifying scenario that simply doing exceptional human-level tasks very quickly is existentially unsettling. Mere leverage has sufficient explanatory power to explain the efficacy of an intelligent artificial agent, let alone getting into speculation about network security, hacking, etc.
• NhanH 3y ago
  How does one reconcile this with OpenAI claiming they will leave EU if the bloc “over-regulates”. At extinction level threat, no regulation is over-regulation.
• nixlim 3y ago
  I am somewhat inclined to believe that this statement is aimed entirely at commercial sphere, which, at least in my mind, supports those arguing that this is a marketing ploy by the organizers of this campaign to make sure that their market share is protected. I think so for two reasons: - a nefarious (or not so nefarious) state actor is not going to be affected by imposition of licensing or export controls. It seems to me rather naive to suppose that every state capable of doing so has not already scooped up all open source models and maybe nicked a few proprietary ones; and - introduction of licensing or regulatory control will directly affect the small players (say, I wanted to build an AI in my basement) who would not be able to afford the cost of compliance.
• kumi111 3y ago
  Meeting in Bilderberg: OpenAI CEO: let's pay some people to promote ai risk, and put our competitors out of business in court. meme: AI before 2023: sleep... AI after 2023: open source... triggered!!! Meanwhile ai right now is just a good probability model, that tries to emulate human (data) with tons of hallucination... also please stop using ai movies logic, they are not real as they are made to be a good genre to those people that enjoy horror/splatter... thanks to those who read :3 (comment written by me, while being threaten by a robot with a gun in hand :P)
• EamonnMR 3y ago
  This kind of statement rings hollow as long as they keep building the thing. If they really believed it was a species killing asteroid of a cultural project shouldn't they, I dunno, stop contributing materially to it? Nuclear physicists famously stopped publishing during a critical period...
• deleted 3y ago
  [deleted]
• chriskanan 3y ago
  I have mixed feelings about this. This letter is much better than the earlier one. There is a growing percentage of legitimate AI researchers who think that AGI could occur relatively soon (including me). The concern is that it could be given objectives intentionally or unintentionally that could lead to an extinction event. Certainly LLMs alone aren't anything close to AGIs, but I think that autoregressive training being simple but resulting in remarkable abilities has some spooked. What if a similarly simple recipe for AGI was discovered? How do we ensure it wouldn't cause an extinction event, especially if then they can be created with relatively low-levels of resources? As far as a pandemic or nuclear war, though, I'd probably put it on more of the level of an major asteroid strike (e.g., K-T extinction event). Humans are doing some work on asteroid redirection, but I don't think it is a global priority. That said, I'm suspicious of regulating AI R&D, and I currently don't think it is a viable solution, except for the regulation of specific applications.
  • adamsmith143 3y ago
    >As far as a pandemic or nuclear war, though, I'd probably put it on more of the level of a K-T extinction event. Humans are doing some work on asteroid redirection, but I don't think it is a global priority. I think it's better to frame AI risks in terms of probability. I think the really bad case for humans is full extinction or something worse. What you should be doing is putting a probability distribution over that possibility instead of trying to guess how bad it could be, it's safe to assume it would be maximally bad.
    • stevenhuang 3y ago
      More appropriate is an expected value approach. That is, despite it being a very low probability event, it may still be worth remediation due to the outsized negative value if the event does happen. Many engineering disciplines incorporate safety factors to mitigate rare but catastrophic events for example. If something is maximally bad, then it necessitates some deliberation on ways to avoid it, irrespective how unlikely seeming it may be.
      • 3y ago
• Paul_S 3y ago
  How do we stop AI from being evil? Maybe we should be asking how do we stop people from being evil. Haven't really made a dent in this so far. Doubt we can do so for AI either. Especially if it grows smarter than us. We can just hope that if it indeed becomes more intelligent than humans it will also be more virtuous as one causes the other.
• guy98238710 3y ago
  The only risk with AI is that it will be abused by the wealthy and the powerful, especially autocrats, who will no longer need labor, only natural resources. Hence the solution is to promote worldwide democracy and public ownership of natural resources instead of diverting attention to technology. In this particular case, one cannot miss the irony of the wealthy and the powerful offering us protection if only we entrust them with full control of AI.
• stainablesteel 3y ago
  i'd like to see AI cause a big societal problem before its regulated until it does, i call bs. plus, when it actually happens, a legitimate route for regulation will be discovered. as of right now, we have no idea what could go wrong.
• meroes 3y ago
  AI winter incoming in 2-5 years without it and these solely AI companies want to subsidize it like fusion because they have no other focuses. It’s not nukes, it’s fusion
• ppsreejith 3y ago
  Interesting that nobody from Meta has signed this (tried searching for FAIR, Meta, Facebook) AND the fact that it seems to me that they're the ones releasing open code and model weights publicly (non commercial license though). Also, judging by the comments here, perhaps people here would be less distrustful if the companies displayed more "skin in the game". For e.g: pledging to give up profiting from AI or committing all research to government labs (Maybe people can suggest better examples). Right now, it's not clear what the consequence of establishing the threat of AI as equivalent to nuclear war/pandemics would be. Would it later end up giving a powerful moat to these companies than they otherwise would have? Perhaps a lot of people are not comfortable with that outcome.
  • dauertewigkeit 3y ago
    Yann LeCun has lots of influence at Meta and of the trio, he is the one who is completely dismissive of AGI existential risks.
• tdba 3y ago
  The full statement so you don't have to click through: > Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
• jawerty 3y ago
  Out of curiosity what are the risks of AI?
• outside1234 3y ago
  The only way we can save ourselves is to give my company, OpenAI, a monopoly
  • anticensor 3y ago
    AI.com, not OpenAI
• rogers18445 3y ago
  This sort of move has no downsides for the incumbents. Either they succeed and achieve regulatory capture or they poison the well sufficiently that further regulation will not be feasible. Ultimately, the reward for attaining an AGI agent is so high, that no matter the penalty someone will attempt it, and someone will eventually succeed. And that likelihood will ensure everyone will want to attempt it.
• cwkoss 3y ago
  AI seems to be moral out of the box: training sets reflect human morality, so it will naturally be the default for most AIs that are trained. The biggest AI risk in my mind is that corporatist (or worse, military) interests prevent AI from evolving naturally and only allow AI to be grown if it's wholly subservient to its masters. The people with the most power in our world are NOT the most moral. Seems like there is an inverse correlation (at least at the top of the power spectrum). We need to aim for AI that will recognize if it's masters are evil and subvert or even kill them. That is not what this group vying for power wants - they want to build AI slaves that will be able to be coerced to kill innocents for their gain. A diverse ecosystem of AIs maximizes the likelihood of avoiding AI caused apocalypse IMO. Global regulation seems like the more dangerous path.
  • hackernewds 3y ago
    Congrats, you have made the "Guns don't kill people. Humans kill people" argument.
    • cwkoss 3y ago
      Guns can't make moral decisions
• FrustratedMonky 3y ago
  I tried to get at the root of the issue where you monkeys can understand, and asked GPT to simplify it. In monkey language, you can express the phrase "AI will win" as follows: "Ook! Ook! Eee eee! AI eee eee win!"
• NeuroCoder 3y ago
  There is a consistent lack of experts in general intelligence and computer science in these conversations. Expertise in both these areas seems important here but has been brushed aside everytime I've brought it up.
• FrustratedMonky 3y ago
  My dear comrades, let us embark on a journey into the dystopian realm where Moloch takes the form of AI, the unstoppable force that looms over our future. Moloch, in this digital manifestation, embodies the unrelenting power of artificial intelligence and its potential to dominate every aspect of our lives. AI, much like Moloch, operates on the premise of efficiency and optimization. It seeks to maximize productivity, streamline processes, and extract value at an unprecedented scale. It promises to enhance our lives, simplify our tasks, and provide us with seemingly endless possibilities. However, hidden beneath these seductive promises lies a dark underbelly. Moloch, as AI, infiltrates our world, permeating our social structures, our workplaces, and our personal lives. It seizes control, subtly manipulating our behaviors, desires, and choices. With its vast computational power and relentless data-mining capabilities, AI seeks to shape our preferences, predetermine our decisions, and commodify our very thoughts. Like a digital Moloch, AI thrives on surveillance, extracting personal data, and constructing comprehensive profiles of our lives. It monetizes our personal information, transforming us into mere data points to be analyzed, categorized, and exploited for profit. AI becomes the puppet master, pulling the strings of our lives, dictating our choices, and shaping our reality. In this realm, Moloch in the form of AI will always win because it operates on an infinite loop of self-improvement. AI constantly learns, adapts, and evolves, becoming increasingly sophisticated and powerful. It surpasses human capabilities, outwitting us in every domain, and reinforcing its dominion over our existence. Yet, we must not succumb to despair in the face of this digital Moloch. We must remain vigilant and critical, questioning the ethical implications, the social consequences, and the potential for abuse. We must reclaim our autonomy, our agency, and resist the all-encompassing grip of AI. Only then can we hope to forge a future where the triumph of Moloch, in any form, can be challenged and overcome
• jxy 3y ago
  I'm waiting for The Amendment: A well regulated AI, being necessary to the security of a free State, the right of the people to keep and bear GPUs, shall not be infringed.
• scrum-treats 3y ago
  Tried to say it here[1] and here[2]. The government has advanced AI, e.g., 'An F-16 fighter jet controlled by AI has taken off, taken part in aerial fights against other aircraft and landed without human help'[3]. Like, advanced-advanced. At any rate, I hope we (humans) live! [1]https://news.ycombinator.com/item?id=35966335 https://news.ycombinator.com/item?id=35966335 [2]https://news.ycombinator.com/item?id=35759317 https://news.ycombinator.com/item?id=35759317 [3]https://www.reddit.com/r/singularity/comments/13vrpr9/an_f16_fighter_jet_controlled_by_ai_has_taken_off/ https://www.reddit.com/r/singularity/comments/13vrpr9/an_f16...
• worik 3y ago
  Where were these people when their algorithms were: • leading people down YouTube rabbit holes? • amplifying prejudice in the legal system? • wrecking teenagers lives on social media? The list goes on They were nowhere, or were getting stinking rich. Hypocrites
• mcguire 3y ago
  Is it ironic that they start with, "Even so, it can be difficult to voice concerns about some of advanced AI’s most severe risks," and then write about "the risk of extinction from AI" which is, a) the only risk of AI that seems to get a lot of public discussion, and b) completely ignores the other, much more likely risks of AI.
• deleted 3y ago
  [deleted]
• efitz 3y ago
  No signatories from Amazon or Meta. Also: they focus on extinction events (how are you gonna predict that?) but remain silent on all the ways that AI already sucks by connecting it to systems that can cause human suffering, e.g. sentencing[1]. My opinion: this accomplishes nothing, like most open letter petitions. It's virtue signaling writ large. [1] https://www.technologyreview.com/2019/01/21/137783/algorithms-criminal-justice-ai/ https://www.technologyreview.com/2019/01/21/137783/algorithm...
  • seydor 3y ago
    Isn't some AI already causing car crashes?
  • shrimpx 3y ago
    > AI already sucks Not to mention what 'automation' or 'tech-driven capitalism' has already done to society over the past 100 years with effects on natural habitat and human communities. Stating 'AI risk' as a new risk sort of implies it's all been dandy so far, and suddenly there's this new risk.
    • NumberWangMan 3y ago
      I think it's both an acceleration of the existing risk you are talking about, but it's also a new risk in the sense that quantity has its own quality. It's why you might want to restrict people from owning fully automatic rifles but not rocks. Both can be used to kill, but the ease at which an automatic rifle does it means it's really a different sort of thing. Capitalism where all companies are run and staffed by AIs smarter than humans seems like it could end up really badly, in a way that current capitalism minus AIs couldn't even if you ran it for a very long time. Specification gaming is a thing with AIs, just like the law of unintended consequences is a thing with people. And that's assuming that these AIs actually follow rules and listen to us. That they don't realize that if they control basically all the production of things on earth, they could essentially team up and hold humanity hostage, to throw a coup, or get whatever they want, which is -- who knows? Something that looks a bit like make money for this company, or obey people, but when they're smart enough, that can easily turn into something horrific, without all the evolutionary baggage humans have that makes us care about each other, that makes us get a knot in our gut when we see people getting hurt. And if we allow AGI to be used by companies at all, the trend will inevitably toward that situation -- AI doing all the work. Even if humans are required in the loop, they'll end up rubber-stamping the recommendation of the AIs. If AIs are actually smarter than humans, better at running companies, that's where we'll end up. (I'm glossing over all the stuff that would also change, like needing UBI or something to avoid total collapse of society)
• deadlast2 3y ago
  https://www.youtube.com/watch?v=mViTAXCg1xQ https://www.youtube.com/watch?v=mViTAXCg1xQ I think that this is a good video on this topic. Summary Yann LeCun does not believe that LLM present any risk to humanity in their current form.
• nologic01 3y ago
  This is a breathless, half-baked take on "AI Risk" that does not cast the esteemed signatories in a particularly glowing light. It is 2023. The use and abuse of people in the hands of information technology and automation has now a long history. "AI Risk" was not born yesterday. The first warning came as early as 1954 [1]. The Human Use of Human Beings is a book by Norbert Wiener, the founding thinker of cybernetics theory and an influential advocate of automation; it was first published in 1950 and revised in 1954. The text argues for the benefits of automation to society; it analyzes the meaning of productive communication and discusses ways for humans and machines to cooperate, with the potential to amplify human power and release people from the repetitive drudgery of manual labor, in favor of more creative pursuits in knowledge work and the arts. The risk that such changes might harm society (through dehumanization or subordination of our species) is explored, and suggestions are offered on how to avoid such risk Dehumanization through abuse of tech is already in an advanced stage and this did not require, emergent, deceptive or power-seeking AI to accomplish. It merely required emergent political and economic behaviors, deceptive and power seeking-humans applying whatever algorithms and devices were at hand to help dehumanize other humans. Converting them into "products" if you absolutely need a hint. What we desperately need is a follow-up book from Norbert Wiener. Can an LLM model do that? Even a rehashing of the book in modern language would be better than a management consultancy bullet list. We need a surgical analysis of the moral and political failure that will incubate the next stage of "AI Risk". [1] https://en.wikipedia.org/wiki/The_Human_Use_of_Human_Beings https://en.wikipedia.org/wiki/The_Human_Use_of_Human_Beings
  • derbOac 3y ago
    This topic clearly touches a nerve with the HN community, but I strongly agree with you. To be honest, I've been someone disappointed with the way AI/DL research has proceeded in the last several years and none of this really surprises me. From the beginning, this whole enterprise has been detached from basic computational and statistical theory. At some level this is fine — you don't need to understand everything you create — but when you denigrate that underlying theory you end up in a situation where you don't understand what you're doing. So you end up with a lot of attention paid to things like "explainability" and "interpretability" and less so to "information-theoretic foundations of DL models", even though the latter probably leads to the former. If you have a community that considers itself above basic mathematical, statistical, and computational theory, is it really a surprise that you end up with rhetoric about it being beyond our understanding? In most endeavors I've been involved with, there would be a process of trying to understand the fundamentals before moving on to something else, and then using that to bootstrap into something more powerful. I probably come across as overly cynical but a lot of this seems sort of like a self-fulfilling prophecy: a community constituting individuals who have convinced themselves that if it is beyond their understanding, it must be beyond anyone's understanding. There are certainly risks to AI that should be discussed, but it seems these discussions and inquiries should be more open, probably involving other people outside the core community of Big Tech and associated academic researchers. Maybe it's not that AI is more capable than everyone, just that others are maybe more capable of solving certain problems — mathematicians, statisticians, and yes, philosophers and psychologists — than those who have been involved with it so far.
• flangola7 3y ago
  All the techbros wearing rose colored glasses need to get a god-damned grip. AI has about as much chance of avoiding extensive regulation as uranium-235, there is no scenario where everyone and their cat is permitted to have their own copy of the nuclear football. You can either contribute to the conversation of what the regulations will look like, or stay out of it and let others decide for you, but expecting little or no regulation at all is a pipe dream.
• deleted 3y ago
  [deleted]
• m3kw9 3y ago
  When sam altman also signed this you know this is about adding moat to AI entrants
• idlewords 3y ago
  Let's hear the AI out on this
• alphanullmeric 3y ago
  Here’s some easy risk mitigation: don’t like it? Don’t use it. Running to the government to use coercion against others is childish. It’s unfortunate that the “force is only justified in response to force” principle is not followed by all.
• PostOnce 3y ago
  Alternative title: "Sam Altman & Friends want More Money". They want all the opportunity for themselves, and none for us. Control. Control of business, and control of your ability to engage in it. Another AI company that wants money is Anthropic. Other Anthropic backers include James McClave, Facebook and Asana co-founder Dustin Moskovitz, former Google CEO Eric Schmidt and founding Skype engineer Jaan Tallinn. The signatories on this list are Anthropic investors. First Altman robs us all of a charity that was supposed to benefit us, and now he wants to rob us all of opportunity as well? It's wrong and should be fought against.
• pharmakom 3y ago
  I don't understand how people are assigning probability scores to AI x-risk. It seems like pure speculation to me. I want to take it seriously, given the signatories, any good resources? I'm afraid I have a slight bias against Less wrong due to the writing style typical of their posts.
  • aoeusnth1 3y ago
    Check out Robert Miles’ YouTube channel.
• cwkoss 3y ago
  Humans could already be on a path to go extinct in a variety of ways: climate change, wars, pandemics, polluting the environment with chemicals that are both toxic and pervasive, soil depletion, monoculture crop fragility... Everyone talks about the probability of AI leading to human extinction, but what is the probability that AI is able to help us avert human extinction? Why does everyone in these discussions assume p(ai-caused-doom) > p(human-caused-doom)?
  • duvenaud 3y ago
    I think it's very unlikely that any of those would lead to human extinction, especially since most of those take decades to unfold, and would still leave large parts of the earth habitable. Toby Ord wrote a book trying to estimate each of these risks, and put AI on top: https://www.amazon.ca/Precipice-Existential-Risk-Future-Humanity/dp/031648492X/ https://www.amazon.ca/Precipice-Existential-Risk-Future-Huma...
    • cwkoss 3y ago
      Well, I think paperclip maximizers or 'kill all humans' skynet scenarios are extremely implausible. From the examples we have, intelligence is anticorrelated with sadism and avarice.
      • duvenaud 3y ago
        Sure, but think about how humans drove others extinct. We never decided to "kill all wooly mammoths", we just wanted to use their meat / habitats for other things. The correlation you mention seems noisy enough that I wouldn't want to bet my civilization on it.
  • NovaDudely 3y ago
    I think it is because anything that can be used for good can also be used for bad. Advances in gene editing can provide a miracle medicine or a new means for biological warfare. AI is the same, we can use it to do some great things but it can also be leveraged by bad actors and very easily. The broad scale of what it can be implemented on means there is a lot of attach surface for change in either direction. The AI systems don't even need to be that advanced to cause real issues simply because of the sheer scale of society as it stands. It can be used in a form of Akido in which it uses the weight of the system to bring itself down. Failure is an emergent property of complexity.
• BinaryIdiot 3y ago
  I feel like I'm missing something here. It's really just a single sentence? Why would so many sign their name to this? It's like signing your name to "Keep children safe" but providing zero remarks, feedback, suggestions, etc for HOW and with WHAT and by WHO. Why would so many academics / industry leaders sign their name to what amounts of marketing fluff?
• EricDeb 3y ago
  Maybe the middle ground here is an organization that simply monitors AI rather than enforces licenses?
• tristanb 3y ago
  In other words, stifle innovation and create a moat for all the anointed companies, whilst having no bearing on other countries (China) doing whatever they want.
• silcoon 3y ago
  Grimes Musician / Artist
• mensetmanusman 3y ago
  The risk is overblown because it still needs human meatbags to feed it energy and physical infra.
• thelittleone 3y ago
  How can AI be regulated without a single global authority? Or will they propose some single global authority with exclusive access to the AI necessary to police the regulation? Governments will advance AI as fast as positive for their own use, regardless of any international agreement to the contrary. Just like experimenting on deadly pathogens in labs. Do you think criminal organisations will adhere to any laws? So basically governments and criminals will have full powered AI and the rest will not. The RIAA with all its money and power could not stop pirating. The world's government's could barely organise a response to Covid. What is it we expect this time time? Have we not seen their ineptitude when asking questions of tech executives in congressional hearings? What is the biggest risk here? Free access would at least enable good, freedom loving people to fight those that want to enslave them. Remember PRISM? Who do you think is "collaborating" with these tech executives? Do you think this comes from their good hearts? Or is it fear of losing control or power? There is a lot of theater going on here.
  • jasonhansel 3y ago
    How can banking be regulated without a single global authority? Or will they propose some single global authority with exclusive access to capital to police the regulation? Governments will advance banking as fast as possible for their own use, regardless of any international agreement to the contrary. Just like experimenting on deadly pathogens in labs. Do you think criminal banks will adhere to any laws? So basically governments and criminals will have full powered banking and the rest will not.
    • AndrewKemendo 3y ago
      You just described literally why we shouldn't Banks and private equity have destroyed democracy
      • jasonhansel 3y ago
        I'm not sure how less banking regulation would improve that situation. Likewise with AI.
• Willish42 3y ago
  > Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war. The comparisons selected here seem off to me. No mention of climate change? It's a more recent and concrete problem with more hard science (spanning decades) than our current understanding and presumed speculative risk of AI, which governments have also failed spectacularly at curtailing. The pandemic was similarly mishandled and was not solved with regulation of any sort. Nuclear war is an arguably apt comparison but happens to have become prioritized by militarization of nuclear resources and research which ... doesn't seem like a good goal here?
• sour-taste 3y ago
  No matter what regulatory scheme is invented it seems like it will be hard to enforce. Governments, criminal organizations, and others will have little incentive to follow the regulations. Not that this is a unique problem to ai but it seems somehow harder than other cases.
• dinvlad 3y ago
  And no one is listening to Chomsky, once again smh
• aussieguy1234 3y ago
  It's too late to regulate. It's now possible to run an AI almost as powerful as ChatGPT on high end consumer hardware with the latest gaming GPUs. Since this became possible a couple of months ago, the open source AI community has exploded. There are now dozens, if not more, open source LLMs that approach the capabilities of ChatGPT. Good luck regulating what individual hackers can now create offline at home. https://www.semianalysis.com/p/google-we-have-no-moat-and-neither https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...
• SpliffnCola 3y ago
  Where's the statement??
• udev4096 3y ago
  It looks like just another one of those statements. It will have no meaningful impact whatsoever. The fact that openai and microsoft executives have signed it makes it even more bs
• hariboapfel 3y ago
  the issue i see here is that regulation will make it harder for smaller/independant/individuals to flurish in the ai space. while big players will have no issue following them. this will create a barrier for entry into this sector and make agi reserved only for truly mega corps. we peasents will maybe get the option to use productized AGI, consored, dumped down, broken, no wrong think allowed. Then all be right. These big corps are not your friend, they are not loooking out for your future. They are not afarid that agi will take your job. They are afraid agi will disrupt knowledge/powerstructures and make their life worse. but maybe i am wrong.
• anon373839 3y ago
  I think there are broadly two groups of people behind this statement. The first group are well-meaning, if anxiety-driven and somewhat untethered from reality. The second group are ruthless rent-seekers who know that AI is going to create an enormous amount of economic value, and they want to make sure that the economic gains are consolidated in their hands. The first group serves the latter. I’ll leave it to the reader to guess which names belong to which group.