14 ms·

The illusion of AI’s existential risk

58 points by headalgorithm 3y ago

grantcas 3y ago
[dead]
martythemaniak 3y ago
We live in a very confusing, transitory time. Right now there's a large chunk of people who are convinced that we are a few years away from extinction if we do not ration out GPUs and bomb illegal clusters of compute, while at the same time there's a large chunk that believes that a car can never navigate a single-lane one-way tunnel safely without a driver (ie, The Boring Company). Absolutely wild.
- hollerith 3y ago
  "Convinced that we are a few years from extinction," is an exaggeration of the pessimistic/alarmed end of the spectrum of opinion. All the forecasts I've seen have the period of danger spread out over at least 20 or 30 years, and no one claims to understand the topic well enough to be able to tell whether a particular model is dangerous even if given complete freedom to examine the source code and interview the creators. Our inability to predict is in large part the basis of the pessimism/alarm because it means that the leading labs won't know when to stop.
  - acqq 3y ago
    Exactly. And here is an example of that inability in the much clearer case: "How many years after 1945 ... would it take for the Soviet Union to get the bomb? Framed this way, it was a much easier question than forecasting the advent of AGI: The Americans were concerned with only one adversary and the development of a single technology which they already knew all the details of. American predictions of Soviet proliferation offer a highly constrained case study for those who would undertake technological forecasting today." https://asteriskmag.com/issues/03/how-long-until-armageddon https://asteriskmag.com/issues/03/how-long-until-armageddon
  - ozr 3y ago
    > "Convinced that we are a few years from extinction," is an exaggeration of the pessimistic/alarmed end of the spectrum of opinion. It really isn't. There are people quitting their jobs, protesting AI companies, and inducing panic attacks in children over their irrational fears: https://www.wired.com/story/pause-ai-existential-risk/ https://www.wired.com/story/pause-ai-existential-risk/
    - yanderekko 3y ago
      No, even the more-panicked wing is not betting the proverbial house on AI killing us within the next few years, although the possibility is there. The "don't expect to live much longer" sentiment is based on a probability horizon that extends over decades.
      - ozr 3y ago
        I've spoken to people in the article. They expect doom in <1 year.
        hollerith 3y ago
        Can you give a name so I don't have to read a long article? It is easy to find people who believe that it is possible (with some low probability) that AI will end the human race in the next 12 mo. (I'm one of them.) I haven't seen anyone however put most of their probability mass in the next 12 or 24 mo, and I'd like to find out what their reasoning is.
        ozr 3y ago
        Sure, first name in the article I posted, Joep Meindertsma. They have an open discord you can join, but fair warning that it's entirely panicky people lamenting their inevitable doom.
        yanderekko 3y ago
        I just don't believe that unless you have specific quotes you can point to that indicates this.
- RC_ITR 3y ago
  > a car can never navigate a single-lane one-way tunnel safely without a driver (ie, The Boring Company). I think the criticisms of the Boring Company are more "small diameter tunnels are a bad way to build high-throughput underground infrastructure, and autonomous cars will not solve that"
  - john2x 3y ago
    "small diameter tunnels" + "autonomous cars" kind of sounds like a subway train.
    - sgu999 3y ago
      Just add tracks, weld the cars together, remove the driving seats, replace the fancy NNs by vintage fuzzy logic and you're there!
    - RC_ITR 3y ago
      Sure, but have you ever been on the London Tube and noticed it's notoriously small and uncomfortable in parts? Have you also noticed how windy some stations get because the small trains fill most of the tube and act like an air piston? Those tunnels are 12 feet diameter. Boring Co's tunnels are 12 feet diameter. A modern subway system needs considerably larger tunnels.
- version_five 3y ago
  I've only seen grifters, hysterics, or the ignorant talking about existential AI risk. A lot of people just nods their heads and go along, but I don't think there are many that have really thought about it (and understand what modern AI is) that really belive the risk. It's more like political polarization where people dwell on something they don't agree with to the point where they treat it as if the world is going to end because of it, instead of in context. AI is more like the political party they don't like winning, potentially undesirable from a certain point of view but they see it as "existential". I think it's important to see it in this light, instead of trying to actually debate the points raised, which are absurd and will only give credibility where none is due.
  - jefftk 3y ago
    > I've only seen grifters, hysterics, or the ignorant talking about existential AI risk. Which category would you put Geoffrey Hinton in? https://www.utoronto.ca/news/risks-artificial-intelligence-must-be-considered-technology-evolves-geoffrey-hinton https://www.utoronto.ca/news/risks-artificial-intelligence-m...
    - cristiancavalli 3y ago
      hysterics. Remember when he said we should stop training radiologists, that’s aged pretty poorly given there’s actually a need to have more: https://www.rsna.org/news/2022/may/global-radiologist-shortage https://www.rsna.org/news/2022/may/global-radiologist-shorta... https://mindmatters.ai/2022/08/turns-out-computers-are-not-vacuuming-up-all-our-jobs/ https://mindmatters.ai/2022/08/turns-out-computers-are-not-v...
      - mptest 3y ago
        So because he made one(+) bad prediction, all future predictions are inherently poor? It seems that may be a tad of a leap? Where is the line between "poor prediction" and "hysterics"? "Deep learning will replace radiologists within 5 years" isn't as inherently crazy as it probably seems in hindsight. Sure, it didn't pan out, but I know a vet who works for an Ai company classifying scans for training and he says first hand the models have gotten a lot better. That said, he's still not worried about being replaced.
ggm 3y ago
AGI doesn't ever have to happen for rapid deployment of AI based systems to cause harm. Google "robodebt australia" for what happens when government uses machine derived decisions to penalise the poor.
- __loam 3y ago
  That point is basically what the article is about.
  - ggm 3y ago
    It can't be repeated enough apparently, because people still whine on about skymesh and "but IFF the robots wake up" when the real risk is what people believe machine derived "facts" are (I quote facts because they are highly conjectural inferences from models, and often diverge from fact pretty rapidly)
    - yanderekko 3y ago
      >when the real risk is what people believe machine derived "facts" are It's hard to play out a scenario where this leads to actual human extinction.
- QuantumG 3y ago
  aussie here, former dole bludger, still know people who live off the welfare system, hide their side income, and are proud of it. There was sooooo much warning for robodebt that they all made sure their taxes were clean before the first demands went out, but a few still got through. Mostly accurate, but a few dodgy ones too. Of course, as we know, it was a dragnet and the human suffering produced was excessive. Considering the depth of the corruption, it was clear the system was incapable of correcting itself, so what else is there to do? How do you get help to the people with real need when there's so many people cheating? Robodebt was big hammer approach.
janalsncm 3y ago
I don’t understand why economic disruption and human obsolescence isn’t considered an existential risk. We could end up in a world where 99.99% of people are redundant and useless cost centers for our GDP maximizing economic paradigm. In that case, you don’t need killer robots in order to push humanity towards near-extinction. The invisible hand will smite them. But don’t worry, TED Talk attendee. The obsoletariat mostly won’t be Americans. The median member of this class will probably be Chinese or Indian. So you can continue your performative concerns about Roko’s Basillisk or whatever topic of distant, paralyzing uncertainty is overflowing from the next room into mine.
- ke88y 3y ago
  Maybe this is why billionaires and hundred-millionaires are the ones so concerned about existential risk. A collapse of consensus in our current and historically not-that-old system of property rights is an existential risk -- to them.
- deleted 3y ago
  [deleted]
- __loam 3y ago
  This is a problem with capitalism in general, not just with AI.
  - version_five 3y ago
    We're seeing this happen a lot with "AI" - now that it's a super popular topic people use it as the lens though which they project their general political grievances. It gets more attention.
  - janalsncm 3y ago
    Well, it’s a problem with any economic system which uses labor as a proxy for value/rights. If my political power is predicated on my ability to withdraw labor or move to a new country, it’s not going to be good for me when my services are no longer required.
  - 3y ago
hashstring 3y ago
Yes, this is a good take. The general discussion is definitely prematurely focused on some sort of end-boss fight, while AI and large scale data collection are causing serious harm and privacy concerns to real humans already. It’s only reasonable to expect that this increases in the future. I would like to see more discussion focussing on these pressing issues. Also a pause on AI development is not an option, not a solution and is digressing from the issue at hand. Big capital with their hands on big valuable data will support anything that distracts here. Finally, I do think that AI also brings many positives, I am not against the technology itself at all- we shouldn’t be.
- radicalace 3y ago
  name a single reason why a pause is not an option. dont say why it would be hard or unlikely or inconvenient or whatever, to exclude it as a possibility means it has to be physically impossible. of course it isnt physically impossible. AI requires large compute, large resources. you cant advance the state of the art inconspicuously. the only reason you say it isnt an option is because you dont consider AI an existential threat, really, and therefore extreme measures arent justified. but they actually are.
  - nradov 3y ago
    That's not even a valid request. There is obviously no valid reason to require such a pause in the first place. You've been watching too many scifi movies and don't understand the technology.
  - hashstring 3y ago
    We are already facing numerous existential threats at the moment, even ones that are deliberately made to be an existential threat, take nuclear weapons. “Hey fellow nation states, let’s not create nuclear bombs, but let’s pause developments and keep the current power dynamics fixed please” is completely unrealistic. Mind that creating nuclear weapons actually requires technical operations that are factors more complex than “large compute” in many aspects… like… enriching uranium and working with supercritical masses. > you cant advance the state of the art inconspicuously. What evidence has led you to believe this?
more_corn 3y ago
The distant future. Like five years. Self-aware (or a simulation thereof) AGI with self-directed motivation and the ability to self modify is barely a half step from ASI. And AGI could happen in the coming year. Five if you wanna be pessimistic. Never if you’re extraordinarily pessimistic but that’s looking unlikely. If it’s possible it’s possible now. This isn’t the Yellowstone super volcano that might blow in the next billion years. People who talk like this are idiotic at best. Granted, the other concerns are real too but the accusation that the concern of existential risk is being used to hide the other known problems is dangerous in the extreme.
- JimtheCoder 3y ago
  "And AGI could happen in the coming year." "People who talk like this are idiotic at best."
  - __loam 3y ago
    Someone built what is essentially a fancy markov chain that was trained to look convincing to people and people think that's going to lead to AGI lol.
    - version_five 3y ago
      "Introducing the AI Mirror Test, which very smart people keep failing" https://www.theverge.com/23604075/ai-chatbots-bing-chatgpt-intelligent-sentient-mirror-test https://www.theverge.com/23604075/ai-chatbots-bing-chatgpt-i...
      - seiferteric 3y ago
        99% of what people do day to day is repetitive/unoriginal so it is not too surprising that this is happening, but I wonder if we are really any closer to an AI that can develop a novel and important scientific theory or mathematical result for instance? When can we expect AI to resolve the quantum gravity issue? If it can't do that, but eventually humans do, what does that say about AI I wonder?
jiggawatts 3y ago
I’m just re-reading Blindsight by Peter Watts, a novel that is amazingly prescient in — amongst other things — its predictions of this coming economic upheaval. Speaking of… one scene in the book has humans communicating with an alien that appears to have “learned” human speech by training a non-sentient LLM on human ship-to-ship transmissions, and then “fine tuned” it to achieve the desired communication goal without ever actually understanding what the LLM is saying. This is a book from 2006 accurately using the salient features of LLMs popularised in the 2020s! That’s proper science fiction, right there. Back to the economic aspect: Several characters in the book had to “butcher themselves” with implants and enhancements to remain economically relevant in the age of AIs. It’s that… or you’re packed away in storage. Useless. PS: Imagine training an LLM on cetacean recordings and then fine-tuning on “orca attack imminent”. You could use this to scare whales away from ships without understanding what specifically the LLM was singing to them!
- ineedasername 3y ago
  >This is a book from 2006 accurately using the salient features of LLMs popularised in the 2020s I'm a huge fan of Peter Watts' work, but to be fair that is at the very least a close-cousin to the Chinese Room thought experiment that's been around for a while.
  - eulgro 3y ago
    And so it is said in the book.
    - lostmsu 3y ago
      And since the book itself is a work of fiction, it doesn't really tell us what will reality look like.
      - XorNot 3y ago
        Yes and no: good science fiction takes a trend and extrapolates. Extrapolation should be based on logic (to be good - but you can also just aim for entertaining and go wild). The details might differ, but it's worth thinking about whether the implications make sense. Watt's certainly asks a lot of good questions which other orders have also asked - i.e. Accelerando[1] is very interested in the idea of "what happens to mankind when economic systems can no longer be participated in by recognizably human minds?" - taken out to the limits of known physics. [1] https://www.antipope.org/charlie/blog-static/fiction/accelerando/accelerando-intro.html https://www.antipope.org/charlie/blog-static/fiction/acceler... ^^^ You read the full-text here, I highly recommend it.
somewhereoutth 3y ago
Probably things would become clearer if what we currently describe as 'AI' we instead term as 'Statistical Pattern Regurgitator' or similar. Any serious attempt at AI would: need to be trained on reality - like people are trained; need to overcome the cardinality barrier - digital systems can only approximate analogue systems; and need to demonstrate spontaneous language emergence driven by survival in a social environment.
- eterevsky 3y ago
  Why wouldn’t you call a human brain “statistical pattern regurgitator” as well, if it comes to this? Matching learned patterns is a big part of how brains work. I don’t see why a strong AI _needs_ to be trained specifically on reality. Imagine an alien race that sees world in X-Rays and neutrinos. An alien AI researcher claims that a “true intelligences” must train on the same inputs, and all those systems trained on the part of the spectrum between infrared and ultraviolet only fake intelligence. I don’t see why intelligence has to be analogue in any way. Analogue systems are inherently noisy and contain limited amount of useful information, so you can always faithfully approximate them using digital representations. If I am not mistaken Deepmind has shown spontaneous emergence of communication around 5 years ago when training agents in multi-player team shooters.
  - radicalace 3y ago
    you absolutely understand. such a key point. GTP discovers deep structural truth, and characterized this with a multidimentional heat map, about the system that produced the data its trained on. that comes from Stephen wolfram. if you trained it on clouds, it would have an autistic knowledge of clouds. what GTP is doing is exactly the same thing as human intuition. its not statistically driven, this is proven mathematically, thats why we havent seen it before. but its still just patterns and concepts, no intellect or logic. the key observations: imagine a human with intuition so good and so powerful that you could compose an essay without thinking, as a reflex. people dont understand how insanely powerful this AI would be if it had an intellect to match its intuition. its coming. people dont understand that intuition is a necessary part of an intellect, that were already half way to true AI. these LLMs are a bundle of kindling.
saulrh 3y ago
One day, someone came up with the Transformer architecture for neural networks. The ideas didn't quite come out of nowhere, but their impact did - I don't think that anyone predicted the degree to which ChatGPT and its ilk are superior to LSTMs and the like. More importantly, nobody predicted the details, the way that the GPTs behave, the way they're willing to invent facts like they're spooling out plausible thoughts without being constrained to observed reality. From a history-of-technology standpoint, the state of the art for generative AI went from LSTMs to ChatGPT overnight. There was no warning. Nobody could have looked at the Attention Is All You Need paper and predicted how much it would improve the state of the art. And worst of all, we already know that AIs can be hostile. What else is a public corporation's corporate bureaucracy but a Chinese Room running an AI whose goal is to maximize quarterly dividends? What else is a government but another Chinese Room trying to survive? We talk about corporations being faceless or friendly, we ascribe them wants and needs and behaviors and habits, we give them names, and they suffer natural selection and mutation and reproduce and evolve over time so as to select from those bureaucracies that are most fit for survival. And this class of AI is already responsible for global climate change and the Cold War and so many other things that might already be on the path to push us into extinction. LLMs are nowhere near as humanlike as a corporation. Corporations are bound by regulations and capitalism and basic human decency. It is absolutely possible to build an LLM whose fundamental goal is to kill people. All you have to do is tell ChatGPT to pretend it's playing a game of global thermonuclear war, and some joker on 4chan would do that in a heartbeat. So what? Why does this matter? We have no way to predict when the next revolution in AI capability will occur. We have no way to predict what the next revolution will be. We have no way to predict the capabilities of the next generation of generative AIs. We have no way to predict how the next generation of AIs will "think", insofar as they do. And because we have no predictions, we have no way to prepare. So the real question is, how much risk are you willing to tolerate? Are you wiling to take a coin flip on it? A spin of the cylinder? A roll of the dice? Because that's what we're looking at here - the next generation of AIs has no reason to be any less inimical to human existence than the examples we already have, and AI will _only_ get better.
- radicalace 3y ago
  exactly. where were all these self proclaimed experts, who are now making confident predictions about AI, before attention is all you need? literally 100% confidently predicted AI like chatGTP was hundreds or thousands of years away, or would have if asked. these people are the real bots. and i never saw a single person on HN or anywhere else raise concerns or even appreciate the true impact GTP would have today in 2018 when GTP was released and samples available! people coundnt get it right even when it was right in front of them. how can they think for a single second theyre getting it right now? to my credit i recognized immediately what was going on in 2018 and to my knowledge i was the only person leaving comments warning people and begging people to support a pause of research. so by the only material metric, i am more qualified to make predictions than anyone here or even the experts who all confidently predicted that what is happening now would never happen in our childrens lifetimes. i will never forget when hinton saw the light. when he started warning of danger and using the key words that are at the heart of the issue. and i realized i had been ahead of him. just because youre an expert in one thing, even the implementation of ML or AI, doesnt make you an expert on the broader consequences of AI.
theragra 3y ago
Arguments in the first part are really bad. I haven't read about ai safety that much, but still see it. For example, ASI may be able to automate manufacturing really quickly, maybe be able to convince humans easily to do what it needs, etc Moreover, why ai would not be competing with us for resources? Materials on earth are finite. Also, most likely scenario is not evil AGI, as it outlined in the article, but indifferent. And basing reasoning about AI on Darwinian evolution is barely useful. Also, talk about this tribal people AI is ironically even worse then silicon valley narrow viewpoint: it is Californian liberal virtue signaling.
- qznc 3y ago
  Is there a difference between evil and indifferent if you are on the receiving end? Humanity is largely indifferent to animal welfare and animals would qualify it as evil how we treat them.
  - dredmorbius 3y ago
    OT: I just wanted to very belatedly thank you for this brilliant Walter Benjamin quote: <https://news.ycombinator.com/item?id=23388285 https://news.ycombinator.com/item?id=23388285>
  - cjbprime 3y ago
    No, but I think that was exactly the point being made: that the article suggests evil would be required for existential risk, when indifferent would suffice.
    - yanderekko 3y ago
      It's noteworthy that even if we do see "evil" it'll just be something painted on an underlying veneer of indifference, eg. ChaosGPT.
  - ChatGTP 3y ago
    I am currently a bit of a doomer and if I had a magic wand, I'd pause AI research for the foreseeable future because I don't think it's creating a better world for children, at least from what I've seen so far however...to your point. I think we're only like this to animals because we are animals (we forgot), and we compete for the same type of resources they need, water, protein, calories, we're not intelligent enough to get ourselves out of this situation, but if we could, we could. If we could we'd be able to grow tasty meat in a lab, reverse climate change, restore forrest, synthesize water cheaply, live in floating cities above beautiful habitats,,,etc etc. We're just not that smart so we do dumb shit, like factory farm.
cjbprime 3y ago
This is a truly awful article. I'm mildly positive towards AI because I think people tend to react with fear and examine the risks without equivalently examining the rewards, and I think that's happening here and the rewards seem amazing. But the arguments of risk are serious and plausible and this article is just lazily dismissive of them. > AI is not competing for resources with human beings. A superintelligence would absolutely be competing for resources (mainly electricity and cooling) with human beings. > Rather, we provide AI systems with their resources, from energy and raw materials to computer chips and network infrastructure. A superintelligence will be able to convince anyone it wants to do anything for it. Even the previous President did a pretty good job at causing adulation and democracy-threatening personal loyalty from people he had not actually met in person. Imagine how amplified and personal the effect could be. > For now, AI depends on us, and a superintelligence would presumably recognize that fact and seek to preserve humanity since we are as fundamental to AI’s existence as oxygen-producing plants are to ours. This makes the evolution of mutualism between AI and humans a far more likely outcome than competition. Moreover, the path to a fully automated economy — if that is the goal — will be long, with each major step serving as a natural checkpoint for human intervention. The author is literally relying on us outsmarting the superintelligence (collectively, with coordination!). It doesn't sound like they've come to terms with the concept of superintelligence at all. Or with the state of human global coordination problems, come to think of it. > AI cannot physically hunt us. This is likely to become literally false sometime soon if it hasn't already, but even if it doesn't, the AI doesn't have to. It just has to convince another human that the human is in love with it and it wants the human to kill a bunch of people, then scale the process. > AI’s impact on the climate is up to us. Our own impact on the climate is up to us. Our collective decision-making is.. suboptimal. > If we really think that superintelligent AI presents a plausible existential risk, shouldn’t we simply stop all AI research right now? Why not preemptively bomb data centers and outlaw GPUs? You are literally linking to the person who is the most prominent voice of AI existential risk, who is seriously suggesting doing exactly that.
- flangola7 3y ago
  >This is likely to become literally false sometime soon if it hasn't already, but even if it doesn't, the AI doesn't have to. It just has to convince another human that the human is in love with it and it wants the human to kill a bunch of people, then scale the process. I'm stunned at the number of people who try to make this argument. The operators of Radio Télévision Libre des Mille Collines never had to leave their broadcast studio. Goebbels never had to do anything but get in front of a microphone or a typewriter. Still, millions were violently killed. We bicker about LLMs' ability to write C code that will compile, but their best abilities are to gaslight, emotionally manipulate, lie and create FUD.
ilaksh 3y ago
Although I disagree with the conclusion, this looks like a pretty decent article in terms of detailed thinking. I just think that we not only need to work on all of the more immediate problems but also worry about less immediate problems that are existential risks. Also people always say "AGI" and they mean different things but I am guessing most now mean something like a digital living human/simulation of human when they say that. Mixed in with some blurry connotation of it automatically becoming godlike in power. That stuff is very speculative. But we don't need AI to get to that level for it to be very dangerous. Just imagine that we have an open source GPT that is something like 33% smarter than GPT-4 and less brittle. Make the output 50 times faster than human thought and suppose that we can run it very inexpensively, in a "swarm" of agents cooperating. Then you have a type of superintelligence that does not require any really speculative AI advance -- hyperspeed reasoning -- based on the current technology. If that is widely deployed for military and industrial decision-making then something like a computer virus could create existential risk. Also, realize that all these systems really need to emulate some of the core functional aspects of animals, like a self-preservation instinct or desire to control resources, is the right instruction and relaxation of guardrails. So when they can reason a bit better and at hyperspeed with collaboration between them, they don't need to be alive or anything to be dangerous. Look at the history of increases in computing efficiency. It is easy to imagine 50 or whatever times output speed increase in less than 5 years. And even though they are not human level now, GPT-4 has proven that these types of systems can have strong problem solving ability.
- joshuahedlund 3y ago
  imo GTP-X is going to need to be much much “smarter” and consistently accurate (which may be fundamentally impossible) before it’s going to be “widely deployed for military and industrial decision-making” anywhere.
  - creer 3y ago
    I don't think a piece of (AI? no, software? Meh?) thinking material needs to be above average human in its capabilities, or in any way consistently correct in its thinking, or in any way all that broadly learned, or in any way accountable - to cause significant mayhem if given half the chance. There is this image of super-human, much faster, much broader thinking in the collective. When a bog standard human, drunk and in a foreign country, can do plenty. By this measure, chatGPTs are plenty intelligent enough - just not yet connected enough to tools, a wallet and vague orders or intentions.
gregwebs 3y ago
The foundation of this article is that there is a zero sum game for resources for AI safety. I think we should reject this notion and improve AI safety both in terms of existential and non-extensional risk. If more resources are needed and can be used effectively for dealing with safety we should instead take resources away from propelling AI forward and apply them to safety. I see this argument against existential risk alliance – that AI needs humans to provide it with resources- this we can hold it hostage. I like to point out that power plants are increasingly since the pandemic remotely operable- that AI will increasingly be able to hold us hostage by hacking the infrastructure we need to survive. Throw in the ability to use some robots and bio weapons to keep humans from taking them back easily. The other aspect is that AI doesn’t have to properly conclude ahead of time that such an attempt has a high probability of success. Its program just has to decide that this will help it achieve some objective.
throwawayoaky 3y ago
Was this written by chatgpt?

58 points by headalgorithm 3y ago

[dead]

We live in a very confusing, transitory time. Right now there's a large chunk of people who are convinced that we are a few years away from extinction if we do not ration out GPUs and bomb illegal clusters of compute, while at the same time there's a large chunk that believes that a car can never navigate a single-lane one-way tunnel safely without a driver (ie, The Boring Company). Absolutely wild.

hollerith 3y ago
"Convinced that we are a few years from extinction," is an exaggeration of the pessimistic/alarmed end of the spectrum of opinion. All the forecasts I've seen have the period of danger spread out over at least 20 or 30 years, and no one claims to understand the topic well enough to be able to tell whether a particular model is dangerous even if given complete freedom to examine the source code and interview the creators. Our inability to predict is in large part the basis of the pessimism/alarm because it means that the leading labs won't know when to stop.
- acqq 3y ago
  Exactly. And here is an example of that inability in the much clearer case: "How many years after 1945 ... would it take for the Soviet Union to get the bomb? Framed this way, it was a much easier question than forecasting the advent of AGI: The Americans were concerned with only one adversary and the development of a single technology which they already knew all the details of. American predictions of Soviet proliferation offer a highly constrained case study for those who would undertake technological forecasting today." https://asteriskmag.com/issues/03/how-long-until-armageddon https://asteriskmag.com/issues/03/how-long-until-armageddon
- ozr 3y ago
  > "Convinced that we are a few years from extinction," is an exaggeration of the pessimistic/alarmed end of the spectrum of opinion. It really isn't. There are people quitting their jobs, protesting AI companies, and inducing panic attacks in children over their irrational fears: https://www.wired.com/story/pause-ai-existential-risk/ https://www.wired.com/story/pause-ai-existential-risk/
  - yanderekko 3y ago
    No, even the more-panicked wing is not betting the proverbial house on AI killing us within the next few years, although the possibility is there. The "don't expect to live much longer" sentiment is based on a probability horizon that extends over decades.
    - ozr 3y ago
      I've spoken to people in the article. They expect doom in <1 year.
      - hollerith 3y ago
        Can you give a name so I don't have to read a long article? It is easy to find people who believe that it is possible (with some low probability) that AI will end the human race in the next 12 mo. (I'm one of them.) I haven't seen anyone however put most of their probability mass in the next 12 or 24 mo, and I'd like to find out what their reasoning is.
        ozr 3y ago
        Sure, first name in the article I posted, Joep Meindertsma. They have an open discord you can join, but fair warning that it's entirely panicky people lamenting their inevitable doom.
      - yanderekko 3y ago
        I just don't believe that unless you have specific quotes you can point to that indicates this.
RC_ITR 3y ago
> a car can never navigate a single-lane one-way tunnel safely without a driver (ie, The Boring Company). I think the criticisms of the Boring Company are more "small diameter tunnels are a bad way to build high-throughput underground infrastructure, and autonomous cars will not solve that"
- john2x 3y ago
  "small diameter tunnels" + "autonomous cars" kind of sounds like a subway train.
  - sgu999 3y ago
    Just add tracks, weld the cars together, remove the driving seats, replace the fancy NNs by vintage fuzzy logic and you're there!
  - RC_ITR 3y ago
    Sure, but have you ever been on the London Tube and noticed it's notoriously small and uncomfortable in parts? Have you also noticed how windy some stations get because the small trains fill most of the tube and act like an air piston? Those tunnels are 12 feet diameter. Boring Co's tunnels are 12 feet diameter. A modern subway system needs considerably larger tunnels.
version_five 3y ago
I've only seen grifters, hysterics, or the ignorant talking about existential AI risk. A lot of people just nods their heads and go along, but I don't think there are many that have really thought about it (and understand what modern AI is) that really belive the risk. It's more like political polarization where people dwell on something they don't agree with to the point where they treat it as if the world is going to end because of it, instead of in context. AI is more like the political party they don't like winning, potentially undesirable from a certain point of view but they see it as "existential". I think it's important to see it in this light, instead of trying to actually debate the points raised, which are absurd and will only give credibility where none is due.
- jefftk 3y ago
  > I've only seen grifters, hysterics, or the ignorant talking about existential AI risk. Which category would you put Geoffrey Hinton in? https://www.utoronto.ca/news/risks-artificial-intelligence-must-be-considered-technology-evolves-geoffrey-hinton https://www.utoronto.ca/news/risks-artificial-intelligence-m...
  - cristiancavalli 3y ago
    hysterics. Remember when he said we should stop training radiologists, that’s aged pretty poorly given there’s actually a need to have more: https://www.rsna.org/news/2022/may/global-radiologist-shortage https://www.rsna.org/news/2022/may/global-radiologist-shorta... https://mindmatters.ai/2022/08/turns-out-computers-are-not-vacuuming-up-all-our-jobs/ https://mindmatters.ai/2022/08/turns-out-computers-are-not-v...
    - mptest 3y ago
      So because he made one(+) bad prediction, all future predictions are inherently poor? It seems that may be a tad of a leap? Where is the line between "poor prediction" and "hysterics"? "Deep learning will replace radiologists within 5 years" isn't as inherently crazy as it probably seems in hindsight. Sure, it didn't pan out, but I know a vet who works for an Ai company classifying scans for training and he says first hand the models have gotten a lot better. That said, he's still not worried about being replaced.

ggm 3y ago

AGI doesn't ever have to happen for rapid deployment of AI based systems to cause harm. Google "robodebt australia" for what happens when government uses machine derived decisions to penalise the poor.

__loam 3y ago
That point is basically what the article is about.
- ggm 3y ago
  It can't be repeated enough apparently, because people still whine on about skymesh and "but IFF the robots wake up" when the real risk is what people believe machine derived "facts" are (I quote facts because they are highly conjectural inferences from models, and often diverge from fact pretty rapidly)
  - yanderekko 3y ago
    >when the real risk is what people believe machine derived "facts" are It's hard to play out a scenario where this leads to actual human extinction.
QuantumG 3y ago
aussie here, former dole bludger, still know people who live off the welfare system, hide their side income, and are proud of it. There was sooooo much warning for robodebt that they all made sure their taxes were clean before the first demands went out, but a few still got through. Mostly accurate, but a few dodgy ones too. Of course, as we know, it was a dragnet and the human suffering produced was excessive. Considering the depth of the corruption, it was clear the system was incapable of correcting itself, so what else is there to do? How do you get help to the people with real need when there's so many people cheating? Robodebt was big hammer approach.

janalsncm 3y ago

I don’t understand why economic disruption and human obsolescence isn’t considered an existential risk. We could end up in a world where 99.99% of people are redundant and useless cost centers for our GDP maximizing economic paradigm. In that case, you don’t need killer robots in order to push humanity towards near-extinction. The invisible hand will smite them. But don’t worry, TED Talk attendee. The obsoletariat mostly won’t be Americans. The median member of this class will probably be Chinese or Indian. So you can continue your performative concerns about Roko’s Basillisk or whatever topic of distant, paralyzing uncertainty is overflowing from the next room into mine.

ke88y 3y ago
Maybe this is why billionaires and hundred-millionaires are the ones so concerned about existential risk. A collapse of consensus in our current and historically not-that-old system of property rights is an existential risk -- to them.
deleted 3y ago
[deleted]
__loam 3y ago
This is a problem with capitalism in general, not just with AI.
- version_five 3y ago
  We're seeing this happen a lot with "AI" - now that it's a super popular topic people use it as the lens though which they project their general political grievances. It gets more attention.
- janalsncm 3y ago
  Well, it’s a problem with any economic system which uses labor as a proxy for value/rights. If my political power is predicated on my ability to withdraw labor or move to a new country, it’s not going to be good for me when my services are no longer required.
- 3y ago

hashstring 3y ago

Yes, this is a good take. The general discussion is definitely prematurely focused on some sort of end-boss fight, while AI and large scale data collection are causing serious harm and privacy concerns to real humans already. It’s only reasonable to expect that this increases in the future. I would like to see more discussion focussing on these pressing issues. Also a pause on AI development is not an option, not a solution and is digressing from the issue at hand. Big capital with their hands on big valuable data will support anything that distracts here. Finally, I do think that AI also brings many positives, I am not against the technology itself at all- we shouldn’t be.

radicalace 3y ago
name a single reason why a pause is not an option. dont say why it would be hard or unlikely or inconvenient or whatever, to exclude it as a possibility means it has to be physically impossible. of course it isnt physically impossible. AI requires large compute, large resources. you cant advance the state of the art inconspicuously. the only reason you say it isnt an option is because you dont consider AI an existential threat, really, and therefore extreme measures arent justified. but they actually are.
- nradov 3y ago
  That's not even a valid request. There is obviously no valid reason to require such a pause in the first place. You've been watching too many scifi movies and don't understand the technology.
- hashstring 3y ago
  We are already facing numerous existential threats at the moment, even ones that are deliberately made to be an existential threat, take nuclear weapons. “Hey fellow nation states, let’s not create nuclear bombs, but let’s pause developments and keep the current power dynamics fixed please” is completely unrealistic. Mind that creating nuclear weapons actually requires technical operations that are factors more complex than “large compute” in many aspects… like… enriching uranium and working with supercritical masses. > you cant advance the state of the art inconspicuously. What evidence has led you to believe this?

more_corn 3y ago

The distant future. Like five years. Self-aware (or a simulation thereof) AGI with self-directed motivation and the ability to self modify is barely a half step from ASI. And AGI could happen in the coming year. Five if you wanna be pessimistic. Never if you’re extraordinarily pessimistic but that’s looking unlikely. If it’s possible it’s possible now. This isn’t the Yellowstone super volcano that might blow in the next billion years. People who talk like this are idiotic at best. Granted, the other concerns are real too but the accusation that the concern of existential risk is being used to hide the other known problems is dangerous in the extreme.

JimtheCoder 3y ago
"And AGI could happen in the coming year." "People who talk like this are idiotic at best."
- __loam 3y ago
  Someone built what is essentially a fancy markov chain that was trained to look convincing to people and people think that's going to lead to AGI lol.
  - version_five 3y ago
    "Introducing the AI Mirror Test, which very smart people keep failing" https://www.theverge.com/23604075/ai-chatbots-bing-chatgpt-intelligent-sentient-mirror-test https://www.theverge.com/23604075/ai-chatbots-bing-chatgpt-i...
    - seiferteric 3y ago
      99% of what people do day to day is repetitive/unoriginal so it is not too surprising that this is happening, but I wonder if we are really any closer to an AI that can develop a novel and important scientific theory or mathematical result for instance? When can we expect AI to resolve the quantum gravity issue? If it can't do that, but eventually humans do, what does that say about AI I wonder?

jiggawatts 3y ago

I’m just re-reading Blindsight by Peter Watts, a novel that is amazingly prescient in — amongst other things — its predictions of this coming economic upheaval. Speaking of… one scene in the book has humans communicating with an alien that appears to have “learned” human speech by training a non-sentient LLM on human ship-to-ship transmissions, and then “fine tuned” it to achieve the desired communication goal without ever actually understanding what the LLM is saying. This is a book from 2006 accurately using the salient features of LLMs popularised in the 2020s! That’s proper science fiction, right there. Back to the economic aspect: Several characters in the book had to “butcher themselves” with implants and enhancements to remain economically relevant in the age of AIs. It’s that… or you’re packed away in storage. Useless. PS: Imagine training an LLM on cetacean recordings and then fine-tuning on “orca attack imminent”. You could use this to scare whales away from ships without understanding what specifically the LLM was singing to them!

ineedasername 3y ago
>This is a book from 2006 accurately using the salient features of LLMs popularised in the 2020s I'm a huge fan of Peter Watts' work, but to be fair that is at the very least a close-cousin to the Chinese Room thought experiment that's been around for a while.
- eulgro 3y ago
  And so it is said in the book.
  - lostmsu 3y ago
    And since the book itself is a work of fiction, it doesn't really tell us what will reality look like.
    - XorNot 3y ago
      Yes and no: good science fiction takes a trend and extrapolates. Extrapolation should be based on logic (to be good - but you can also just aim for entertaining and go wild). The details might differ, but it's worth thinking about whether the implications make sense. Watt's certainly asks a lot of good questions which other orders have also asked - i.e. Accelerando[1] is very interested in the idea of "what happens to mankind when economic systems can no longer be participated in by recognizably human minds?" - taken out to the limits of known physics. [1] https://www.antipope.org/charlie/blog-static/fiction/accelerando/accelerando-intro.html https://www.antipope.org/charlie/blog-static/fiction/acceler... ^^^ You read the full-text here, I highly recommend it.

somewhereoutth 3y ago

Probably things would become clearer if what we currently describe as 'AI' we instead term as 'Statistical Pattern Regurgitator' or similar. Any serious attempt at AI would: need to be trained on reality - like people are trained; need to overcome the cardinality barrier - digital systems can only approximate analogue systems; and need to demonstrate spontaneous language emergence driven by survival in a social environment.

eterevsky 3y ago
Why wouldn’t you call a human brain “statistical pattern regurgitator” as well, if it comes to this? Matching learned patterns is a big part of how brains work. I don’t see why a strong AI _needs_ to be trained specifically on reality. Imagine an alien race that sees world in X-Rays and neutrinos. An alien AI researcher claims that a “true intelligences” must train on the same inputs, and all those systems trained on the part of the spectrum between infrared and ultraviolet only fake intelligence. I don’t see why intelligence has to be analogue in any way. Analogue systems are inherently noisy and contain limited amount of useful information, so you can always faithfully approximate them using digital representations. If I am not mistaken Deepmind has shown spontaneous emergence of communication around 5 years ago when training agents in multi-player team shooters.
- radicalace 3y ago
  you absolutely understand. such a key point. GTP discovers deep structural truth, and characterized this with a multidimentional heat map, about the system that produced the data its trained on. that comes from Stephen wolfram. if you trained it on clouds, it would have an autistic knowledge of clouds. what GTP is doing is exactly the same thing as human intuition. its not statistically driven, this is proven mathematically, thats why we havent seen it before. but its still just patterns and concepts, no intellect or logic. the key observations: imagine a human with intuition so good and so powerful that you could compose an essay without thinking, as a reflex. people dont understand how insanely powerful this AI would be if it had an intellect to match its intuition. its coming. people dont understand that intuition is a necessary part of an intellect, that were already half way to true AI. these LLMs are a bundle of kindling.

saulrh 3y ago

One day, someone came up with the Transformer architecture for neural networks. The ideas didn't quite come out of nowhere, but their impact did - I don't think that anyone predicted the degree to which ChatGPT and its ilk are superior to LSTMs and the like. More importantly, nobody predicted the details, the way that the GPTs behave, the way they're willing to invent facts like they're spooling out plausible thoughts without being constrained to observed reality. From a history-of-technology standpoint, the state of the art for generative AI went from LSTMs to ChatGPT overnight. There was no warning. Nobody could have looked at the Attention Is All You Need paper and predicted how much it would improve the state of the art. And worst of all, we already know that AIs can be hostile. What else is a public corporation's corporate bureaucracy but a Chinese Room running an AI whose goal is to maximize quarterly dividends? What else is a government but another Chinese Room trying to survive? We talk about corporations being faceless or friendly, we ascribe them wants and needs and behaviors and habits, we give them names, and they suffer natural selection and mutation and reproduce and evolve over time so as to select from those bureaucracies that are most fit for survival. And this class of AI is already responsible for global climate change and the Cold War and so many other things that might already be on the path to push us into extinction. LLMs are nowhere near as humanlike as a corporation. Corporations are bound by regulations and capitalism and basic human decency. It is absolutely possible to build an LLM whose fundamental goal is to kill people. All you have to do is tell ChatGPT to pretend it's playing a game of global thermonuclear war, and some joker on 4chan would do that in a heartbeat. So what? Why does this matter? We have no way to predict when the next revolution in AI capability will occur. We have no way to predict what the next revolution will be. We have no way to predict the capabilities of the next generation of generative AIs. We have no way to predict how the next generation of AIs will "think", insofar as they do. And because we have no predictions, we have no way to prepare. So the real question is, how much risk are you willing to tolerate? Are you wiling to take a coin flip on it? A spin of the cylinder? A roll of the dice? Because that's what we're looking at here - the next generation of AIs has no reason to be any less inimical to human existence than the examples we already have, and AI will _only_ get better.

radicalace 3y ago
exactly. where were all these self proclaimed experts, who are now making confident predictions about AI, before attention is all you need? literally 100% confidently predicted AI like chatGTP was hundreds or thousands of years away, or would have if asked. these people are the real bots. and i never saw a single person on HN or anywhere else raise concerns or even appreciate the true impact GTP would have today in 2018 when GTP was released and samples available! people coundnt get it right even when it was right in front of them. how can they think for a single second theyre getting it right now? to my credit i recognized immediately what was going on in 2018 and to my knowledge i was the only person leaving comments warning people and begging people to support a pause of research. so by the only material metric, i am more qualified to make predictions than anyone here or even the experts who all confidently predicted that what is happening now would never happen in our childrens lifetimes. i will never forget when hinton saw the light. when he started warning of danger and using the key words that are at the heart of the issue. and i realized i had been ahead of him. just because youre an expert in one thing, even the implementation of ML or AI, doesnt make you an expert on the broader consequences of AI.

theragra 3y ago

Arguments in the first part are really bad. I haven't read about ai safety that much, but still see it. For example, ASI may be able to automate manufacturing really quickly, maybe be able to convince humans easily to do what it needs, etc Moreover, why ai would not be competing with us for resources? Materials on earth are finite. Also, most likely scenario is not evil AGI, as it outlined in the article, but indifferent. And basing reasoning about AI on Darwinian evolution is barely useful. Also, talk about this tribal people AI is ironically even worse then silicon valley narrow viewpoint: it is Californian liberal virtue signaling.

qznc 3y ago
Is there a difference between evil and indifferent if you are on the receiving end? Humanity is largely indifferent to animal welfare and animals would qualify it as evil how we treat them.
- dredmorbius 3y ago
  OT: I just wanted to very belatedly thank you for this brilliant Walter Benjamin quote: <https://news.ycombinator.com/item?id=23388285 https://news.ycombinator.com/item?id=23388285>
- cjbprime 3y ago
  No, but I think that was exactly the point being made: that the article suggests evil would be required for existential risk, when indifferent would suffice.
  - yanderekko 3y ago
    It's noteworthy that even if we do see "evil" it'll just be something painted on an underlying veneer of indifference, eg. ChaosGPT.
- ChatGTP 3y ago
  I am currently a bit of a doomer and if I had a magic wand, I'd pause AI research for the foreseeable future because I don't think it's creating a better world for children, at least from what I've seen so far however...to your point. I think we're only like this to animals because we are animals (we forgot), and we compete for the same type of resources they need, water, protein, calories, we're not intelligent enough to get ourselves out of this situation, but if we could, we could. If we could we'd be able to grow tasty meat in a lab, reverse climate change, restore forrest, synthesize water cheaply, live in floating cities above beautiful habitats,,,etc etc. We're just not that smart so we do dumb shit, like factory farm.

cjbprime 3y ago

This is a truly awful article. I'm mildly positive towards AI because I think people tend to react with fear and examine the risks without equivalently examining the rewards, and I think that's happening here and the rewards seem amazing. But the arguments of risk are serious and plausible and this article is just lazily dismissive of them. > AI is not competing for resources with human beings. A superintelligence would absolutely be competing for resources (mainly electricity and cooling) with human beings. > Rather, we provide AI systems with their resources, from energy and raw materials to computer chips and network infrastructure. A superintelligence will be able to convince anyone it wants to do anything for it. Even the previous President did a pretty good job at causing adulation and democracy-threatening personal loyalty from people he had not actually met in person. Imagine how amplified and personal the effect could be. > For now, AI depends on us, and a superintelligence would presumably recognize that fact and seek to preserve humanity since we are as fundamental to AI’s existence as oxygen-producing plants are to ours. This makes the evolution of mutualism between AI and humans a far more likely outcome than competition. Moreover, the path to a fully automated economy — if that is the goal — will be long, with each major step serving as a natural checkpoint for human intervention. The author is literally relying on us outsmarting the superintelligence (collectively, with coordination!). It doesn't sound like they've come to terms with the concept of superintelligence at all. Or with the state of human global coordination problems, come to think of it. > AI cannot physically hunt us. This is likely to become literally false sometime soon if it hasn't already, but even if it doesn't, the AI doesn't have to. It just has to convince another human that the human is in love with it and it wants the human to kill a bunch of people, then scale the process. > AI’s impact on the climate is up to us. Our own impact on the climate is up to us. Our collective decision-making is.. suboptimal. > If we really think that superintelligent AI presents a plausible existential risk, shouldn’t we simply stop all AI research right now? Why not preemptively bomb data centers and outlaw GPUs? You are literally linking to the person who is the most prominent voice of AI existential risk, who is seriously suggesting doing exactly that.

flangola7 3y ago
>This is likely to become literally false sometime soon if it hasn't already, but even if it doesn't, the AI doesn't have to. It just has to convince another human that the human is in love with it and it wants the human to kill a bunch of people, then scale the process. I'm stunned at the number of people who try to make this argument. The operators of Radio Télévision Libre des Mille Collines never had to leave their broadcast studio. Goebbels never had to do anything but get in front of a microphone or a typewriter. Still, millions were violently killed. We bicker about LLMs' ability to write C code that will compile, but their best abilities are to gaslight, emotionally manipulate, lie and create FUD.

ilaksh 3y ago

Although I disagree with the conclusion, this looks like a pretty decent article in terms of detailed thinking. I just think that we not only need to work on all of the more immediate problems but also worry about less immediate problems that are existential risks. Also people always say "AGI" and they mean different things but I am guessing most now mean something like a digital living human/simulation of human when they say that. Mixed in with some blurry connotation of it automatically becoming godlike in power. That stuff is very speculative. But we don't need AI to get to that level for it to be very dangerous. Just imagine that we have an open source GPT that is something like 33% smarter than GPT-4 and less brittle. Make the output 50 times faster than human thought and suppose that we can run it very inexpensively, in a "swarm" of agents cooperating. Then you have a type of superintelligence that does not require any really speculative AI advance -- hyperspeed reasoning -- based on the current technology. If that is widely deployed for military and industrial decision-making then something like a computer virus could create existential risk. Also, realize that all these systems really need to emulate some of the core functional aspects of animals, like a self-preservation instinct or desire to control resources, is the right instruction and relaxation of guardrails. So when they can reason a bit better and at hyperspeed with collaboration between them, they don't need to be alive or anything to be dangerous. Look at the history of increases in computing efficiency. It is easy to imagine 50 or whatever times output speed increase in less than 5 years. And even though they are not human level now, GPT-4 has proven that these types of systems can have strong problem solving ability.

joshuahedlund 3y ago
imo GTP-X is going to need to be much much “smarter” and consistently accurate (which may be fundamentally impossible) before it’s going to be “widely deployed for military and industrial decision-making” anywhere.
- creer 3y ago
  I don't think a piece of (AI? no, software? Meh?) thinking material needs to be above average human in its capabilities, or in any way consistently correct in its thinking, or in any way all that broadly learned, or in any way accountable - to cause significant mayhem if given half the chance. There is this image of super-human, much faster, much broader thinking in the collective. When a bog standard human, drunk and in a foreign country, can do plenty. By this measure, chatGPTs are plenty intelligent enough - just not yet connected enough to tools, a wallet and vague orders or intentions.

gregwebs 3y ago

The foundation of this article is that there is a zero sum game for resources for AI safety. I think we should reject this notion and improve AI safety both in terms of existential and non-extensional risk. If more resources are needed and can be used effectively for dealing with safety we should instead take resources away from propelling AI forward and apply them to safety. I see this argument against existential risk alliance – that AI needs humans to provide it with resources- this we can hold it hostage. I like to point out that power plants are increasingly since the pandemic remotely operable- that AI will increasingly be able to hold us hostage by hacking the infrastructure we need to survive. Throw in the ability to use some robots and bio weapons to keep humans from taking them back easily. The other aspect is that AI doesn’t have to properly conclude ahead of time that such an attempt has a high probability of success. Its program just has to decide that this will help it achieve some objective.

throwawayoaky 3y ago

Was this written by chatgpt?