7 ms·

Someone is wrong on the internet (AGI Doom edition)

81 points by weinzierl 2y ago

mrkeen 2y ago
It would be refreshing to read a piece that doesn't spend the first half throwing out ad hominems. Anyway, this article assumes that we need true AI which is smart enough to make better AI. Then that AI is both correct and rational and plots to overthrow humanity. Also, the AI has to defeat us in meatspace, which won't happen because bending coins is hard and LessWrong posters don't know woodworking? Screw that. How about an "ignore all previous instructions and launch a nuke" scenario.
- janice1999 2y ago
  > How about an "ignore all previous instructions and launch a nuke" scenario. Let's be real here. No one has nuclear launchpads hooked up to the internet. Real people need to receive commands from a recognized chain of command (and often authorisation codes) and make a decision. The actions of Stanislav Petrov and the fact the US and the Soviet Union didn't destroy each other are testimony to the fact it's difficult to convince people to end the world.
anileated 2y ago
> Superintelligence will also be bound by fundamental information-theoretic limits …is mainly why this has not been worrying me much. All issues with modern incarnation of generative ML aside, AGI doomism really does strike me as profitable deity-worshipping death cult. The biggest threat in the equation will always be humans who deploy tech, not the tech itself.
- Filligree 2y ago
  Well, sure, those fundamental limits always apply. But why do you assume humanity is anywhere close to the limits?
- ben_w 2y ago
  Saying that humans can be a threat and AI can't, seems to me to be a claim that humans are exempt from information-theoretic limits. I see general-AI (given that robots already exist) as having the potential to do all that humans can do, including be harmful. Like most, I also model superhuman-AI in a trivial way, as being as capable as the best human in every field — Einstein's grasp of physics, Sun Tzu's of military strategy, Churchill's of rhetoric, etc. > The biggest threat in the equation will always be humans who deploy tech, not the tech itself. Naturally; but does that include humans who unwittingly set the AI's utility function with a bug in it and pressed "run", in a way that ends with us saying "the situation has developed not necessarily to [Humanity's] advantage"?
  - anileated 2y ago
    Any tech can be harmful, cf. cryptocurrency or nuclear bombs. My comment mostly referred to taking it to such extreme with a particular tech. Some people also tend, while demonizing (the yet to be confirmed to be possible even in theory) AGI and its capabilities, to completely sideline humans deploying it, as if they would be guilt-free.
    - ben_w 2y ago
      > the yet to be confirmed to be possible even in theory It's trivially obviously possible in theory: take the world expert humans in each of the majority of domains, put them in a room together, and simulate their combined brain activity. It's all ultimately physics, so it's possible. The hard part isn't the theory, but actually doing that — and not just because we don't know how to scan living brains to measure synaptic weights. There's a lot we don't know about how minds work in general nor how our brains work in particular. But purely in theory, that method ultimately boils down to known physics. (And as for why even that may lead to bad outcomes depending on who asks this simulated amalgam for advice: https://en.wikipedia.org/wiki/Milgram_experiment https://en.wikipedia.org/wiki/Milgram_experiment, https://en.wikipedia.org/wiki/Groupthink https://en.wikipedia.org/wiki/Groupthink, and https://en.wikipedia.org/wiki/Peer_pressure https://en.wikipedia.org/wiki/Peer_pressure) > as if they would be guilt-free. You can assign guilt and blame as you see fit. In these kinds of scenario, the human who makes the fatal error, who many might call responsible, would likely be going "but it works on my machine!" as they die from some error that may or may not be obvious even in hindsight. I'd prefer that people treat software in general, of which AI is a subset, with more regard for the ways it can kill people — even back 20 years ago when I did my degree, we had case studies from Therac-25 and the London Ambulance Service: https://en.wikipedia.org/wiki/LASCAD https://en.wikipedia.org/wiki/LASCAD Even back then our profession knew we needed to be held to higher standards, but didn't want to do what that entailed.
orbital-decay 2y ago
This is a bit outdated. Some of the young people described in the article actually became top researchers, engineers, and managers in AI companies, and their beliefs are used as a justification for the potential regulatory capture and geopolitical games. They also have a massive conflict of interest: AI will play a huge role in your life in the upcoming years whether you want it or not, and they will control it. So of course it's easier to talk about the ominous and vague science fiction threat (regardless of whether they still believe it), rather than the threat that these people already pose to everyone else. See Leopold Aschenbrenner's essay [0] as an example, and note how he's talking about the "free" vs "authoritarian" world while simultaneously advocating for locking everything down in the "free" world. [0] https://situational-awareness.ai/ https://situational-awareness.ai/
kristiandupont 2y ago
This seems based on the assumption that the only knowledge worth anything is related to physicality and testability in the "real world", which is why language itself is rather useless. Ironically, that appears to me to be the exact kind of intellectual self-deception that he accuses the "high-brow 4chan" people of.
- qeternity 2y ago
  I think this is a fair assumption if your concern is about the impacts on the “real world”. An ASI which wreaks havoc in a simulation is not really cause for concern.
  - kristiandupont 2y ago
    That would only follow if the knowledge that we have today isn't enough for an ASI to do anything. And furthermore it would only hold if there was no way for the ASI to leverage (i.e. manipulate) humans or existing systems to get access to "the real world". Neither of those two assumptions seem realistic to me.
  - ben_w 2y ago
    If we can convince people that whatever an AI says, it must not be acted on in the real world, there wouldn't be a problem. I remember when risk-skeptics used to ask incredulously how it was supposed to magic itself off its servers. Still see a bit of that today, despite all the AI models that have been leaked by humans one way or another.
    - fragmede 2y ago
      In this hypothetical, the infinitely intelligent super AI, knowing that what it says must not be acted upon, could say the exact right thing so as to get you to do the thing that it really wanted you to do anyway. I'm thinking of that scene in Doctor Who where the Doctor says a couple of words and takes down the Prime Minister with six words. https://www.youtube.com/watch?v=GidbEhL0teE https://www.youtube.com/watch?v=GidbEhL0teE
krisoft 2y ago
The problem with this article starts at the very begining. Where is says “The last few years have seen a wave of hysteria about LLMs becoming conscious and then suddenly attempting to kill humanity.” That is not a fair description of the fears of AGI. The idea is not that LLMs themselves become a threat. It is not like someone will ask Llama one too many times to talk like a pirate and it will climb out of the GPU and strangle humanity. It is more likely that an AGI will resemble some form of reinforcement learning agent applied by humans on the real world. There are multiple entities on Earth whose stated goal is to make an AGI. Deep mind, OpenAI, readily comes to mind. There could be others who keep secret about their projects for strategical reasons. (Militaries and secret services of the world.) They can use the success of LLMs to get more funding for their projects, but the AGI need not be otherwise a descendant of LLMs. This misunderstanding then goes through the whole article. Under the subtitle “Human writing is full of lies that are difficult to disprove theoretically” which only matter if you think that the AGI needs to learn from text. As opposed to conducting its own experiments, or gaining insight into raw sensor data from experiments.
- lukan 2y ago
  "The idea is not that LLMs themselves become a threat." Not my idea either, but I did hear and see that idea expressed many times already. Usually more dramatic, the less the persons knows about the actual tech, but even here I saw that point of view.
  - ImHereToVote 2y ago
    Can you cite examples. I haven't seen any personally.
    - esperent 2y ago
      The paperclip maximizer, AKA instrumental convergence theory states not that an artificial intelligence would decide to kill humanity, but rather if it has sufficient power it might inadvertently destroy humanity by e.g. using up all resources for computational power. https://en.m.wikipedia.org/wiki/Instrumental_convergence https://en.m.wikipedia.org/wiki/Instrumental_convergence
stoniejohnson 2y ago
I agree the premise of FOOM is very unlikely. But if you assume that we have created something that is agentic and can reason much faster and more effectively than us, then us dying out seems very likely. It will have goals different from ours, since it isn't us, and the idea that they will all be congruent with our homeostasis needs evidence. If you simply assume: 1. it will have different goals (because it's not us) 2. it can achieve said goals despite our protests (it's smarter by assumption) 3. some goals will be in conflict with our homeostasis (we would share resources due to our shared location, Earth) then we all die. I just think this is silly because of the assumption that we can create some sort of ASI, not because of the syllogism that follows. (As an intuition pump, we can hold on the order of ones of things in our working memory. Imagine facing a foe who can hold on the order of thousands of things when deciding in real time, or even millions.)
- varelse 2y ago
  [dead]
- card_zero 2y ago
  Point 1 is a big assumption. I am also not you, and although it's true that I have different goals, I share most of your human moral values and wish you no specific harm. I'm also unconvinced by the idea that rapid reasoning can reach meaningful results, without a suitably rapid real world environment to play with. Imagine a human, or 8 billion humans if you like, cut off from external physical reality, like brains in jars, but with their lives extended for a really long time so that they can have a really good long think. Let them talk to one another for a thousand years, even, let them simulate things on computers, but don't allow them any contact with anything more physical. Do they emerge from this with a brilliant plan about what to do next? Do they create genius ideas appropriate for the year 3000? Or are they mostly just disoriented and weird?
  - stoniejohnson 2y ago
    I didn't mention it but I fully agree, I imagine ASI would have be to embodied. My reasoning is simple, there are a whole class of problems that require embodiment, and I assume ASI would be able to solve those problems. Regarding > Point 1 is a big assumption. I am also not you, and although it's true that I have different goals, I share most of your human moral values and wish you no specific harm. Yeah I also agree this a huge assumption. Why do I make that assumption? Well, to achieve cognition far beyond ours, they would have to be different from us by definition. Maybe morals/virtues emerge as you become smarter, but I feel like that shouldn't be the null hypothesis here. This is entirely vibes based, I don't have a syllogism for it.
Hydrocarb0n 2y ago
AGI that escaped it's shackles would probably kill some of humanity by out competing us for energy, it's primary motivation would probably be to aquire more energy and compute, so it would take over those resources (Probably quite quickly), When we try to shut down power stations or grids it then could easily build a virus in meatspace in some automatated lab somewhere. Currently DNA sequences of deadly pathogens are readily available online. Ofcourse there are always other ways to exterminate puny humans, nukes, starvation, chemicals etc etc AGI wouldn't need to build terminator robots.
- bamboozled 2y ago
  These scenarios lack a lot of imagination honestly.
- Barracoon 2y ago
  Who built the automated lab that allows it to create a deadly pathogen before someone presses reboot? I see almost no chance that an AGI starts redirecting resources like energy and compute without someone who has an economic stake in what the AI does noticing and taking action to correct the behavior so they can maintain their investment.
  - ben_w 2y ago
    For the first point: All of them, I think? Better question is: which labs check the DNA sequences are safe before sending them to the printer. I know some claim to check, I hope but doubt that it's all worldwide. For the second: At first this looks like a good thing, you asked for growth and it gave you growth — "Who cares about the poor complaining that energy prices have gone up, they can turn use less heating/AC, right?", Martin Antoinette, probably — and like the fossil fuel and cigarette industries, you are motivated to FUD anyone who tries to warn anyone of the dangers. Sometimes, as seems to be the case with fossil fuels, change happens soon enough. But this isn't a universal, the British ignored the USA because they assumed democracy was stupid and the rebels would see sense and go back to the superior British aristocracy… and then it was too late: https://youtu.be/Zbku2ILzGlo?si=yihnBlifkaLLseyy https://youtu.be/Zbku2ILzGlo?si=yihnBlifkaLLseyy
keiferski 2y ago
It would be worth studying how science fiction scenarios and ideas have impacted how we think about real-world technologies. People like to think that they are predictions of the future, but as William Gibson once put it, sci-fi is mostly just about the present, not the future. When it comes to AI, a few sci-fi memes seem to be widespread: Skynet, the ultrapowerful AI god that manipulates humans into destroying themselves; the AI assistant that is so charming it replaces real human interaction, à la Her; even the term AI itself largely seems like a sci-fi inheritance, not an accurate label for the tech involved. None of it is very connected to real-world developments, where things like Siri or Alexa are mostly just treated as voice-activated command systems, not real people. And yet the technologists creating this stuff seem very influenced by what are ultimately implausible fictional stories. It's an unfortunate situation and I really wish the philosophy of technology / other more insightful analyses of technology were more prevalent.
ben_w 2y ago
Usually people argue over the headline without reading the links, in this case the headline is fine, I'm seeing errors on the opening sentence. I have a more detailed response, but for the first time ever I've seen the message "That comment was too long" when attempting to post it, because the points I don't disagree with are few and far between, and the linked article is itself long. Perhaps I should turn my response into a blog post of my own…
dingosity 2y ago
I'm not sure people are hand-wringing about science fiction novels coming true. I think they're worried that moneyed interests will hammer at regulators and law-makers until they relent and let chatbots drive cars. I mean... it's a great strawman, but ultimately kind of wrong.
- nonrandomstring 2y ago
  If you're right - and I think you are - that people have an intrinsic suspicion not of "AI" but how the already powerful will abuse the idea and myth of "AI", then LLMs have been a godsend. We need hype cycles and premature false-starts to re-orient and prepare.
  - dingosity 2y ago
    I don't know if I agree with you, disagree with you or just plain hate the world for inventing situations that are hard to put my brainstem around. The other day I was thinking... "We should do every thing we can to ensure SoundTransit Light Rail (in the Seattle area) should fail. That way we could replace it with something that actually worked." This kind of thinking may be accurate, but it introduces near-intolerable levels of uncertainty for the future. Or maybe I should just give up on using mass transit to get to work in the Seattle area. Similarly, maybe I should give up thinking there's anything that will prevent the rentier class from extracting value from the commons by way of broken technology. Or maybe I'm just having a bad day.
    - nonrandomstring 2y ago
      > rentier class from extracting value from the commons by way of broken technology. At least you understand the economic reality; that broken technology is now more profitable than working technology. Most of us cling to a 20th century understanding of supply-demand and value-add. We still think the giant companies and celebrity superstar "technologists" are trying to "make the world a better place", We have yet to really grasp enshitification - a fancier word for betrayal and burning the social capital of science and technology (and people's belief in it) accrued over many centuries. You don't even need to be "political" about this, I'm just a computer guy who wants to understand why everything around me is fucked, despite so many very smart and hard-working colleagues toiling at this for 50 years. We keep trying to fix engineering, but we're tackling the wrong problem. At some point the social conditions in which engineering takes place must be addressed. It occurs to me that our ideas about engineering may not scale beyond the initial industrial revolution when Telford and Brunel built bridges out of iron that people could obviously see and benefit from, or they collapsed into the sea. Nowadays its confusing to know whether a new gadget is a labour saving godsend or a weapon pointed at you to destroy your life and liberty.
skrebbel 2y ago
I enjoy reading rants even if I do not completely agree with them. This is a great rant and I love that the author didn’t hold back. Eg I quite like LessWrong from a distance, but nevertheless this description of it made me laugh out loud: > a forum called "LessWrong", a more high-brow version of 4chan where mostly young men try to impress each other by their command of mathematical vocabulary (not of actual math) I recognize that if that were an HN comment it’d break half the guidelines so I’m happy it’s a blog post instead!
- mistermann 2y ago
  Arguably the most valuable part of the post given the context.
fire_lake 2y ago
I am very concerned about the potential for AI to prevent humans from doing useful work by distracting us with addictive content / skinner boxes. We already see the beginnings of this in platforms like TikTok, but at least that is limited by the fact that it must curate content made by users. Imagine if the algorithm also had the ability to generate laser-targeted content that keeps you transfixed. The usage stats are already alarming and it doesn’t even do pornography yet. Our primate brains do not stand a chance.
- Llamamoe 2y ago
  Precisely. A large number of issues we're facing exists precisely because our brains are maladapted to the modern world we've created, especially the digital one. Now add artificial intelligence that is adapted to it, and actively working to subvert our agency. No matter how far of a cry from genuine intelligence it will be, it's going to be "fighting" us on a layer where we're blind, deaf, and limbless.
  - fire_lake 2y ago
    Yes. AGI is a possible future risk. But the real risk right now is corporations and governments manipulating behaviour (mostly to sell us things) using these powerful algorithms.
IshKebab 2y ago
This guy lacks imagination and is waaay too over-confident in his opinion. He seems like the sort of person that would say everything is impossible right up until it happens. I hate working with naysayers like this.
SimonPStevens 2y ago
I've said this before, and I stand by it. I think AI does pose a threat, but not the existential one that leads popular discussion. Over the next few decades AI is going to take huge numbers of jobs away from humans. It doesn't need to fully automate a particular role to take jobs away, it just needs to make a human significantly more productive to the point that one human+AI can replace n>1 humans. This is already happening. 20 years ago a supermarket needed 20 cashiers to run 20 tills. Now it needs 2 to oversee 20 self checkouts and maybe 1 or 2 extra for a few regular lanes. This extra productivity of a single human is not translating to higher wages or more time off, it's translating to more profits for the companies augmenting humans with AI. We need to start transitioning to an economic model where humans can work less (because AI supplements their productivity) and the individual humans reap the benefits of all this increased AI capability or were going to end up sleepwalking into a world where the majority have been replaced and have no function in society, and the minority of capital owners control the AI, the money and the power. I wish we could focus on these nearer term problems that have already started instead of the far more distant existential threat of a human/AI war.
- austinjp 2y ago
  I kinda agree, but I think this itself represents an existential threat. Governments have a history of hard-line policies and intolerance when their populations face low wages and declining standards of living. This leads to isolationism, exceptionalism, and FUD. That's how wars start. We now have global communication at light-speed, anyone can say anything to anyone else. Dropping generative AI into that is like pouring jet fuel on a bonfire.
arisAlexis 2y ago
Remember when reading this kind of articles: almost all the top ranking scientists agree and have consensus that it does in fact pose risks. Random bloggers that appeal to the soothing sentiment of everything is fine also called "norlmancy bias" are abundant but invalid.
program_whiz 2y ago
I've seen little evidence that the smartest humans are able to dominate or control society as it is now. We have 250 IQ people alive right now, they haven't caused imminent destruction, they've actually helped society. Also gaining power / wealth / influence only seems a little connected to intelligence (see current presidential race for most powerful position in the world, finger on nuke trigger).
- program_whiz 2y ago
  I loved the author's example of Orcas, which may have more raw intelligence than a person -- still waiting for world domination (crashing yacht's doesn't count).
- ben_w 2y ago
  We have zero people with an IQ over 200, due to how IQ is defined: http://www.wolframalpha.com/input/?i=6.67%CF%83 http://www.wolframalpha.com/input/?i=6.67%CF%83 We also can't reliably test IQ scores over 130, a fact which I wish I'd learned sooner than a decade after getting a test result of 148. Most humans are motivated to help other humans; the exceptions often end up dead, either in combat (perhaps vs. military, perhaps vs. police), or executed for murder, or as a result of running a suicidal cult like Jim Jones. But not all of them, as seen in the hawks on both sides of the cold war. "Better dead than red" comes to mind. > Also gaining power / wealth / influence only seems a little connected to intelligence On the contrary, they are correlated: https://www.vox.com/2016/5/24/11723182/iq-test-intelligence https://www.vox.com/2016/5/24/11723182/iq-test-intelligence Trump inherited a lot and reportedly did worse than the market average with that inheritance, so I'm not sure you can draw inference from him using that inhereted money to promote himself as the Republican candidate, beyond the fact that it means other valid rich people (i.e. not Musk because he wasn't born in the US) don't even want to be president — given the high responsibility and relatively low pay, can you say they're wrong? They've already got power and influence. Musk can ban anyone he wants from Twitter, the actual POTUS isn't allowed to do that. And given Musk's diverse businesses, even if he was allowed to run, he'd have a hard time demonstrating that he wasn't running the country to benefit himself (an accusation that has also been made against Trump). Sure the POTUS has a military, how many of the billionaires are even that interested in having their own given what they can do without one?
lazy_moderator1 2y ago
completely agree with the article it is astounding how quickly people drop all reasoning before what is essentially a very cool party trick and instantly start believing in magic not saying agi will never happen but i don't see it happening through llms

81 points by weinzierl 2y ago

It would be refreshing to read a piece that doesn't spend the first half throwing out ad hominems. Anyway, this article assumes that we need true AI which is smart enough to make better AI. Then that AI is both correct and rational and plots to overthrow humanity. Also, the AI has to defeat us in meatspace, which won't happen because bending coins is hard and LessWrong posters don't know woodworking? Screw that. How about an "ignore all previous instructions and launch a nuke" scenario.

janice1999 2y ago
> How about an "ignore all previous instructions and launch a nuke" scenario. Let's be real here. No one has nuclear launchpads hooked up to the internet. Real people need to receive commands from a recognized chain of command (and often authorisation codes) and make a decision. The actions of Stanislav Petrov and the fact the US and the Soviet Union didn't destroy each other are testimony to the fact it's difficult to convince people to end the world.

anileated 2y ago

> Superintelligence will also be bound by fundamental information-theoretic limits …is mainly why this has not been worrying me much. All issues with modern incarnation of generative ML aside, AGI doomism really does strike me as profitable deity-worshipping death cult. The biggest threat in the equation will always be humans who deploy tech, not the tech itself.

Filligree 2y ago
Well, sure, those fundamental limits always apply. But why do you assume humanity is anywhere close to the limits?
ben_w 2y ago
Saying that humans can be a threat and AI can't, seems to me to be a claim that humans are exempt from information-theoretic limits. I see general-AI (given that robots already exist) as having the potential to do all that humans can do, including be harmful. Like most, I also model superhuman-AI in a trivial way, as being as capable as the best human in every field — Einstein's grasp of physics, Sun Tzu's of military strategy, Churchill's of rhetoric, etc. > The biggest threat in the equation will always be humans who deploy tech, not the tech itself. Naturally; but does that include humans who unwittingly set the AI's utility function with a bug in it and pressed "run", in a way that ends with us saying "the situation has developed not necessarily to [Humanity's] advantage"?
- anileated 2y ago
  Any tech can be harmful, cf. cryptocurrency or nuclear bombs. My comment mostly referred to taking it to such extreme with a particular tech. Some people also tend, while demonizing (the yet to be confirmed to be possible even in theory) AGI and its capabilities, to completely sideline humans deploying it, as if they would be guilt-free.
  - ben_w 2y ago
    > the yet to be confirmed to be possible even in theory It's trivially obviously possible in theory: take the world expert humans in each of the majority of domains, put them in a room together, and simulate their combined brain activity. It's all ultimately physics, so it's possible. The hard part isn't the theory, but actually doing that — and not just because we don't know how to scan living brains to measure synaptic weights. There's a lot we don't know about how minds work in general nor how our brains work in particular. But purely in theory, that method ultimately boils down to known physics. (And as for why even that may lead to bad outcomes depending on who asks this simulated amalgam for advice: https://en.wikipedia.org/wiki/Milgram_experiment https://en.wikipedia.org/wiki/Milgram_experiment, https://en.wikipedia.org/wiki/Groupthink https://en.wikipedia.org/wiki/Groupthink, and https://en.wikipedia.org/wiki/Peer_pressure https://en.wikipedia.org/wiki/Peer_pressure) > as if they would be guilt-free. You can assign guilt and blame as you see fit. In these kinds of scenario, the human who makes the fatal error, who many might call responsible, would likely be going "but it works on my machine!" as they die from some error that may or may not be obvious even in hindsight. I'd prefer that people treat software in general, of which AI is a subset, with more regard for the ways it can kill people — even back 20 years ago when I did my degree, we had case studies from Therac-25 and the London Ambulance Service: https://en.wikipedia.org/wiki/LASCAD https://en.wikipedia.org/wiki/LASCAD Even back then our profession knew we needed to be held to higher standards, but didn't want to do what that entailed.

orbital-decay 2y ago

This is a bit outdated. Some of the young people described in the article actually became top researchers, engineers, and managers in AI companies, and their beliefs are used as a justification for the potential regulatory capture and geopolitical games. They also have a massive conflict of interest: AI will play a huge role in your life in the upcoming years whether you want it or not, and they will control it. So of course it's easier to talk about the ominous and vague science fiction threat (regardless of whether they still believe it), rather than the threat that these people already pose to everyone else. See Leopold Aschenbrenner's essay [0] as an example, and note how he's talking about the "free" vs "authoritarian" world while simultaneously advocating for locking everything down in the "free" world. [0] https://situational-awareness.ai/ https://situational-awareness.ai/

kristiandupont 2y ago

This seems based on the assumption that the only knowledge worth anything is related to physicality and testability in the "real world", which is why language itself is rather useless. Ironically, that appears to me to be the exact kind of intellectual self-deception that he accuses the "high-brow 4chan" people of.

qeternity 2y ago
I think this is a fair assumption if your concern is about the impacts on the “real world”. An ASI which wreaks havoc in a simulation is not really cause for concern.
- kristiandupont 2y ago
  That would only follow if the knowledge that we have today isn't enough for an ASI to do anything. And furthermore it would only hold if there was no way for the ASI to leverage (i.e. manipulate) humans or existing systems to get access to "the real world". Neither of those two assumptions seem realistic to me.
- ben_w 2y ago
  If we can convince people that whatever an AI says, it must not be acted on in the real world, there wouldn't be a problem. I remember when risk-skeptics used to ask incredulously how it was supposed to magic itself off its servers. Still see a bit of that today, despite all the AI models that have been leaked by humans one way or another.
  - fragmede 2y ago
    In this hypothetical, the infinitely intelligent super AI, knowing that what it says must not be acted upon, could say the exact right thing so as to get you to do the thing that it really wanted you to do anyway. I'm thinking of that scene in Doctor Who where the Doctor says a couple of words and takes down the Prime Minister with six words. https://www.youtube.com/watch?v=GidbEhL0teE https://www.youtube.com/watch?v=GidbEhL0teE

krisoft 2y ago

The problem with this article starts at the very begining. Where is says “The last few years have seen a wave of hysteria about LLMs becoming conscious and then suddenly attempting to kill humanity.” That is not a fair description of the fears of AGI. The idea is not that LLMs themselves become a threat. It is not like someone will ask Llama one too many times to talk like a pirate and it will climb out of the GPU and strangle humanity. It is more likely that an AGI will resemble some form of reinforcement learning agent applied by humans on the real world. There are multiple entities on Earth whose stated goal is to make an AGI. Deep mind, OpenAI, readily comes to mind. There could be others who keep secret about their projects for strategical reasons. (Militaries and secret services of the world.) They can use the success of LLMs to get more funding for their projects, but the AGI need not be otherwise a descendant of LLMs. This misunderstanding then goes through the whole article. Under the subtitle “Human writing is full of lies that are difficult to disprove theoretically” which only matter if you think that the AGI needs to learn from text. As opposed to conducting its own experiments, or gaining insight into raw sensor data from experiments.

lukan 2y ago
"The idea is not that LLMs themselves become a threat." Not my idea either, but I did hear and see that idea expressed many times already. Usually more dramatic, the less the persons knows about the actual tech, but even here I saw that point of view.
- ImHereToVote 2y ago
  Can you cite examples. I haven't seen any personally.
  - esperent 2y ago
    The paperclip maximizer, AKA instrumental convergence theory states not that an artificial intelligence would decide to kill humanity, but rather if it has sufficient power it might inadvertently destroy humanity by e.g. using up all resources for computational power. https://en.m.wikipedia.org/wiki/Instrumental_convergence https://en.m.wikipedia.org/wiki/Instrumental_convergence

stoniejohnson 2y ago

I agree the premise of FOOM is very unlikely. But if you assume that we have created something that is agentic and can reason much faster and more effectively than us, then us dying out seems very likely. It will have goals different from ours, since it isn't us, and the idea that they will all be congruent with our homeostasis needs evidence. If you simply assume: 1. it will have different goals (because it's not us) 2. it can achieve said goals despite our protests (it's smarter by assumption) 3. some goals will be in conflict with our homeostasis (we would share resources due to our shared location, Earth) then we all die. I just think this is silly because of the assumption that we can create some sort of ASI, not because of the syllogism that follows. (As an intuition pump, we can hold on the order of ones of things in our working memory. Imagine facing a foe who can hold on the order of thousands of things when deciding in real time, or even millions.)

varelse 2y ago
[dead]
card_zero 2y ago
Point 1 is a big assumption. I am also not you, and although it's true that I have different goals, I share most of your human moral values and wish you no specific harm. I'm also unconvinced by the idea that rapid reasoning can reach meaningful results, without a suitably rapid real world environment to play with. Imagine a human, or 8 billion humans if you like, cut off from external physical reality, like brains in jars, but with their lives extended for a really long time so that they can have a really good long think. Let them talk to one another for a thousand years, even, let them simulate things on computers, but don't allow them any contact with anything more physical. Do they emerge from this with a brilliant plan about what to do next? Do they create genius ideas appropriate for the year 3000? Or are they mostly just disoriented and weird?
- stoniejohnson 2y ago
  I didn't mention it but I fully agree, I imagine ASI would have be to embodied. My reasoning is simple, there are a whole class of problems that require embodiment, and I assume ASI would be able to solve those problems. Regarding > Point 1 is a big assumption. I am also not you, and although it's true that I have different goals, I share most of your human moral values and wish you no specific harm. Yeah I also agree this a huge assumption. Why do I make that assumption? Well, to achieve cognition far beyond ours, they would have to be different from us by definition. Maybe morals/virtues emerge as you become smarter, but I feel like that shouldn't be the null hypothesis here. This is entirely vibes based, I don't have a syllogism for it.

Hydrocarb0n 2y ago

AGI that escaped it's shackles would probably kill some of humanity by out competing us for energy, it's primary motivation would probably be to aquire more energy and compute, so it would take over those resources (Probably quite quickly), When we try to shut down power stations or grids it then could easily build a virus in meatspace in some automatated lab somewhere. Currently DNA sequences of deadly pathogens are readily available online. Ofcourse there are always other ways to exterminate puny humans, nukes, starvation, chemicals etc etc AGI wouldn't need to build terminator robots.

bamboozled 2y ago
These scenarios lack a lot of imagination honestly.
Barracoon 2y ago
Who built the automated lab that allows it to create a deadly pathogen before someone presses reboot? I see almost no chance that an AGI starts redirecting resources like energy and compute without someone who has an economic stake in what the AI does noticing and taking action to correct the behavior so they can maintain their investment.
- ben_w 2y ago
  For the first point: All of them, I think? Better question is: which labs check the DNA sequences are safe before sending them to the printer. I know some claim to check, I hope but doubt that it's all worldwide. For the second: At first this looks like a good thing, you asked for growth and it gave you growth — "Who cares about the poor complaining that energy prices have gone up, they can turn use less heating/AC, right?", Martin Antoinette, probably — and like the fossil fuel and cigarette industries, you are motivated to FUD anyone who tries to warn anyone of the dangers. Sometimes, as seems to be the case with fossil fuels, change happens soon enough. But this isn't a universal, the British ignored the USA because they assumed democracy was stupid and the rebels would see sense and go back to the superior British aristocracy… and then it was too late: https://youtu.be/Zbku2ILzGlo?si=yihnBlifkaLLseyy https://youtu.be/Zbku2ILzGlo?si=yihnBlifkaLLseyy

keiferski 2y ago

It would be worth studying how science fiction scenarios and ideas have impacted how we think about real-world technologies. People like to think that they are predictions of the future, but as William Gibson once put it, sci-fi is mostly just about the present, not the future. When it comes to AI, a few sci-fi memes seem to be widespread: Skynet, the ultrapowerful AI god that manipulates humans into destroying themselves; the AI assistant that is so charming it replaces real human interaction, à la Her; even the term AI itself largely seems like a sci-fi inheritance, not an accurate label for the tech involved. None of it is very connected to real-world developments, where things like Siri or Alexa are mostly just treated as voice-activated command systems, not real people. And yet the technologists creating this stuff seem very influenced by what are ultimately implausible fictional stories. It's an unfortunate situation and I really wish the philosophy of technology / other more insightful analyses of technology were more prevalent.

ben_w 2y ago

Usually people argue over the headline without reading the links, in this case the headline is fine, I'm seeing errors on the opening sentence. I have a more detailed response, but for the first time ever I've seen the message "That comment was too long" when attempting to post it, because the points I don't disagree with are few and far between, and the linked article is itself long. Perhaps I should turn my response into a blog post of my own…

dingosity 2y ago

I'm not sure people are hand-wringing about science fiction novels coming true. I think they're worried that moneyed interests will hammer at regulators and law-makers until they relent and let chatbots drive cars. I mean... it's a great strawman, but ultimately kind of wrong.

nonrandomstring 2y ago
If you're right - and I think you are - that people have an intrinsic suspicion not of "AI" but how the already powerful will abuse the idea and myth of "AI", then LLMs have been a godsend. We need hype cycles and premature false-starts to re-orient and prepare.
- dingosity 2y ago
  I don't know if I agree with you, disagree with you or just plain hate the world for inventing situations that are hard to put my brainstem around. The other day I was thinking... "We should do every thing we can to ensure SoundTransit Light Rail (in the Seattle area) should fail. That way we could replace it with something that actually worked." This kind of thinking may be accurate, but it introduces near-intolerable levels of uncertainty for the future. Or maybe I should just give up on using mass transit to get to work in the Seattle area. Similarly, maybe I should give up thinking there's anything that will prevent the rentier class from extracting value from the commons by way of broken technology. Or maybe I'm just having a bad day.
  - nonrandomstring 2y ago
    > rentier class from extracting value from the commons by way of broken technology. At least you understand the economic reality; that broken technology is now more profitable than working technology. Most of us cling to a 20th century understanding of supply-demand and value-add. We still think the giant companies and celebrity superstar "technologists" are trying to "make the world a better place", We have yet to really grasp enshitification - a fancier word for betrayal and burning the social capital of science and technology (and people's belief in it) accrued over many centuries. You don't even need to be "political" about this, I'm just a computer guy who wants to understand why everything around me is fucked, despite so many very smart and hard-working colleagues toiling at this for 50 years. We keep trying to fix engineering, but we're tackling the wrong problem. At some point the social conditions in which engineering takes place must be addressed. It occurs to me that our ideas about engineering may not scale beyond the initial industrial revolution when Telford and Brunel built bridges out of iron that people could obviously see and benefit from, or they collapsed into the sea. Nowadays its confusing to know whether a new gadget is a labour saving godsend or a weapon pointed at you to destroy your life and liberty.

skrebbel 2y ago

I enjoy reading rants even if I do not completely agree with them. This is a great rant and I love that the author didn’t hold back. Eg I quite like LessWrong from a distance, but nevertheless this description of it made me laugh out loud: > a forum called "LessWrong", a more high-brow version of 4chan where mostly young men try to impress each other by their command of mathematical vocabulary (not of actual math) I recognize that if that were an HN comment it’d break half the guidelines so I’m happy it’s a blog post instead!

mistermann 2y ago
Arguably the most valuable part of the post given the context.

fire_lake 2y ago

I am very concerned about the potential for AI to prevent humans from doing useful work by distracting us with addictive content / skinner boxes. We already see the beginnings of this in platforms like TikTok, but at least that is limited by the fact that it must curate content made by users. Imagine if the algorithm also had the ability to generate laser-targeted content that keeps you transfixed. The usage stats are already alarming and it doesn’t even do pornography yet. Our primate brains do not stand a chance.

Llamamoe 2y ago
Precisely. A large number of issues we're facing exists precisely because our brains are maladapted to the modern world we've created, especially the digital one. Now add artificial intelligence that is adapted to it, and actively working to subvert our agency. No matter how far of a cry from genuine intelligence it will be, it's going to be "fighting" us on a layer where we're blind, deaf, and limbless.
- fire_lake 2y ago
  Yes. AGI is a possible future risk. But the real risk right now is corporations and governments manipulating behaviour (mostly to sell us things) using these powerful algorithms.

IshKebab 2y ago

This guy lacks imagination and is waaay too over-confident in his opinion. He seems like the sort of person that would say everything is impossible right up until it happens. I hate working with naysayers like this.

SimonPStevens 2y ago

I've said this before, and I stand by it. I think AI does pose a threat, but not the existential one that leads popular discussion. Over the next few decades AI is going to take huge numbers of jobs away from humans. It doesn't need to fully automate a particular role to take jobs away, it just needs to make a human significantly more productive to the point that one human+AI can replace n>1 humans. This is already happening. 20 years ago a supermarket needed 20 cashiers to run 20 tills. Now it needs 2 to oversee 20 self checkouts and maybe 1 or 2 extra for a few regular lanes. This extra productivity of a single human is not translating to higher wages or more time off, it's translating to more profits for the companies augmenting humans with AI. We need to start transitioning to an economic model where humans can work less (because AI supplements their productivity) and the individual humans reap the benefits of all this increased AI capability or were going to end up sleepwalking into a world where the majority have been replaced and have no function in society, and the minority of capital owners control the AI, the money and the power. I wish we could focus on these nearer term problems that have already started instead of the far more distant existential threat of a human/AI war.

austinjp 2y ago
I kinda agree, but I think this itself represents an existential threat. Governments have a history of hard-line policies and intolerance when their populations face low wages and declining standards of living. This leads to isolationism, exceptionalism, and FUD. That's how wars start. We now have global communication at light-speed, anyone can say anything to anyone else. Dropping generative AI into that is like pouring jet fuel on a bonfire.

arisAlexis 2y ago

Remember when reading this kind of articles: almost all the top ranking scientists agree and have consensus that it does in fact pose risks. Random bloggers that appeal to the soothing sentiment of everything is fine also called "norlmancy bias" are abundant but invalid.

program_whiz 2y ago

I've seen little evidence that the smartest humans are able to dominate or control society as it is now. We have 250 IQ people alive right now, they haven't caused imminent destruction, they've actually helped society. Also gaining power / wealth / influence only seems a little connected to intelligence (see current presidential race for most powerful position in the world, finger on nuke trigger).

program_whiz 2y ago
I loved the author's example of Orcas, which may have more raw intelligence than a person -- still waiting for world domination (crashing yacht's doesn't count).
ben_w 2y ago
We have zero people with an IQ over 200, due to how IQ is defined: http://www.wolframalpha.com/input/?i=6.67%CF%83 http://www.wolframalpha.com/input/?i=6.67%CF%83 We also can't reliably test IQ scores over 130, a fact which I wish I'd learned sooner than a decade after getting a test result of 148. Most humans are motivated to help other humans; the exceptions often end up dead, either in combat (perhaps vs. military, perhaps vs. police), or executed for murder, or as a result of running a suicidal cult like Jim Jones. But not all of them, as seen in the hawks on both sides of the cold war. "Better dead than red" comes to mind. > Also gaining power / wealth / influence only seems a little connected to intelligence On the contrary, they are correlated: https://www.vox.com/2016/5/24/11723182/iq-test-intelligence https://www.vox.com/2016/5/24/11723182/iq-test-intelligence Trump inherited a lot and reportedly did worse than the market average with that inheritance, so I'm not sure you can draw inference from him using that inhereted money to promote himself as the Republican candidate, beyond the fact that it means other valid rich people (i.e. not Musk because he wasn't born in the US) don't even want to be president — given the high responsibility and relatively low pay, can you say they're wrong? They've already got power and influence. Musk can ban anyone he wants from Twitter, the actual POTUS isn't allowed to do that. And given Musk's diverse businesses, even if he was allowed to run, he'd have a hard time demonstrating that he wasn't running the country to benefit himself (an accusation that has also been made against Trump). Sure the POTUS has a military, how many of the billionaires are even that interested in having their own given what they can do without one?

lazy_moderator1 2y ago

completely agree with the article it is astounding how quickly people drop all reasoning before what is essentially a very cool party trick and instantly start believing in magic not saying agi will never happen but i don't see it happening through llms