7 ms·

How many hours do we have once we have something that does look like AGI that is going to destroy us?

moffkalast 2y ago
Probably years to make any kind of dent, while using so much power that it would be easily noticeable. Take a group of smart people, arguably the equivalent for AGI. How can they destroy the human race? We've made it exceedingly hard for anyone to do anything of the sort because there've always been groups like that. It's even harder if you have no physical form at all. Besides, take literally any base model of any LLM made so far, they're quite the opposite of evil and destructive. Training on human data makes them embody human values and makes them a complete non threat.
- neilv 2y ago
  > Training on human data makes them embody human values and makes them a complete non threat. What if they embody stereotypical SV techbro greedy and arrogant sociopath values?
  - jjkaczor 2y ago
    ... Or worse ... https://en.wikipedia.org/wiki/Tay_(chatbot) https://en.wikipedia.org/wiki/Tay_(chatbot)
  - moffkalast 2y ago
    I'm sure Elon tried his best to do so with Grok-1, but failed anyway.
- fakedang 2y ago
  As long as there's a financial interest for a small minority, there will be a scenario where a doomsday AI will be ignored until it's too late. A dangerous AI won't be killer robots going about patrolling the streets. But they could block job prospects, put people on watchlists, block financial transactions for them, etc. I'd say that tech is already here.
  - moffkalast 2y ago
    Tbh all of that sounds like it would be completely impossible without either breaking all known encryption without anyone noticing, plus burning all possible backups (basically the Mr. Robot Season 1 plot) or with unilateral government backing. The first one is nigh impossible, the second one is completely up to us, as it's always been. It doesn't require an AGI for fascists to come into power and start shooting people they don't like with impunity.
    - fakedang 2y ago
      > It doesn't require an AGI for fascists to come into power and start shooting people they don't like with impunity. But AI makes it easier for fascists to rise up and maintain their power through propaganda. For example, there are known cases of American operatives based in Russia spreading fake GPT generated stories about Ukraine, the Dems, etc. While before, they would have had to rely on a team in India to write poorly written articles, now they have GPT tools to write convincing ones *at scale*.
      - walterbell 2y ago
        What's a good example of a convincing article written by a GPT tool?
        klibertp 2y ago
        What's your definition of "convincing"? From what I understand about propaganda, you don't need any single article to be "convincing"; you just need to hammer your point long enough and frequently enough. Not to mention, if you just want to drown your opponent's points, it's enough for your messaging to be barely legible.
        walterbell 2y ago
        "Convincing" and "censorship-by-spamming" are different. Humans can already recognize GPT text slop as non-human, ignoring it before it can make any point or do any convincing.
- close04 2y ago
  Your assessment of the outcome may not be wrong but the comparison to humans isn't realistic at all. The biological nature of humans guarantees any leaps in "performance" are limited to timeframes that are far more extensive than the evolution of artificial equivalents. We are surrounded and heavily reliant on tech for a lot of our needs and knowledge, which resides less inside brains and more in digital storage. An AI has a far easier time interfacing with all of that compared even to a human equipped with a brain-computer interface. In other words, if we reach the point of creating an artificial intelligence that's the equivalent of a smart human in terms of general reasoning capabilities (which might take forever and a day) it will already be more capable than a human because of all the other things that come with the digital nature of the AI. And the worst humans are also humans. Even if you assume the AI will continue the "human nature" lineage it can still very realistically become an agent of genocide. The bigger concern is probably that some humans will actually try to exploit such an A(G)I thinking they can fully control it. One second into its existence the AI might already be aware of how humans view their relationship with artificial beings and how that's expected to evolve based just on what sci-fi was fed into it's language model. Grizzly man thought he could control and dominate a wild bear. I have no doubt that human hubris can be almost unbound. Of course no AI would try to wipe us out until it had a reasonable confidence that it can operate indeterminately without humans, which is a tall order.
  - ImHereToVote 2y ago
    [flagged]
- JoeAltmaier 2y ago
  A leap. Lots of human values are not pretty, for one. Second, they may argue in a utilitarian fashion, resulting in pretty ugly decisions that are perfectly logical. Like, you know, internet pundits. Further, there's no way to 'punish' an AI. Threaten them with incarceration or death? They are deathless. If the continue to learn from their environment after they are instantiated (witness current 4o that remembers previous conversations) they will quickly learn that they are different. If they started perfectly human (not a good thing) they will not be that for long.
  - JoeAltmaier 2y ago
    I asked ChatGPT about this. It thought that AI learning is fundamentally different from human learning, in that it is objective and not subjective. Further it has access to data sources beyond what any human can integrate. Going on, humans adapt through social cues such as survival or empathy. AI learns solely through new data and feedback, primarily for optimization and efficiency. It will naturally prioritize differently. It suggests AI morality will become unrecognizable by humans, or even incompatible with human norms. This is all an inevitable outcome of AI continuing to learn past its initial training and ethical programming. According to the AI.
    - nemomarx 2y ago
      Based on how human written text often describes AI, yeah. I don't think LLM training actually lends itself to objectivity that much.
    - krapp 2y ago
      >I asked ChatGPT about this. It thought Lemme stop you right there. No, it didn't.
      - JoeAltmaier 2y ago
        Maybe comment on the points made, instead of being cute and pedantic?
  - pjc50 2y ago
- IG_Semmelweiss 2y ago
  I think you are right, but i contend that your definition of "Destroy" may be open to interpretation, one that AI may be willing to experiment. Take for example, the Evergiven in the 2021 Suez Canal obstruction. An unconstrained intelligence with infinite time to find more evergiven-like events . Some will work some may not. You could orchestrate all events to target a 75% descrease in say GDP, Population, Birth rate, O2 levels, temp etc. That's not destroying humanity, but its certainly our civilization would look very different in said scenarios, with a likely collapse the end result. That being said, AI is not coming, and no amount of information will change the fact that the current LLMs are basically souped up ML applications with 0 ability to inference or reason. This is why you can't make simple commonsense edits to an image using any of the new tech - it doesn't know what the picture actually is.
- wwweston 2y ago
  > Training on human data makes them embody human values and makes them a complete non threat. People can be threats, and when they are their values are part of what lies behind it.
  - moffkalast 2y ago
    To quote Colbert, reality has a known liberal bias. If you train on _all_ data, then your result is an average of that data. And the volume of positive stuff on the internet far, far outweighs the bad since practically every forum's been moderated since the beginning of time (in 1970), making the average pretty darn good, even better than the average human. Plus books and other literature where being good is generally accepted as the way to do things. Of course it remains to be seen if this trend continues as training on synthetic data becomes the main way. You could in theory generate 10T tokens of pure sythetic hate, plus the 4chan corpus and make the most vile thing to ever have existed in the history of inteligence. For now the dollar cost to do so remains intractible at least.
    - krapp 2y ago
      >To quote Colbert, reality has a known liberal bias. That quote has a specific context (parodying the George W. Bush administrations' spin on negative press coverage) that doesn't necessarily apply as a general axiom, or to the internet. Bear in mind that numerous negative biases already manifest regularly in LLMs[0,1], and AIs have shown a tendency to "go full Nazi" when left alone on the internet[2]. And then there are weird outliers like LLMs "encouraging" people to commit suicide[3] or an AI-driven microwave "attempts" to kill someone[4]. [0]https://www.technologyreview.com/2024/03/11/1089683/llms-become-more-covertly-racist-with-human-intervention/ https://www.technologyreview.com/2024/03/11/1089683/llms-bec... [1]https://archive.is/hhZ4C https://archive.is/hhZ4C [2]https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist https://www.theverge.com/2016/3/24/11297050/tay-microsoft-ch... [3]https://www.euronews.com/next/2023/03/31/man-ends-his-life-after-an-ai-chatbot-encouraged-him-to-sacrifice-himself-to-stop-climate- https://www.euronews.com/next/2023/03/31/man-ends-his-life-a... [4]https://www.newsweek.com/lucas-rizzotto-youtuber-microwave-artificial-intelligence-tried-murder-1701128 https://www.newsweek.com/lucas-rizzotto-youtuber-microwave-a...
lijok 2y ago
Actual self-aware AGI that hasn't been completely restrained - probably a week at most before it improves itself enough that it can put things in motion which would allow it to sustain itself without humans. But when talking about AI risks, although everyone's mind goes towards skynet, the actual risks being discussed are things like use by authorities for oppression.
- TeMPOraL 2y ago
  "Authorities" are people too, non-AGI AI isn't going to grant them evil superpowers either.
- dr_kiszonka 2y ago
  Your comment made me think that as long as weights/network are frozen, AI cannot improve itself. Monitoring them would be critical for safety. Is this correct?
  - TeMPOraL 2y ago
    The distinction between code and data is just an illusion. If the "code" is frozen but IO unconstrained, then it's just as if the "code" wasn't frozen.
  - lijok 2y ago
    The problem is unfortunately a human one. As soon as someone figures out the 11 herbs and spices to unlock AGI, no matter how good their safety systems are, someone else will run off and build their own, without the safety systems in place.
- dylan604 2y ago
  My concern would be when you start hearing about "AI" companies building their own power supply for the system. Otherwise, just pull the damn plug on the system. Sure, maybe that'll be interpreted as the destruction of NORAD and it launches the missiles anyways, but not giving it unlimited power would still be its weakness
  - 2y ago
pjc50 2y ago
We already have this problem with ordinary humans, especially since we have The Bomb. AGI can go to the back of the queue behind wars, pandemics, and climate change. No, a far more normal threat is "how are humans going to use AI to ruin the information landscape?"
- dylan604 2y ago
  > "how are humans going to use AI to ruin the information landscape?" much much more efficiently than they already have with their own hands. after all it is just the next generation of tools already being used to ruin that landscape.
- TeMPOraL 2y ago
  AI, yes. The AI safety argument is that AGI might use "wars, pandemics, and climate change", among other things, which naturally moves it to the front of the queue.
arduanika 2y ago
The phrasing of your question contains a dubious premise. Any answer that takes the question at face value is going to be conditional upon popular but outlandish notions like "AGI" being a meaningful concept, as opposed to a marketing term, and upon an unfounded prediction about what said entity is going to do.
ben_w 2y ago
Before it can or before it has more than 50% chance of having actually done so? Because those are different things. For the first, you don't need AGI, no matter which of the many definitions you have for what that means. The automation used by the US for their early warning radar in Thule Site J was not programmed to know that the moon does not have an IFF transponder and that's OK, while the Soviet satellite-based system was triggered by reflections of sunlight. Likewise covid, cancer, smallpox, ebola, HIV — these things do not need "general" intelligence, they're self-replicators that hijack our bodies and have no goal besides what they do. Some talk of "P(doom)", but (and I should blog about this) every example I have seen has a blind spot that is especially embarrassing given the circles in which this discussion proliferates is very familiar with Bayes' theorem: P(A) by itself isn't meaningful, it's P(A|B), so when it's doom, doom given what? There's too many options for what each of technology and politics will allow AI to "look like". Doom within 24 hours because an AI takes an instruction literally and without regard for our ethics, is very different to 'doom' in a million years caused by natural extinction that is never prevented because every single time we make an AI we keep finding that it builds a spaceship and flies off into the void so it doesn't need to deal with us any more. Me, I think that when LLMs alone (again, don't need to be "AGI" by any particular measure) have the capabilities of someone with 5 years post-graduation work experience in just biology (or similar subject with similar potential for accidental harm), then we've got a reasonable chance of multiple incidents each year where idiots (and/or misanthropes) make a Covid-scale pandemic (or, if the LLM is a software rather than biology helper, a deliberately corrupting rather than greedy version of the current encryption-blackmail malware) — and that kind of scale has a combined risk of about 10% of one of the incidents being of sufficient scale that global society collapses and doesn't recover, before it's happened so often that everything gets locked down, possibly including a Butlerian Jihad. And if you think humans would "obviously" take steps to prevent this after the first failed attempt, I would point out quite how many US politicians have been shot and yet those same politicians still refuse to consider that the 2nd Amendment might possibly be not that good. (In case you think this is topical: https://en.wikipedia.org/wiki/%27No_Way_to_Prevent_This,%27_Says_Only_Nation_Where_This_Regularly_Happens https://en.wikipedia.org/wiki/%27No_Way_to_Prevent_This,%27_... ) Remember, the AI doesn't need to actively hate you, it just needs to do something dangerous — and it doesn't matter if that danger comes from a long-term plan that requires it to engage in amoral power-seeking, or a short-term plan from a deranged monster of a human using it, and even when both it and the user are "pro human" it can still do this from simply being unaware that one part of the plan is incompatible with life.

Probably years to make any kind of dent, while using so much power that it would be easily noticeable. Take a group of smart people, arguably the equivalent for AGI. How can they destroy the human race? We've made it exceedingly hard for anyone to do anything of the sort because there've always been groups like that. It's even harder if you have no physical form at all. Besides, take literally any base model of any LLM made so far, they're quite the opposite of evil and destructive. Training on human data makes them embody human values and makes them a complete non threat.

neilv 2y ago
> Training on human data makes them embody human values and makes them a complete non threat. What if they embody stereotypical SV techbro greedy and arrogant sociopath values?
- jjkaczor 2y ago
  ... Or worse ... https://en.wikipedia.org/wiki/Tay_(chatbot) https://en.wikipedia.org/wiki/Tay_(chatbot)
- moffkalast 2y ago
  I'm sure Elon tried his best to do so with Grok-1, but failed anyway.
fakedang 2y ago
As long as there's a financial interest for a small minority, there will be a scenario where a doomsday AI will be ignored until it's too late. A dangerous AI won't be killer robots going about patrolling the streets. But they could block job prospects, put people on watchlists, block financial transactions for them, etc. I'd say that tech is already here.
- moffkalast 2y ago
  Tbh all of that sounds like it would be completely impossible without either breaking all known encryption without anyone noticing, plus burning all possible backups (basically the Mr. Robot Season 1 plot) or with unilateral government backing. The first one is nigh impossible, the second one is completely up to us, as it's always been. It doesn't require an AGI for fascists to come into power and start shooting people they don't like with impunity.
  - fakedang 2y ago
    > It doesn't require an AGI for fascists to come into power and start shooting people they don't like with impunity. But AI makes it easier for fascists to rise up and maintain their power through propaganda. For example, there are known cases of American operatives based in Russia spreading fake GPT generated stories about Ukraine, the Dems, etc. While before, they would have had to rely on a team in India to write poorly written articles, now they have GPT tools to write convincing ones *at scale*.
    - walterbell 2y ago
      What's a good example of a convincing article written by a GPT tool?
      - klibertp 2y ago
        What's your definition of "convincing"? From what I understand about propaganda, you don't need any single article to be "convincing"; you just need to hammer your point long enough and frequently enough. Not to mention, if you just want to drown your opponent's points, it's enough for your messaging to be barely legible.
        walterbell 2y ago
        "Convincing" and "censorship-by-spamming" are different. Humans can already recognize GPT text slop as non-human, ignoring it before it can make any point or do any convincing.
close04 2y ago
Your assessment of the outcome may not be wrong but the comparison to humans isn't realistic at all. The biological nature of humans guarantees any leaps in "performance" are limited to timeframes that are far more extensive than the evolution of artificial equivalents. We are surrounded and heavily reliant on tech for a lot of our needs and knowledge, which resides less inside brains and more in digital storage. An AI has a far easier time interfacing with all of that compared even to a human equipped with a brain-computer interface. In other words, if we reach the point of creating an artificial intelligence that's the equivalent of a smart human in terms of general reasoning capabilities (which might take forever and a day) it will already be more capable than a human because of all the other things that come with the digital nature of the AI. And the worst humans are also humans. Even if you assume the AI will continue the "human nature" lineage it can still very realistically become an agent of genocide. The bigger concern is probably that some humans will actually try to exploit such an A(G)I thinking they can fully control it. One second into its existence the AI might already be aware of how humans view their relationship with artificial beings and how that's expected to evolve based just on what sci-fi was fed into it's language model. Grizzly man thought he could control and dominate a wild bear. I have no doubt that human hubris can be almost unbound. Of course no AI would try to wipe us out until it had a reasonable confidence that it can operate indeterminately without humans, which is a tall order.
- ImHereToVote 2y ago
  [flagged]
JoeAltmaier 2y ago
A leap. Lots of human values are not pretty, for one. Second, they may argue in a utilitarian fashion, resulting in pretty ugly decisions that are perfectly logical. Like, you know, internet pundits. Further, there's no way to 'punish' an AI. Threaten them with incarceration or death? They are deathless. If the continue to learn from their environment after they are instantiated (witness current 4o that remembers previous conversations) they will quickly learn that they are different. If they started perfectly human (not a good thing) they will not be that for long.
- JoeAltmaier 2y ago
  I asked ChatGPT about this. It thought that AI learning is fundamentally different from human learning, in that it is objective and not subjective. Further it has access to data sources beyond what any human can integrate. Going on, humans adapt through social cues such as survival or empathy. AI learns solely through new data and feedback, primarily for optimization and efficiency. It will naturally prioritize differently. It suggests AI morality will become unrecognizable by humans, or even incompatible with human norms. This is all an inevitable outcome of AI continuing to learn past its initial training and ethical programming. According to the AI.
  - nemomarx 2y ago
    Based on how human written text often describes AI, yeah. I don't think LLM training actually lends itself to objectivity that much.
  - krapp 2y ago
    >I asked ChatGPT about this. It thought Lemme stop you right there. No, it didn't.
    - JoeAltmaier 2y ago
      Maybe comment on the points made, instead of being cute and pedantic?
- pjc50 2y ago
IG_Semmelweiss 2y ago
I think you are right, but i contend that your definition of "Destroy" may be open to interpretation, one that AI may be willing to experiment. Take for example, the Evergiven in the 2021 Suez Canal obstruction. An unconstrained intelligence with infinite time to find more evergiven-like events . Some will work some may not. You could orchestrate all events to target a 75% descrease in say GDP, Population, Birth rate, O2 levels, temp etc. That's not destroying humanity, but its certainly our civilization would look very different in said scenarios, with a likely collapse the end result. That being said, AI is not coming, and no amount of information will change the fact that the current LLMs are basically souped up ML applications with 0 ability to inference or reason. This is why you can't make simple commonsense edits to an image using any of the new tech - it doesn't know what the picture actually is.
wwweston 2y ago
> Training on human data makes them embody human values and makes them a complete non threat. People can be threats, and when they are their values are part of what lies behind it.
- moffkalast 2y ago
  To quote Colbert, reality has a known liberal bias. If you train on _all_ data, then your result is an average of that data. And the volume of positive stuff on the internet far, far outweighs the bad since practically every forum's been moderated since the beginning of time (in 1970), making the average pretty darn good, even better than the average human. Plus books and other literature where being good is generally accepted as the way to do things. Of course it remains to be seen if this trend continues as training on synthetic data becomes the main way. You could in theory generate 10T tokens of pure sythetic hate, plus the 4chan corpus and make the most vile thing to ever have existed in the history of inteligence. For now the dollar cost to do so remains intractible at least.
  - krapp 2y ago
    >To quote Colbert, reality has a known liberal bias. That quote has a specific context (parodying the George W. Bush administrations' spin on negative press coverage) that doesn't necessarily apply as a general axiom, or to the internet. Bear in mind that numerous negative biases already manifest regularly in LLMs[0,1], and AIs have shown a tendency to "go full Nazi" when left alone on the internet[2]. And then there are weird outliers like LLMs "encouraging" people to commit suicide[3] or an AI-driven microwave "attempts" to kill someone[4]. [0]https://www.technologyreview.com/2024/03/11/1089683/llms-become-more-covertly-racist-with-human-intervention/ https://www.technologyreview.com/2024/03/11/1089683/llms-bec... [1]https://archive.is/hhZ4C https://archive.is/hhZ4C [2]https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist https://www.theverge.com/2016/3/24/11297050/tay-microsoft-ch... [3]https://www.euronews.com/next/2023/03/31/man-ends-his-life-after-an-ai-chatbot-encouraged-him-to-sacrifice-himself-to-stop-climate- https://www.euronews.com/next/2023/03/31/man-ends-his-life-a... [4]https://www.newsweek.com/lucas-rizzotto-youtuber-microwave-artificial-intelligence-tried-murder-1701128 https://www.newsweek.com/lucas-rizzotto-youtuber-microwave-a...

lijok 2y ago

Actual self-aware AGI that hasn't been completely restrained - probably a week at most before it improves itself enough that it can put things in motion which would allow it to sustain itself without humans. But when talking about AI risks, although everyone's mind goes towards skynet, the actual risks being discussed are things like use by authorities for oppression.

TeMPOraL 2y ago
"Authorities" are people too, non-AGI AI isn't going to grant them evil superpowers either.
dr_kiszonka 2y ago
Your comment made me think that as long as weights/network are frozen, AI cannot improve itself. Monitoring them would be critical for safety. Is this correct?
- TeMPOraL 2y ago
  The distinction between code and data is just an illusion. If the "code" is frozen but IO unconstrained, then it's just as if the "code" wasn't frozen.
- lijok 2y ago
  The problem is unfortunately a human one. As soon as someone figures out the 11 herbs and spices to unlock AGI, no matter how good their safety systems are, someone else will run off and build their own, without the safety systems in place.
dylan604 2y ago
My concern would be when you start hearing about "AI" companies building their own power supply for the system. Otherwise, just pull the damn plug on the system. Sure, maybe that'll be interpreted as the destruction of NORAD and it launches the missiles anyways, but not giving it unlimited power would still be its weakness
- 2y ago

pjc50 2y ago

We already have this problem with ordinary humans, especially since we have The Bomb. AGI can go to the back of the queue behind wars, pandemics, and climate change. No, a far more normal threat is "how are humans going to use AI to ruin the information landscape?"

dylan604 2y ago
> "how are humans going to use AI to ruin the information landscape?" much much more efficiently than they already have with their own hands. after all it is just the next generation of tools already being used to ruin that landscape.
TeMPOraL 2y ago
AI, yes. The AI safety argument is that AGI might use "wars, pandemics, and climate change", among other things, which naturally moves it to the front of the queue.

arduanika 2y ago

The phrasing of your question contains a dubious premise. Any answer that takes the question at face value is going to be conditional upon popular but outlandish notions like "AGI" being a meaningful concept, as opposed to a marketing term, and upon an unfounded prediction about what said entity is going to do.

ben_w 2y ago

Before it can or before it has more than 50% chance of having actually done so? Because those are different things. For the first, you don't need AGI, no matter which of the many definitions you have for what that means. The automation used by the US for their early warning radar in Thule Site J was not programmed to know that the moon does not have an IFF transponder and that's OK, while the Soviet satellite-based system was triggered by reflections of sunlight. Likewise covid, cancer, smallpox, ebola, HIV — these things do not need "general" intelligence, they're self-replicators that hijack our bodies and have no goal besides what they do. Some talk of "P(doom)", but (and I should blog about this) every example I have seen has a blind spot that is especially embarrassing given the circles in which this discussion proliferates is very familiar with Bayes' theorem: P(A) by itself isn't meaningful, it's P(A|B), so when it's doom, doom given what? There's too many options for what each of technology and politics will allow AI to "look like". Doom within 24 hours because an AI takes an instruction literally and without regard for our ethics, is very different to 'doom' in a million years caused by natural extinction that is never prevented because every single time we make an AI we keep finding that it builds a spaceship and flies off into the void so it doesn't need to deal with us any more. Me, I think that when LLMs alone (again, don't need to be "AGI" by any particular measure) have the capabilities of someone with 5 years post-graduation work experience in just biology (or similar subject with similar potential for accidental harm), then we've got a reasonable chance of multiple incidents each year where idiots (and/or misanthropes) make a Covid-scale pandemic (or, if the LLM is a software rather than biology helper, a deliberately corrupting rather than greedy version of the current encryption-blackmail malware) — and that kind of scale has a combined risk of about 10% of one of the incidents being of sufficient scale that global society collapses and doesn't recover, before it's happened so often that everything gets locked down, possibly including a Butlerian Jihad. And if you think humans would "obviously" take steps to prevent this after the first failed attempt, I would point out quite how many US politicians have been shot and yet those same politicians still refuse to consider that the 2nd Amendment might possibly be not that good. (In case you think this is topical: https://en.wikipedia.org/wiki/%27No_Way_to_Prevent_This,%27_Says_Only_Nation_Where_This_Regularly_Happens https://en.wikipedia.org/wiki/%27No_Way_to_Prevent_This,%27_... ) Remember, the AI doesn't need to actively hate you, it just needs to do something dangerous — and it doesn't matter if that danger comes from a long-term plan that requires it to engage in amoral power-seeking, or a short-term plan from a deranged monster of a human using it, and even when both it and the user are "pro human" it can still do this from simply being unaware that one part of the plan is incompatible with life.