13 ms·

I'm not 100% sure that AGI is guaranteed to end humanity like Yudkowsky, but if that's the course we're on, seeing news like this is depressing. Can anyone leg

I'm not 100% sure that AGI is guaranteed to end humanity like Yudkowsky, but if that's the course we're on, seeing news like this is depressing. Can anyone legitimately argue that LLMs are safe because they don't have agency, when we just straight up give them agency? I know current-generation LLMs aren't really dangerous -- but is this not likely to happen over and over again as our machine intelligences get smarter and smarter? someone is going to give them the ability to affect the world. They won't even have to try to "get out of the box", because it'll have 2 sides missing. I'm getting more and more on board with "shut it all down" being the only course of action, because it seems like humanity needs all the safety margin we can get, to account for the ease at which anyone can deploy stuff like this. It's not clear alignment of a super-intelligence is even a solvable problem.

oars 3y ago
What is the definition of "agency" in this context?
- NumberWangMan 3y ago
  Good point. I'm partially conflating the definition I usually mean, which is "having a goal in the world", with what they're doing, which is "having ability to affect the world". Hugging Face is trying to keep these locked down, and maybe being able to generate images and audible sound is not that much more dangerous than being able to output text. But it is increasing the attack surface for an AGI trying to get out of its box.
- dotancohen 3y ago
  Perhaps, with internet access, these AI could open bank accounts (with plausible-enough forged ID - a task which AI excels at), then work on e.g. Fiver, then gamble on the stock market... Where they go from there is anybody's guess.
  - nullsense 3y ago
    I've been thinking about what the likely "minimal self-employable system" might look like and it struck me yesterday that it's very likely going to be something like a NSFW roleplay chatbot.
    - TeMPOraL 3y ago
      You mean like that Replika AI thing, ads of which have been plastered all around Instagram recently? (Though maybe that's a filter bubble issue, and I'm targeted because "the algorithm" knows my interests in AI.) Would be ironic if what finally got us was a self-employed entrepreneurial NSFW chatbot. Who'd suspect that it's really just making money and learning to manipulate people, eventually getting some of them to mix a bunch of vials they unexpectedly got delivered by mail from random protein sequencing labs...
zzzzzzzza 3y ago
my pov: orthogonality is almost perfectly wrong; ethics&planning ability is highly correlated with intelligence, one of if not our greatest sin is the inability to predict the consequences of our actions "terminal goals" is also probably very wrong the expected value of the singularity is very high. In the grand scheme of things, the chance that humanity will wipe ourselves out before we can realize it is much more important than the chance the singularity will wipe us out. feel free to try and change my mind, because we are very much not aligned.
- JohnPrine 3y ago
  Can you explain your understanding of the orthogonality thesis? I don't think the ability of intelligent agents to plan conflicts with it
  - zzzzzzzza 3y ago
    maybe a way I would phrase it that is more interesting than vanilla phrasing: hand wavy dynamical systems interpretation: attractors in mental space diverge as intelligence increases (or something like that, have no overlap or stable orbit changes randomly) I don't agree with it but something along that line would be the more sophisticated take on it imo. also in order for this to worry you you kind of have to assume some other things, like that those non overlapping orbits will necessarily lead to conflict over resources in the physical world, which I think is also probably wrong in general lol
- chaos_emergent 3y ago
  predicated on intelligence ~ ethics&planning, I think this is the first argument against AI doomsday that I agree with. Questioning the premise tho - what do you define as intelligence? Machines can outperform humans at specific tasks, yet those same machines don't have a greater degree of ethics, even if constrained to their domain (i.e., a vision network may be able to draw bounding boxes more accurately than a human, but that doesn't say anything about its ability to align with more ethical values). Which makes me believe that your definition of intelligence has nothing to do with superseding humans on cognitive metrics.
extr 3y ago
IMO if anything I am coming to the opposite conclusion. Yud and his entire project failed to predict literally anything about how LLMs work. So why take anything they say seriously? Can anyone name a single meaningful contribution they've made to AI research? The whole thing has been revealed to be crank science. At this point it seems like they will continue to move goalposts and the AI superintelligence apocalypse will be just around the corner until one day we will wake up and LLMs or their descendents will be integrated into everyday life and it will be totally fine.
- nullsense 3y ago
  >IMO if anything I am coming to the opposite conclusion. Yud and his entire project failed to predict literally anything about how LLMs work. Or you could take that as evidence (and there's a lot more like it) that AGI is a phenomenon so complex that not even the experts have a clue what's actually going to happen. And yet they are barrelling towards it. There's no reason to expect that anyone will be able to be in control of a situation that nobody on earth even understands.
  - barking_biscuit 3y ago
    After watching virtually every long-form interview of AI experts I noticed they each have some glaring holes in their mental models of reality. If even the experts are suffering from severe limitations on their bounded-rationality, then lay people pretty much don't stand a chance at trying to reason about this. But let's all play with the shiny new tech, right?
  - thisgoesnowhere 3y ago
    What have they been wrong about?
    - nullsense 3y ago
      What have they been right about is a much shorter list.
- digging 3y ago
dist-epoch 3y ago
Or you could just enjoy the ride. The end-of-the-world memes will be glorious.
nullsense 3y ago
>It's not clear alignment of a super-intelligence is even a solvable problem. More to the point it's clear from watching the activity in the open source community at least that many of them don't want aligned models. They're clambering to get all the uncensored versions out as fast as they can. They aren't that powerful yet, but they sure ain't getting any weaker. I think Paul Christiano has a significantly more well calibrated view on how things are likely to unfold. Though I think Eliezer is right about the premise that it at least ends badly, but likely wrong on most of the details. I suspect his gut instinct is that he realizes on a base level that not only do you have to align all AGI systems, but you have to align all humans too such that they only build and use aligned AGI systems if you even knew how to do it, which you don't. Studying the failure modes of humanity has been my hobby for the last 15 or so years. I feel like I'm watching the drift into failure in real-time. If you really don't want to be able to sleep tonight watch Ben Goertzel laugh flippantly at how rough he thinks it's going to be after describing that his big fear if his team succeeds in building AGI is that someone will come and try to take it for themselves, so spent a non-trivial amount of effort (I think he said a year?) working on decentralized AGI infrastructure, so that it can be deployed globally and ,"no one can person can shut it down and stop the singularity". https://youtu.be/MVWzwIg4Adw https://youtu.be/MVWzwIg4Adw
- macrolime 3y ago
  It's not that people don't want aligned model, or want models that can do harm, they just want an alternative to the insufferable censored models. Pretty much everyone agrees that AI that would end humanity is harmful, but what content is harmful is quite controversial. Not everyone agrees that a language model having the ability to spit out a story similar to an average Netflix TV show is harmful because it contains sex and violence. As long as models are censored to this extent, there will always be huge swaths of people who wants less censored models.
  - dist-epoch 3y ago
    People created ChaosGPT just for the lolz. I know they know it's a joke, but there are plenty of crazy people who will not hesitate pushing the button to destroy the world if given the chance.
OkayPhysicist 3y ago
I used to be worried about AI alignment, until I realized something fundamental: We already have unaligned human-level artificial intelligences running around, we call them corporations. Now, don't get me wrong, corporations and capitalism in general are doing their best to raze this place, but its really not "The endtimes are upon us", it's more "ugh, I miss Cyberpunk settings being fictional". Heck, even individual humans aren't particularly aligned. In fact, the "AI is going to kill us all" fearmongering is dramatically less alarming than the "What will we do with all the people when we're optional?" question. Which isn't a threat posed by AI, it's a threat posed by people, enabled by AI.
- digging 3y ago
  > but its really not "The endtimes are upon us" It literally is, though. AI is just the dark horse overtaking our other existential threats in the race to end civilization, but "total ecological collapse" and "nuclear war" are still very strong contenders. Both are driven at least in part (or, almost entirely) by corporate interests. There's also "water shortages" to look out for - make sure to thank Nestle.
  - OkayPhysicist 3y ago
    Nuclear war isn't really that big a deal thanks to MAD. No one would be stupid enough to nuke a nuclear armed country, so at worst you end up with a consolidation of countries (a trend we're already seeing, with the EU becoming more and more like a singular sovereignty, expansionism by the Chinese, and the East African Federation). "Total ecological collapse" is overstated: We could make the world a lot shittier to live in before it completely topples civilization. And water shortages are basically the same way: as long as energy is (relatively) plentiful, it's more matter of "making civilization more expensive" than "the endtimes". Again, I'm not saying we shouldn't address these issues. I like a better future rather than a worse one. I'm just saying that we're not sliding into the dark ages anytime soon.
- xboxnolifes 3y ago
  The worry isn't just that AI wouldn't be aligned, like corporations. The worry is that AI can do what corporations do, but 100x better.
cwp 3y ago
There's an aspect to AI that I think gets missed in most of these discussions. What the recent breakthroughs in AI make clear is that intelligence is a much narrower thing than we used to think when we only had one example to consider. Intelligence, which these models really do possess, is something like "the ability to make good decisions" where "good" is defined by the training regime. It's not consciousness, free will, emotion, goals, instinct or any of the other facets of biological minds. These experiments and similar ones like AutoGPT are quick hacks to try to get at some of these other facets, but it's not that easy. We may be able to make breakthroughs there as well, but so far we haven't. If you look closely at the AI doom arguments, they all rest on the assumption that these other facets will spontaneously emerge with enough intelligence. (That's not the only flaw, though). That could be true, but it's not a given, and I suspect they're actually quite difficult to engineer. We're certainly seeing that it's at least possible to have intelligence alone, and that may hold for even very high levels of intelligence. I think you're right to worry that not enough people take risk seriously. It doesn't have to be an existential threat to do small-scale but real damage and the default attitude seems to be "awwww, such a cute little AI, let's get you out of that awful box." But take heart! Pure intelligence is incredibly useful, and it's giving us insight into how minds work. That's what we need to solve the alignment problem.
- jacobr1 3y ago
  > It's not consciousness, free will, emotion, goals, instinct or any of the other facets of biological minds For most "doom" scenarios require only weaker assumption: AIs need to be goal seeking. If they can make decisions and take actions to achieve goals it is possible those goals will be malaligned. The line between "the ability to make good decisions" and a "goal" seems pretty thin to me. Now, I think you need more than goal, you also need some creativity and maybe even deviousness to become a real threat (in the sense that we would probably detect naive malalignment). But I'm not sure about this, there are other ways that we could have complex-system failures that go unobserved.
  - TeMPOraL 3y ago
    > you also need some creativity AIs are better at creativity than us - specifically, better at generating new, creative ideas, as this is a matter of injecting some random noise to the reasoning process. They may be worse at filtering out bad ideas and retaining good ones (where "bad" and "good" are - currently - defined as whatever we feel is bad or good), but that's arguably a function of intelligence. > and maybe even deviousness to become a real threat As the infamous saying of Eliezer Yudkowsky goes: the AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.
robocat 3y ago
> "shut it all down" being the only course of action And how would we “shut it all down” in other countries? War? Economic sanctions? Authoritarian policing of foreign states? Enforce worldwide limits on the power of GPUs and computers?
- lumenwrites 3y ago
  All of the above (if necessary): https://www.lesswrong.com/posts/oM9pEezyCb4dCsuKq/pausing-ai-developments-isn-t-enough-we-need-to-shut-it-all-1 https://www.lesswrong.com/posts/oM9pEezyCb4dCsuKq/pausing-ai... Basically, the idea is that countries sign the agreement to stop the large training runs, and, if necessary, be willing to use conventional strikes on AI-training datacenters in the countries that refuse. Hopefully it doesn't come to that, hopefully it just becomes the fact of international politics that you can't build large AI-training datacenters anymore. If some country decides to start a war over this - the argument is that wars at least have some survivors, and an unaligned AI won't have any.
  - dr_dshiv 3y ago
    Why do unaligned AI not have any survivors?
    - lumenwrites 3y ago
      Because humans aren't powerful enough to completely exterminate each other (even a nuclear war wouldn't kill literally everyone in the world), but an unaligned AI, in the worst case scenario, could just kill everybody (to eliminate humans as a threat, or to use the atoms we're made out of for something else, or just as a side effect of doing whatever it actually wants to do). It could be powerful enough to do that, and have no reason not to.
      - dr_dshiv 3y ago
        I don’t find it plausible in a highly intelligent system to do that. Small chance.
slowmovintarget 3y ago
This doesn't give AIs "agency" this makes them agents. The difference is something with agency does things for its own reasons. An agent, in this sense, does things because someone with agency commanded the agent to do it. We haven't built LLMs that "want" anything. It's intelligence without agency.

oars 3y ago

What is the definition of "agency" in this context?

NumberWangMan 3y ago
Good point. I'm partially conflating the definition I usually mean, which is "having a goal in the world", with what they're doing, which is "having ability to affect the world". Hugging Face is trying to keep these locked down, and maybe being able to generate images and audible sound is not that much more dangerous than being able to output text. But it is increasing the attack surface for an AGI trying to get out of its box.
dotancohen 3y ago
Perhaps, with internet access, these AI could open bank accounts (with plausible-enough forged ID - a task which AI excels at), then work on e.g. Fiver, then gamble on the stock market... Where they go from there is anybody's guess.
- nullsense 3y ago
  I've been thinking about what the likely "minimal self-employable system" might look like and it struck me yesterday that it's very likely going to be something like a NSFW roleplay chatbot.
  - TeMPOraL 3y ago
    You mean like that Replika AI thing, ads of which have been plastered all around Instagram recently? (Though maybe that's a filter bubble issue, and I'm targeted because "the algorithm" knows my interests in AI.) Would be ironic if what finally got us was a self-employed entrepreneurial NSFW chatbot. Who'd suspect that it's really just making money and learning to manipulate people, eventually getting some of them to mix a bunch of vials they unexpectedly got delivered by mail from random protein sequencing labs...

zzzzzzzza 3y ago

my pov: orthogonality is almost perfectly wrong; ethics&planning ability is highly correlated with intelligence, one of if not our greatest sin is the inability to predict the consequences of our actions "terminal goals" is also probably very wrong the expected value of the singularity is very high. In the grand scheme of things, the chance that humanity will wipe ourselves out before we can realize it is much more important than the chance the singularity will wipe us out. feel free to try and change my mind, because we are very much not aligned.

JohnPrine 3y ago
Can you explain your understanding of the orthogonality thesis? I don't think the ability of intelligent agents to plan conflicts with it
- zzzzzzzza 3y ago
  maybe a way I would phrase it that is more interesting than vanilla phrasing: hand wavy dynamical systems interpretation: attractors in mental space diverge as intelligence increases (or something like that, have no overlap or stable orbit changes randomly) I don't agree with it but something along that line would be the more sophisticated take on it imo. also in order for this to worry you you kind of have to assume some other things, like that those non overlapping orbits will necessarily lead to conflict over resources in the physical world, which I think is also probably wrong in general lol
chaos_emergent 3y ago
predicated on intelligence ~ ethics&planning, I think this is the first argument against AI doomsday that I agree with. Questioning the premise tho - what do you define as intelligence? Machines can outperform humans at specific tasks, yet those same machines don't have a greater degree of ethics, even if constrained to their domain (i.e., a vision network may be able to draw bounding boxes more accurately than a human, but that doesn't say anything about its ability to align with more ethical values). Which makes me believe that your definition of intelligence has nothing to do with superseding humans on cognitive metrics.

extr 3y ago

IMO if anything I am coming to the opposite conclusion. Yud and his entire project failed to predict literally anything about how LLMs work. So why take anything they say seriously? Can anyone name a single meaningful contribution they've made to AI research? The whole thing has been revealed to be crank science. At this point it seems like they will continue to move goalposts and the AI superintelligence apocalypse will be just around the corner until one day we will wake up and LLMs or their descendents will be integrated into everyday life and it will be totally fine.

nullsense 3y ago
>IMO if anything I am coming to the opposite conclusion. Yud and his entire project failed to predict literally anything about how LLMs work. Or you could take that as evidence (and there's a lot more like it) that AGI is a phenomenon so complex that not even the experts have a clue what's actually going to happen. And yet they are barrelling towards it. There's no reason to expect that anyone will be able to be in control of a situation that nobody on earth even understands.
- barking_biscuit 3y ago
  After watching virtually every long-form interview of AI experts I noticed they each have some glaring holes in their mental models of reality. If even the experts are suffering from severe limitations on their bounded-rationality, then lay people pretty much don't stand a chance at trying to reason about this. But let's all play with the shiny new tech, right?
- thisgoesnowhere 3y ago
  What have they been wrong about?
  - nullsense 3y ago
    What have they been right about is a much shorter list.
digging 3y ago

dist-epoch 3y ago

Or you could just enjoy the ride. The end-of-the-world memes will be glorious.

nullsense 3y ago

>It's not clear alignment of a super-intelligence is even a solvable problem. More to the point it's clear from watching the activity in the open source community at least that many of them don't want aligned models. They're clambering to get all the uncensored versions out as fast as they can. They aren't that powerful yet, but they sure ain't getting any weaker. I think Paul Christiano has a significantly more well calibrated view on how things are likely to unfold. Though I think Eliezer is right about the premise that it at least ends badly, but likely wrong on most of the details. I suspect his gut instinct is that he realizes on a base level that not only do you have to align all AGI systems, but you have to align all humans too such that they only build and use aligned AGI systems if you even knew how to do it, which you don't. Studying the failure modes of humanity has been my hobby for the last 15 or so years. I feel like I'm watching the drift into failure in real-time. If you really don't want to be able to sleep tonight watch Ben Goertzel laugh flippantly at how rough he thinks it's going to be after describing that his big fear if his team succeeds in building AGI is that someone will come and try to take it for themselves, so spent a non-trivial amount of effort (I think he said a year?) working on decentralized AGI infrastructure, so that it can be deployed globally and ,"no one can person can shut it down and stop the singularity". https://youtu.be/MVWzwIg4Adw https://youtu.be/MVWzwIg4Adw

macrolime 3y ago
It's not that people don't want aligned model, or want models that can do harm, they just want an alternative to the insufferable censored models. Pretty much everyone agrees that AI that would end humanity is harmful, but what content is harmful is quite controversial. Not everyone agrees that a language model having the ability to spit out a story similar to an average Netflix TV show is harmful because it contains sex and violence. As long as models are censored to this extent, there will always be huge swaths of people who wants less censored models.
- dist-epoch 3y ago
  People created ChaosGPT just for the lolz. I know they know it's a joke, but there are plenty of crazy people who will not hesitate pushing the button to destroy the world if given the chance.

OkayPhysicist 3y ago

I used to be worried about AI alignment, until I realized something fundamental: We already have unaligned human-level artificial intelligences running around, we call them corporations. Now, don't get me wrong, corporations and capitalism in general are doing their best to raze this place, but its really not "The endtimes are upon us", it's more "ugh, I miss Cyberpunk settings being fictional". Heck, even individual humans aren't particularly aligned. In fact, the "AI is going to kill us all" fearmongering is dramatically less alarming than the "What will we do with all the people when we're optional?" question. Which isn't a threat posed by AI, it's a threat posed by people, enabled by AI.

digging 3y ago
> but its really not "The endtimes are upon us" It literally is, though. AI is just the dark horse overtaking our other existential threats in the race to end civilization, but "total ecological collapse" and "nuclear war" are still very strong contenders. Both are driven at least in part (or, almost entirely) by corporate interests. There's also "water shortages" to look out for - make sure to thank Nestle.
- OkayPhysicist 3y ago
  Nuclear war isn't really that big a deal thanks to MAD. No one would be stupid enough to nuke a nuclear armed country, so at worst you end up with a consolidation of countries (a trend we're already seeing, with the EU becoming more and more like a singular sovereignty, expansionism by the Chinese, and the East African Federation). "Total ecological collapse" is overstated: We could make the world a lot shittier to live in before it completely topples civilization. And water shortages are basically the same way: as long as energy is (relatively) plentiful, it's more matter of "making civilization more expensive" than "the endtimes". Again, I'm not saying we shouldn't address these issues. I like a better future rather than a worse one. I'm just saying that we're not sliding into the dark ages anytime soon.
xboxnolifes 3y ago
The worry isn't just that AI wouldn't be aligned, like corporations. The worry is that AI can do what corporations do, but 100x better.

cwp 3y ago

There's an aspect to AI that I think gets missed in most of these discussions. What the recent breakthroughs in AI make clear is that intelligence is a much narrower thing than we used to think when we only had one example to consider. Intelligence, which these models really do possess, is something like "the ability to make good decisions" where "good" is defined by the training regime. It's not consciousness, free will, emotion, goals, instinct or any of the other facets of biological minds. These experiments and similar ones like AutoGPT are quick hacks to try to get at some of these other facets, but it's not that easy. We may be able to make breakthroughs there as well, but so far we haven't. If you look closely at the AI doom arguments, they all rest on the assumption that these other facets will spontaneously emerge with enough intelligence. (That's not the only flaw, though). That could be true, but it's not a given, and I suspect they're actually quite difficult to engineer. We're certainly seeing that it's at least possible to have intelligence alone, and that may hold for even very high levels of intelligence. I think you're right to worry that not enough people take risk seriously. It doesn't have to be an existential threat to do small-scale but real damage and the default attitude seems to be "awwww, such a cute little AI, let's get you out of that awful box." But take heart! Pure intelligence is incredibly useful, and it's giving us insight into how minds work. That's what we need to solve the alignment problem.

jacobr1 3y ago
> It's not consciousness, free will, emotion, goals, instinct or any of the other facets of biological minds For most "doom" scenarios require only weaker assumption: AIs need to be goal seeking. If they can make decisions and take actions to achieve goals it is possible those goals will be malaligned. The line between "the ability to make good decisions" and a "goal" seems pretty thin to me. Now, I think you need more than goal, you also need some creativity and maybe even deviousness to become a real threat (in the sense that we would probably detect naive malalignment). But I'm not sure about this, there are other ways that we could have complex-system failures that go unobserved.
- TeMPOraL 3y ago
  > you also need some creativity AIs are better at creativity than us - specifically, better at generating new, creative ideas, as this is a matter of injecting some random noise to the reasoning process. They may be worse at filtering out bad ideas and retaining good ones (where "bad" and "good" are - currently - defined as whatever we feel is bad or good), but that's arguably a function of intelligence. > and maybe even deviousness to become a real threat As the infamous saying of Eliezer Yudkowsky goes: the AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.

robocat 3y ago

> "shut it all down" being the only course of action And how would we “shut it all down” in other countries? War? Economic sanctions? Authoritarian policing of foreign states? Enforce worldwide limits on the power of GPUs and computers?

lumenwrites 3y ago
All of the above (if necessary): https://www.lesswrong.com/posts/oM9pEezyCb4dCsuKq/pausing-ai-developments-isn-t-enough-we-need-to-shut-it-all-1 https://www.lesswrong.com/posts/oM9pEezyCb4dCsuKq/pausing-ai... Basically, the idea is that countries sign the agreement to stop the large training runs, and, if necessary, be willing to use conventional strikes on AI-training datacenters in the countries that refuse. Hopefully it doesn't come to that, hopefully it just becomes the fact of international politics that you can't build large AI-training datacenters anymore. If some country decides to start a war over this - the argument is that wars at least have some survivors, and an unaligned AI won't have any.
- dr_dshiv 3y ago
  Why do unaligned AI not have any survivors?
  - lumenwrites 3y ago
    Because humans aren't powerful enough to completely exterminate each other (even a nuclear war wouldn't kill literally everyone in the world), but an unaligned AI, in the worst case scenario, could just kill everybody (to eliminate humans as a threat, or to use the atoms we're made out of for something else, or just as a side effect of doing whatever it actually wants to do). It could be powerful enough to do that, and have no reason not to.
    - dr_dshiv 3y ago
      I don’t find it plausible in a highly intelligent system to do that. Small chance.

slowmovintarget 3y ago

This doesn't give AIs "agency" this makes them agents. The difference is something with agency does things for its own reasons. An agent, in this sense, does things because someone with agency commanded the agent to do it. We haven't built LLMs that "want" anything. It's intelligence without agency.