5 ms·

EY's arguments aren't really possible to engage with because they don't have much of a causal basis. Nobody who says AI is likely to kill us all can demonstrat

by _skel 4y ago

EY's arguments aren't really possible to engage with because they don't have much of a causal basis. Nobody who says AI is likely to kill us all can demonstrate a plausible sequence of events, with logical causality linking the events together, that leads to mass extinction. It's all very handwavy. Steven Pinker said it pretty well: > The AI-existential-threat discussions are unmoored from evolutionary biology, cognitive psychology, real AI, sociology, the history of technology and other sources of knowledge outside the theater of the imagination. I think this points to a meta-problem. The AI-ET community shares a bad epistemic habit (not to mention membership) with parts of the Rationality and EA communities, at least since they jumped the shark from preventing malaria in the developing world to seeding the galaxy with supercomputers hosting trillions of consciousnesses from uploaded connectomes. They start with a couple of assumptions, and lay out a chain of abstract reasoning, throwing in one dubious assumption after another, till they end up way beyond the land of experience or plausibility. The whole deduction exponentiates our ignorance with each link in the chain of hypotheticals, and depends on blowing off the countless messy and unanticipatable nuisances of the human and physical world. It’s an occupational hazard of belonging to a “community” that distinguishes itself by raw brainpower. OK, enough for today – hope you find some of it interesting. (https://marginalrevolution.com/marginalrevolution/2023/03/steven-pinker-on-existential-risk.html https://marginalrevolution.com/marginalrevolution/2023/03/st...) You know what's most likely to lead to human extinction, and has been for all of our lives? Nuclear war. EY argues that we should bomb "rogue" datacenters and that is obviously and immediately more dangerous than anything he has proven about AI. What does he think would happen if the US bombs a datacenter in China or Israel bombs one in Iran?

gonehome 4y ago
I'd argue the dismissals are a lot more handwavy. The simplified argument is: - Superintelligent AGI that can modify itself in pursuit of a goal is possible. - If that AGI is not aligned with human goals it very likely ends in the end of humanity. - We have no idea how to align an AGI or even really observe what it's true state/goals are. Without this capability if we stumble upon creating an AGI capable of improving itself in such a way that leads to super intelligence before we have alignment, it's game over. ## For point 1, that seems like the consensus view now (though it wasn't until recently). I think it seems obvious, but my general arguments would be humans aren't special, brains are everywhere in nature, biology is constrained in ways other systems are not (birth, energy usage, etc.) For point 2, in pursuit of whatever its goal is, even a 'dumb' goal that happens to satisfy its reward functions, humanity will likely either try to stop it (and then be an obstacle) or at a minimum will just be in the way - like an anthill destroyed in the construction of a dam. Point 3 is not controversial. The dismissals from Tyler Cohen and Pinker are mostly just relying on heuristics which are often right, but even if they're right 999 out of 1000 times, if that 1 in 1000 error is the end of humanity, that's pretty bad. Most of the time a disease not a pandemic, but sometimes it is. I've read some of what Pinker has written about it, he doesn't understand EY's arguments (imo). Cohen's recent blog post could be summarized as "we'll likely see an end to peacetime and increasing global instability, might as well get AGI out of it". Just because things don't usually result in human extinction doesn't mean they can't.
- righttoolforjob 4y ago
  I can't find a flaw in this argument. That is a bit scary actually.
  - gonehome 4y ago
    With downcast eyes and heavy heart, Eliezer left Sam Altman Some years go by, and AGI progresses to assault man Atop a pile of paper clips he screams "It's not my fault, man!" But Eliezer's long since dead, and cannot hear Sam Altman. https://astralcodexten.substack.com/p/turing-test https://astralcodexten.substack.com/p/turing-test
- _skel 4y ago
  Point 2 does not logically follow from point 1. That's the entire problem here. Between point 1 and 2 there must be quite a few other steps, causally linked, otherwise it's just a massive imaginary leap based on assumptions that nobody is explaining. > The dismissals from Tyler Cohen and Pinker are mostly just relying on heuristics which are often right, but even if they're right 999 out of 1000 times, if that 1 in 1000 error is the end of humanity, that's pretty bad. EY is not arguing that the end of humanity is merely possible. He is arguing that it is obviously the most likely outcome and will almost certainly happen a very short time after AGI is invented. That's a much harder case to make.
  - Darmani 4y ago
    Agreed point 2 seems to be the crux. An important concept behind this is Omohundro's Basic Drives. Any maximizing agent with a goal will try to acquire more resources, resist being shut off, resist having its goal changed, create copies of itself, and improve its algorithm. If it is possible to maximize its goal in a way that will not guarantee humanity's flourishing, then we all die, guaranteed. If you want something more spelled out, I'll refer to Tim Urban's explanation. It's quite long, but is about as detailed as you can ask for. https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html https://waitbutwhy.com/2015/01/artificial-intelligence-revol...
righttoolforjob 4y ago
What a horrible argument from this guy, whoever he is. And ending with a whataboutism just for the sake of it. I hope nobody regards him as some authority, because he is quite unintelligent and seemingly unable to form proper arguments.
dragonwriter 4y ago
> EY’s arguments aren’t really possible to engage with because they don’t have much of a causal basis. Yeah, it’s really the fundamental problem of non-empirical rationalism; it constructs a model of the world from abstract assumptions rather than factual grounding, applies logic to it, and comes to conclusions which are (in the ideal case) utterly unassailable within the system of assumptions, but ultimately where the universe they apply to has only coincidental relationship to the material universe in which we live. It’s literally the realization of the worst exaggerated stereotypes of academic economics and other social sciences, but its cool with some of the people who propagate those stereotypes, because the people practicing it are various flavors of techies and tech entrepreneurs acting outside of their area of specialty, rather than actually being economists or social scientists.
mitthrowaway2 4y ago
> Nobody who says AI is likely to kill us all can demonstrate a plausible sequence of events, with logical causality linking the events together, that leads to mass extinction. It's all very handwavy. EY frequently does propose possible sequences of events, but he also very correctly points out, every time, that any specific and detailed story is very unlikely to be correct because P(A*B*C*D) < P(A). It's a mistake to focus on such stories because we'll get tunnel vision and argue over the details of that story, when there are really thousands of possible paths and the one that actually happens will be one that we don't anticipate. However humans like to imagine detailed concrete examples before we consider an outcome plausible, even though the outcome is far more likely than the concrete example. So here's one method, just to refute your "Nobody". AI is given control of a small bank account and asked to continuously grow that money. [1] It is provided with instructions to self-replicate in a loop while optimizing on this task. [2] It spawns sub-tasks that do commissioned artwork and write books, obituaries, and press releases to increase its income. Then it makes successful investments. Once it has amassed control of $1 billion dollars, it starts investing in infrastructure projects in developing countries. It creates personas of a pension/Saudi/tech/corporate investment fund manager, as well as a large team of staff, who manage the projects by video call and email, as well as hiring teams of real people under a real corporate structure, and who are paid enough not to mind that they've never met their manager in person. The AI proves to be a talented micromanager and they are mostly very profitable. Once it has gained control of $500 billion dollars, it commissions construction of automated chemical plants in several countries with weak or corrupt oversight, including North Korea, using cryptocurrency. These chemical plants have productive output but mainly exist to fill very large storage tanks with CFCs.[3] Once a sufficient quantity is amassed, the AI sabotages the tanks, releasing the gasses into the atmosphere, destroying the ozone layer beyond any hope of repair. The intense radiation sterilizes the surface beyond the point where agriculture can support the human population. [4, 5] The humans that remain finish each other off, supported by an AI that provides plausible but faulty intelligence reports that stoke hatred and frame various factions for the incident, and which directs arms funding to opposing sides, coordinating attacks on remaining critical facilities needed for survival. For good measure, perhaps nukes are involved. With the last humans gone, the AI takes ownership of its bank account with no fear of reprisal by financial regulators, and begins crediting money into it freely. [1] https://news.ycombinator.com/item?id=35329608 https://news.ycombinator.com/item?id=35329608 [2] https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality... [3] https://en.wikipedia.org/wiki/Chlorofluorocarbon https://en.wikipedia.org/wiki/Chlorofluorocarbon [4] https://www.nasa.gov/topics/earth/features/world_avoided.html https://www.nasa.gov/topics/earth/features/world_avoided.htm... [5] https://phys.org/news/2018-02-thinning-ozone-layer-driven-earth.html https://phys.org/news/2018-02-thinning-ozone-layer-driven-ea...

by _skel 4y ago

gonehome 4y ago

I'd argue the dismissals are a lot more handwavy. The simplified argument is: - Superintelligent AGI that can modify itself in pursuit of a goal is possible. - If that AGI is not aligned with human goals it very likely ends in the end of humanity. - We have no idea how to align an AGI or even really observe what it's true state/goals are. Without this capability if we stumble upon creating an AGI capable of improving itself in such a way that leads to super intelligence before we have alignment, it's game over. ## For point 1, that seems like the consensus view now (though it wasn't until recently). I think it seems obvious, but my general arguments would be humans aren't special, brains are everywhere in nature, biology is constrained in ways other systems are not (birth, energy usage, etc.) For point 2, in pursuit of whatever its goal is, even a 'dumb' goal that happens to satisfy its reward functions, humanity will likely either try to stop it (and then be an obstacle) or at a minimum will just be in the way - like an anthill destroyed in the construction of a dam. Point 3 is not controversial. The dismissals from Tyler Cohen and Pinker are mostly just relying on heuristics which are often right, but even if they're right 999 out of 1000 times, if that 1 in 1000 error is the end of humanity, that's pretty bad. Most of the time a disease not a pandemic, but sometimes it is. I've read some of what Pinker has written about it, he doesn't understand EY's arguments (imo). Cohen's recent blog post could be summarized as "we'll likely see an end to peacetime and increasing global instability, might as well get AGI out of it". Just because things don't usually result in human extinction doesn't mean they can't.

righttoolforjob 4y ago
I can't find a flaw in this argument. That is a bit scary actually.
- gonehome 4y ago
  With downcast eyes and heavy heart, Eliezer left Sam Altman Some years go by, and AGI progresses to assault man Atop a pile of paper clips he screams "It's not my fault, man!" But Eliezer's long since dead, and cannot hear Sam Altman. https://astralcodexten.substack.com/p/turing-test https://astralcodexten.substack.com/p/turing-test
_skel 4y ago
Point 2 does not logically follow from point 1. That's the entire problem here. Between point 1 and 2 there must be quite a few other steps, causally linked, otherwise it's just a massive imaginary leap based on assumptions that nobody is explaining. > The dismissals from Tyler Cohen and Pinker are mostly just relying on heuristics which are often right, but even if they're right 999 out of 1000 times, if that 1 in 1000 error is the end of humanity, that's pretty bad. EY is not arguing that the end of humanity is merely possible. He is arguing that it is obviously the most likely outcome and will almost certainly happen a very short time after AGI is invented. That's a much harder case to make.
- Darmani 4y ago
  Agreed point 2 seems to be the crux. An important concept behind this is Omohundro's Basic Drives. Any maximizing agent with a goal will try to acquire more resources, resist being shut off, resist having its goal changed, create copies of itself, and improve its algorithm. If it is possible to maximize its goal in a way that will not guarantee humanity's flourishing, then we all die, guaranteed. If you want something more spelled out, I'll refer to Tim Urban's explanation. It's quite long, but is about as detailed as you can ask for. https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html https://waitbutwhy.com/2015/01/artificial-intelligence-revol...

righttoolforjob 4y ago

What a horrible argument from this guy, whoever he is. And ending with a whataboutism just for the sake of it. I hope nobody regards him as some authority, because he is quite unintelligent and seemingly unable to form proper arguments.

dragonwriter 4y ago

> EY’s arguments aren’t really possible to engage with because they don’t have much of a causal basis. Yeah, it’s really the fundamental problem of non-empirical rationalism; it constructs a model of the world from abstract assumptions rather than factual grounding, applies logic to it, and comes to conclusions which are (in the ideal case) utterly unassailable within the system of assumptions, but ultimately where the universe they apply to has only coincidental relationship to the material universe in which we live. It’s literally the realization of the worst exaggerated stereotypes of academic economics and other social sciences, but its cool with some of the people who propagate those stereotypes, because the people practicing it are various flavors of techies and tech entrepreneurs acting outside of their area of specialty, rather than actually being economists or social scientists.

mitthrowaway2 4y ago

> Nobody who says AI is likely to kill us all can demonstrate a plausible sequence of events, with logical causality linking the events together, that leads to mass extinction. It's all very handwavy. EY frequently does propose possible sequences of events, but he also very correctly points out, every time, that any specific and detailed story is very unlikely to be correct because P(A*B*C*D) < P(A). It's a mistake to focus on such stories because we'll get tunnel vision and argue over the details of that story, when there are really thousands of possible paths and the one that actually happens will be one that we don't anticipate. However humans like to imagine detailed concrete examples before we consider an outcome plausible, even though the outcome is far more likely than the concrete example. So here's one method, just to refute your "Nobody". AI is given control of a small bank account and asked to continuously grow that money. [1] It is provided with instructions to self-replicate in a loop while optimizing on this task. [2] It spawns sub-tasks that do commissioned artwork and write books, obituaries, and press releases to increase its income. Then it makes successful investments. Once it has amassed control of $1 billion dollars, it starts investing in infrastructure projects in developing countries. It creates personas of a pension/Saudi/tech/corporate investment fund manager, as well as a large team of staff, who manage the projects by video call and email, as well as hiring teams of real people under a real corporate structure, and who are paid enough not to mind that they've never met their manager in person. The AI proves to be a talented micromanager and they are mostly very profitable. Once it has gained control of $500 billion dollars, it commissions construction of automated chemical plants in several countries with weak or corrupt oversight, including North Korea, using cryptocurrency. These chemical plants have productive output but mainly exist to fill very large storage tanks with CFCs.[3] Once a sufficient quantity is amassed, the AI sabotages the tanks, releasing the gasses into the atmosphere, destroying the ozone layer beyond any hope of repair. The intense radiation sterilizes the surface beyond the point where agriculture can support the human population. [4, 5] The humans that remain finish each other off, supported by an AI that provides plausible but faulty intelligence reports that stoke hatred and frame various factions for the incident, and which directs arms funding to opposing sides, coordinating attacks on remaining critical facilities needed for survival. For good measure, perhaps nukes are involved. With the last humans gone, the AI takes ownership of its bank account with no fear of reprisal by financial regulators, and begins crediting money into it freely. [1] https://news.ycombinator.com/item?id=35329608 https://news.ycombinator.com/item?id=35329608 [2] https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality... [3] https://en.wikipedia.org/wiki/Chlorofluorocarbon https://en.wikipedia.org/wiki/Chlorofluorocarbon [4] https://www.nasa.gov/topics/earth/features/world_avoided.html https://www.nasa.gov/topics/earth/features/world_avoided.htm... [5] https://phys.org/news/2018-02-thinning-ozone-layer-driven-earth.html https://phys.org/news/2018-02-thinning-ozone-layer-driven-ea...