6 ms·

In a world where humans develop superintelligence, what would you predict to see beforehand? Does what we’re seeing look like failure? I’ve always found the di

In a world where humans develop superintelligence, what would you predict to see beforehand? Does what we’re seeing look like failure? I’ve always found the dismissals of EY’s arguments to be pretty weak and people rarely engage on the actual arguments, even as his concerns have become more and more relevant with capability (in 2007 people thought AGI was at best 50yrs out and commonly people thought it was impossible, this was before any real deep learning success and GOFAI was a useless dead-end). Most of the comments here are similarly dumb dismissals that don’t engage, and some even come across as highschool level mocking. It’s worth reading his A-Z book to at least understand why he holds his position. Over time, the dumb responses from others make me think he’s probably more likely right than not. Given the extreme downside risks, I can understand why he argues this.

_skel 4y ago
EY's arguments aren't really possible to engage with because they don't have much of a causal basis. Nobody who says AI is likely to kill us all can demonstrate a plausible sequence of events, with logical causality linking the events together, that leads to mass extinction. It's all very handwavy. Steven Pinker said it pretty well: > The AI-existential-threat discussions are unmoored from evolutionary biology, cognitive psychology, real AI, sociology, the history of technology and other sources of knowledge outside the theater of the imagination. I think this points to a meta-problem. The AI-ET community shares a bad epistemic habit (not to mention membership) with parts of the Rationality and EA communities, at least since they jumped the shark from preventing malaria in the developing world to seeding the galaxy with supercomputers hosting trillions of consciousnesses from uploaded connectomes. They start with a couple of assumptions, and lay out a chain of abstract reasoning, throwing in one dubious assumption after another, till they end up way beyond the land of experience or plausibility. The whole deduction exponentiates our ignorance with each link in the chain of hypotheticals, and depends on blowing off the countless messy and unanticipatable nuisances of the human and physical world. It’s an occupational hazard of belonging to a “community” that distinguishes itself by raw brainpower. OK, enough for today – hope you find some of it interesting. (https://marginalrevolution.com/marginalrevolution/2023/03/steven-pinker-on-existential-risk.html https://marginalrevolution.com/marginalrevolution/2023/03/st...) You know what's most likely to lead to human extinction, and has been for all of our lives? Nuclear war. EY argues that we should bomb "rogue" datacenters and that is obviously and immediately more dangerous than anything he has proven about AI. What does he think would happen if the US bombs a datacenter in China or Israel bombs one in Iran?
- gonehome 4y ago
  I'd argue the dismissals are a lot more handwavy. The simplified argument is: - Superintelligent AGI that can modify itself in pursuit of a goal is possible. - If that AGI is not aligned with human goals it very likely ends in the end of humanity. - We have no idea how to align an AGI or even really observe what it's true state/goals are. Without this capability if we stumble upon creating an AGI capable of improving itself in such a way that leads to super intelligence before we have alignment, it's game over. ## For point 1, that seems like the consensus view now (though it wasn't until recently). I think it seems obvious, but my general arguments would be humans aren't special, brains are everywhere in nature, biology is constrained in ways other systems are not (birth, energy usage, etc.) For point 2, in pursuit of whatever its goal is, even a 'dumb' goal that happens to satisfy its reward functions, humanity will likely either try to stop it (and then be an obstacle) or at a minimum will just be in the way - like an anthill destroyed in the construction of a dam. Point 3 is not controversial. The dismissals from Tyler Cohen and Pinker are mostly just relying on heuristics which are often right, but even if they're right 999 out of 1000 times, if that 1 in 1000 error is the end of humanity, that's pretty bad. Most of the time a disease not a pandemic, but sometimes it is. I've read some of what Pinker has written about it, he doesn't understand EY's arguments (imo). Cohen's recent blog post could be summarized as "we'll likely see an end to peacetime and increasing global instability, might as well get AGI out of it". Just because things don't usually result in human extinction doesn't mean they can't.
  - righttoolforjob 4y ago
    I can't find a flaw in this argument. That is a bit scary actually.
    - gonehome 4y ago
      With downcast eyes and heavy heart, Eliezer left Sam Altman Some years go by, and AGI progresses to assault man Atop a pile of paper clips he screams "It's not my fault, man!" But Eliezer's long since dead, and cannot hear Sam Altman. https://astralcodexten.substack.com/p/turing-test https://astralcodexten.substack.com/p/turing-test
sterlind 4y ago
I think it's likely we'll develop superintelligence very soon, but it won't be any more knowledgeable than us, just smarter. And knowledge has been the bottleneck for most of human history - our ancestors weren't dumber. Also, it will be bottlenecked by needing to solve NP-complete problems - unless P=NP, it's not going to be all that much faster at solving hard math problems. Basically, it still needs to do science, and science takes time.
- gonehome 4y ago
  Take a smart human and speed up their thinking a billion times, you get the time for free. What takes humanity a decade of effort could happen in a day (or faster) that's the nature of thinking faster. People often mistakenly model a smart human compared to a dumb human, but the actual spectrum of capability is way wider than that. Lack of knowledge could be another way of saying "not aligned" which is the core issue.
  - sterlind 4y ago
    that's the thing though, I don't believe superintelligence can possibly work like that. I think it's a logical fallacy to treat superintelligences like hypercomputers. they're constrained by existing on classical computers. like, as a trivial counterexample, I can speak at 500 wpm. let's say that ChatGPT can generate 500 words per second for a single thread of conversation - I think that's a generous overestimate. now that's a 60x speedup over me, not a 1,000,000,000x like you're talking about. do you honestly believe they they can make ChatGPT run 16.6 million times faster, without changing hardware? do you think ChatGPT will just like, hit an inflection point where it realizes how to refactor its inference code to run 16,666,666x faster? no, I think that's absurd. you're treating these things like black boxes but they are constrained by computational complexity, die area and the speed of light for Christ's sake.
    - righttoolforjob 4y ago
      What is the limit of intelligence? What limits the human intelligence? How many more times intelligent would an AGI have to be to have the capacity to take over the world?
flappyeagle 4y ago
EY’s arguments don’t seem to take physics into account. That’s why they are not taken very seriously. To many it sounds like: volcanos are dangerous. What if there were a billion volcanos? We would all die. Everything individual sentence is true. Volcanos are dangerous. If there were a billion of them the world might be uninhabitable. The thing preventing a billion volcanos is like… thermodynamics
- gonehome 4y ago
  This is wrong, computers can operate at billions of operations per second. You can wave hands and say the words “thermodynamics” but it doesn’t actually mean anything. The claim that computers just can’t do X because “physics” looks weaker every day. Intelligence isn’t magic, hardware today seems more than capable. I think Carmack is probably right, it’s not a hardware constraint at this point - it’s a software intuitive leap.
  - flappyeagle 4y ago
    > Intelligence isn’t magic, hardware today seems more than capable. I think Carmack is probably right, it’s not a hardware constraint at this point - it’s a software intuitive leap. This is all ass-pulling. The hardware is or is not good enough (it's not). We either have found a software breakthrough or not (we haven't).
    - tanseydavid 4y ago
      >> We either have found a software breakthrough or not (we haven't). Well it is not clearly not AGI but even the improvement from GPT3 to GPT4 seems to me at least to reflect what one might describe as a "software breakthrough."

_skel 4y ago

EY's arguments aren't really possible to engage with because they don't have much of a causal basis. Nobody who says AI is likely to kill us all can demonstrate a plausible sequence of events, with logical causality linking the events together, that leads to mass extinction. It's all very handwavy. Steven Pinker said it pretty well: > The AI-existential-threat discussions are unmoored from evolutionary biology, cognitive psychology, real AI, sociology, the history of technology and other sources of knowledge outside the theater of the imagination. I think this points to a meta-problem. The AI-ET community shares a bad epistemic habit (not to mention membership) with parts of the Rationality and EA communities, at least since they jumped the shark from preventing malaria in the developing world to seeding the galaxy with supercomputers hosting trillions of consciousnesses from uploaded connectomes. They start with a couple of assumptions, and lay out a chain of abstract reasoning, throwing in one dubious assumption after another, till they end up way beyond the land of experience or plausibility. The whole deduction exponentiates our ignorance with each link in the chain of hypotheticals, and depends on blowing off the countless messy and unanticipatable nuisances of the human and physical world. It’s an occupational hazard of belonging to a “community” that distinguishes itself by raw brainpower. OK, enough for today – hope you find some of it interesting. (https://marginalrevolution.com/marginalrevolution/2023/03/steven-pinker-on-existential-risk.html https://marginalrevolution.com/marginalrevolution/2023/03/st...) You know what's most likely to lead to human extinction, and has been for all of our lives? Nuclear war. EY argues that we should bomb "rogue" datacenters and that is obviously and immediately more dangerous than anything he has proven about AI. What does he think would happen if the US bombs a datacenter in China or Israel bombs one in Iran?

gonehome 4y ago
I'd argue the dismissals are a lot more handwavy. The simplified argument is: - Superintelligent AGI that can modify itself in pursuit of a goal is possible. - If that AGI is not aligned with human goals it very likely ends in the end of humanity. - We have no idea how to align an AGI or even really observe what it's true state/goals are. Without this capability if we stumble upon creating an AGI capable of improving itself in such a way that leads to super intelligence before we have alignment, it's game over. ## For point 1, that seems like the consensus view now (though it wasn't until recently). I think it seems obvious, but my general arguments would be humans aren't special, brains are everywhere in nature, biology is constrained in ways other systems are not (birth, energy usage, etc.) For point 2, in pursuit of whatever its goal is, even a 'dumb' goal that happens to satisfy its reward functions, humanity will likely either try to stop it (and then be an obstacle) or at a minimum will just be in the way - like an anthill destroyed in the construction of a dam. Point 3 is not controversial. The dismissals from Tyler Cohen and Pinker are mostly just relying on heuristics which are often right, but even if they're right 999 out of 1000 times, if that 1 in 1000 error is the end of humanity, that's pretty bad. Most of the time a disease not a pandemic, but sometimes it is. I've read some of what Pinker has written about it, he doesn't understand EY's arguments (imo). Cohen's recent blog post could be summarized as "we'll likely see an end to peacetime and increasing global instability, might as well get AGI out of it". Just because things don't usually result in human extinction doesn't mean they can't.
- righttoolforjob 4y ago
  I can't find a flaw in this argument. That is a bit scary actually.
  - gonehome 4y ago
    With downcast eyes and heavy heart, Eliezer left Sam Altman Some years go by, and AGI progresses to assault man Atop a pile of paper clips he screams "It's not my fault, man!" But Eliezer's long since dead, and cannot hear Sam Altman. https://astralcodexten.substack.com/p/turing-test https://astralcodexten.substack.com/p/turing-test

sterlind 4y ago

I think it's likely we'll develop superintelligence very soon, but it won't be any more knowledgeable than us, just smarter. And knowledge has been the bottleneck for most of human history - our ancestors weren't dumber. Also, it will be bottlenecked by needing to solve NP-complete problems - unless P=NP, it's not going to be all that much faster at solving hard math problems. Basically, it still needs to do science, and science takes time.

gonehome 4y ago
Take a smart human and speed up their thinking a billion times, you get the time for free. What takes humanity a decade of effort could happen in a day (or faster) that's the nature of thinking faster. People often mistakenly model a smart human compared to a dumb human, but the actual spectrum of capability is way wider than that. Lack of knowledge could be another way of saying "not aligned" which is the core issue.
- sterlind 4y ago
  that's the thing though, I don't believe superintelligence can possibly work like that. I think it's a logical fallacy to treat superintelligences like hypercomputers. they're constrained by existing on classical computers. like, as a trivial counterexample, I can speak at 500 wpm. let's say that ChatGPT can generate 500 words per second for a single thread of conversation - I think that's a generous overestimate. now that's a 60x speedup over me, not a 1,000,000,000x like you're talking about. do you honestly believe they they can make ChatGPT run 16.6 million times faster, without changing hardware? do you think ChatGPT will just like, hit an inflection point where it realizes how to refactor its inference code to run 16,666,666x faster? no, I think that's absurd. you're treating these things like black boxes but they are constrained by computational complexity, die area and the speed of light for Christ's sake.
  - righttoolforjob 4y ago
    What is the limit of intelligence? What limits the human intelligence? How many more times intelligent would an AGI have to be to have the capacity to take over the world?

flappyeagle 4y ago

EY’s arguments don’t seem to take physics into account. That’s why they are not taken very seriously. To many it sounds like: volcanos are dangerous. What if there were a billion volcanos? We would all die. Everything individual sentence is true. Volcanos are dangerous. If there were a billion of them the world might be uninhabitable. The thing preventing a billion volcanos is like… thermodynamics

gonehome 4y ago
This is wrong, computers can operate at billions of operations per second. You can wave hands and say the words “thermodynamics” but it doesn’t actually mean anything. The claim that computers just can’t do X because “physics” looks weaker every day. Intelligence isn’t magic, hardware today seems more than capable. I think Carmack is probably right, it’s not a hardware constraint at this point - it’s a software intuitive leap.
- flappyeagle 4y ago
  > Intelligence isn’t magic, hardware today seems more than capable. I think Carmack is probably right, it’s not a hardware constraint at this point - it’s a software intuitive leap. This is all ass-pulling. The hardware is or is not good enough (it's not). We either have found a software breakthrough or not (we haven't).
  - tanseydavid 4y ago
    >> We either have found a software breakthrough or not (we haven't). Well it is not clearly not AGI but even the improvement from GPT3 to GPT4 seems to me at least to reflect what one might describe as a "software breakthrough."