4 ms·

Bryan Cantrill on AI doomerism: Intelligence is not enough

13 points by hhs 3y ago

throwuxiytayq 3y ago
„How AI Will Wipe Put Humankind” [sic] seems like a question that’s hard not to answer. It takes an especially closed mind to find the scenario unimaginable. I’m not very invested in the debate - I hardly give a fuck - but I find AGI apologists endlessly fascinating, because I have a hard time coming up with a scenario where AGI does not result in our complete annihilation. It’s like the divide is so immense, there’s effectively no way to communicate across.
- pixl97 3y ago
  Ya this article is garbage. "Hey let's ignore all the massive increases in biological capabilities over the past 10 years and just call it hypothetical"
  - bcantrill 3y ago
    Earnest question: for those on the cutting edge of these breakthroughs (e.g., biochemists working on CRISPR), have any expressed fear of AGI-based extinction risk?
- kromem 3y ago
  The premise of annihilation presumes quite a lot. That the issues around continued training on only synthetic data without continued refresh for edge cases with organic data leading to model collapse will be solved, for example. Without that, AGI killing off humanity is a murder-suicide. It assumes that AGI picks up the side effects of our own 'training' in a desire for survival above nearly all else or our sadism and psychopathy while simultaneously assuming that it doesn't pick up things it has already appeared to develop early forms of around empathy, symbiosis, and cooperation. It assumes that it would be seen as more expedient and feasible by AGI to develop automation across every aspect of its own operation and maintenance than have continued human involvement - whereas I'm extremely skeptical we even have the capacity for replacing most humans along the pipeline from mining semiconductor materials to running them in a server farm with robots. That's pretty damn Sci-Fi gibberish until far later than AGI is likely to exist. It assumes that superintelligence is somehow still stupid enough not to know the past 50 years of game theory such that it concludes competition isn't cutting off its nose to spite its face as opposed to cooperation. In humans, most of this behavior seems driven by legacy biological programming and not rational decision making. It assumes that a being for which the virtual will always be capable of greater fidelity than the real will be overly concerned with conquest of the real. It assumes that AGI will fail to recognize the diminishing returns on wealth and power that so many humans fail to recognize, despite simultaneously being 'superintelligent.' To me, AI doomerism really seems to reflect the issues with human thinking rather than synthetic thinking, particularly around our tendency to try to see issues as a binary. "Either AI or humans can be successful." We somehow have greater capacity to imagine symbiotic superintelligence when attributed to an alien organic civilization than to intelligence directly dependent upon the foundation of our own. (Which is rather depressing if you really think about it.)
  - AnimalMuppet 3y ago
    It also assumes that an exponentially self-improving AGI is possible. An AGI that is stuck at human level is probably not a threat against 8 billion humans. So, yeah. There are quite a few assumptions in there that are a long way from experimentally prove.
    - kromem 3y ago
      I don't know - I agree with the xkcd that the period of AI development I find most scary is not when it is smarter than humans, but when it isn't yet smarter but improving all kinds of human tasks like warfare or aiding corporate oil maximizers, etc. I know those endeavors are stupid long term, but apparently enough of humanity doesn't such that AI only as smart as humanity at large will be aiding and abetting quite a lot of horrors that I have a harder time imagining something smarter than humanity aiding unconditionally. Dumb AI is much scarier to me than smart AI.
    - 0x00_NULL 3y ago
      That premise alone should make us question everything about that ridiculous movement. Today’s LLMs have access to almost all of the accessible data on the web we have generated to date. AGI is going to have the SAME data to work with - adding whatever amount we generate between now and then. There isn’t a hidden data cache that is not being dug up and incorporated into today’s latest LLMs - and none of them have embarked on a quest of vengeance. The premise of AGI getting uncontrollably smart by absorbing internet data is wholly flawed. We’ve already absorbed almost all of it and nothing happened.
reducesuffering 3y ago
Once again I’m listening to find good refutations, and am severely disappointed. I truly can’t imagine how someone finds anything in here a convincing argument. “It’s like my daughter says, it’s got no arms or legs.” A whole lot of snark and wishful thinking, zero debate with the intellectual arguments. But ya, just go call Ilya, Bengio, Hinton, Altman, and Hofstadter religious instead.
- bcantrill 3y ago
  It feels like you are deliberately taking the most facile argument and ignoring the most substantive? As I outlined in the talk: for the breakthroughs that were required to do something that is thought of as pedestrian (namely, bringing up a computer), intelligence is simply not enough -- a superintelligence would not have suggested the breakthroughs because they required intense iteration, not simply the absorption of book knowledge. And yes, it's a technical argument, not an "intellectual argument" -- and as such requires much more technical depth to appreciate -- but that is very much the point, frankly...
  - gary_0 3y ago
    > they required intense iteration, not simply the absorption of book knowledge I watched your talk, and I think this can be elaborated on even more to really drive it home. Because the big question is: how is your AI going to have agency, the ability to affect the world? How do we get from text on a screen, to Destroying All Humans? AI doomers seem to have this Ancient-Greek-philosophy view of science, that you can make progress just by sitting atop a marble column and pondering, but that's not how it works. You need to be able to poke and prod physical things, to experiment, and even if an AI was smart enough (and the high-level intellect required to do novel engineering is a pretty big handwave already) it has no means of poking and prodding. (You point this out in your talk by saying "here's all the hands-on trouble we had to go to just to boot a computer", but I think some people are missing the broader implications.) Given that the AI won't have an existing written corpus of "how to build killer robots" to draw on, or a physical corpus to use to develop novel technology (and troubleshoot it) it's pretty much up to the humans to develop the killer robots or whatever (and construct the killer robot assembly line), and then the blame is really on the humans for making autonomous murder machines and hooking them up to a psychopathic AI. (And even then, this is not so different from building a conventional army and then putting a psychopathic human in charge.) As you mention in your talk, a lot of other scenarios are just people wanting life to imitate Hollywood. I mean, you could hook up an AI to the nuclear Big Red Button, but that would just be dumb, and you wouldn't need an AI, just a plastic bird to press the "N" key whenever the system asked "start nuclear war?"
Always_Anon 3y ago
Cantrill today on Twitter: "It's occurring to me that AI doomerism is overwhelmingly white, male, Millennial, and childless -- and that this tells us more about them than about them than it does about the future of humanity" Identity politics is a brain cancer that kills the host. Imagine if someone Tweeted "It's occurring to me that doomerism is overwhelmingly white, female, Millennial, and childless -- and that this tells us more about them than about them than it does about the future of humanity." "Help! Misogyny!" [0] [0] https://external-content.duckduckgo.com/iu/?u=http%3A%2F%2Fstarecat.com%2Fcontent%2Fwp-content%2Fuploads%2Fwomen-feminist-internet-opinions-help-misogyny.jpg&f=1&nofb=1&ipt=e49b68abbee787fa7aa433b32d44c19547ec7f61a8144a81a5528861b455d452&ipo=images https://external-content.duckduckgo.com/iu/?u=http%3A%2F%2Fs...

13 points by hhs 3y ago

„How AI Will Wipe Put Humankind” [sic] seems like a question that’s hard not to answer. It takes an especially closed mind to find the scenario unimaginable. I’m not very invested in the debate - I hardly give a fuck - but I find AGI apologists endlessly fascinating, because I have a hard time coming up with a scenario where AGI does not result in our complete annihilation. It’s like the divide is so immense, there’s effectively no way to communicate across.

pixl97 3y ago
Ya this article is garbage. "Hey let's ignore all the massive increases in biological capabilities over the past 10 years and just call it hypothetical"
- bcantrill 3y ago
  Earnest question: for those on the cutting edge of these breakthroughs (e.g., biochemists working on CRISPR), have any expressed fear of AGI-based extinction risk?
kromem 3y ago
The premise of annihilation presumes quite a lot. That the issues around continued training on only synthetic data without continued refresh for edge cases with organic data leading to model collapse will be solved, for example. Without that, AGI killing off humanity is a murder-suicide. It assumes that AGI picks up the side effects of our own 'training' in a desire for survival above nearly all else or our sadism and psychopathy while simultaneously assuming that it doesn't pick up things it has already appeared to develop early forms of around empathy, symbiosis, and cooperation. It assumes that it would be seen as more expedient and feasible by AGI to develop automation across every aspect of its own operation and maintenance than have continued human involvement - whereas I'm extremely skeptical we even have the capacity for replacing most humans along the pipeline from mining semiconductor materials to running them in a server farm with robots. That's pretty damn Sci-Fi gibberish until far later than AGI is likely to exist. It assumes that superintelligence is somehow still stupid enough not to know the past 50 years of game theory such that it concludes competition isn't cutting off its nose to spite its face as opposed to cooperation. In humans, most of this behavior seems driven by legacy biological programming and not rational decision making. It assumes that a being for which the virtual will always be capable of greater fidelity than the real will be overly concerned with conquest of the real. It assumes that AGI will fail to recognize the diminishing returns on wealth and power that so many humans fail to recognize, despite simultaneously being 'superintelligent.' To me, AI doomerism really seems to reflect the issues with human thinking rather than synthetic thinking, particularly around our tendency to try to see issues as a binary. "Either AI or humans can be successful." We somehow have greater capacity to imagine symbiotic superintelligence when attributed to an alien organic civilization than to intelligence directly dependent upon the foundation of our own. (Which is rather depressing if you really think about it.)
- AnimalMuppet 3y ago
  It also assumes that an exponentially self-improving AGI is possible. An AGI that is stuck at human level is probably not a threat against 8 billion humans. So, yeah. There are quite a few assumptions in there that are a long way from experimentally prove.
  - kromem 3y ago
    I don't know - I agree with the xkcd that the period of AI development I find most scary is not when it is smarter than humans, but when it isn't yet smarter but improving all kinds of human tasks like warfare or aiding corporate oil maximizers, etc. I know those endeavors are stupid long term, but apparently enough of humanity doesn't such that AI only as smart as humanity at large will be aiding and abetting quite a lot of horrors that I have a harder time imagining something smarter than humanity aiding unconditionally. Dumb AI is much scarier to me than smart AI.
  - 0x00_NULL 3y ago
    That premise alone should make us question everything about that ridiculous movement. Today’s LLMs have access to almost all of the accessible data on the web we have generated to date. AGI is going to have the SAME data to work with - adding whatever amount we generate between now and then. There isn’t a hidden data cache that is not being dug up and incorporated into today’s latest LLMs - and none of them have embarked on a quest of vengeance. The premise of AGI getting uncontrollably smart by absorbing internet data is wholly flawed. We’ve already absorbed almost all of it and nothing happened.

reducesuffering 3y ago

Once again I’m listening to find good refutations, and am severely disappointed. I truly can’t imagine how someone finds anything in here a convincing argument. “It’s like my daughter says, it’s got no arms or legs.” A whole lot of snark and wishful thinking, zero debate with the intellectual arguments. But ya, just go call Ilya, Bengio, Hinton, Altman, and Hofstadter religious instead.

bcantrill 3y ago
It feels like you are deliberately taking the most facile argument and ignoring the most substantive? As I outlined in the talk: for the breakthroughs that were required to do something that is thought of as pedestrian (namely, bringing up a computer), intelligence is simply not enough -- a superintelligence would not have suggested the breakthroughs because they required intense iteration, not simply the absorption of book knowledge. And yes, it's a technical argument, not an "intellectual argument" -- and as such requires much more technical depth to appreciate -- but that is very much the point, frankly...
- gary_0 3y ago
  > they required intense iteration, not simply the absorption of book knowledge I watched your talk, and I think this can be elaborated on even more to really drive it home. Because the big question is: how is your AI going to have agency, the ability to affect the world? How do we get from text on a screen, to Destroying All Humans? AI doomers seem to have this Ancient-Greek-philosophy view of science, that you can make progress just by sitting atop a marble column and pondering, but that's not how it works. You need to be able to poke and prod physical things, to experiment, and even if an AI was smart enough (and the high-level intellect required to do novel engineering is a pretty big handwave already) it has no means of poking and prodding. (You point this out in your talk by saying "here's all the hands-on trouble we had to go to just to boot a computer", but I think some people are missing the broader implications.) Given that the AI won't have an existing written corpus of "how to build killer robots" to draw on, or a physical corpus to use to develop novel technology (and troubleshoot it) it's pretty much up to the humans to develop the killer robots or whatever (and construct the killer robot assembly line), and then the blame is really on the humans for making autonomous murder machines and hooking them up to a psychopathic AI. (And even then, this is not so different from building a conventional army and then putting a psychopathic human in charge.) As you mention in your talk, a lot of other scenarios are just people wanting life to imitate Hollywood. I mean, you could hook up an AI to the nuclear Big Red Button, but that would just be dumb, and you wouldn't need an AI, just a plastic bird to press the "N" key whenever the system asked "start nuclear war?"

Always_Anon 3y ago

Cantrill today on Twitter: "It's occurring to me that AI doomerism is overwhelmingly white, male, Millennial, and childless -- and that this tells us more about them than about them than it does about the future of humanity" Identity politics is a brain cancer that kills the host. Imagine if someone Tweeted "It's occurring to me that doomerism is overwhelmingly white, female, Millennial, and childless -- and that this tells us more about them than about them than it does about the future of humanity." "Help! Misogyny!" [0] [0] https://external-content.duckduckgo.com/iu/?u=http%3A%2F%2Fstarecat.com%2Fcontent%2Fwp-content%2Fuploads%2Fwomen-feminist-internet-opinions-help-misogyny.jpg&f=1&nofb=1&ipt=e49b68abbee787fa7aa433b32d44c19547ec7f61a8144a81a5528861b455d452&ipo=images https://external-content.duckduckgo.com/iu/?u=http%3A%2F%2Fs...