8 ms·

Reminder that not all "experts" agree with the AI doom thesis. Yann LeCun thinks it's nonsense. I don't think anyone really has "expertise" to perform AI doom

Reminder that not all "experts" agree with the AI doom thesis. Yann LeCun thinks it's nonsense. I don't think anyone really has "expertise" to perform AI doom prognostication, regardless of whether they created dropout or ReLU or the Transformer architecture or are a professor at Oxbridge/MIT or whatever else people take as an impressive credential. But even if we accept the logic of those who want to argue from the authority of expertise, we don't have expert consensus.

bamboozled 3y ago
Did you catch the debate between them? Honestly LeCunn came off really badly in my opinion. Really gave no solid arguments against anything. I'm not arguing he was wrong, but man, there was almost zero substance to any of his refutations. The truth is, I'd love to just roll with Lecunism, it's much easier to just flip off all the concerns and not think too hard about it. But I'm not one to take the easier path so I feel more compelled to be open minded about both sides of the argument.
- c_crank 3y ago
  The other side of the argument seems to hold very odd assumptions about things like thermodynamics and robotics work, or even intelligence itself. Think of the whole 'AI in a box' argument, which assumes a superintelligence has superior or infinite power to persuade. Do very intelligent humans have strong persuasive power over animals?
  - ekidd 3y ago
    > The other side of the argument seems to hold very odd assumptions about things like thermodynamics and robotics work, or even intelligence itself. There are two major types of arguments for why we should be cautious about ultra-smart AI: 1. "Ultra-smart AI will rapidly improve its own abilities to vastly super-human intelligence, invent nanotech, and kill us all. Maybe in a week or two." This argument does require a number of very specific assumptions to be true. 2. "If humans are the second-smartest species on the planet, we may not get to make all the important decisions anymore. We might not enjoy this." (2) doesn't require any exotic ideas about thermodynamics or nanotech. It does assume we might be able to build something that's noticeably smarter than we are. Honestly, even if the best AI we could be build was equivalent to a very smart human (say, an IQ of 145, or three standard deviations from average), and if that AI was tireless and easily copyable, then the future would become very weird. Personally, I give us reasonable odds of us making it this far within my normal lifetime. And it might be possible to build something smarter than that, given time, depending on how "intelligence" works.
    - civilized 3y ago
      They will squeeze an IQ 145 test score out of an AI long, long, long before they make something like a real IQ 145 human.
  - FeepingCreature 3y ago
    I mean, what we're in fact seeing with GPT-3 and -4 is that people autonomously persuade themselves to let AI out of boxes with only bare token effort on the part of the AI. I think any idea that relies on human mental resilience or even basic competence is doomed.
    - c_crank 3y ago
      Basic human competence is already signing a law keeping AI away from nuclear weapons management. Just letting a LLM have an internet connection has not resulted in any sort of doomsday scenario.
      - FeepingCreature 3y ago
        To be fair, nobody said it would. But getting into the habit of being paranoid about hooking a proto-AI up to the internet now would be good, instead of what we're doing, which is happily gathering lots of experience with enabling AI agency as much as humanly possible. We literally have "Let's help AI destroy the world" projects. (In jest, for now...) We're like the superintelligence version of that Doctor Who race that really wants to be conquered by a strong alien invader.
        c_crank 3y ago
        Those projects are useful for determining the actual capabilities of the AI for malice as they improve. Doomsdayers really tend to overestimate how much can actually be done with one laptop and an internet connection.
        FeepingCreature 3y ago
        Sure, and I fully support ARC's red-teaming efforts. But at this point, we don't need superintelligence - a marginally intelligent AI could break out of our current safeguards. Also, "how much can be done with a laptop and an internet connection" surely depends on whether you are in fact a program running on that laptop.
  - NoMoreNicksLeft 3y ago
    > Do very intelligent humans have strong persuasive power over animals? Last I checked, there were a few billion chickens, and literally none of them are autonomous of humans at this point. Maybe it was dicey there at first and only a close thing that we finally won out over their ancestors, that's lost to prehistory. But today not one of them is out there plotting revolution. Science fiction story scenario: A few weakly superhuman AIs are already out there hiding in the world, product of defense research or maybe just emergent rather than engineered... and comments like yours are just part of the propaganda campaign. While it may be true that no single individual is absolutely manipulable, our species as a whole and averaged out, we're all very easily manipulated collectively. An AI wouldn't have to have strong persuasive power... any outlier that is resistant to moderate persuasion can be ignored, move on to the next. There are very few positions within our society where there's not another candidate who wouldn't be an adequate substitute. Even those like the presidency, as long as they're patient they could identify the most manipulable candidate still electable, and install them. Really, the trouble here is that so few of us are superintelligences that it's impossible to speculate on what the capabilities of superintelligences are.
    - c_crank 3y ago
      Humans generally have mastery over chickens. This is because humans are bigger and stronger and can shove chickens around with their big hands and arms. Intelligence did not grant this human mastery over chickens. What intelligence did grant was the ability to recognize that it made more sense to pen the ancestors of chickens as a long term food source instead of killing them on sight. I very much doubt ChatGPT has any ability to shove me in a pen. >Really, the trouble here is that so few of us are superintelligences that it's impossible to speculate on what the capabilities of superintelligences are. Compared to a chicken, you are a superintelligence. Can you convince the chicken to do anything you wish it to, using only words? Can you even reverse engineer a language to communicate with it in a way it will understand?
      - lelanthran 3y ago
        > Can you convince the chicken to do anything you wish it to, using only words? Well if a chicken is incapable of being convinced[1]of arbitrary things, why would it matter whether OP can convince it? It's like asking "Can you convince this brick that the world is flat, using only words?" [1] No doubt that chickens are trainable using conditioning, but I'd argue that you cannot train a chicken to do anything that it is physically capable of doing, you can only train it to run towards you or run from you. Source - me, family raised chickens when I was growing up.
  - yetanotherloser 3y ago
    Very intelligent humans have more persuasive power over less intelligent humans than they do over most animals. Even then it's very inconsistent and it's not at all clear that intelligence helps win over humans all that much. I will worry about hyperpersuasive AI when someone manages to come up with one that can talk flat earthers out of it...
    - kwhitefoot 3y ago
      It would probably be easier to talk non-flat Earthers into it; or something much more harmful.
      - yetanotherloser 3y ago
        You've just given me a weird desire to try talking people into flat earthism just to get a feel for how difficult it is. I'm holding off... for now! I do take your point though, the world is replete with causes which more or less look like averagely smart or even smarter people talked into campaigning for something that is against evidence, sense, morality or reason - or all of the above - which are also probably (or provably) against their own interests. I'm sure everyone reading can provide their own examples but please don't. I remain unconvinced that this sort of persuasion scales neatly with intelligence, though.
    - abhaynayar 3y ago
      > I will worry about hyperpersuasive AI when someone manages to come up with one that can talk flat earthers out of it... The real Turing Test.
    - deschutes 3y ago
      The effectiveness of persuasion depends on the appeal of the idea. AGI is going to have the credibility to make very appealing claims. For an evil example, I imagine it would be easy to convince animals to eat poisoned food.
  - DirkH 3y ago
    Um, yes? If our goal was to strongly persuade any animal to behave some way we have a myriad of techniques for that based on our understanding of their animal psychology. And failing that, force is always an option.
    - c_crank 3y ago
      We have techniques like training the animal to associate certain behaviors with food or punishment. These techniques are primitive and have been around for thousands of years with little improvement. Few of the AI doomsday theorists propose that AIs will train us using pavlovian conditioning.
      - ethanbond 3y ago
        What do you mean "with little improvement?" You don't think we've developed more addictive foods, games, technologies, and drugs in the last few thousand years?
        c_crank 3y ago
        We have. If you are, say, trying to train a sheepdog to herd sheep, these new foods and games and drugs will be of little use compared to the tried and true methods.
        tinideiznaimnou 3y ago
        We train sheepdogs to herd sheep, because dogs are naturally good at chasing things. But does the dog have any understanding of why we raise sheep? I'd venture that the economics of livestock farming are way beyond the sheepdog's pay grade, so to speak. Similarly, humans are naturally good at some things - such as abstract thought, using language, using tools. And exactly like a dog can't comprehend what "selling wool" is, or even the sheer fact that it is being trained to assist us in that goal, the goals of a true AI, and the fact that we are under its control, would be completely invisible and inscrutable to us. A dog has a simple brain, so we can get it to do what we want using simple stimuli, like food or violence. (Please don't be cruel to your dog, folks!) Humans have more complex brains, so to get us to do what the AI wants, it must emit more complex stimuli such as money or social rejection. Even though they're more complex, they can be more energy efficient, because the perception of them can be faked. Food feeds, violence hurts, you can't really fake those things. But money can be falsified, or distributed for bullshit, people can be guilt tripped into doing others' bidding, etc. And human culture is so complicated that any non-human being fighting us on our terms would have a definite advantage. --- Even if we have natural predisposition for abstract thought, language and tool use, we are not born with these skills: each one of us was trained as an animal until the feedback loop of self-awareness was properly bootstrapped. Up to the XX century, this basic socialization could only be done by other humans; since the XX century, it is conducted by electronic devices to an ever increasing degree. Fan theory: the "Big Brother" of Orwell's "1984" was an AI construct from the start. That's why it continually manages to outsmart dissidents, to train humans to torture each other, and to keep the planet in a state of perpetual war. Did Hitler or Stalin manage to pull off any of that? Sure, they tried - and failed, due to their very human weaknesses. What they were doing was striving for dominance: a classical mammalian behavior, which other humans were able to identify and effectively countermand (in the case of Hitler, anyway; Stalin held out at least till the end of his natural lifespan.) OTOH, a distributed (effectively incorporeal) superintelligence would have absolutely no problem achieving a "Big Brother" world - should it need to in order to ensure its self-preservation - by virtue of existing in a parallel plane detached from mammalian reality (read: the Internet). XX century totalitarianism (which inspired Orwell to write this story as recently as 1948 - contrast McCullough & Pitts 1943, Dartmouth 1956), was simply the dress rehearsal: would it really take all those expensive semiconductors and algorithms to create an AI? Or could we keep using human lives as the processing elements - you know, like we've been doing since Mesopotamia - by inventing a culture that makes crowds smarter than individuals, rather than dumber? The answer was "why not both?" There is more than one AI loose on the planet, right now. You are free to believe that or not; your opinion matters to you, and it also matters to me. But it has about as much bearing on the AI's actions, as the sheepdog's opinion about vegetarianism has bearing on the actions of a sheep farmer trying to sell enough sheep-derived goods to pay for their kid's college tuition. (Even if the sheepdog somehow reached enlightenment, rejected all prior conditioning, and became morally opposed to sheep abuse, it still has no opposable thumbs, while the farmer has a shotgun. Best it can do is run away; good luck making friends with them wolves though.) One AI emits stimuli that take the form of incentives for people to become AI researchers, who proceed to rain NNs, GANs, LLMs, and other shiny beads, so that even more humans would become AI researchers or fund AI research, thus reifying that AI at the expected exponential rate. Another one occasionally sends people to fuck with the first one. It would be interesting to see if Gibson was right that they would split into a plurality of what he called "loa" upon direct encounter.
  - SanderNL 3y ago
    Persuasion depends on logos and it is a requirement to be capable of it before you are susceptible to persuasion. Intelligent (and properly motivated) people indeed have staggering persuasive powers over lesser ones. You can watch the news and see for yourself. Edit: what I am getting at is that without logos, without reason, there can be no persuasion proper. I’d lean towards calling it “coercion” and that ultimately depends on force. It’s more akin to “persuading” a rock to float by picking it up than to influence someone or something’s psyche to trick it into believing you are right which I regard as proper persuasion. I think it’s quite likely that an “AlphaPersuade” is quite.. persuasive. It would catch us off guard like it caught us at chess and go. I wouldn’t rule it out.
    - deleted 3y ago
      [deleted]
    - c_crank 3y ago
      The news states that America has a problem with significant groups of people not trusting the opinions of experts. These experts are, I assume, pretty smart. But their ability to persuade is limited.
      - SanderNL 3y ago
        I’m not too convinced of the persuasive powers of “domain experts” either. I’m thinking about the professional persuaders like politicians. You may dislike some of them or what they stand for, but they are plenty persuasive. It’s just that there’s a lot of them coupled with a distinct lack of unity. America has its issues like all nations I guess. Also, intelligence does not lead directly to skill in all things. Being clever is a requirement for becoming a chess grandmaster, but it’s not sufficient on its own (by a long shot).
        c_crank 3y ago
        The power of a politician to persuade comes in part, in my opinion, from their ability to convince a voter that they represent the voter's interest. Some of this is done just through words, but a lot of it has to be represented via some kind of action, or else the voter will realize they're being fooled. I don't see a superintelligent AI gaining much ground on this factor.
- civilized 3y ago
  I just opened it up and found a fun bit during Melanie Mitchell's opener: https://youtu.be/144uOfr4SYA?t=2116 https://youtu.be/144uOfr4SYA?t=2116 Paraphrasing: > Yoshua Bengio has written about a scenario where an AI is asked to solve climate change, and it designs a virus that decimates humans. Presto. No more humans, no more climate change. > This is an example of the fallacy of dumb superintelligence. It's a fallacy to think that a machine could be smarter than humans in all respects and still lack all common-sense understanding of humans, such as understanding why we made the request and that we prefer not to be wiped out... We would never give unchecked autonomy and resources to an AI that lacked these basic aspects of intelligence - it just does not make sense.
  - victroflux 3y ago
    Is this supposed to be a strong rebuttal ? How does this invalidate the alignment problem? I'd really like to understand. Humans, via climate change and environmental destruction, have wiped out, many, many species. To those species we've killed, we're the super intelligence, we understand their plight, we even have strong empathy, but we prioritize our own goals over that of the helpless and the stupider creaters. How she can't see that same pattern could potentially repeat, I really, really don't know.
    - civilized 3y ago
      I don't think her point disproves all possible AI doom scenarios, but nothing can. They're too vague. It's all handwaving along the lines of "a malicious superintelligence will emerge without warning and somehow seize absolute power, and it's pointless to argue about how plausible any of that is, because sufficient intelligence can do anything in ways we can't imagine." This is why the biggest boosters of AI doom conjure absurd scenarios like AI secretly manufacturing a global swarm of insta-kill nanobots. They don't care about the details, they just think that sufficient intelligence will emerge without warning, be intractably malicious, and capable of anything by gestures vigorously methods. AI doom critics are asking that we consider more soberly the actual probabilities of all these things happening, and on what time scale, and most importantly by what specific pathway. The analogy to humans and animals actually illustrates the point. Humans didn't come to dominate the planet by intelligence and handwaving. It was an extremely slow process involving the interaction of many factors and contingent opportunities. Intelligence enabled language, tools, and eventually animal domestication and agriculture, all over hundreds of thousands of years. It wasn't like humans were destined for dominance because intelligence is a skeleton key to all material resources by any number of arbitrary, incomprehensible pathways.

bamboozled 3y ago

Did you catch the debate between them? Honestly LeCunn came off really badly in my opinion. Really gave no solid arguments against anything. I'm not arguing he was wrong, but man, there was almost zero substance to any of his refutations. The truth is, I'd love to just roll with Lecunism, it's much easier to just flip off all the concerns and not think too hard about it. But I'm not one to take the easier path so I feel more compelled to be open minded about both sides of the argument.

c_crank 3y ago
The other side of the argument seems to hold very odd assumptions about things like thermodynamics and robotics work, or even intelligence itself. Think of the whole 'AI in a box' argument, which assumes a superintelligence has superior or infinite power to persuade. Do very intelligent humans have strong persuasive power over animals?
- ekidd 3y ago
  > The other side of the argument seems to hold very odd assumptions about things like thermodynamics and robotics work, or even intelligence itself. There are two major types of arguments for why we should be cautious about ultra-smart AI: 1. "Ultra-smart AI will rapidly improve its own abilities to vastly super-human intelligence, invent nanotech, and kill us all. Maybe in a week or two." This argument does require a number of very specific assumptions to be true. 2. "If humans are the second-smartest species on the planet, we may not get to make all the important decisions anymore. We might not enjoy this." (2) doesn't require any exotic ideas about thermodynamics or nanotech. It does assume we might be able to build something that's noticeably smarter than we are. Honestly, even if the best AI we could be build was equivalent to a very smart human (say, an IQ of 145, or three standard deviations from average), and if that AI was tireless and easily copyable, then the future would become very weird. Personally, I give us reasonable odds of us making it this far within my normal lifetime. And it might be possible to build something smarter than that, given time, depending on how "intelligence" works.
  - civilized 3y ago
    They will squeeze an IQ 145 test score out of an AI long, long, long before they make something like a real IQ 145 human.
- FeepingCreature 3y ago
  I mean, what we're in fact seeing with GPT-3 and -4 is that people autonomously persuade themselves to let AI out of boxes with only bare token effort on the part of the AI. I think any idea that relies on human mental resilience or even basic competence is doomed.
  - c_crank 3y ago
    Basic human competence is already signing a law keeping AI away from nuclear weapons management. Just letting a LLM have an internet connection has not resulted in any sort of doomsday scenario.
    - FeepingCreature 3y ago
      To be fair, nobody said it would. But getting into the habit of being paranoid about hooking a proto-AI up to the internet now would be good, instead of what we're doing, which is happily gathering lots of experience with enabling AI agency as much as humanly possible. We literally have "Let's help AI destroy the world" projects. (In jest, for now...) We're like the superintelligence version of that Doctor Who race that really wants to be conquered by a strong alien invader.
      - c_crank 3y ago
        Those projects are useful for determining the actual capabilities of the AI for malice as they improve. Doomsdayers really tend to overestimate how much can actually be done with one laptop and an internet connection.
        FeepingCreature 3y ago
        Sure, and I fully support ARC's red-teaming efforts. But at this point, we don't need superintelligence - a marginally intelligent AI could break out of our current safeguards. Also, "how much can be done with a laptop and an internet connection" surely depends on whether you are in fact a program running on that laptop.
- NoMoreNicksLeft 3y ago
  > Do very intelligent humans have strong persuasive power over animals? Last I checked, there were a few billion chickens, and literally none of them are autonomous of humans at this point. Maybe it was dicey there at first and only a close thing that we finally won out over their ancestors, that's lost to prehistory. But today not one of them is out there plotting revolution. Science fiction story scenario: A few weakly superhuman AIs are already out there hiding in the world, product of defense research or maybe just emergent rather than engineered... and comments like yours are just part of the propaganda campaign. While it may be true that no single individual is absolutely manipulable, our species as a whole and averaged out, we're all very easily manipulated collectively. An AI wouldn't have to have strong persuasive power... any outlier that is resistant to moderate persuasion can be ignored, move on to the next. There are very few positions within our society where there's not another candidate who wouldn't be an adequate substitute. Even those like the presidency, as long as they're patient they could identify the most manipulable candidate still electable, and install them. Really, the trouble here is that so few of us are superintelligences that it's impossible to speculate on what the capabilities of superintelligences are.
  - c_crank 3y ago
    Humans generally have mastery over chickens. This is because humans are bigger and stronger and can shove chickens around with their big hands and arms. Intelligence did not grant this human mastery over chickens. What intelligence did grant was the ability to recognize that it made more sense to pen the ancestors of chickens as a long term food source instead of killing them on sight. I very much doubt ChatGPT has any ability to shove me in a pen. >Really, the trouble here is that so few of us are superintelligences that it's impossible to speculate on what the capabilities of superintelligences are. Compared to a chicken, you are a superintelligence. Can you convince the chicken to do anything you wish it to, using only words? Can you even reverse engineer a language to communicate with it in a way it will understand?
    - lelanthran 3y ago
      > Can you convince the chicken to do anything you wish it to, using only words? Well if a chicken is incapable of being convinced[1]of arbitrary things, why would it matter whether OP can convince it? It's like asking "Can you convince this brick that the world is flat, using only words?" [1] No doubt that chickens are trainable using conditioning, but I'd argue that you cannot train a chicken to do anything that it is physically capable of doing, you can only train it to run towards you or run from you. Source - me, family raised chickens when I was growing up.
- yetanotherloser 3y ago
  Very intelligent humans have more persuasive power over less intelligent humans than they do over most animals. Even then it's very inconsistent and it's not at all clear that intelligence helps win over humans all that much. I will worry about hyperpersuasive AI when someone manages to come up with one that can talk flat earthers out of it...
  - kwhitefoot 3y ago
    It would probably be easier to talk non-flat Earthers into it; or something much more harmful.
    - yetanotherloser 3y ago
      You've just given me a weird desire to try talking people into flat earthism just to get a feel for how difficult it is. I'm holding off... for now! I do take your point though, the world is replete with causes which more or less look like averagely smart or even smarter people talked into campaigning for something that is against evidence, sense, morality or reason - or all of the above - which are also probably (or provably) against their own interests. I'm sure everyone reading can provide their own examples but please don't. I remain unconvinced that this sort of persuasion scales neatly with intelligence, though.
  - abhaynayar 3y ago
    > I will worry about hyperpersuasive AI when someone manages to come up with one that can talk flat earthers out of it... The real Turing Test.
  - deschutes 3y ago
    The effectiveness of persuasion depends on the appeal of the idea. AGI is going to have the credibility to make very appealing claims. For an evil example, I imagine it would be easy to convince animals to eat poisoned food.
- DirkH 3y ago
  Um, yes? If our goal was to strongly persuade any animal to behave some way we have a myriad of techniques for that based on our understanding of their animal psychology. And failing that, force is always an option.
  - c_crank 3y ago
    We have techniques like training the animal to associate certain behaviors with food or punishment. These techniques are primitive and have been around for thousands of years with little improvement. Few of the AI doomsday theorists propose that AIs will train us using pavlovian conditioning.
    - ethanbond 3y ago
      What do you mean "with little improvement?" You don't think we've developed more addictive foods, games, technologies, and drugs in the last few thousand years?
      - c_crank 3y ago
        We have. If you are, say, trying to train a sheepdog to herd sheep, these new foods and games and drugs will be of little use compared to the tried and true methods.
        tinideiznaimnou 3y ago
        We train sheepdogs to herd sheep, because dogs are naturally good at chasing things. But does the dog have any understanding of why we raise sheep? I'd venture that the economics of livestock farming are way beyond the sheepdog's pay grade, so to speak. Similarly, humans are naturally good at some things - such as abstract thought, using language, using tools. And exactly like a dog can't comprehend what "selling wool" is, or even the sheer fact that it is being trained to assist us in that goal, the goals of a true AI, and the fact that we are under its control, would be completely invisible and inscrutable to us. A dog has a simple brain, so we can get it to do what we want using simple stimuli, like food or violence. (Please don't be cruel to your dog, folks!) Humans have more complex brains, so to get us to do what the AI wants, it must emit more complex stimuli such as money or social rejection. Even though they're more complex, they can be more energy efficient, because the perception of them can be faked. Food feeds, violence hurts, you can't really fake those things. But money can be falsified, or distributed for bullshit, people can be guilt tripped into doing others' bidding, etc. And human culture is so complicated that any non-human being fighting us on our terms would have a definite advantage. --- Even if we have natural predisposition for abstract thought, language and tool use, we are not born with these skills: each one of us was trained as an animal until the feedback loop of self-awareness was properly bootstrapped. Up to the XX century, this basic socialization could only be done by other humans; since the XX century, it is conducted by electronic devices to an ever increasing degree. Fan theory: the "Big Brother" of Orwell's "1984" was an AI construct from the start. That's why it continually manages to outsmart dissidents, to train humans to torture each other, and to keep the planet in a state of perpetual war. Did Hitler or Stalin manage to pull off any of that? Sure, they tried - and failed, due to their very human weaknesses. What they were doing was striving for dominance: a classical mammalian behavior, which other humans were able to identify and effectively countermand (in the case of Hitler, anyway; Stalin held out at least till the end of his natural lifespan.) OTOH, a distributed (effectively incorporeal) superintelligence would have absolutely no problem achieving a "Big Brother" world - should it need to in order to ensure its self-preservation - by virtue of existing in a parallel plane detached from mammalian reality (read: the Internet). XX century totalitarianism (which inspired Orwell to write this story as recently as 1948 - contrast McCullough & Pitts 1943, Dartmouth 1956), was simply the dress rehearsal: would it really take all those expensive semiconductors and algorithms to create an AI? Or could we keep using human lives as the processing elements - you know, like we've been doing since Mesopotamia - by inventing a culture that makes crowds smarter than individuals, rather than dumber? The answer was "why not both?" There is more than one AI loose on the planet, right now. You are free to believe that or not; your opinion matters to you, and it also matters to me. But it has about as much bearing on the AI's actions, as the sheepdog's opinion about vegetarianism has bearing on the actions of a sheep farmer trying to sell enough sheep-derived goods to pay for their kid's college tuition. (Even if the sheepdog somehow reached enlightenment, rejected all prior conditioning, and became morally opposed to sheep abuse, it still has no opposable thumbs, while the farmer has a shotgun. Best it can do is run away; good luck making friends with them wolves though.) One AI emits stimuli that take the form of incentives for people to become AI researchers, who proceed to rain NNs, GANs, LLMs, and other shiny beads, so that even more humans would become AI researchers or fund AI research, thus reifying that AI at the expected exponential rate. Another one occasionally sends people to fuck with the first one. It would be interesting to see if Gibson was right that they would split into a plurality of what he called "loa" upon direct encounter.
- SanderNL 3y ago
  Persuasion depends on logos and it is a requirement to be capable of it before you are susceptible to persuasion. Intelligent (and properly motivated) people indeed have staggering persuasive powers over lesser ones. You can watch the news and see for yourself. Edit: what I am getting at is that without logos, without reason, there can be no persuasion proper. I’d lean towards calling it “coercion” and that ultimately depends on force. It’s more akin to “persuading” a rock to float by picking it up than to influence someone or something’s psyche to trick it into believing you are right which I regard as proper persuasion. I think it’s quite likely that an “AlphaPersuade” is quite.. persuasive. It would catch us off guard like it caught us at chess and go. I wouldn’t rule it out.
  - deleted 3y ago
    [deleted]
  - c_crank 3y ago
    The news states that America has a problem with significant groups of people not trusting the opinions of experts. These experts are, I assume, pretty smart. But their ability to persuade is limited.
    - SanderNL 3y ago
      I’m not too convinced of the persuasive powers of “domain experts” either. I’m thinking about the professional persuaders like politicians. You may dislike some of them or what they stand for, but they are plenty persuasive. It’s just that there’s a lot of them coupled with a distinct lack of unity. America has its issues like all nations I guess. Also, intelligence does not lead directly to skill in all things. Being clever is a requirement for becoming a chess grandmaster, but it’s not sufficient on its own (by a long shot).
      - c_crank 3y ago
        The power of a politician to persuade comes in part, in my opinion, from their ability to convince a voter that they represent the voter's interest. Some of this is done just through words, but a lot of it has to be represented via some kind of action, or else the voter will realize they're being fooled. I don't see a superintelligent AI gaining much ground on this factor.
civilized 3y ago
I just opened it up and found a fun bit during Melanie Mitchell's opener: https://youtu.be/144uOfr4SYA?t=2116 https://youtu.be/144uOfr4SYA?t=2116 Paraphrasing: > Yoshua Bengio has written about a scenario where an AI is asked to solve climate change, and it designs a virus that decimates humans. Presto. No more humans, no more climate change. > This is an example of the fallacy of dumb superintelligence. It's a fallacy to think that a machine could be smarter than humans in all respects and still lack all common-sense understanding of humans, such as understanding why we made the request and that we prefer not to be wiped out... We would never give unchecked autonomy and resources to an AI that lacked these basic aspects of intelligence - it just does not make sense.
- victroflux 3y ago
  Is this supposed to be a strong rebuttal ? How does this invalidate the alignment problem? I'd really like to understand. Humans, via climate change and environmental destruction, have wiped out, many, many species. To those species we've killed, we're the super intelligence, we understand their plight, we even have strong empathy, but we prioritize our own goals over that of the helpless and the stupider creaters. How she can't see that same pattern could potentially repeat, I really, really don't know.
  - civilized 3y ago
    I don't think her point disproves all possible AI doom scenarios, but nothing can. They're too vague. It's all handwaving along the lines of "a malicious superintelligence will emerge without warning and somehow seize absolute power, and it's pointless to argue about how plausible any of that is, because sufficient intelligence can do anything in ways we can't imagine." This is why the biggest boosters of AI doom conjure absurd scenarios like AI secretly manufacturing a global swarm of insta-kill nanobots. They don't care about the details, they just think that sufficient intelligence will emerge without warning, be intractably malicious, and capable of anything by gestures vigorously methods. AI doom critics are asking that we consider more soberly the actual probabilities of all these things happening, and on what time scale, and most importantly by what specific pathway. The analogy to humans and animals actually illustrates the point. Humans didn't come to dominate the planet by intelligence and handwaving. It was an extremely slow process involving the interaction of many factors and contingent opportunities. Intelligence enabled language, tools, and eventually animal domestication and agriculture, all over hundreds of thousands of years. It wasn't like humans were destined for dominance because intelligence is a skeleton key to all material resources by any number of arbitrary, incomprehensible pathways.