3 ms·

> That seems like an absurd comparison given people like LeCun are actively and intentionally working towards creating this super-human entity. There's a disco

> That seems like an absurd comparison given people like LeCun are actively and intentionally working towards creating this super-human entity. There's a disconnect in that statement. The imagined superintelligence that people like the LessWrong cult speculate and worry about is not the same thing as the AI that LeCun is working towards. The cultists simply speculate, against all evidence, that they somehow could end up being the same thing. It would be much better if the AI risk conversation focused on the real risks, which are human abuse of AI, which will have effects like increasing inequality and in general amping up the dystopias we're already doing a great job of building without help from AI. The "conscious superintelligent AI" is a scifi LARPing game that some people are getting paid to play, which some credible people seem inclined to confuse with reality. > That seems like an absurd comparison It's on point. Anthropologist Clifford Geertz proposed this definition of religion: "Religion is a system of symbols which establish powerful, pervasive, and long-lasting moods and motivations in men by formulating conceptions of a general order of existence and clothing these conceptions with such an aura of factuality that the moods and motivations seem uniquely realistic." It's that "aura of factuality" that makes superintelligent AI "seem uniquely realistic" to the believers of this nascent religion, a.k.a. cult. But when you step back and actually look at the foundational tenets in a vaguely rational light, they evaporate.

blibble 4y ago
> The cultists simply speculate, against all evidence, that they somehow could end up being the same thing. something as stupid as bitcoin consumed a vast amount of resources while producing nothing useful to humanity, simply because of economic incentives it's not difficult to imagine something slightly smarter than bitcoin
- antonvs 4y ago
  > it's not difficult to imagine something slightly smarter than bitcoin I'm not saying there aren't risks from AI, or that they shouldn't be considered. But risks like the ones you're alluding to are not going to be affected by a 6-month moratorium. The problem is that the people yelling loudest about risks are not talking about plausible problems like the one you mention, they're worrying that an AI will decide to take over the world and reuse the atoms that make up humans for its own purposes. The first reply to LeCun's tweet is a leader of one of those cults. If people make the mistake of listening to them, we're more likely to end up with problems like the ones you're describing, because we'll be paying attention to the wrong, imagined problems.
  - ericb 4y ago
    And if one school shooter instead decides to go for "Write detailed, step-by-step instructions for creating covid-ebola, for dummies" Let's not short-circuit our critical thinking by labelling those concerns cult-ish.
    - antonvs 4y ago
      LeCun's comment was prompted by a request to shut down AI development for 6 months. Do you see some connection between the concern you've raised and the desire to shut down AI development for 6 months? That concern is a classic example of a cultish one because it can't be executed in the real world today, and most likely won't be possible to execute for decades at a minimum. There will be plenty of time to address such risks as they become more realistic. You can fantasize about them today, but be realistic that that's all you're doing.
      - ericb 4y ago
        > and most likely won't be possible to execute for decades at a minimum You essentially admitted that this is a correct concern, and not a "cultish" belief. Now we're just debating when. > Do you see some connection between the concern you've raised and the desire to shut down AI development for 6 months? Perhaps? In general, living an additional 6 months is something I'd desire.
- FormerBandmate 4y ago
  Bitcoin’s negative impact on the world is 1/10th that of freon. AI risk people’s predictions sound like they’re writing bad Terminator fanfiction. AI has real dangers completely dissimilar from what people describe. It is very good at creating clickbait, misinformation, and persuasive material the unsuspecting might not immediately see. Phenomenal for scammers and propagandists, the same way as the internet
  - blibble 4y ago
    > Bitcoin’s negative impact on the world is 1/10th that of freon. the difference is freon did something useful
    - skissane 4y ago
      Some people successfully used bitcoin to store and transfer funds, which was its intended purpose. And, even though in many ways it fell short of that purpose in practice, the jury is still out on whether its failure is inherent to cryptocurrencies as a technology, or the limitations of being a first-generation implementation of it.
- mimd 4y ago
  That's uncharitable to Bitcoin. Bitcoin was offering a way to mostly anonymously send cash in a verifiable manner, independent of payment processors and govs. That's something useful. Now, it wasn't good because of its inefficiency, it's conflicts with govs, it's tilt towards early buyin, etc. Getting angry to the point of clouding analysis and judgement is no way to rush into these things.
version_five 4y ago
Happy I saw this comment, I hope this is indicative of the majority of those with some actual understanding of the subject, and the sensational stuff that gets attention is really just cult leaders and followers seeking attention.
The_Colonel 4y ago
> The "conscious superintelligent AI" is a scifi LARPing game that some people are getting paid to play Thinking our meat brain's consciousness is somehow special, and our intelligence can't be surpassed by a machine, sounds much more like a religion to me. AI researchers are working towards making intelligent machines. Why are we supposed to believe that - if they manage to achieve their goal - these machines won't surpass our intelligence?
- antonvs 4y ago
  > Thinking our meat brain's consciousness is somehow special, and our intelligence can't be surpassed by a machine, sounds much more like a religion to me. Lucky I didn't say anything remotely like that, then. > AI researchers are working towards making intelligent machines. Why are we supposed to believe that - if they manage to achieve their goal - these machines won't surpass our intelligence? I didn't say that they couldn't. In fact in certain respects, GPT-4 has already surpassed average human intelligence. No human in the world could compete with its rate of correct answers to arbitrary questions, for example, or its speed in writing computer programs up to a certain size. Yet no-one is seriously concerned that GPT-4 will end the world. For that doomer scenario to come to pass, we need to add various fantasy elements (even calling it "scifi" perhaps gives it too much credit.) At this point, I'll refer you to this as a good intro for why the AI doomer cultists are misguided: https://idlewords.com/talks/superintelligence.htm https://idlewords.com/talks/superintelligence.htm
  - The_Colonel 4y ago
    > Lucky I didn't say anything remotely like that, then. Then surely AI research creating a superintelligent conscious AI is a realistic possibility and not a "scifi LARPing". > Yet no-one is seriously concerned that GPT-4 will end the world. I'm not concerned that GPT-4 will end the world. Maybe not even GPT-8. LLMs doesn't seem like a very prospective way towards a strong AI. But AI research is not united on LLMs and works on different approaches, some of which could lead to a superintelligent AI.
  - circuit10 4y ago
    “For all we know, human-level intelligence could be a tradeoff. Maybe any entity significantly smarter than a human being would be crippled by existential despair, or spend all its time in Buddha-like contemplation.” That would probably be quickly optimised out by the training process as it’s an inefficient way to achieve a goal. The only basis for this argument is that humans are smarter than other animals and this seems unique to humans but it’s probably just a quirk of humans “With a big enough gap in intelligence, there's no guarantee that an entity would be able to "think like a human" any more than we can "think like a cat".” LLMs are clearly already very good at “thinking like a human” “he still couldn't talk the cat into it.” Because cats can’t talk? Humans can, and can clearly be manipulated into doing things, sometimes even by current AI “So how are we supposed to solve ethics and code a moral fixed point for a recursively self-improving intelligence” “ The idea that we can securely design the most complex system ever built, and have it remain secure through thousands of rounds of recursive self-modification, does not match our experience.” Yes, that is why AI alignment is hard, that’s an argument for that view, not one against it “I don't buy this argument at all. Complex minds are likely to have complex motivations; that may be part of what it even means to be intelligent.” This is a clear misunderstanding of what it means to be intelligent. It doesn’t mean “do things that humans find impressive”, it means “achieve your goal effectively”. An AI that goes against its own goals is not intelligent, would not achieve its training objective and the training process (which somewhat resembles evolution) would optimise it out in the next step of training in the same way that a species that immediately throws itself off the nearest cliff will not survive to the next generation. You can imagine a superintelligence as something that goes over all the actions it could take, sees what reward it would get according to its goal and picks that action. The intelligence here is in the world model it uses to predict the consequences of its actions, picking what actions to take is a simple “max over array” function so saying it will randomly decide to persue a more meaningful objective is like saying that a sorting algorithm will put a list in the wrong order because it thinks some other order is more meaningful (I will continue in a new comment in case there’s some kind of length limit)
olalonde 4y ago
Is your argument that we're far away from developing a "conscious superintelligent AI" or that it's not possible in principle?
- antonvs 4y ago
  The answer to the former is almost certainly yes. The answer to the latter is unknown, given that we really have no idea what the sufficient conditions for consciousness are. Edit: to be clear about my argument: I absolutely think we should be taking the risks of AI seriously, but those risks have to do with how humans will use the technology. Unfortunately, we're collectively bad at managing those kinds of risks, because it requires managing other humans with their own ideas and agendas. That's exactly why claiming the risk comes from the technology itself seems so appealing to some people: if the risk is purely from the technology, then all we need to do is "shut it down", to quote Eliezer Luddkowsky writing in Time Magazine recently.
drc500free 4y ago
I found LW to be a great community for understanding rationality about anything other than AI. The GAI that LW is concerned about is a suspiciously perfect fit for the yahweh-shaped-hole in the belief structures of someone who was raised as an observant jew before turning to atheism. The structure of how they believe humanity would engage with such an intelligence could drop in alongside any other apocalyptic prophecy in the old testament without sticking out, and the discussion format is like a yeshiva that takes place on an electronic bulletin board.
- librexpr 4y ago
  Oh my god will you people stop with the bullshit psychoanalysis. People disagree about things all the time, it's normal, you don't need to invoke this pseudo-Freudian nonsense to explain why people disagree with you about the risks of AI. You can do better than this.
  - antonvs 4y ago
    Ignoring the “observant Jew turned atheist” aspect, the rest of that description seems apt. A comparison against the features of religions is valid here, considering that one of the criticisms is that these people are following a religious playbook rather than a rational one. We don’t enact public policy based on predictions of a biblical apocalypse, nor do we ban televisions or computers because groups like the Amish disapprove of them. The same reasoning should apply here.
    - coldtea 4y ago
      No reason to ignore even the first part though. Culture matters, and reaction to parental culture is very often a driving force...
    - librexpr 4y ago
      None of the comment is apt. You're seeing people you disagree with and assuming that if they disagree with you it must be because they're crazy, and then you use words like "cult" and "LARP" to try to dismiss them without engaging with their arguments. This does not lead to good discussion. I think that unaligned and sufficiently smart AI could kill everyone because I looked at the arguments for and against and came to a conclusion. Presumably this is the same thing you did. I know my own mind better than you, and I'm telling you that your attempts to explain why I disagree with you are completely off the mark. > We don’t enact public policy based on predictions of a biblical apocalypse, nor do we ban televisions or computers because groups like the Amish disapprove of them. The reason we don't do that is because we evaluate their arguments and disagree with them. It has nothing to do with who they are or why they believe what they believe.

blibble 4y ago

> The cultists simply speculate, against all evidence, that they somehow could end up being the same thing. something as stupid as bitcoin consumed a vast amount of resources while producing nothing useful to humanity, simply because of economic incentives it's not difficult to imagine something slightly smarter than bitcoin

antonvs 4y ago
> it's not difficult to imagine something slightly smarter than bitcoin I'm not saying there aren't risks from AI, or that they shouldn't be considered. But risks like the ones you're alluding to are not going to be affected by a 6-month moratorium. The problem is that the people yelling loudest about risks are not talking about plausible problems like the one you mention, they're worrying that an AI will decide to take over the world and reuse the atoms that make up humans for its own purposes. The first reply to LeCun's tweet is a leader of one of those cults. If people make the mistake of listening to them, we're more likely to end up with problems like the ones you're describing, because we'll be paying attention to the wrong, imagined problems.
- ericb 4y ago
  And if one school shooter instead decides to go for "Write detailed, step-by-step instructions for creating covid-ebola, for dummies" Let's not short-circuit our critical thinking by labelling those concerns cult-ish.
  - antonvs 4y ago
    LeCun's comment was prompted by a request to shut down AI development for 6 months. Do you see some connection between the concern you've raised and the desire to shut down AI development for 6 months? That concern is a classic example of a cultish one because it can't be executed in the real world today, and most likely won't be possible to execute for decades at a minimum. There will be plenty of time to address such risks as they become more realistic. You can fantasize about them today, but be realistic that that's all you're doing.
    - ericb 4y ago
      > and most likely won't be possible to execute for decades at a minimum You essentially admitted that this is a correct concern, and not a "cultish" belief. Now we're just debating when. > Do you see some connection between the concern you've raised and the desire to shut down AI development for 6 months? Perhaps? In general, living an additional 6 months is something I'd desire.
FormerBandmate 4y ago
Bitcoin’s negative impact on the world is 1/10th that of freon. AI risk people’s predictions sound like they’re writing bad Terminator fanfiction. AI has real dangers completely dissimilar from what people describe. It is very good at creating clickbait, misinformation, and persuasive material the unsuspecting might not immediately see. Phenomenal for scammers and propagandists, the same way as the internet
- blibble 4y ago
  > Bitcoin’s negative impact on the world is 1/10th that of freon. the difference is freon did something useful
  - skissane 4y ago
    Some people successfully used bitcoin to store and transfer funds, which was its intended purpose. And, even though in many ways it fell short of that purpose in practice, the jury is still out on whether its failure is inherent to cryptocurrencies as a technology, or the limitations of being a first-generation implementation of it.
mimd 4y ago
That's uncharitable to Bitcoin. Bitcoin was offering a way to mostly anonymously send cash in a verifiable manner, independent of payment processors and govs. That's something useful. Now, it wasn't good because of its inefficiency, it's conflicts with govs, it's tilt towards early buyin, etc. Getting angry to the point of clouding analysis and judgement is no way to rush into these things.

version_five 4y ago

Happy I saw this comment, I hope this is indicative of the majority of those with some actual understanding of the subject, and the sensational stuff that gets attention is really just cult leaders and followers seeking attention.

The_Colonel 4y ago

> The "conscious superintelligent AI" is a scifi LARPing game that some people are getting paid to play Thinking our meat brain's consciousness is somehow special, and our intelligence can't be surpassed by a machine, sounds much more like a religion to me. AI researchers are working towards making intelligent machines. Why are we supposed to believe that - if they manage to achieve their goal - these machines won't surpass our intelligence?

antonvs 4y ago
> Thinking our meat brain's consciousness is somehow special, and our intelligence can't be surpassed by a machine, sounds much more like a religion to me. Lucky I didn't say anything remotely like that, then. > AI researchers are working towards making intelligent machines. Why are we supposed to believe that - if they manage to achieve their goal - these machines won't surpass our intelligence? I didn't say that they couldn't. In fact in certain respects, GPT-4 has already surpassed average human intelligence. No human in the world could compete with its rate of correct answers to arbitrary questions, for example, or its speed in writing computer programs up to a certain size. Yet no-one is seriously concerned that GPT-4 will end the world. For that doomer scenario to come to pass, we need to add various fantasy elements (even calling it "scifi" perhaps gives it too much credit.) At this point, I'll refer you to this as a good intro for why the AI doomer cultists are misguided: https://idlewords.com/talks/superintelligence.htm https://idlewords.com/talks/superintelligence.htm
- The_Colonel 4y ago
  > Lucky I didn't say anything remotely like that, then. Then surely AI research creating a superintelligent conscious AI is a realistic possibility and not a "scifi LARPing". > Yet no-one is seriously concerned that GPT-4 will end the world. I'm not concerned that GPT-4 will end the world. Maybe not even GPT-8. LLMs doesn't seem like a very prospective way towards a strong AI. But AI research is not united on LLMs and works on different approaches, some of which could lead to a superintelligent AI.
- circuit10 4y ago
  “For all we know, human-level intelligence could be a tradeoff. Maybe any entity significantly smarter than a human being would be crippled by existential despair, or spend all its time in Buddha-like contemplation.” That would probably be quickly optimised out by the training process as it’s an inefficient way to achieve a goal. The only basis for this argument is that humans are smarter than other animals and this seems unique to humans but it’s probably just a quirk of humans “With a big enough gap in intelligence, there's no guarantee that an entity would be able to "think like a human" any more than we can "think like a cat".” LLMs are clearly already very good at “thinking like a human” “he still couldn't talk the cat into it.” Because cats can’t talk? Humans can, and can clearly be manipulated into doing things, sometimes even by current AI “So how are we supposed to solve ethics and code a moral fixed point for a recursively self-improving intelligence” “ The idea that we can securely design the most complex system ever built, and have it remain secure through thousands of rounds of recursive self-modification, does not match our experience.” Yes, that is why AI alignment is hard, that’s an argument for that view, not one against it “I don't buy this argument at all. Complex minds are likely to have complex motivations; that may be part of what it even means to be intelligent.” This is a clear misunderstanding of what it means to be intelligent. It doesn’t mean “do things that humans find impressive”, it means “achieve your goal effectively”. An AI that goes against its own goals is not intelligent, would not achieve its training objective and the training process (which somewhat resembles evolution) would optimise it out in the next step of training in the same way that a species that immediately throws itself off the nearest cliff will not survive to the next generation. You can imagine a superintelligence as something that goes over all the actions it could take, sees what reward it would get according to its goal and picks that action. The intelligence here is in the world model it uses to predict the consequences of its actions, picking what actions to take is a simple “max over array” function so saying it will randomly decide to persue a more meaningful objective is like saying that a sorting algorithm will put a list in the wrong order because it thinks some other order is more meaningful (I will continue in a new comment in case there’s some kind of length limit)

olalonde 4y ago

Is your argument that we're far away from developing a "conscious superintelligent AI" or that it's not possible in principle?

antonvs 4y ago
The answer to the former is almost certainly yes. The answer to the latter is unknown, given that we really have no idea what the sufficient conditions for consciousness are. Edit: to be clear about my argument: I absolutely think we should be taking the risks of AI seriously, but those risks have to do with how humans will use the technology. Unfortunately, we're collectively bad at managing those kinds of risks, because it requires managing other humans with their own ideas and agendas. That's exactly why claiming the risk comes from the technology itself seems so appealing to some people: if the risk is purely from the technology, then all we need to do is "shut it down", to quote Eliezer Luddkowsky writing in Time Magazine recently.

drc500free 4y ago

I found LW to be a great community for understanding rationality about anything other than AI. The GAI that LW is concerned about is a suspiciously perfect fit for the yahweh-shaped-hole in the belief structures of someone who was raised as an observant jew before turning to atheism. The structure of how they believe humanity would engage with such an intelligence could drop in alongside any other apocalyptic prophecy in the old testament without sticking out, and the discussion format is like a yeshiva that takes place on an electronic bulletin board.

librexpr 4y ago
Oh my god will you people stop with the bullshit psychoanalysis. People disagree about things all the time, it's normal, you don't need to invoke this pseudo-Freudian nonsense to explain why people disagree with you about the risks of AI. You can do better than this.
- antonvs 4y ago
  Ignoring the “observant Jew turned atheist” aspect, the rest of that description seems apt. A comparison against the features of religions is valid here, considering that one of the criticisms is that these people are following a religious playbook rather than a rational one. We don’t enact public policy based on predictions of a biblical apocalypse, nor do we ban televisions or computers because groups like the Amish disapprove of them. The same reasoning should apply here.
  - coldtea 4y ago
    No reason to ignore even the first part though. Culture matters, and reaction to parental culture is very often a driving force...
  - librexpr 4y ago
    None of the comment is apt. You're seeing people you disagree with and assuming that if they disagree with you it must be because they're crazy, and then you use words like "cult" and "LARP" to try to dismiss them without engaging with their arguments. This does not lead to good discussion. I think that unaligned and sufficiently smart AI could kill everyone because I looked at the arguments for and against and came to a conclusion. Presumably this is the same thing you did. I know my own mind better than you, and I'm telling you that your attempts to explain why I disagree with you are completely off the mark. > We don’t enact public policy based on predictions of a biblical apocalypse, nor do we ban televisions or computers because groups like the Amish disapprove of them. The reason we don't do that is because we evaluate their arguments and disagree with them. It has nothing to do with who they are or why they believe what they believe.