5 ms·

AI Companies and Advocates Are Becoming More Cult-Like

44 points by legrande 3y ago

LiquidSky 3y ago
Becoming?
- lainga 3y ago
  SV Pivots from Bunkers in Nauru to Calling Enemies Murderers I mean, really? "effective accelerationism"?
  - 15457345234 3y ago
    They're effectively accelerating how quickly people recognise that they're loons, which I think is a huge plus.
rightbyte 3y ago
“Deaths that were preventable by the AI that was prevented from existing is a form of murder.” Is he trying to dehumanize critics?
- floren 3y ago
  Give me money or you're a murderer.
- JohnFen 3y ago
  It's just a pure emotional manipulation attempt. The statement itself is illogical on its face, and presents the tech as having only upsides. It ignores the meaning of deaths caused by AI. Any time I see appeals like this, my takeaway is that the person has no actual argument.
  - rightbyte 3y ago
    Ye. I mean, I saw now that in another quote he seems to advocate using AI in war to "save lives". I don't think his reasoning holds up for any inspection.
- eli_gottlieb 3y ago
  Well yeah, it's just plain moral blackmail. If it wasn't so common in the discourse I'd say he should actually be stopped from doing it.
- add-sub-mul-div 3y ago
  I have murdered so many people by not becoming a doctor.
- AnimalMuppet 3y ago
  Two can play that game. "Taking all this money you're wasting on AI and not giving it to buy mosquito nets is a form of murder."
- deleted 3y ago
  [deleted]
EA-3167 3y ago
Without an affective death spiral, how can an overpromised, overfunded technology keep a dominant market position? Think of the shareholders!
lumost 3y ago
One emerging challenge with AI is that we are not seeing strong adoption outside of Chat applications. While every company has pivoted to trying to leverage GenAI - applications outside of OpenAI have ... mixed adoption. This is not dissimilar to mobile in 2008, or cloud circa 2006. It's ok that new applications take time to emerge, but the valuation of firms implies that these technologies have already emerged. Perhaps this is fine for core AI development, but placing these valuations on SaaS firms could be problematic in the future.
- floren 3y ago
  Driving 101 through the peninsula, it's shocking how many billboards are advertising AI <something>. I think the article correctly diagnoses it as a desperate attempt to tap VC money that's getting scarcer and scarcer.
- linkjuice4all 3y ago
  When Siri and Alexa first came out there was another chatbot gold rush but it obviously didn’t work out for a number of reasons. LLMs and other recent (in the last year) advances have certainly improved the understanding and response for chat agents so the conversation aspect has gotten better (fewer dead ends or “I don’t understand”) - but these newer agents still can’t really “do” anything e.g. give a refund. There’s still the dead end when you need an actual human to do something or approve the chatbot’s recommended action.
  - lumost 3y ago
    Aye - this would form the counter-point to the revolutionary thesis. ChatGPT might be the LLM product, and everything else is small potatoes. This would still leave a 30-100 billion dollar in revenue industry - but that’s about the size where the current combined Nvidia/MSFT/Openai valuation has no room to grow.
kaycebasques 3y ago
I was expecting more analysis of specific cult-like behavior, which I agree there is quite a lot of. After a close call with a borderline cult in the 2010s I spent a fair bit of time processing that experience and reading up on cult dynamics. Some of the stuff OpenAI employees were saying on Twitter during the weekend crisis raised a lot of yellow flags for me. We should not fool ourselves into thinking that cult-like behavior is isolated to AI, though. It often pops up alongside new tech. On that note this is a pretty remarkable quote: > “If your product isn’t amenable to spontaneously producing a cult, it’s probably not impactful enough.”
- EA-3167 3y ago
  I'm curious, given your experiences, if you've read 'Terror, Love and Brainwashing' by Stein? And if so, what did you think of it?
  - kaycebasques 3y ago
    Have not read it, will check it out
- aredox 3y ago
  Is there a cult of the wheel? Of penicillin? Of the Haber-Bosch process? A tool is just a tool. If one turns it into an Idol, they have a giant void in your life. They should seek therapy. Those AI doomsayers or zealots are all dull people, who missed on life overworking, have more money than sense, and can't accept not being relevant all the time.
RGamma 3y ago
People are scary :/
fjoireoipe 3y ago
Can we finally talk about this site, and how it relates to these cults? There's a number of lesswrong, e/acc, and other pseudo "rationalist" blogs that get shared and upvoted on this site. Most of their assumptions go unchallenged. Not saying they shouldn't be read, or debated. But their writing should be viewed in context - it's fringe stuff, written by people from a peculiar subculture with values out-of-whack with most people. I'd like to make some modest assertions, to push back on fringe ideas I've seen repeated here: 1. For "the singularity" to happen, we probably need something more to happen than just chatGPT to ingest more data or use more processing power. 2. Even if, somehow, chatGPT turned into skynet, we'd hopefully be able to unplug the computer. 3. If you want to save lives, it's probably more useful to think about lives saved today than hypothetical lives you could save 100 years from now. Not that you shouldn't consider the long term, but your ability to predict the future gets worse and worse the farther you project out. 4. If you want to save lives, it's probably more useful to save actual lives, than say, hypothetical simulated lives that exist inside of a computer. 5. The argument that "we're killing more people by delaying time inventing the hypothetical life saving technology" is not very useful either, because you can't actually say how many lives would be saved versus harmed. And mostly it just sounds like a front for "give me more money and less regulations". 6. Reading a bunch of science fiction and getting into fights on an internet forum is not a substitute for education and experience. Unless you've spent a good time studying viruses, you are not a virologist, and while the consensus among virologists can be wrong, you should have the intellectual humility to realize that you are probably not equipped to tell, unless you have expertise in the field. 7. Anything that smacks of eugenics is most likely pseudoscience. 8. If someone talks like a racist / sexist / nazi, or acts like a racist / sexist / nazi, they probably are one. It's probably not a joke, or a test.
- AnimalMuppet 3y ago
  Well, this site often deals with fringe stuff. Take Lisp, for example. It is a fringe language, no matter how zealous its advocates are. So is Haskell. But you see articles about them posted here far out of proportion to their real-world use and impact. And I disagree that "most of their assumptions go unchallenged". I see challenges to the assumptions of lesswrong, e/acc, and other such stuff often here. (Maybe less often than is warranted, but still fairly often.) "It is the mark of an educated mind to be able to entertain a thought without accepting it." - often attributed to Aristotle, but apparently not actually from him. Still a good thought. We can post about this stuff, and discuss it, without buying it. I agree with your pushbacks.
frogamel 3y ago
I've worked in many research scientist/MLE roles over the years and I haven't met any IC who has this much of a fixation on AI being a moral evil/good. The ones who do are inevitably nontechnical hucksters, usually just trying to get money or self-promote.
- PeterStuer 3y ago
  Or trying to abuse regulation to digg a mote and draw up the bridge behind them.
ChrisArchitect 3y ago
"Related" rollingstone, from 3 years ago: Welcome to the Church of Bitcoin https://news.ycombinator.com/item?id=27871038 https://news.ycombinator.com/item?id=27871038
anonreeeeplor 3y ago
Part of it is ultra concentration of white men with few or no women in the Bay Area imo. If you have zero social life and zero chance of a social life and the aperture of available activities is purely constrained to ultra pure social media groups of similar white men and you never interact with women individually or as a group … Society is generally breaking down and falling apart at that point. Normally I would expect men to moderate their behavior if there are women going: “eghhhhhh idk that is tooooooo far guys you need to chill out.” That isn’t happening anymore, and hasn’t been happening for a while. So now you are getting a sort of ultra distilled white male in group effect. Women civilize men, generally, and that hasn’t been happening for years. You are going to get cults and they are going to get more crazy. Things like imagining a heaven where you get 77 virgins (Muslim religion) are byproducts of highly stressed out societies where men have next to zero prospects. It’s not good. The media isn’t going to fix it tho. My life personally post covid has declined considerably in the social domain. My circle has gotten more and more narrow and I feel more and more weird and isolated. I don’t want to be weird and isolated. I would rather have friends and a community. So what do I do? I actually go to church now. That’s literally my only option and I’m not even that religious. Remote work and the isolation has put incredible stress on me as an animal. I don’t like this.
- 15457345234 3y ago
  You make a good overall point here but you've really compromised it by saying 'white men' because: 1. That particular phrase has been massively overused in a disparaging way by 'wokists' and a lot of people are going to see those words and just assume you're 'one of them' and ignore you and 2. The SV AI set isn't particularly white.
  - Nasrudith 3y ago
    Of course they should ignore them. It is literally pure sexism and racism.
  - anonreeeeplor 3y ago
    I have never seen a group of Asians organizing an AI cult. All the OpenAI people doing cults are white. Have you seen any Indian or Asian people or women building a cult?
kopirgan 3y ago
I remember listening to a panel discussing block chains in a user group meeting of a software company, decade or so ago. Mind you, this was not some cutting edge technology company it was boring accounting stuff. All panelists had zero clue and were talking just general vague terms. Of course not giving too much, being politically correct etc put additional constraints over what you can say in such meetings. So whatever evangelical zeal you see in these tech shows are not worth the time. I think most people know. They just play their role, nodding heads waiting for next coffee break. But it's a very interesting article that will sure stir a lot of thought.. Usual high standards of this magazine.
dventimi 3y ago
https://archive.ph/2024.01.29-165444/https://www.rollingstone.com/culture/culture-features/ai-companies-advocates-cult-1234954528/ https://archive.ph/2024.01.29-165444/https://www.rollingston...

44 points by legrande 3y ago

LiquidSky 3y ago

Becoming?

lainga 3y ago
SV Pivots from Bunkers in Nauru to Calling Enemies Murderers I mean, really? "effective accelerationism"?
- 15457345234 3y ago
  They're effectively accelerating how quickly people recognise that they're loons, which I think is a huge plus.

rightbyte 3y ago

“Deaths that were preventable by the AI that was prevented from existing is a form of murder.” Is he trying to dehumanize critics?

floren 3y ago
Give me money or you're a murderer.
JohnFen 3y ago
It's just a pure emotional manipulation attempt. The statement itself is illogical on its face, and presents the tech as having only upsides. It ignores the meaning of deaths caused by AI. Any time I see appeals like this, my takeaway is that the person has no actual argument.
- rightbyte 3y ago
  Ye. I mean, I saw now that in another quote he seems to advocate using AI in war to "save lives". I don't think his reasoning holds up for any inspection.
eli_gottlieb 3y ago
Well yeah, it's just plain moral blackmail. If it wasn't so common in the discourse I'd say he should actually be stopped from doing it.
add-sub-mul-div 3y ago
I have murdered so many people by not becoming a doctor.
AnimalMuppet 3y ago
Two can play that game. "Taking all this money you're wasting on AI and not giving it to buy mosquito nets is a form of murder."
deleted 3y ago
[deleted]

EA-3167 3y ago

Without an affective death spiral, how can an overpromised, overfunded technology keep a dominant market position? Think of the shareholders!

lumost 3y ago

One emerging challenge with AI is that we are not seeing strong adoption outside of Chat applications. While every company has pivoted to trying to leverage GenAI - applications outside of OpenAI have ... mixed adoption. This is not dissimilar to mobile in 2008, or cloud circa 2006. It's ok that new applications take time to emerge, but the valuation of firms implies that these technologies have already emerged. Perhaps this is fine for core AI development, but placing these valuations on SaaS firms could be problematic in the future.

floren 3y ago
Driving 101 through the peninsula, it's shocking how many billboards are advertising AI <something>. I think the article correctly diagnoses it as a desperate attempt to tap VC money that's getting scarcer and scarcer.
linkjuice4all 3y ago
When Siri and Alexa first came out there was another chatbot gold rush but it obviously didn’t work out for a number of reasons. LLMs and other recent (in the last year) advances have certainly improved the understanding and response for chat agents so the conversation aspect has gotten better (fewer dead ends or “I don’t understand”) - but these newer agents still can’t really “do” anything e.g. give a refund. There’s still the dead end when you need an actual human to do something or approve the chatbot’s recommended action.
- lumost 3y ago
  Aye - this would form the counter-point to the revolutionary thesis. ChatGPT might be the LLM product, and everything else is small potatoes. This would still leave a 30-100 billion dollar in revenue industry - but that’s about the size where the current combined Nvidia/MSFT/Openai valuation has no room to grow.

kaycebasques 3y ago

I was expecting more analysis of specific cult-like behavior, which I agree there is quite a lot of. After a close call with a borderline cult in the 2010s I spent a fair bit of time processing that experience and reading up on cult dynamics. Some of the stuff OpenAI employees were saying on Twitter during the weekend crisis raised a lot of yellow flags for me. We should not fool ourselves into thinking that cult-like behavior is isolated to AI, though. It often pops up alongside new tech. On that note this is a pretty remarkable quote: > “If your product isn’t amenable to spontaneously producing a cult, it’s probably not impactful enough.”

EA-3167 3y ago
I'm curious, given your experiences, if you've read 'Terror, Love and Brainwashing' by Stein? And if so, what did you think of it?
- kaycebasques 3y ago
  Have not read it, will check it out
aredox 3y ago
Is there a cult of the wheel? Of penicillin? Of the Haber-Bosch process? A tool is just a tool. If one turns it into an Idol, they have a giant void in your life. They should seek therapy. Those AI doomsayers or zealots are all dull people, who missed on life overworking, have more money than sense, and can't accept not being relevant all the time.

RGamma 3y ago

People are scary :/

fjoireoipe 3y ago

Can we finally talk about this site, and how it relates to these cults? There's a number of lesswrong, e/acc, and other pseudo "rationalist" blogs that get shared and upvoted on this site. Most of their assumptions go unchallenged. Not saying they shouldn't be read, or debated. But their writing should be viewed in context - it's fringe stuff, written by people from a peculiar subculture with values out-of-whack with most people. I'd like to make some modest assertions, to push back on fringe ideas I've seen repeated here: 1. For "the singularity" to happen, we probably need something more to happen than just chatGPT to ingest more data or use more processing power. 2. Even if, somehow, chatGPT turned into skynet, we'd hopefully be able to unplug the computer. 3. If you want to save lives, it's probably more useful to think about lives saved today than hypothetical lives you could save 100 years from now. Not that you shouldn't consider the long term, but your ability to predict the future gets worse and worse the farther you project out. 4. If you want to save lives, it's probably more useful to save actual lives, than say, hypothetical simulated lives that exist inside of a computer. 5. The argument that "we're killing more people by delaying time inventing the hypothetical life saving technology" is not very useful either, because you can't actually say how many lives would be saved versus harmed. And mostly it just sounds like a front for "give me more money and less regulations". 6. Reading a bunch of science fiction and getting into fights on an internet forum is not a substitute for education and experience. Unless you've spent a good time studying viruses, you are not a virologist, and while the consensus among virologists can be wrong, you should have the intellectual humility to realize that you are probably not equipped to tell, unless you have expertise in the field. 7. Anything that smacks of eugenics is most likely pseudoscience. 8. If someone talks like a racist / sexist / nazi, or acts like a racist / sexist / nazi, they probably are one. It's probably not a joke, or a test.

AnimalMuppet 3y ago
Well, this site often deals with fringe stuff. Take Lisp, for example. It is a fringe language, no matter how zealous its advocates are. So is Haskell. But you see articles about them posted here far out of proportion to their real-world use and impact. And I disagree that "most of their assumptions go unchallenged". I see challenges to the assumptions of lesswrong, e/acc, and other such stuff often here. (Maybe less often than is warranted, but still fairly often.) "It is the mark of an educated mind to be able to entertain a thought without accepting it." - often attributed to Aristotle, but apparently not actually from him. Still a good thought. We can post about this stuff, and discuss it, without buying it. I agree with your pushbacks.

frogamel 3y ago

I've worked in many research scientist/MLE roles over the years and I haven't met any IC who has this much of a fixation on AI being a moral evil/good. The ones who do are inevitably nontechnical hucksters, usually just trying to get money or self-promote.

PeterStuer 3y ago
Or trying to abuse regulation to digg a mote and draw up the bridge behind them.

ChrisArchitect 3y ago

"Related" rollingstone, from 3 years ago: Welcome to the Church of Bitcoin https://news.ycombinator.com/item?id=27871038 https://news.ycombinator.com/item?id=27871038

anonreeeeplor 3y ago

Part of it is ultra concentration of white men with few or no women in the Bay Area imo. If you have zero social life and zero chance of a social life and the aperture of available activities is purely constrained to ultra pure social media groups of similar white men and you never interact with women individually or as a group … Society is generally breaking down and falling apart at that point. Normally I would expect men to moderate their behavior if there are women going: “eghhhhhh idk that is tooooooo far guys you need to chill out.” That isn’t happening anymore, and hasn’t been happening for a while. So now you are getting a sort of ultra distilled white male in group effect. Women civilize men, generally, and that hasn’t been happening for years. You are going to get cults and they are going to get more crazy. Things like imagining a heaven where you get 77 virgins (Muslim religion) are byproducts of highly stressed out societies where men have next to zero prospects. It’s not good. The media isn’t going to fix it tho. My life personally post covid has declined considerably in the social domain. My circle has gotten more and more narrow and I feel more and more weird and isolated. I don’t want to be weird and isolated. I would rather have friends and a community. So what do I do? I actually go to church now. That’s literally my only option and I’m not even that religious. Remote work and the isolation has put incredible stress on me as an animal. I don’t like this.

15457345234 3y ago
You make a good overall point here but you've really compromised it by saying 'white men' because: 1. That particular phrase has been massively overused in a disparaging way by 'wokists' and a lot of people are going to see those words and just assume you're 'one of them' and ignore you and 2. The SV AI set isn't particularly white.
- Nasrudith 3y ago
  Of course they should ignore them. It is literally pure sexism and racism.
- anonreeeeplor 3y ago
  I have never seen a group of Asians organizing an AI cult. All the OpenAI people doing cults are white. Have you seen any Indian or Asian people or women building a cult?

kopirgan 3y ago

I remember listening to a panel discussing block chains in a user group meeting of a software company, decade or so ago. Mind you, this was not some cutting edge technology company it was boring accounting stuff. All panelists had zero clue and were talking just general vague terms. Of course not giving too much, being politically correct etc put additional constraints over what you can say in such meetings. So whatever evangelical zeal you see in these tech shows are not worth the time. I think most people know. They just play their role, nodding heads waiting for next coffee break. But it's a very interesting article that will sure stir a lot of thought.. Usual high standards of this magazine.

dventimi 3y ago

https://archive.ph/2024.01.29-165444/https://www.rollingstone.com/culture/culture-features/ai-companies-advocates-cult-1234954528/ https://archive.ph/2024.01.29-165444/https://www.rollingston...