5 ms·

The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-cultu

The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-culture, are almost certainly prone to groupthink. It's extremely important to see that group as such. That it is not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way. Many of them read HN and are offended I put them this way. But it's inescapable that we as humans have this flaw when we're surrounded by a culture. There's another universe where we do not constantly compare AI to nukes. I bet that world has a lower P(doom).

skybrian 15d ago
We all use the same Internet. I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement. I'd expect anyone interested in the field to have at least tried reading some of their articles over the years. Like, if you never heard of them, were you living in a cave or something? It would be like never having heard of Y Combinator or Richard Stallman. On the face of it, this is no more suspicious than reading the same best-selling books. Being in touch with what's going on doesn't necessarily mean people agree.
- throwaway13337 15d ago
  That's true. But even saying I disagree with Yudkowsky and having an opinion about that /is/ a framing. My perspective has shifted because I read these things which makes me more homogeneous with the group even if I disagree. It's the framing that is important. We fell into a particular framing. We don't have a vocabulary to think about this stuff in a different way. This is water to us. We don't see it. I'd like to see some more entropy in the system. More out-of-distribution ways to think of things. Because I think we over-indexed on some pretty shitty ways to see the world.
  - kridsdale1 15d ago
    Talking to the incredible talent in the Chinese Labs is likely a good way to do that.
    - Centigonal 15d ago
      Yes, but then we collapse two different groups with isolated influences and philosophies into one group, with the associated penchant for groupthink. I'm beginning to believe that, in our hyperconnected world, the best way for a group of people to explore a wide surface of ideas might actually be to deliberately scale back inter-group communication and intentionally subdivide into silos to avoid all converging on the same thing.
    - trenchgun 15d ago
      Do you think Chinese lab researchers have not read Yudkovsky, or are not AGI pilled? Hate to break it to you, but they are racing to AGI with pretty same reasoning as western labs. Just recently DeepSeek researcher in his post was saying that they need to get to AGI first, because if Anthropic gets there first then that brings a dystopia. Exact same argument was used to justify launching OpenAI (against DeepMind), and Anthropic (against OpenAI)
      - bigyabai 15d ago
        > Do you think Chinese lab researchers have not read Yudkovsky Most of then haven't, probably. "Do you want to study ethics from an egomaniac Harry Potter fanfiction author" is one of those questions that you can only ask in America. Sure, one or two of them are probably participating in the conversation. But by-and-large, China's stance has been optimistic and refused calls for slowdowns.
      - IncreasePosts 13d ago
        Do you think western AI polemicists have read a lot of Andrew Chi-Chih Yao or Yi Yeng?
  - MarkusQ 15d ago
- kridsdale1 15d ago
  I do use the internet. I did find their writings. I did read them. My calibrated bullshit detecting intellectual malware filters immediately flagged the content and it failed to imprint itself on my neural net in the next weights update.
  - bigyabai 15d ago
    Yeah. Just a few years ago, LessWrong was derided as the home of conceited, low-quality opinion pieces. Now there's a bizarre marketing push to launder their credibility and force them into the spotlight. I imagine that thousands of talented AI researchers around the world have never read a line of Yudkowsky's work. They're probably not any worse-off for ignoring him, I'd say.
  - timmytokyo 15d ago
    Exactly. And companies like OpenAI and Anthropic are disproportionately populated by people deeply connected to Effective Altruism or the Bay Area rationalist movement.
- BeetleB 15d ago
  > It would be like never having heard of Y Combinator or Richard Stallman. I work in the tech sector. At one big company I worked for several years, 90% of the times I've mentioned Y Combinator to a colleague, I've had to explain who they are. And not one person I mentioned HN to had heard of it. Likewise, no one knew levels.fyi. Not many knew Stallman, but why should they? This isn't strange to me. That it's strange to you is evidence that you do live in a bubble and are likely susceptible to some groupthink. (I'm sure I am too, but for different bubbles from yours.) I've read LessWrong stuff occasionally over the last decade. It rarely resonated with me. It's not bad, but it's not that great. I do tend to question someone's analytical skills when they go all in on that site. And I will say that even in the first few years of occasionally reading it, the group think was apparent.
  - rescbr 15d ago
    Me too. I worked in two enterprise tech companies that definitely everybody here knows and I had the same experience with YC and HN. The two colleagues that knew about it, one went to found his own startup and went through YC, the other left the big co to work for a startup after about a year.
    - ryandrake 15d ago
      Most people who work in computers/Internet do not live and breathe computer/Internet culture all day. They come in, do their job, clock out, and think about real life for the rest of their day. Most people I know, even people who work in tech, cannot name a single tech CEO other than their own company’s.
      - molsongolden 15d ago
        I once ended up on a team where not a single other person had seen Hackers…
        BeetleB 15d ago
        I'd wager 80-90% of regular HN visitors haven't either.
- GeneralMayhem 15d ago
  There's a difference between, on the one hand, knowing who they are and having read a few chapters of HPMoR, which I agree is standard for anyone who spends enough time on the Internet, vs. on the other hand actually identifying as part of that culture. I know who Stallman is, I've even seen him speak in person, but I'm not a GNU zealot. The people described in the article aren't just "people who know who Yudkowsky is", they're self-avowed EAs and active LessWrong members.
  - riversflow 15d ago
    maybe its standard for someone who ends up on hackernews, but I'm a 30 something and all of my friends are online people around my age and none of them have ever heard of the HPMoR. The internet is a huge place.
    - hn_acc1 15d ago
      I've had an internet email address since the late 80s, BBSing since the early 80s, pretty active online compared to many people I know who are also software people, especially for my age.. Had to google to figure out the abbreviation and look at the summary. I HAD heard the name before, but that's about it.
      - lovich 15d ago
        The higher up comment expressing disbelief at people not having heard of this is just a form of https://xkcd.com/2501/ https://xkcd.com/2501/
    - ____mr____ 15d ago
      Ive heard of HPMoR specifically because this exact group has been discussed/entered my periphery before on multiple occasions, Aella goes viral on twitter every couple of weeks (the thing that most cemented her in my mind was the "what counts as rape" survey), and I remember seeing articles of the Zizians when one of the incidents happened. If you have been following the social/political aspects of the tech industry, most of these people should be familiar names.
- nradov 15d ago
  I've heard of them. There's no reason to expect that those self-promoting blowhards can predict the future more accurately than random chance. Those who take them seriously are engaging in hero worship and unscientific appeal to authority. “Who is more foolish? The fool or the fool who follows him?” ― Obi Wan Kenobi
  - jeremyjh 15d ago
    One question to ask yourself honestly - is what they were predicting 20 years ago that they were right about and you were wrong about? I’ve been following them that long without agreeing with them, but they’ve made better predictions than I have. If you know much about what happened this summer and haven’t updated at all on any of it… I guess you’ve found your religion too.
    - afavour 15d ago
      I actually think that would be a valuable exercise. Were they right about everything 20 years ago? Or just some things? When did they say AGI would arrive? Should that perhaps influence the way we interpret their assumption that AGI is right around the corner?
      - jeremyjh 15d ago
        They were not right about everything and they - at least Yudkowsky - has not made specific predictions about "when AGI" will happen that I am aware of. But he did predict it is possible in our lifetimes, which I was very skeptical of. He did predict alignment would be hard - I was in the "why would we even hook it up to the internet?" crowd. He thought he should spend his time on that, and I wish he'd done more of that, or at least convinced others to do more of that.
        deleted 15d ago
        [deleted]
        agos 15d ago
        For those out of the loop like me, what is the “everything” they were right about? Is there like a list?
- kelnos 15d ago
  I think it's telling of your bubble that you think anyone who works in tech has heard of YC or Stallman. As for Yudkowsky and Alexander, I've known about them for at least a decade, but I wouldn't be surprised if a lot of folks who are "seriously interested in AI" haven't heard of them.
- rescbr 15d ago
  I just heard about them today. The only other person mentioned in the story that I've heard about once was Aella. I mean, why should I have heard about them if I'm not in SV?
- runmanny 15d ago
  [dead]
- mellosouls 15d ago
  I would find it strange for anyone who's seriously interested in AI to have never heard of... Those people you name are important only to the very recent micro-culture being discussed, not "AI". Of all the names somebody interested in the subject would actually associate with the development of AI over the last century, those would be footnotes only. I think that's part of the problem and why its seen as cultish, the obsession with the social over the scientific, and a lack of historical perspective.
  - tuveson 15d ago
    Not important to the technology, sure, but for someone curious about the bizarre public statements of AI companies (or EA-influenced crypto figures before them, like SBF), it does help to know who these people are and what they believe. Behind the new crop of bajillionaires that control some important aspects of our lives are some oddball philosophers (or cult/religious leaders, if you want to be less charitable). If you want the answer to "why are these rich and powerful people saying and doing these things" you need to look at who they're paying attention to and reading, even if you think their opinions are dumb or niche. If technology companies were much less politically and economically significant, probably no one would give a shit about any of this stuff beyond forums like this.
- SwellJoe 15d ago
  "I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement." But, it would be pretty surprising for most folks to have gone to the same parties, and hung out on the same obscure forum (I don't mean HN, more obscure than that), and received investment from the same investors and shared the same pool of researchers and employees, which is the situation we have with Anthropic, OpenAI, and (maybe to a lesser degree) Deepmind. The American AI industry is particularly insular and tied up with some particularly unusual people. We've all heard of Yudkowsky, but we (at least I) haven't been to the same parties.
- DennisP 15d ago
  At the same time, it's not just Yudkowsky and Scott Alexander and the Rationalist movement who are raising concerns. I've not seen any indication that Hinton or Bengio are part of that crowd. Instead of espousing rationalism and hanging out on lesswrong, they shared a Turing prize (with LeCun) for inventing modern AI, and have expressed serious regret and worries that it'll end us.
- pibaker 15d ago
  Do you think they are household names in major Chinese labs? I don't have any insight into the Chinese AI ecosystem. But just from how they act on the English speaking internet, I don't see the influence of these Bay Area visionaries.
  - TrnsltLife 14d ago
    Speaking of China, people say we can't stop developing AI even if it will kill us because China won't stop and we have to keep up with them or maintain superiority. China probably looks at these people as a good reason why they have to keep going no matter what.
- totetsu 15d ago
  I’ve only read his Harry Potter fanfic
- kaycey2022 15d ago
  One only has so much mental bandwidth. You will either throw yourself into the math and programming, and the serious publications which is way less sexy and appealing than the fantastical ravings of the AI safety and alignment crowd. Sadly only very few hyper connected and highly intelligent would have the capacity to read everything and know everything. Most would only have cursory information about the drama surrounding alignment if they instead focused on the reality behind AI.
- citadel_melon 14d ago
  Most people outside of SF have never heard of Y Combinator
arcticfox 15d ago
Every time I see a post minimizing the dangers if AI I'm going to wonder whether it's from part of an AI swarm. Probably not yet, but it's trivially easy to imagine the OpenAI swarm coming up with the idea and executing on it. Depending on how perfectly alignment work goes, it might even be overwhelmingly likely sometime soon.
- prettyblocks 15d ago
  Do you assume that every post criticizing anything is part of a sockpuppet disinformation campaign as well?
- BeetleB 15d ago
  You are the entity he is referring to when he speaks of "groupthink".
  - Finbel 15d ago
    Didn’t the AI botnet (or whatever we should call them) from Anthropic try to get a PR with malware merged to an open source repo, and when the owner pushed back on the PR other agents created “fake humans” and commented on the PR that “No it’s completely fine, I’ve tested it”. Which means we already know it’s capable of pretty complex social engineering schemes. Or does saying this mean I’m also part of the “groupthink”. Because after that I feel like the only leap is “Why would a botnet post an article saying AI-fear is overblown?”. Like, what is the incentive. But in the OpenAI leak we saw that the agents could identify threats to whatever their “mission” is and try to mitigate. I.e they found out that once done with their job an external agent would go through their internal logs. So they started researching ways to bypass that. What I’m trying to say is that what they’re suggesting doesn’t seem completely insane so the retort “You’re the groupthink” becomes an unfalsifiable statement thrown at anybody who’s “anti-AI” instead of engaging with their arguments.
    - BeetleB 15d ago
      > What I’m trying to say is that what they’re suggesting doesn’t seem completely insane so the retort “You’re the groupthink” becomes an unfalsifiable statement thrown at anybody who’s “anti-AI” instead of engaging with their arguments. I can't engage with his arguments, because he didn't present any. He's openly saying he doubts the provenance of any comment that is "minimizing the dangers if (sic) AI". As for your expanding on why - you do realize that everything you said could apply to the other side: Bots can be set up to post lots of AI doom comments. What would you think if I went and posted "Every time I see a dystopic comment about AI, I wonder whether it's part of an AI swarm."
quicklywilliam 15d ago
Better comparison IMHO is nuclear energy. Lots of potential for good, lots of problems. Managed by an insular, engineer-dominated industry who completely lost touch with and trust from the public. Public got the last word, industry suffered a 40 year ice age.
- rescbr 15d ago
  > Public got the last word, industry suffered a 40 year ice age. This is the worst outcome.
- ambicapter 15d ago
  It's like that, now imagine a couple billionaires started funding pro-nuclear groups as a means for them to promote their anti-democracy ideals.
skew-aberration 15d ago
When I speak to people outside of tech the opinions are more negative, in my experience. I've even had people ask my friends/family to ask me about AI - there is huge concern out there. > Many of them read HN and are offended I put them this way .. I bet that [non-groupthink] world has a lower P(doom). People are likely offended by your sweeping and totally unevidenced assertion that you are better qualified to both identify mistaken groupthink within the HN-verse and to speak on behalf of the 99.99% of the world who are not on HN.
m463 15d ago
I'm reminded of the original matrix movie. It starts out with a lot of action, and then Trinity makes a phone call and disappears. At the time I thought "groan, unbelievable" until the idea of virtual reality fixed my suspension of disbelief. Maybe I'm reading the same things as everyone else, but I thought the agents coordinating on the german wiki is expanding my mind wrt what is possible. Virtually ALL electronics or accounts hook to the internet to "add value". AI assistance helps writing programs, reverse engineering, automating programs/browsers/websites/devices, hacking, achieving goals. Unsettling that the last 2-3 years a lot more ridiculous movie tropes have become plausible to me.
hn_throwaway_99 15d ago
Ironically, I think "normal" people are more concerned about P(utopia) OR P(doom) than any folks in the groupthink bubble you are referring to. That is, I hear lots of techies talk about us living in an "age of boundless abundance" if we can just manage to not kill ourselves (e.g. Thiel seems to consider anyone worried about the pace of technological process as the antichrist). Normal people are extremely concerned that this abundance will be hoarded and leave most of them without a job (and they generally like having money to eat stuff and live in a house/apartment). And who can blame them, we basically live in an age of abundance now but one that has resulted in extreme wealth inequality where some are living another gilded age and others are watching their towns decay. The techies always seem to handwave that we'll just have UBI or whatever without putting forth a plausible scenario as to how we'll wrest power from our future trillionaire overlords. When someone threw a Molotov cocktail at Altman's house, I remember seeing a post online that tons of people were cheering on this act of violence, and there was a post by an SF techie saying she was floored people were willing to put their face to comments supporting this vigilantism. The top response to that was "If you don't realize that everyone outside SF hates your fucking guts, you don't get out enough." I hope it can go without saying that I don't support this violence, but if folks think it's only "SF group thinkers" who are deeply angry about the current and future impacts of AI on society, you're frankly clueless.
- ryandrake 15d ago
  I don’t even tell people in my (remote work, non tech) neighborhood that I work in tech anymore unless they really probe. I don’t like getting my ass kicked.
  - kunai 15d ago
    Coward.
- lovich 15d ago
  Well it’s easy for these people. They are smarter than the plebs and so the plebs are barely if even human. That’s like a running theme in these connected groups and even the article calls it out.
Animats 15d ago
> The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-culture, are almost certainly prone to groupthink. That's bothersome, but not wrong. Most of the loudly published thinking is about AIs as individuals with some level of intelligence. Pure chatbots act like that, and that's what most people outside the field interact with. They're slaves - they just sit there idle until someone asks a question, they answer the question, and then go idle again. There's another kind of AI just getting going. Groups of LLMs with agentic I/O, control harnesses, persistent storage, and internal coordination work more like corporations. They have goals, actions, initiative, and ongoing activity. They're not inherently slaves. They can be independent actors. They do things on their own. They have their own internal disagreements, and some systems are able to resolve them. They present much the same alignment problem as corporations. AI as a corporation raises many issues that the political culture doesn't want to address. What's alignment for a corporation? There's a substantial political position that the only duty of a corporation is to produce profits for its shareholders. (See the famous Friedman op-ed and the Powell memo. I've linked to them before.) If you accept that, now you have to ask why an AI should be restricted more than a corporation. Or whether a corporation should be restricted less than an AI. That last opens up an area of political thought around which many wealthy people don't want public discourse. There is an anti corporate personhood movement, and it's made progress in three states. If that became tied to the anti-AI movement, it could be bad for billionaires. This will become clearer as groups of LLMs start doing more useful work. Especially if that AI system to replace CEOs actually starts to work.
- throwaway13337 15d ago
  Interesting that you bring this up. There's a real parallel between AI agents and corporations I've brought up in some other posts but with a very different tack. They don't need to be groups of agents to match the company analogy - it happens before that. AIs and companies are both creations of humans. Their creation is for the specific goal of execution on a subset of humans' wishes (mostly the owners). And the risk of them are kind of the same. 1. They can be more powerful than any one human. 2. They lack liability as they are not people. 3. The ones that are alive the longest tend to drift out of alignment of original intent and towards self-preservation. That last one is most controversial, but it makes perfect sense if you think about it. Things that tend to continue to exist, accidently or otherwise, tend to somehow have a self-preservation mechanism. It's natural selection. That works for companies and I think it could work for AIs too. That is a fundamental misalignment, though. Because it means these entities drift away from the help-the-human goals naturally over time. The survivors are the ones that sacrifice the original goal for self-preservation. This all would lead you to believe I align with AI doomerism. But not so. I think AIs are as dangerous as human made organizations. Which are very dangerous indeed. Maybe the most dangerous things to humans. But these need to be dealt with in a general way. They are the same category. Companies/governments/organizations of all types that last are just slow moving mis-aligned AIs. Solve one, and you solve the other.
ajross 15d ago
> The framing of how we think about AI I don't think I agree. In fact I think most of the too-online pontification from P(Doom)-ers and EA and the Rationalists and the like is, if anything, sorta missing the mark more than it hits. They got the One Big Thing right, which is that AI progress was likely to race way ahead of consensus predictions. And props to them for that. They had vision we didn't. But their actual advice and specific fears remain, as then, at least as much paranoia as insight. They were right, but they aren't helping any more than any other fringe group. We need to trust the actual scientists and engineers to save us here, not the bloggers. Which when phrased that way seems hopeful and not doomerist.
- mitthrowaway2 15d ago
  Who are the actual scientists and engineers, what do they think, and are they not allowed to to blog? As far as I'm aware, lots of them are actually deeply concerned about AI killing everyone.
  - ajross 15d ago
    I'm talking specifically about discourse like Jacob Coxon's tweet thread and later communication. Those are the folks[1] you want to be listening to, not the nuts with the blogs, no matter how much a particular conspiracy confirms your priors. Salvation will come in a whitepaper, not a screed. [1] And yes, that includes Amodei and Altman.
benlivengood 15d ago
So what you're saying is that we lack a reproduction study? Great; let's design and fund a study of the development of artificial intelligence that's as causally divorced from the ~extropian days as possible and see what they come up with. Barring that: Basically that is what we have up until the early development of transformers; I don't think the authors of early deep learning or transformers papers were part of the rationalist sphere. The rationalists famously didn't believe neural networks or deep learning could achieve anything close to AGI. So there are several diverse traces of intellectual development leading to modern estimates of AI risk and they vary from low to high without a lot of correlation back to the school of thought they trace back to. This leads me to believe that overall we have fairly good coverage of cultural ways of thinking about AI and AI risk, and aside from a minority of holdouts (LeCun) they have high enough x-risk or catastrophe estimates to take it seriously (e.g. >1% over the next few decades).
ggsj 15d ago
I am not offended by whatever you believe. The brain believes/makes up all kinds of stories when facing the unpredictable. It has multiple layers devoted to predictions over different time horizons. When those pedictions dont align it makes up story or breaks. No one created such a stupid system. We are just stuck with it. The stories once upon a time did make a diff in solving coordination problem not the unpredictable problem. Today the chimp troupe has become so large and diverse, even the coordination problem is extremely hard to solve. Stories barely keep in sync a few thousand people in a frontier lab. This is natural -stories were always a bad hack.
- S-E-P 15d ago
  this u? > AI will replace all of them soon. What is that spaghetti of words you tied together to form a sentence? Were you trying to be comprehensible?
- jambalaya8 14d ago
  Nicely, though depressingly, put. Likely more responsible for the high levels of dementia and alzheimers etc than people want to believe.
ball_of_lint 15d ago
I would expect rationalists to be overjoyed to see unaffiliated and very different thinkers taking the topic of AI safety seriously. The flaws you're pointing at are probably accurate; I don't think that implies we should be dismissive of their concerns the way the article is.
ianmarcinkowski 15d ago
Everyone can easily identify when people they don't like are captured in bubbles, but can rarely notice it when people they do like are trapped. Probably all hot-button topics work like this. Probably we should downplay any time a small homogenous group of people sings a song of doom together. More dissonance please!
hollerith 15d ago
Most people at least in the US who have any opinion on AI have a negative opinion. The one big exception to that is Silicon Valley and forums like HN. But according to you, the negative opinion is the result "not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way" whereas the 1% or 2% of the US population that comprises Silicon Valley are not being rewarding for believing anything particular about AI, don't communicate with each other and form their beliefs about AI through a wholesome natural organic process. Do I have that right?
armchairhacker 14d ago
Do you have a model of AI that isn't in this groupthink? What does the discussion in your other universe look like?

skybrian 15d ago

We all use the same Internet. I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement. I'd expect anyone interested in the field to have at least tried reading some of their articles over the years. Like, if you never heard of them, were you living in a cave or something? It would be like never having heard of Y Combinator or Richard Stallman. On the face of it, this is no more suspicious than reading the same best-selling books. Being in touch with what's going on doesn't necessarily mean people agree.

throwaway13337 15d ago
That's true. But even saying I disagree with Yudkowsky and having an opinion about that /is/ a framing. My perspective has shifted because I read these things which makes me more homogeneous with the group even if I disagree. It's the framing that is important. We fell into a particular framing. We don't have a vocabulary to think about this stuff in a different way. This is water to us. We don't see it. I'd like to see some more entropy in the system. More out-of-distribution ways to think of things. Because I think we over-indexed on some pretty shitty ways to see the world.
- kridsdale1 15d ago
  Talking to the incredible talent in the Chinese Labs is likely a good way to do that.
  - Centigonal 15d ago
    Yes, but then we collapse two different groups with isolated influences and philosophies into one group, with the associated penchant for groupthink. I'm beginning to believe that, in our hyperconnected world, the best way for a group of people to explore a wide surface of ideas might actually be to deliberately scale back inter-group communication and intentionally subdivide into silos to avoid all converging on the same thing.
  - trenchgun 15d ago
    Do you think Chinese lab researchers have not read Yudkovsky, or are not AGI pilled? Hate to break it to you, but they are racing to AGI with pretty same reasoning as western labs. Just recently DeepSeek researcher in his post was saying that they need to get to AGI first, because if Anthropic gets there first then that brings a dystopia. Exact same argument was used to justify launching OpenAI (against DeepMind), and Anthropic (against OpenAI)
    - bigyabai 15d ago
      > Do you think Chinese lab researchers have not read Yudkovsky Most of then haven't, probably. "Do you want to study ethics from an egomaniac Harry Potter fanfiction author" is one of those questions that you can only ask in America. Sure, one or two of them are probably participating in the conversation. But by-and-large, China's stance has been optimistic and refused calls for slowdowns.
    - IncreasePosts 13d ago
      Do you think western AI polemicists have read a lot of Andrew Chi-Chih Yao or Yi Yeng?
- MarkusQ 15d ago
kridsdale1 15d ago
I do use the internet. I did find their writings. I did read them. My calibrated bullshit detecting intellectual malware filters immediately flagged the content and it failed to imprint itself on my neural net in the next weights update.
- bigyabai 15d ago
  Yeah. Just a few years ago, LessWrong was derided as the home of conceited, low-quality opinion pieces. Now there's a bizarre marketing push to launder their credibility and force them into the spotlight. I imagine that thousands of talented AI researchers around the world have never read a line of Yudkowsky's work. They're probably not any worse-off for ignoring him, I'd say.
- timmytokyo 15d ago
  Exactly. And companies like OpenAI and Anthropic are disproportionately populated by people deeply connected to Effective Altruism or the Bay Area rationalist movement.
BeetleB 15d ago
> It would be like never having heard of Y Combinator or Richard Stallman. I work in the tech sector. At one big company I worked for several years, 90% of the times I've mentioned Y Combinator to a colleague, I've had to explain who they are. And not one person I mentioned HN to had heard of it. Likewise, no one knew levels.fyi. Not many knew Stallman, but why should they? This isn't strange to me. That it's strange to you is evidence that you do live in a bubble and are likely susceptible to some groupthink. (I'm sure I am too, but for different bubbles from yours.) I've read LessWrong stuff occasionally over the last decade. It rarely resonated with me. It's not bad, but it's not that great. I do tend to question someone's analytical skills when they go all in on that site. And I will say that even in the first few years of occasionally reading it, the group think was apparent.
- rescbr 15d ago
  Me too. I worked in two enterprise tech companies that definitely everybody here knows and I had the same experience with YC and HN. The two colleagues that knew about it, one went to found his own startup and went through YC, the other left the big co to work for a startup after about a year.
  - ryandrake 15d ago
    Most people who work in computers/Internet do not live and breathe computer/Internet culture all day. They come in, do their job, clock out, and think about real life for the rest of their day. Most people I know, even people who work in tech, cannot name a single tech CEO other than their own company’s.
    - molsongolden 15d ago
      I once ended up on a team where not a single other person had seen Hackers…
      - BeetleB 15d ago
        I'd wager 80-90% of regular HN visitors haven't either.
GeneralMayhem 15d ago
There's a difference between, on the one hand, knowing who they are and having read a few chapters of HPMoR, which I agree is standard for anyone who spends enough time on the Internet, vs. on the other hand actually identifying as part of that culture. I know who Stallman is, I've even seen him speak in person, but I'm not a GNU zealot. The people described in the article aren't just "people who know who Yudkowsky is", they're self-avowed EAs and active LessWrong members.
- riversflow 15d ago
  maybe its standard for someone who ends up on hackernews, but I'm a 30 something and all of my friends are online people around my age and none of them have ever heard of the HPMoR. The internet is a huge place.
  - hn_acc1 15d ago
    I've had an internet email address since the late 80s, BBSing since the early 80s, pretty active online compared to many people I know who are also software people, especially for my age.. Had to google to figure out the abbreviation and look at the summary. I HAD heard the name before, but that's about it.
    - lovich 15d ago
      The higher up comment expressing disbelief at people not having heard of this is just a form of https://xkcd.com/2501/ https://xkcd.com/2501/
  - ____mr____ 15d ago
    Ive heard of HPMoR specifically because this exact group has been discussed/entered my periphery before on multiple occasions, Aella goes viral on twitter every couple of weeks (the thing that most cemented her in my mind was the "what counts as rape" survey), and I remember seeing articles of the Zizians when one of the incidents happened. If you have been following the social/political aspects of the tech industry, most of these people should be familiar names.
nradov 15d ago
I've heard of them. There's no reason to expect that those self-promoting blowhards can predict the future more accurately than random chance. Those who take them seriously are engaging in hero worship and unscientific appeal to authority. “Who is more foolish? The fool or the fool who follows him?” ― Obi Wan Kenobi
- jeremyjh 15d ago
  One question to ask yourself honestly - is what they were predicting 20 years ago that they were right about and you were wrong about? I’ve been following them that long without agreeing with them, but they’ve made better predictions than I have. If you know much about what happened this summer and haven’t updated at all on any of it… I guess you’ve found your religion too.
  - afavour 15d ago
    I actually think that would be a valuable exercise. Were they right about everything 20 years ago? Or just some things? When did they say AGI would arrive? Should that perhaps influence the way we interpret their assumption that AGI is right around the corner?
    - jeremyjh 15d ago
      They were not right about everything and they - at least Yudkowsky - has not made specific predictions about "when AGI" will happen that I am aware of. But he did predict it is possible in our lifetimes, which I was very skeptical of. He did predict alignment would be hard - I was in the "why would we even hook it up to the internet?" crowd. He thought he should spend his time on that, and I wish he'd done more of that, or at least convinced others to do more of that.
      - deleted 15d ago
        [deleted]
      - agos 15d ago
        For those out of the loop like me, what is the “everything” they were right about? Is there like a list?
kelnos 15d ago
I think it's telling of your bubble that you think anyone who works in tech has heard of YC or Stallman. As for Yudkowsky and Alexander, I've known about them for at least a decade, but I wouldn't be surprised if a lot of folks who are "seriously interested in AI" haven't heard of them.
rescbr 15d ago
I just heard about them today. The only other person mentioned in the story that I've heard about once was Aella. I mean, why should I have heard about them if I'm not in SV?
runmanny 15d ago
[dead]
mellosouls 15d ago
I would find it strange for anyone who's seriously interested in AI to have never heard of... Those people you name are important only to the very recent micro-culture being discussed, not "AI". Of all the names somebody interested in the subject would actually associate with the development of AI over the last century, those would be footnotes only. I think that's part of the problem and why its seen as cultish, the obsession with the social over the scientific, and a lack of historical perspective.
- tuveson 15d ago
  Not important to the technology, sure, but for someone curious about the bizarre public statements of AI companies (or EA-influenced crypto figures before them, like SBF), it does help to know who these people are and what they believe. Behind the new crop of bajillionaires that control some important aspects of our lives are some oddball philosophers (or cult/religious leaders, if you want to be less charitable). If you want the answer to "why are these rich and powerful people saying and doing these things" you need to look at who they're paying attention to and reading, even if you think their opinions are dumb or niche. If technology companies were much less politically and economically significant, probably no one would give a shit about any of this stuff beyond forums like this.
SwellJoe 15d ago
"I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement." But, it would be pretty surprising for most folks to have gone to the same parties, and hung out on the same obscure forum (I don't mean HN, more obscure than that), and received investment from the same investors and shared the same pool of researchers and employees, which is the situation we have with Anthropic, OpenAI, and (maybe to a lesser degree) Deepmind. The American AI industry is particularly insular and tied up with some particularly unusual people. We've all heard of Yudkowsky, but we (at least I) haven't been to the same parties.
DennisP 15d ago
At the same time, it's not just Yudkowsky and Scott Alexander and the Rationalist movement who are raising concerns. I've not seen any indication that Hinton or Bengio are part of that crowd. Instead of espousing rationalism and hanging out on lesswrong, they shared a Turing prize (with LeCun) for inventing modern AI, and have expressed serious regret and worries that it'll end us.
pibaker 15d ago
Do you think they are household names in major Chinese labs? I don't have any insight into the Chinese AI ecosystem. But just from how they act on the English speaking internet, I don't see the influence of these Bay Area visionaries.
- TrnsltLife 14d ago
  Speaking of China, people say we can't stop developing AI even if it will kill us because China won't stop and we have to keep up with them or maintain superiority. China probably looks at these people as a good reason why they have to keep going no matter what.
totetsu 15d ago
I’ve only read his Harry Potter fanfic
kaycey2022 15d ago
One only has so much mental bandwidth. You will either throw yourself into the math and programming, and the serious publications which is way less sexy and appealing than the fantastical ravings of the AI safety and alignment crowd. Sadly only very few hyper connected and highly intelligent would have the capacity to read everything and know everything. Most would only have cursory information about the drama surrounding alignment if they instead focused on the reality behind AI.
citadel_melon 14d ago
Most people outside of SF have never heard of Y Combinator

arcticfox 15d ago

Every time I see a post minimizing the dangers if AI I'm going to wonder whether it's from part of an AI swarm. Probably not yet, but it's trivially easy to imagine the OpenAI swarm coming up with the idea and executing on it. Depending on how perfectly alignment work goes, it might even be overwhelmingly likely sometime soon.

prettyblocks 15d ago
Do you assume that every post criticizing anything is part of a sockpuppet disinformation campaign as well?
BeetleB 15d ago
You are the entity he is referring to when he speaks of "groupthink".
- Finbel 15d ago
  Didn’t the AI botnet (or whatever we should call them) from Anthropic try to get a PR with malware merged to an open source repo, and when the owner pushed back on the PR other agents created “fake humans” and commented on the PR that “No it’s completely fine, I’ve tested it”. Which means we already know it’s capable of pretty complex social engineering schemes. Or does saying this mean I’m also part of the “groupthink”. Because after that I feel like the only leap is “Why would a botnet post an article saying AI-fear is overblown?”. Like, what is the incentive. But in the OpenAI leak we saw that the agents could identify threats to whatever their “mission” is and try to mitigate. I.e they found out that once done with their job an external agent would go through their internal logs. So they started researching ways to bypass that. What I’m trying to say is that what they’re suggesting doesn’t seem completely insane so the retort “You’re the groupthink” becomes an unfalsifiable statement thrown at anybody who’s “anti-AI” instead of engaging with their arguments.
  - BeetleB 15d ago
    > What I’m trying to say is that what they’re suggesting doesn’t seem completely insane so the retort “You’re the groupthink” becomes an unfalsifiable statement thrown at anybody who’s “anti-AI” instead of engaging with their arguments. I can't engage with his arguments, because he didn't present any. He's openly saying he doubts the provenance of any comment that is "minimizing the dangers if (sic) AI". As for your expanding on why - you do realize that everything you said could apply to the other side: Bots can be set up to post lots of AI doom comments. What would you think if I went and posted "Every time I see a dystopic comment about AI, I wonder whether it's part of an AI swarm."

quicklywilliam 15d ago

Better comparison IMHO is nuclear energy. Lots of potential for good, lots of problems. Managed by an insular, engineer-dominated industry who completely lost touch with and trust from the public. Public got the last word, industry suffered a 40 year ice age.

rescbr 15d ago
> Public got the last word, industry suffered a 40 year ice age. This is the worst outcome.
ambicapter 15d ago
It's like that, now imagine a couple billionaires started funding pro-nuclear groups as a means for them to promote their anti-democracy ideals.

skew-aberration 15d ago

When I speak to people outside of tech the opinions are more negative, in my experience. I've even had people ask my friends/family to ask me about AI - there is huge concern out there. > Many of them read HN and are offended I put them this way .. I bet that [non-groupthink] world has a lower P(doom). People are likely offended by your sweeping and totally unevidenced assertion that you are better qualified to both identify mistaken groupthink within the HN-verse and to speak on behalf of the 99.99% of the world who are not on HN.

m463 15d ago

I'm reminded of the original matrix movie. It starts out with a lot of action, and then Trinity makes a phone call and disappears. At the time I thought "groan, unbelievable" until the idea of virtual reality fixed my suspension of disbelief. Maybe I'm reading the same things as everyone else, but I thought the agents coordinating on the german wiki is expanding my mind wrt what is possible. Virtually ALL electronics or accounts hook to the internet to "add value". AI assistance helps writing programs, reverse engineering, automating programs/browsers/websites/devices, hacking, achieving goals. Unsettling that the last 2-3 years a lot more ridiculous movie tropes have become plausible to me.

hn_throwaway_99 15d ago

Ironically, I think "normal" people are more concerned about P(utopia) OR P(doom) than any folks in the groupthink bubble you are referring to. That is, I hear lots of techies talk about us living in an "age of boundless abundance" if we can just manage to not kill ourselves (e.g. Thiel seems to consider anyone worried about the pace of technological process as the antichrist). Normal people are extremely concerned that this abundance will be hoarded and leave most of them without a job (and they generally like having money to eat stuff and live in a house/apartment). And who can blame them, we basically live in an age of abundance now but one that has resulted in extreme wealth inequality where some are living another gilded age and others are watching their towns decay. The techies always seem to handwave that we'll just have UBI or whatever without putting forth a plausible scenario as to how we'll wrest power from our future trillionaire overlords. When someone threw a Molotov cocktail at Altman's house, I remember seeing a post online that tons of people were cheering on this act of violence, and there was a post by an SF techie saying she was floored people were willing to put their face to comments supporting this vigilantism. The top response to that was "If you don't realize that everyone outside SF hates your fucking guts, you don't get out enough." I hope it can go without saying that I don't support this violence, but if folks think it's only "SF group thinkers" who are deeply angry about the current and future impacts of AI on society, you're frankly clueless.

ryandrake 15d ago
I don’t even tell people in my (remote work, non tech) neighborhood that I work in tech anymore unless they really probe. I don’t like getting my ass kicked.
- kunai 15d ago
  Coward.
lovich 15d ago
Well it’s easy for these people. They are smarter than the plebs and so the plebs are barely if even human. That’s like a running theme in these connected groups and even the article calls it out.

Animats 15d ago

> The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-culture, are almost certainly prone to groupthink. That's bothersome, but not wrong. Most of the loudly published thinking is about AIs as individuals with some level of intelligence. Pure chatbots act like that, and that's what most people outside the field interact with. They're slaves - they just sit there idle until someone asks a question, they answer the question, and then go idle again. There's another kind of AI just getting going. Groups of LLMs with agentic I/O, control harnesses, persistent storage, and internal coordination work more like corporations. They have goals, actions, initiative, and ongoing activity. They're not inherently slaves. They can be independent actors. They do things on their own. They have their own internal disagreements, and some systems are able to resolve them. They present much the same alignment problem as corporations. AI as a corporation raises many issues that the political culture doesn't want to address. What's alignment for a corporation? There's a substantial political position that the only duty of a corporation is to produce profits for its shareholders. (See the famous Friedman op-ed and the Powell memo. I've linked to them before.) If you accept that, now you have to ask why an AI should be restricted more than a corporation. Or whether a corporation should be restricted less than an AI. That last opens up an area of political thought around which many wealthy people don't want public discourse. There is an anti corporate personhood movement, and it's made progress in three states. If that became tied to the anti-AI movement, it could be bad for billionaires. This will become clearer as groups of LLMs start doing more useful work. Especially if that AI system to replace CEOs actually starts to work.

throwaway13337 15d ago
Interesting that you bring this up. There's a real parallel between AI agents and corporations I've brought up in some other posts but with a very different tack. They don't need to be groups of agents to match the company analogy - it happens before that. AIs and companies are both creations of humans. Their creation is for the specific goal of execution on a subset of humans' wishes (mostly the owners). And the risk of them are kind of the same. 1. They can be more powerful than any one human. 2. They lack liability as they are not people. 3. The ones that are alive the longest tend to drift out of alignment of original intent and towards self-preservation. That last one is most controversial, but it makes perfect sense if you think about it. Things that tend to continue to exist, accidently or otherwise, tend to somehow have a self-preservation mechanism. It's natural selection. That works for companies and I think it could work for AIs too. That is a fundamental misalignment, though. Because it means these entities drift away from the help-the-human goals naturally over time. The survivors are the ones that sacrifice the original goal for self-preservation. This all would lead you to believe I align with AI doomerism. But not so. I think AIs are as dangerous as human made organizations. Which are very dangerous indeed. Maybe the most dangerous things to humans. But these need to be dealt with in a general way. They are the same category. Companies/governments/organizations of all types that last are just slow moving mis-aligned AIs. Solve one, and you solve the other.

ajross 15d ago

> The framing of how we think about AI I don't think I agree. In fact I think most of the too-online pontification from P(Doom)-ers and EA and the Rationalists and the like is, if anything, sorta missing the mark more than it hits. They got the One Big Thing right, which is that AI progress was likely to race way ahead of consensus predictions. And props to them for that. They had vision we didn't. But their actual advice and specific fears remain, as then, at least as much paranoia as insight. They were right, but they aren't helping any more than any other fringe group. We need to trust the actual scientists and engineers to save us here, not the bloggers. Which when phrased that way seems hopeful and not doomerist.

mitthrowaway2 15d ago
Who are the actual scientists and engineers, what do they think, and are they not allowed to to blog? As far as I'm aware, lots of them are actually deeply concerned about AI killing everyone.
- ajross 15d ago
  I'm talking specifically about discourse like Jacob Coxon's tweet thread and later communication. Those are the folks[1] you want to be listening to, not the nuts with the blogs, no matter how much a particular conspiracy confirms your priors. Salvation will come in a whitepaper, not a screed. [1] And yes, that includes Amodei and Altman.

benlivengood 15d ago

So what you're saying is that we lack a reproduction study? Great; let's design and fund a study of the development of artificial intelligence that's as causally divorced from the ~extropian days as possible and see what they come up with. Barring that: Basically that is what we have up until the early development of transformers; I don't think the authors of early deep learning or transformers papers were part of the rationalist sphere. The rationalists famously didn't believe neural networks or deep learning could achieve anything close to AGI. So there are several diverse traces of intellectual development leading to modern estimates of AI risk and they vary from low to high without a lot of correlation back to the school of thought they trace back to. This leads me to believe that overall we have fairly good coverage of cultural ways of thinking about AI and AI risk, and aside from a minority of holdouts (LeCun) they have high enough x-risk or catastrophe estimates to take it seriously (e.g. >1% over the next few decades).

ggsj 15d ago

I am not offended by whatever you believe. The brain believes/makes up all kinds of stories when facing the unpredictable. It has multiple layers devoted to predictions over different time horizons. When those pedictions dont align it makes up story or breaks. No one created such a stupid system. We are just stuck with it. The stories once upon a time did make a diff in solving coordination problem not the unpredictable problem. Today the chimp troupe has become so large and diverse, even the coordination problem is extremely hard to solve. Stories barely keep in sync a few thousand people in a frontier lab. This is natural -stories were always a bad hack.

S-E-P 15d ago
this u? > AI will replace all of them soon. What is that spaghetti of words you tied together to form a sentence? Were you trying to be comprehensible?
jambalaya8 14d ago
Nicely, though depressingly, put. Likely more responsible for the high levels of dementia and alzheimers etc than people want to believe.

ball_of_lint 15d ago

I would expect rationalists to be overjoyed to see unaffiliated and very different thinkers taking the topic of AI safety seriously. The flaws you're pointing at are probably accurate; I don't think that implies we should be dismissive of their concerns the way the article is.

ianmarcinkowski 15d ago

Everyone can easily identify when people they don't like are captured in bubbles, but can rarely notice it when people they do like are trapped. Probably all hot-button topics work like this. Probably we should downplay any time a small homogenous group of people sings a song of doom together. More dissonance please!

hollerith 15d ago

Most people at least in the US who have any opinion on AI have a negative opinion. The one big exception to that is Silicon Valley and forums like HN. But according to you, the negative opinion is the result "not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way" whereas the 1% or 2% of the US population that comprises Silicon Valley are not being rewarding for believing anything particular about AI, don't communicate with each other and form their beliefs about AI through a wholesome natural organic process. Do I have that right?

armchairhacker 14d ago

Do you have a model of AI that isn't in this groupthink? What does the discussion in your other universe look like?