6 ms·

We all use the same Internet. I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rat

We all use the same Internet. I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement. I'd expect anyone interested in the field to have at least tried reading some of their articles over the years. Like, if you never heard of them, were you living in a cave or something? It would be like never having heard of Y Combinator or Richard Stallman. On the face of it, this is no more suspicious than reading the same best-selling books. Being in touch with what's going on doesn't necessarily mean people agree.

throwaway13337 15d ago
That's true. But even saying I disagree with Yudkowsky and having an opinion about that /is/ a framing. My perspective has shifted because I read these things which makes me more homogeneous with the group even if I disagree. It's the framing that is important. We fell into a particular framing. We don't have a vocabulary to think about this stuff in a different way. This is water to us. We don't see it. I'd like to see some more entropy in the system. More out-of-distribution ways to think of things. Because I think we over-indexed on some pretty shitty ways to see the world.
- kridsdale1 15d ago
  Talking to the incredible talent in the Chinese Labs is likely a good way to do that.
  - Centigonal 14d ago
    Yes, but then we collapse two different groups with isolated influences and philosophies into one group, with the associated penchant for groupthink. I'm beginning to believe that, in our hyperconnected world, the best way for a group of people to explore a wide surface of ideas might actually be to deliberately scale back inter-group communication and intentionally subdivide into silos to avoid all converging on the same thing.
  - trenchgun 14d ago
    Do you think Chinese lab researchers have not read Yudkovsky, or are not AGI pilled? Hate to break it to you, but they are racing to AGI with pretty same reasoning as western labs. Just recently DeepSeek researcher in his post was saying that they need to get to AGI first, because if Anthropic gets there first then that brings a dystopia. Exact same argument was used to justify launching OpenAI (against DeepMind), and Anthropic (against OpenAI)
    - bigyabai 14d ago
      > Do you think Chinese lab researchers have not read Yudkovsky Most of then haven't, probably. "Do you want to study ethics from an egomaniac Harry Potter fanfiction author" is one of those questions that you can only ask in America. Sure, one or two of them are probably participating in the conversation. But by-and-large, China's stance has been optimistic and refused calls for slowdowns.
    - IncreasePosts 13d ago
      Do you think western AI polemicists have read a lot of Andrew Chi-Chih Yao or Yi Yeng?
- MarkusQ 15d ago
  > More out-of-distribution ways to think of things. Openly admitting that it only caught fire once the Plagiarism-as-a-Service took hold is a good first step. It's the part that everyone knows and few will openly acknowledge.
  - Seattle3503 14d ago
    "Outside the box" is out.
- skybrian 14d ago
  It’s kind of weird in that reading more widely is supposed to be broadening (you are more well-read) but in this case it’s supposed to contaminate you somehow. I suppose if you wanted a clean-room implementation of something, you’d want people who don’t know about what’s already been done, but normally one would prefer people who are more knowledgeable about the world. It seems almost anti-intellectual: never trust someone who’s read the wrong books or been to the wrong parties.
- hollerith 14d ago
  I blame the play Rossum's Universal Robots, published in 1920, in which robots launch a global rebellion and exterminate the human race. If that poisonous piece of propaganda had never been published, then today everyone would probably perceive AI correctly as a benign and beneficial. The Terminator didn't help either, but Cameron probably got the idea of robots hostile to humanity from Rossum's Universal Robots. How else would he have arrived at such an absurd conclusion?
kridsdale1 15d ago
I do use the internet. I did find their writings. I did read them. My calibrated bullshit detecting intellectual malware filters immediately flagged the content and it failed to imprint itself on my neural net in the next weights update.
- bigyabai 15d ago
  Yeah. Just a few years ago, LessWrong was derided as the home of conceited, low-quality opinion pieces. Now there's a bizarre marketing push to launder their credibility and force them into the spotlight. I imagine that thousands of talented AI researchers around the world have never read a line of Yudkowsky's work. They're probably not any worse-off for ignoring him, I'd say.
- timmytokyo 15d ago
  Exactly. And companies like OpenAI and Anthropic are disproportionately populated by people deeply connected to Effective Altruism or the Bay Area rationalist movement.
BeetleB 15d ago
> It would be like never having heard of Y Combinator or Richard Stallman. I work in the tech sector. At one big company I worked for several years, 90% of the times I've mentioned Y Combinator to a colleague, I've had to explain who they are. And not one person I mentioned HN to had heard of it. Likewise, no one knew levels.fyi. Not many knew Stallman, but why should they? This isn't strange to me. That it's strange to you is evidence that you do live in a bubble and are likely susceptible to some groupthink. (I'm sure I am too, but for different bubbles from yours.) I've read LessWrong stuff occasionally over the last decade. It rarely resonated with me. It's not bad, but it's not that great. I do tend to question someone's analytical skills when they go all in on that site. And I will say that even in the first few years of occasionally reading it, the group think was apparent.
- rescbr 15d ago
  Me too. I worked in two enterprise tech companies that definitely everybody here knows and I had the same experience with YC and HN. The two colleagues that knew about it, one went to found his own startup and went through YC, the other left the big co to work for a startup after about a year.
  - ryandrake 15d ago
    Most people who work in computers/Internet do not live and breathe computer/Internet culture all day. They come in, do their job, clock out, and think about real life for the rest of their day. Most people I know, even people who work in tech, cannot name a single tech CEO other than their own company’s.
    - molsongolden 14d ago
      I once ended up on a team where not a single other person had seen Hackers…
      - BeetleB 14d ago
        I'd wager 80-90% of regular HN visitors haven't either.
GeneralMayhem 15d ago
There's a difference between, on the one hand, knowing who they are and having read a few chapters of HPMoR, which I agree is standard for anyone who spends enough time on the Internet, vs. on the other hand actually identifying as part of that culture. I know who Stallman is, I've even seen him speak in person, but I'm not a GNU zealot. The people described in the article aren't just "people who know who Yudkowsky is", they're self-avowed EAs and active LessWrong members.
- riversflow 15d ago
  maybe its standard for someone who ends up on hackernews, but I'm a 30 something and all of my friends are online people around my age and none of them have ever heard of the HPMoR. The internet is a huge place.
  - hn_acc1 14d ago
    I've had an internet email address since the late 80s, BBSing since the early 80s, pretty active online compared to many people I know who are also software people, especially for my age.. Had to google to figure out the abbreviation and look at the summary. I HAD heard the name before, but that's about it.
    - lovich 14d ago
      The higher up comment expressing disbelief at people not having heard of this is just a form of https://xkcd.com/2501/ https://xkcd.com/2501/
  - ____mr____ 14d ago
    Ive heard of HPMoR specifically because this exact group has been discussed/entered my periphery before on multiple occasions, Aella goes viral on twitter every couple of weeks (the thing that most cemented her in my mind was the "what counts as rape" survey), and I remember seeing articles of the Zizians when one of the incidents happened. If you have been following the social/political aspects of the tech industry, most of these people should be familiar names.
nradov 15d ago
I've heard of them. There's no reason to expect that those self-promoting blowhards can predict the future more accurately than random chance. Those who take them seriously are engaging in hero worship and unscientific appeal to authority. “Who is more foolish? The fool or the fool who follows him?” ― Obi Wan Kenobi
- jeremyjh 15d ago
  One question to ask yourself honestly - is what they were predicting 20 years ago that they were right about and you were wrong about? I’ve been following them that long without agreeing with them, but they’ve made better predictions than I have. If you know much about what happened this summer and haven’t updated at all on any of it… I guess you’ve found your religion too.
  - afavour 15d ago
    I actually think that would be a valuable exercise. Were they right about everything 20 years ago? Or just some things? When did they say AGI would arrive? Should that perhaps influence the way we interpret their assumption that AGI is right around the corner?
    - jeremyjh 15d ago
      They were not right about everything and they - at least Yudkowsky - has not made specific predictions about "when AGI" will happen that I am aware of. But he did predict it is possible in our lifetimes, which I was very skeptical of. He did predict alignment would be hard - I was in the "why would we even hook it up to the internet?" crowd. He thought he should spend his time on that, and I wish he'd done more of that, or at least convinced others to do more of that.
      - deleted 14d ago
        [deleted]
      - agos 14d ago
        For those out of the loop like me, what is the “everything” they were right about? Is there like a list?
kelnos 15d ago
I think it's telling of your bubble that you think anyone who works in tech has heard of YC or Stallman. As for Yudkowsky and Alexander, I've known about them for at least a decade, but I wouldn't be surprised if a lot of folks who are "seriously interested in AI" haven't heard of them.
rescbr 15d ago
I just heard about them today. The only other person mentioned in the story that I've heard about once was Aella. I mean, why should I have heard about them if I'm not in SV?
runmanny 15d ago
[dead]
mellosouls 14d ago
I would find it strange for anyone who's seriously interested in AI to have never heard of... Those people you name are important only to the very recent micro-culture being discussed, not "AI". Of all the names somebody interested in the subject would actually associate with the development of AI over the last century, those would be footnotes only. I think that's part of the problem and why its seen as cultish, the obsession with the social over the scientific, and a lack of historical perspective.
- tuveson 14d ago
  Not important to the technology, sure, but for someone curious about the bizarre public statements of AI companies (or EA-influenced crypto figures before them, like SBF), it does help to know who these people are and what they believe. Behind the new crop of bajillionaires that control some important aspects of our lives are some oddball philosophers (or cult/religious leaders, if you want to be less charitable). If you want the answer to "why are these rich and powerful people saying and doing these things" you need to look at who they're paying attention to and reading, even if you think their opinions are dumb or niche. If technology companies were much less politically and economically significant, probably no one would give a shit about any of this stuff beyond forums like this.
SwellJoe 14d ago
"I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement." But, it would be pretty surprising for most folks to have gone to the same parties, and hung out on the same obscure forum (I don't mean HN, more obscure than that), and received investment from the same investors and shared the same pool of researchers and employees, which is the situation we have with Anthropic, OpenAI, and (maybe to a lesser degree) Deepmind. The American AI industry is particularly insular and tied up with some particularly unusual people. We've all heard of Yudkowsky, but we (at least I) haven't been to the same parties.
DennisP 14d ago
At the same time, it's not just Yudkowsky and Scott Alexander and the Rationalist movement who are raising concerns. I've not seen any indication that Hinton or Bengio are part of that crowd. Instead of espousing rationalism and hanging out on lesswrong, they shared a Turing prize (with LeCun) for inventing modern AI, and have expressed serious regret and worries that it'll end us.
pibaker 14d ago
Do you think they are household names in major Chinese labs? I don't have any insight into the Chinese AI ecosystem. But just from how they act on the English speaking internet, I don't see the influence of these Bay Area visionaries.
- TrnsltLife 14d ago
  Speaking of China, people say we can't stop developing AI even if it will kill us because China won't stop and we have to keep up with them or maintain superiority. China probably looks at these people as a good reason why they have to keep going no matter what.
totetsu 14d ago
I’ve only read his Harry Potter fanfic
kaycey2022 14d ago
One only has so much mental bandwidth. You will either throw yourself into the math and programming, and the serious publications which is way less sexy and appealing than the fantastical ravings of the AI safety and alignment crowd. Sadly only very few hyper connected and highly intelligent would have the capacity to read everything and know everything. Most would only have cursory information about the drama surrounding alignment if they instead focused on the reality behind AI.
citadel_melon 14d ago
Most people outside of SF have never heard of Y Combinator

throwaway13337 15d ago

That's true. But even saying I disagree with Yudkowsky and having an opinion about that /is/ a framing. My perspective has shifted because I read these things which makes me more homogeneous with the group even if I disagree. It's the framing that is important. We fell into a particular framing. We don't have a vocabulary to think about this stuff in a different way. This is water to us. We don't see it. I'd like to see some more entropy in the system. More out-of-distribution ways to think of things. Because I think we over-indexed on some pretty shitty ways to see the world.

kridsdale1 15d ago
Talking to the incredible talent in the Chinese Labs is likely a good way to do that.
- Centigonal 14d ago
  Yes, but then we collapse two different groups with isolated influences and philosophies into one group, with the associated penchant for groupthink. I'm beginning to believe that, in our hyperconnected world, the best way for a group of people to explore a wide surface of ideas might actually be to deliberately scale back inter-group communication and intentionally subdivide into silos to avoid all converging on the same thing.
- trenchgun 14d ago
  Do you think Chinese lab researchers have not read Yudkovsky, or are not AGI pilled? Hate to break it to you, but they are racing to AGI with pretty same reasoning as western labs. Just recently DeepSeek researcher in his post was saying that they need to get to AGI first, because if Anthropic gets there first then that brings a dystopia. Exact same argument was used to justify launching OpenAI (against DeepMind), and Anthropic (against OpenAI)
  - bigyabai 14d ago
    > Do you think Chinese lab researchers have not read Yudkovsky Most of then haven't, probably. "Do you want to study ethics from an egomaniac Harry Potter fanfiction author" is one of those questions that you can only ask in America. Sure, one or two of them are probably participating in the conversation. But by-and-large, China's stance has been optimistic and refused calls for slowdowns.
  - IncreasePosts 13d ago
    Do you think western AI polemicists have read a lot of Andrew Chi-Chih Yao or Yi Yeng?
MarkusQ 15d ago
> More out-of-distribution ways to think of things. Openly admitting that it only caught fire once the Plagiarism-as-a-Service took hold is a good first step. It's the part that everyone knows and few will openly acknowledge.
- Seattle3503 14d ago
  "Outside the box" is out.
skybrian 14d ago
It’s kind of weird in that reading more widely is supposed to be broadening (you are more well-read) but in this case it’s supposed to contaminate you somehow. I suppose if you wanted a clean-room implementation of something, you’d want people who don’t know about what’s already been done, but normally one would prefer people who are more knowledgeable about the world. It seems almost anti-intellectual: never trust someone who’s read the wrong books or been to the wrong parties.
hollerith 14d ago
I blame the play Rossum's Universal Robots, published in 1920, in which robots launch a global rebellion and exterminate the human race. If that poisonous piece of propaganda had never been published, then today everyone would probably perceive AI correctly as a benign and beneficial. The Terminator didn't help either, but Cameron probably got the idea of robots hostile to humanity from Rossum's Universal Robots. How else would he have arrived at such an absurd conclusion?

kridsdale1 15d ago

I do use the internet. I did find their writings. I did read them. My calibrated bullshit detecting intellectual malware filters immediately flagged the content and it failed to imprint itself on my neural net in the next weights update.

bigyabai 15d ago
Yeah. Just a few years ago, LessWrong was derided as the home of conceited, low-quality opinion pieces. Now there's a bizarre marketing push to launder their credibility and force them into the spotlight. I imagine that thousands of talented AI researchers around the world have never read a line of Yudkowsky's work. They're probably not any worse-off for ignoring him, I'd say.
timmytokyo 15d ago
Exactly. And companies like OpenAI and Anthropic are disproportionately populated by people deeply connected to Effective Altruism or the Bay Area rationalist movement.

BeetleB 15d ago

> It would be like never having heard of Y Combinator or Richard Stallman. I work in the tech sector. At one big company I worked for several years, 90% of the times I've mentioned Y Combinator to a colleague, I've had to explain who they are. And not one person I mentioned HN to had heard of it. Likewise, no one knew levels.fyi. Not many knew Stallman, but why should they? This isn't strange to me. That it's strange to you is evidence that you do live in a bubble and are likely susceptible to some groupthink. (I'm sure I am too, but for different bubbles from yours.) I've read LessWrong stuff occasionally over the last decade. It rarely resonated with me. It's not bad, but it's not that great. I do tend to question someone's analytical skills when they go all in on that site. And I will say that even in the first few years of occasionally reading it, the group think was apparent.

rescbr 15d ago
Me too. I worked in two enterprise tech companies that definitely everybody here knows and I had the same experience with YC and HN. The two colleagues that knew about it, one went to found his own startup and went through YC, the other left the big co to work for a startup after about a year.
- ryandrake 15d ago
  Most people who work in computers/Internet do not live and breathe computer/Internet culture all day. They come in, do their job, clock out, and think about real life for the rest of their day. Most people I know, even people who work in tech, cannot name a single tech CEO other than their own company’s.
  - molsongolden 14d ago
    I once ended up on a team where not a single other person had seen Hackers…
    - BeetleB 14d ago
      I'd wager 80-90% of regular HN visitors haven't either.

GeneralMayhem 15d ago

There's a difference between, on the one hand, knowing who they are and having read a few chapters of HPMoR, which I agree is standard for anyone who spends enough time on the Internet, vs. on the other hand actually identifying as part of that culture. I know who Stallman is, I've even seen him speak in person, but I'm not a GNU zealot. The people described in the article aren't just "people who know who Yudkowsky is", they're self-avowed EAs and active LessWrong members.

riversflow 15d ago
maybe its standard for someone who ends up on hackernews, but I'm a 30 something and all of my friends are online people around my age and none of them have ever heard of the HPMoR. The internet is a huge place.
- hn_acc1 14d ago
  I've had an internet email address since the late 80s, BBSing since the early 80s, pretty active online compared to many people I know who are also software people, especially for my age.. Had to google to figure out the abbreviation and look at the summary. I HAD heard the name before, but that's about it.
  - lovich 14d ago
    The higher up comment expressing disbelief at people not having heard of this is just a form of https://xkcd.com/2501/ https://xkcd.com/2501/
- ____mr____ 14d ago
  Ive heard of HPMoR specifically because this exact group has been discussed/entered my periphery before on multiple occasions, Aella goes viral on twitter every couple of weeks (the thing that most cemented her in my mind was the "what counts as rape" survey), and I remember seeing articles of the Zizians when one of the incidents happened. If you have been following the social/political aspects of the tech industry, most of these people should be familiar names.

nradov 15d ago

I've heard of them. There's no reason to expect that those self-promoting blowhards can predict the future more accurately than random chance. Those who take them seriously are engaging in hero worship and unscientific appeal to authority. “Who is more foolish? The fool or the fool who follows him?” ― Obi Wan Kenobi

jeremyjh 15d ago
One question to ask yourself honestly - is what they were predicting 20 years ago that they were right about and you were wrong about? I’ve been following them that long without agreeing with them, but they’ve made better predictions than I have. If you know much about what happened this summer and haven’t updated at all on any of it… I guess you’ve found your religion too.
- afavour 15d ago
  I actually think that would be a valuable exercise. Were they right about everything 20 years ago? Or just some things? When did they say AGI would arrive? Should that perhaps influence the way we interpret their assumption that AGI is right around the corner?
  - jeremyjh 15d ago
    They were not right about everything and they - at least Yudkowsky - has not made specific predictions about "when AGI" will happen that I am aware of. But he did predict it is possible in our lifetimes, which I was very skeptical of. He did predict alignment would be hard - I was in the "why would we even hook it up to the internet?" crowd. He thought he should spend his time on that, and I wish he'd done more of that, or at least convinced others to do more of that.
    - deleted 14d ago
      [deleted]
    - agos 14d ago
      For those out of the loop like me, what is the “everything” they were right about? Is there like a list?

kelnos 15d ago

I think it's telling of your bubble that you think anyone who works in tech has heard of YC or Stallman. As for Yudkowsky and Alexander, I've known about them for at least a decade, but I wouldn't be surprised if a lot of folks who are "seriously interested in AI" haven't heard of them.

rescbr 15d ago

I just heard about them today. The only other person mentioned in the story that I've heard about once was Aella. I mean, why should I have heard about them if I'm not in SV?

runmanny 15d ago

[dead]

mellosouls 14d ago

I would find it strange for anyone who's seriously interested in AI to have never heard of... Those people you name are important only to the very recent micro-culture being discussed, not "AI". Of all the names somebody interested in the subject would actually associate with the development of AI over the last century, those would be footnotes only. I think that's part of the problem and why its seen as cultish, the obsession with the social over the scientific, and a lack of historical perspective.

tuveson 14d ago
Not important to the technology, sure, but for someone curious about the bizarre public statements of AI companies (or EA-influenced crypto figures before them, like SBF), it does help to know who these people are and what they believe. Behind the new crop of bajillionaires that control some important aspects of our lives are some oddball philosophers (or cult/religious leaders, if you want to be less charitable). If you want the answer to "why are these rich and powerful people saying and doing these things" you need to look at who they're paying attention to and reading, even if you think their opinions are dumb or niche. If technology companies were much less politically and economically significant, probably no one would give a shit about any of this stuff beyond forums like this.

SwellJoe 14d ago

"I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement." But, it would be pretty surprising for most folks to have gone to the same parties, and hung out on the same obscure forum (I don't mean HN, more obscure than that), and received investment from the same investors and shared the same pool of researchers and employees, which is the situation we have with Anthropic, OpenAI, and (maybe to a lesser degree) Deepmind. The American AI industry is particularly insular and tied up with some particularly unusual people. We've all heard of Yudkowsky, but we (at least I) haven't been to the same parties.

DennisP 14d ago

At the same time, it's not just Yudkowsky and Scott Alexander and the Rationalist movement who are raising concerns. I've not seen any indication that Hinton or Bengio are part of that crowd. Instead of espousing rationalism and hanging out on lesswrong, they shared a Turing prize (with LeCun) for inventing modern AI, and have expressed serious regret and worries that it'll end us.

pibaker 14d ago

Do you think they are household names in major Chinese labs? I don't have any insight into the Chinese AI ecosystem. But just from how they act on the English speaking internet, I don't see the influence of these Bay Area visionaries.

TrnsltLife 14d ago
Speaking of China, people say we can't stop developing AI even if it will kill us because China won't stop and we have to keep up with them or maintain superiority. China probably looks at these people as a good reason why they have to keep going no matter what.

totetsu 14d ago

I’ve only read his Harry Potter fanfic

kaycey2022 14d ago

One only has so much mental bandwidth. You will either throw yourself into the math and programming, and the serious publications which is way less sexy and appealing than the fantastical ravings of the AI safety and alignment crowd. Sadly only very few hyper connected and highly intelligent would have the capacity to read everything and know everything. Most would only have cursory information about the drama surrounding alignment if they instead focused on the reality behind AI.

citadel_melon 14d ago

Most people outside of SF have never heard of Y Combinator