20 ms·

Damn, there's a lot of nonsense in that essay. "Rationalists believed that Bayesian statistics and decision theory could de-bias human thinking and model the b

Damn, there's a lot of nonsense in that essay. "Rationalists believed that Bayesian statistics and decision theory could de-bias human thinking and model the behaviour of godlike intelligences" Lol

nullc 3y ago
That's actually the lite version told to non-believers, the full Xenu version is that they're going to construct an AI God that embodies "human" (their) values which will instantly take over the world and maximizes all the positive things, protect everyone from harm, and (most importantly) prohibit the formation of any competing super-intelligence. And if they fail to do this, all life on earth will soon be extinguished (maybe in the next few years, probably not more than 50 years), and potentially not just destroyed but converted into an unspeakable hell. An offshoot of the group started protesting the main group for not doing enough to stop the apocalypse, but has been set back a bit by the attempted murder and felony-murder charges they're dealing with. Both the protests and the murder made the news a bit but the media hasn't managed to note the connections between the events or to AI doomer cult yet. I worry that it's going to evolve to outright terrorist attacks, esp with their community moralizing the use of nuclear weapons to shut down GPU clusters... but even it doesn't its still harming people by convincing them that they likely have no future and that they're murdering trillions of future humans by not doing everything they can to stop the doom and by influencing product and policy discussions in weird ways.
- turnsout 3y ago
  Uh, what? Please write about this somewhere.
  - jodrellblank 3y ago
    A series of blog posts about it[1]: https://aiascendant.substack.com/p/extropias-children-chapter-1-the-wunderkind https://aiascendant.substack.com/p/extropias-children-chapte... [1] I think, this is Yudkowsky, Lesswrong, MIRI, the doom cult it became, the crazy revelations like https://www.lesswrong.com/posts/MnFqyPLqbiKL8nSR7/my-experience-at-and-around-miri-and-cfar-inspired-by-zoe https://www.lesswrong.com/posts/MnFqyPLqbiKL8nSR7/my-experie... and https://old.reddit.com/r/slatestarcodex/comments/qa14kg/my_experience_at_and_around_miri_and_cfar/ https://old.reddit.com/r/slatestarcodex/comments/qa14kg/my_e...
  - concordDance 3y ago
    As someone fairly close to the thing it's quite safe to ignore this guy as he doesn't have much understanding of anything in this space.
- ttt11199907 3y ago
  > their community moralizing the use of nuclear weapons to shut down GPU clusters Did I miss some news?
  - makeworld 3y ago
    > be willing to destroy a rogue datacenter by airstrike https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/ https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...
  - deleted 3y ago
    [deleted]
- dylan604 3y ago
  This sounds very familiar to the Republic of Texas movement in the mid-90s. The notion that Texas was not legally ratified into the Union was making its way through the courts by winning its cases. Some activists thought it was moving too slowly and skipped a few steps. A new fiat money was involved, other countries started recognizing the Republic of Texas government, liens were being placed on property, oh, and then the kidnappings started happening. The gung-ho crowd will always look like crazy types to the outsiders
- reducesuffering 3y ago
  > An offshoot of the group started protesting the main group for not doing enough to stop the apocalypse, but has been set back a bit by the attempted murder and felony-murder charges they're dealing with. Both the protests and the murder made the news a bit but the media hasn't managed to note the connections between the events or to AI doomer cult yet. Are you talking about anyone but the Ziz cult? They're very far removed from typical rationalist sphere think. Just because a schizophrenic psychopath reads a blog and spends all their time trying to manipulate other vulnerable isolated readers doesn't mean it's because of the bigger group they were trying to associate with. Those people are homeless self-proclaimed "vegan siths," hardly the employed MD / graduate types that ACX / lesswrong / EA is made of. If I go do something terrible, you and HN are hardly to blame.
  - nullc 3y ago
    I and HN haven't been brainwashing you that the world is sure to end because of AI doom for the last 5 years. And yeah, they're extreme, but also made them more vulnerable to it and more likely to act out. (And also, thus, more likely to form a plot to murder their landlord) When some psycho goes to an AI conference with a bomb vest on and their views are directly traceable to EA and LW then it absolutely will be their fault as it is the expected outcome from years of brainwashing that AI will bring about the end of the world and potentially eternal torment. It's the expected outcome from saying that using violence to stop it would be morally justified (which is, incidentally, also inherent to the AI God story: the idea is that it will forcibly prevent the creation of other super intelligences, and not by talking people out of it, it's always been a power and control fantasy). It's unlikely to be the people at the top that will be pulling the trigger, they're going to be too busy enjoying the traditional fringe benefits of having a cult. It's the people at the bottom that haven't figured out that the purpose of any cult is locking other people into extreme views so that the people at the top of the power structure can control them and profit from them, so they can enjoy relative fame and having their interests catered to... the people who are marginal and at the edge, not too valuable to the cult themselves but electrified by its message. It'll be that marginal member who's group approved drug use wasn't kept under control, who didn't find success in the cult group sex harem, etc. It'll always be the fault of the person who pulled the trigger, but also the fault of the people who put their finger on it.
- drdeca 3y ago
  > that embodies "human" (their) values They have made posts (well, I can remember at least one, which I think was well-received) about how people working on AI safety/alignment should avoid having “leaving their fingerprints on the future”, meaning, making the AI aligned to specifically their values, rather than an impartial one. So, I think they generally believe they oughtn’t have it aligned specifically to their values, but that instead they should try to make it aligned with a more impartial aggregate of human values. (Though they might imagine that a “coherent extrapolation” of typical human values, will be pretty close to their own. But, I suppose most people think that their values are the correct ones, or something else playing the role of “correct” for the people who don’t think there are “correct values”.) Do they put enough of an emphasis on this? I’m not sure. I can say that I certainly don’t look forwards to the development of such a superintelligence (partially on account of how my values differ from theirs and from what I imagine to be “the average”). But, still, “aligned with some people “ still sounds a fair bit better than “not aligned with anyone”. So, I’m mostly hoping that either superintelligence won’t be developed, or that if it is, God intervenes and prevents it from causing too much of a problem.
  - nullc 3y ago
    Still, when you find yourself trying to develop God in order to take over the world (or even sometimes universe), you should really be saying "Wait. Are we the baddies?" and not respond that that with "oh no, we'll give other people's ideas of 'value' consideration too, especially if they're not intellectually disabled by our judgement". > But, still, “aligned with some people “ still sounds a fair bit better than “not aligned with anyone”. I'm not sure about that. Part of the capability to do great harm is from the near miss. Fighting off Mindless grey good is better than some techbro trying to "optimize" you against your will. Of course this line of thinking requires accepting their inane premises, but it's worrysome that even if they were right the astonishing hubris would make them the existential threat more than anyone else. I think the greater relevance is the underlying unhealthy mental states underlying the doom cult. A person living a life in service of a very fringe idea we'll all likely suffer ultimate doom beyond measure unless their social group gets its way is going to exhibit all manner of compromised judgement.
- concordDance 3y ago
  More bullshit. The AI X-risk statements are public. > has been set back a bit by the attempted murder and felony-murder charges they're dealing with You have no idea what you're talking about.
- matthewdgreen 3y ago
  To add a citation to the "bombing of data centers" bit: https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/ https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...
EamonnMR 3y ago
That's a pretty fair characterization of the Roko's Basilisk crowd though, isn't it?
- drdeca 3y ago
  What “Roko’s basilisk crowd”? Afaiui, there’s no significant group that believes that the Roko’s basilisk argument actually works.
ImaCake 3y ago
That seems like a pretty accurate take on the rationalist movement though? My skepticism with it is that awareness is not enough to overcome bias.
- concordDance 3y ago
  Nah, that's not how you debias, you mostly doing by practicing calibration (give odds for things and then checking what proportion of the 90% probability things actually happened). The reasoning improvements primarily cone from a greater awareness of words (ambiguous meaning, context sensitivity, considering things as if they were statements in set theory) and tools like word tabooing or outside view. I doubt the author has had much contact with rationalists (who the hell says "rationslism"?!).

Damn, there's a lot of nonsense in that essay. "Rationalists believed that Bayesian statistics and decision theory could de-bias human thinking and model the behaviour of godlike intelligences" Lol

nullc 3y ago

That's actually the lite version told to non-believers, the full Xenu version is that they're going to construct an AI God that embodies "human" (their) values which will instantly take over the world and maximizes all the positive things, protect everyone from harm, and (most importantly) prohibit the formation of any competing super-intelligence. And if they fail to do this, all life on earth will soon be extinguished (maybe in the next few years, probably not more than 50 years), and potentially not just destroyed but converted into an unspeakable hell. An offshoot of the group started protesting the main group for not doing enough to stop the apocalypse, but has been set back a bit by the attempted murder and felony-murder charges they're dealing with. Both the protests and the murder made the news a bit but the media hasn't managed to note the connections between the events or to AI doomer cult yet. I worry that it's going to evolve to outright terrorist attacks, esp with their community moralizing the use of nuclear weapons to shut down GPU clusters... but even it doesn't its still harming people by convincing them that they likely have no future and that they're murdering trillions of future humans by not doing everything they can to stop the doom and by influencing product and policy discussions in weird ways.

turnsout 3y ago
Uh, what? Please write about this somewhere.
- jodrellblank 3y ago
  A series of blog posts about it[1]: https://aiascendant.substack.com/p/extropias-children-chapter-1-the-wunderkind https://aiascendant.substack.com/p/extropias-children-chapte... [1] I think, this is Yudkowsky, Lesswrong, MIRI, the doom cult it became, the crazy revelations like https://www.lesswrong.com/posts/MnFqyPLqbiKL8nSR7/my-experience-at-and-around-miri-and-cfar-inspired-by-zoe https://www.lesswrong.com/posts/MnFqyPLqbiKL8nSR7/my-experie... and https://old.reddit.com/r/slatestarcodex/comments/qa14kg/my_experience_at_and_around_miri_and_cfar/ https://old.reddit.com/r/slatestarcodex/comments/qa14kg/my_e...
- concordDance 3y ago
  As someone fairly close to the thing it's quite safe to ignore this guy as he doesn't have much understanding of anything in this space.
ttt11199907 3y ago
> their community moralizing the use of nuclear weapons to shut down GPU clusters Did I miss some news?
- makeworld 3y ago
  > be willing to destroy a rogue datacenter by airstrike https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/ https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...
- deleted 3y ago
  [deleted]
dylan604 3y ago
This sounds very familiar to the Republic of Texas movement in the mid-90s. The notion that Texas was not legally ratified into the Union was making its way through the courts by winning its cases. Some activists thought it was moving too slowly and skipped a few steps. A new fiat money was involved, other countries started recognizing the Republic of Texas government, liens were being placed on property, oh, and then the kidnappings started happening. The gung-ho crowd will always look like crazy types to the outsiders
reducesuffering 3y ago
> An offshoot of the group started protesting the main group for not doing enough to stop the apocalypse, but has been set back a bit by the attempted murder and felony-murder charges they're dealing with. Both the protests and the murder made the news a bit but the media hasn't managed to note the connections between the events or to AI doomer cult yet. Are you talking about anyone but the Ziz cult? They're very far removed from typical rationalist sphere think. Just because a schizophrenic psychopath reads a blog and spends all their time trying to manipulate other vulnerable isolated readers doesn't mean it's because of the bigger group they were trying to associate with. Those people are homeless self-proclaimed "vegan siths," hardly the employed MD / graduate types that ACX / lesswrong / EA is made of. If I go do something terrible, you and HN are hardly to blame.
- nullc 3y ago
  I and HN haven't been brainwashing you that the world is sure to end because of AI doom for the last 5 years. And yeah, they're extreme, but also made them more vulnerable to it and more likely to act out. (And also, thus, more likely to form a plot to murder their landlord) When some psycho goes to an AI conference with a bomb vest on and their views are directly traceable to EA and LW then it absolutely will be their fault as it is the expected outcome from years of brainwashing that AI will bring about the end of the world and potentially eternal torment. It's the expected outcome from saying that using violence to stop it would be morally justified (which is, incidentally, also inherent to the AI God story: the idea is that it will forcibly prevent the creation of other super intelligences, and not by talking people out of it, it's always been a power and control fantasy). It's unlikely to be the people at the top that will be pulling the trigger, they're going to be too busy enjoying the traditional fringe benefits of having a cult. It's the people at the bottom that haven't figured out that the purpose of any cult is locking other people into extreme views so that the people at the top of the power structure can control them and profit from them, so they can enjoy relative fame and having their interests catered to... the people who are marginal and at the edge, not too valuable to the cult themselves but electrified by its message. It'll be that marginal member who's group approved drug use wasn't kept under control, who didn't find success in the cult group sex harem, etc. It'll always be the fault of the person who pulled the trigger, but also the fault of the people who put their finger on it.
drdeca 3y ago
> that embodies "human" (their) values They have made posts (well, I can remember at least one, which I think was well-received) about how people working on AI safety/alignment should avoid having “leaving their fingerprints on the future”, meaning, making the AI aligned to specifically their values, rather than an impartial one. So, I think they generally believe they oughtn’t have it aligned specifically to their values, but that instead they should try to make it aligned with a more impartial aggregate of human values. (Though they might imagine that a “coherent extrapolation” of typical human values, will be pretty close to their own. But, I suppose most people think that their values are the correct ones, or something else playing the role of “correct” for the people who don’t think there are “correct values”.) Do they put enough of an emphasis on this? I’m not sure. I can say that I certainly don’t look forwards to the development of such a superintelligence (partially on account of how my values differ from theirs and from what I imagine to be “the average”). But, still, “aligned with some people “ still sounds a fair bit better than “not aligned with anyone”. So, I’m mostly hoping that either superintelligence won’t be developed, or that if it is, God intervenes and prevents it from causing too much of a problem.
- nullc 3y ago
  Still, when you find yourself trying to develop God in order to take over the world (or even sometimes universe), you should really be saying "Wait. Are we the baddies?" and not respond that that with "oh no, we'll give other people's ideas of 'value' consideration too, especially if they're not intellectually disabled by our judgement". > But, still, “aligned with some people “ still sounds a fair bit better than “not aligned with anyone”. I'm not sure about that. Part of the capability to do great harm is from the near miss. Fighting off Mindless grey good is better than some techbro trying to "optimize" you against your will. Of course this line of thinking requires accepting their inane premises, but it's worrysome that even if they were right the astonishing hubris would make them the existential threat more than anyone else. I think the greater relevance is the underlying unhealthy mental states underlying the doom cult. A person living a life in service of a very fringe idea we'll all likely suffer ultimate doom beyond measure unless their social group gets its way is going to exhibit all manner of compromised judgement.
concordDance 3y ago
More bullshit. The AI X-risk statements are public. > has been set back a bit by the attempted murder and felony-murder charges they're dealing with You have no idea what you're talking about.
matthewdgreen 3y ago
To add a citation to the "bombing of data centers" bit: https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/ https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...

EamonnMR 3y ago

That's a pretty fair characterization of the Roko's Basilisk crowd though, isn't it?

drdeca 3y ago
What “Roko’s basilisk crowd”? Afaiui, there’s no significant group that believes that the Roko’s basilisk argument actually works.

ImaCake 3y ago

That seems like a pretty accurate take on the rationalist movement though? My skepticism with it is that awareness is not enough to overcome bias.

concordDance 3y ago
Nah, that's not how you debias, you mostly doing by practicing calibration (give odds for things and then checking what proportion of the 90% probability things actually happened). The reasoning improvements primarily cone from a greater awareness of words (ambiguous meaning, context sensitivity, considering things as if they were statements in set theory) and tools like word tabooing or outside view. I doubt the author has had much contact with rationalists (who the hell says "rationslism"?!).