4 ms·

I've tried to read this, but somehow the article doesn't really come to a point. Is there any point besides Yud not being perfect, and having connections to peo

I've tried to read this, but somehow the article doesn't really come to a point. Is there any point besides Yud not being perfect, and having connections to people who openly talk about their sex life (and that it's somewhat non-standard)?

sauwan 16d ago
I think the point is that Yud and the rationalist community have put themselves in an awkward position where their lifestyle now depends on "p(doom) is big" being true. Which means they can't necessarily be trusted to give unbiased updates on the true level of risk we're facing as new data and information comes up. Can "ASI will inevitably kill us all" be falsifiable before we create ASI? If not, then what separates it from religion? Not sure how much I agree with that position, but that's just how I understood it.
- Kinrany 16d ago
  > Can "ASI will inevitably kill us all" be falsifiable before we create ASI? No, but there are decent proxies, like "Alice who is X standard deviations smarter than Bob will win >50% of the time at game Y even given a disadvantage that would make her chances <5% if she was playing against someone equally smart". I've thought about this analogy for less than five minutes, so it's far from perfect. Point is, ASI being made of silicon isn't very central to the problem. There are lots of analogies that are falsifiable, and "ASI will kill us all" can be decomposed into a simple and bulletproof composition of falsifiable beliefs, even if it is too specific to be itself directly falsifiable.
  - mcint 15d ago
    Uh oh, your opinions are merely part of the funnel to the sex cult. Can we really trust them? /s
- Viliam1234 16d ago
  > their lifestyle now depends on "p(doom) is big" being true How? Suppose that for the following 20 or even 100 years nothing happens. The AIs miraculously stay as they are today, the further progress turns out to be impossible, there will be no more agents hacking various websites, no cyber warfare involving AI agents, nothing. There will be the year 2026, forever. How would any of that impact the lifestyle of anyone in the rationalist community? Please be more specific with your accusation. (Seems like the rationalists are simultaneously the dark lords of TESCREAL, pulling the strings of the world's most powerful billionaires and politicians, but also irrelevant weird nerds who can be ruined at any moment by a smart blogger explaining how their opinions are silly, or even that they know each other.)
  - nullc 15d ago
    As an example, in that example who pays for the orgy compound on Telegraph Ave?
- rphv 15d ago
  > what separates it from religion? The rational methods used to establish a large P(doom). To definitively falsify the claim, humanity would need to live alongside superintelligence forever without it wiping us out. That doesn't mean we can't be pretty darn sure though, in the same way that we can be pretty darn sure that nuclear war could end civilization.
  - nullc 15d ago
    Those "rational methods" are essentially just pascal's wager. You stick some ~infinitely bad outcome (nasal daemons torturing simulations of you and everyone you love for all time) and/or ~infinitely good outcome (perfect paradise for as many sentient minds as could fit in the physical universe), then you can get whatever conclusion you want by multiplying any finite probability by these things, and even give yourself bonus asspats for being conservative by adding several order of magnitude fudge factors against your claims. The rationalist doom and sex cult uses convoluted mathematics inspired language to justify their faith-- but at the end of the day it's just faith no matter how they dress it up.
    - hollerith 14d ago
      No, it's not Pascal's wager because Pascal's wager works even when the probabilities are tiny whereas Eliezer's p(doom) is about 99% unless AI "progress" is shut down and kept shut long enough that the current crop of AI researchers is going to have to find something else to do for a career. In other words, it's not Pascal's wager if the very bad outcome is what you naturally expect to happen. And what I just wrote applies to not just Eliezer and his allies: I don't know a single person or a single published argument that has ever used Pascal's-wager-like argumentation to argue that AI "progress" needs to be stopped. Eliezer is the author of a blog post titled "Pascal's mugging", but that blog post and any follow-up discussion are completely silent on the dangers of AI.
  - hollerith 14d ago
    I wish you'd used "very large asteroid strike on Earth" rather than "nuclear war" because although nuclear war will probably make thing "uncivilized" for a while, I'm pretty sure human civilization bounces right back after even the largest realistic nuclear war.
- joshjob42 15d ago
  Why does their lifestyle require p(doom) to be large? Because they work at orgs focused on AI safety? I'm not sure what the claim is here.
  - nullc 15d ago
    Because they depend on a supply of anxiety fueled doomers to both fund their lifestyle and provide a supply of willing participants for their control and fetish games.
- tripzilch 15d ago
  you say "falsifiable" as if it holds some important key, but the concept is really not applicable to predictions that by their very nature, can only ever have one data point (doesn't mean that it's not a religion or it is, but rather I mean it being "falsifiable" or not, doesn't carry any information to that regard)
- hollerith 14d ago
  There's are many people with high p(human extinction) whose salary does not depend on their continuing to profess a high p(extinction) -- me for example. Also Yud's friends have promised to pay him what he is earning now if the organization he works for ever has to reduce his salary because the organization lacks funding. They did this to neutralize the financial incentive he would otherwise have to hesitate to announce that his p(extinction) had dropped.
xeyownt 16d ago
I tried too, but found no sentence that makes any sense. It's all gibberish from top to bottom. I'm not a native speaker but my only reaction was "wtf this is all about and wtf are you even trying to say???"
- pas 16d ago
  it's the Ed Zitron style of arguing that something big/expensive/unprecedented/hard is bad, going to fail, because [____________]. and if you put something trivially true into the blank the implication seems logical, and the argument feels powerful. but simply putting a true claim at the end doesn't mean that everything before follows from this claim. (and even if they do the argument is not a total description of all the causal chains that influence the outcomes.) of course it's hard to do a careful survey of the facts and weigh them and then deduce some likelihoods for possibilities. because some aspects are unprecedented (investment ferocity, AI results/progress) while others are very much precedented (collapse of investment bubbles, doomsayers' wrong predictions).
  - Yizahi 16d ago
    Just because something was called bad, predicted to fail and actually didn't fail, doesn't make it "definitively not bad, because it didn't fail, I told you so". Case in point - Bitfinex was and is absolutely a criminal enterprise and scam, and it didn't have any backing for years in row. But it didn't fail, because many different criminals in the world had decided that it is quite convenient tool and put a monetary IV line in it. And now ruzzians are buying kamikaze drones with it, selling oil to India with it, drug cartels are shuffling around their money with it and so on. Does it mean it wasn't and isn't bad? Of course no. They simply found a rich sponsor to bail them out. Or the whole 2008 crisis is another popular example.
    - pas 15d ago
      I'm not saying absence of evidence of wrongdoing is evidence of absence of wrongdoing. But note, you have very concrete things to claim about Bitfinex. Criminal enterprise, scam, organized crime funding, facilitating arms dealing and embargo avoidance, and so on. Then you throw in 2008. It's definitely popular but example of what? Bad thing that didn't fail? (Things did fail, hence the crisis? Or you mean the too big to fail attitude?) Policy proposals (eg. "coordinated AI safety R&D should be exempt from antitrust regulations" or "TARP made money on paper, but the financial sector should pay for the total cost of the crisis") cannot be divorced from the provenance of the source (sex cult, goldbugs, anticapitalists), but the same is true of bluesky screeds.
slowmovintarget 16d ago
Read to the end... The front matter is context to the conclusion: > This all most recently came together over the weekend, and they got Dario Amodei and Sam Altman and Elon Musk and Demis Hassabis all onto the same page to say they wanted to WAIVE ANTI-TRUST LAWS so that they could "pace", or slow down, research. > I feel about this roughly the same way I would feel about all the pharmaceutical companies getting together to say they wanted to waive anti-trust laws so they could form a cartel to ban medical research: > these people are the enemies of humanity. > Also: Much of AI research IS medical research. They are very literally proposing to stop a pretty good chunk of medical research in its tracks so that they don't have to fear the paperclipper. > There are many regulations I would be in favor of, but they only ever want to form a cartel or a world government against the paperclipper. > No other industry has ever framed any regulation in terms of "stop", "pause", or "slow down". Those are insane ways to consider any form of policy. Policy should prevent bad outcomes and encourage good ones, it is never about some specific speed. > This industry frames its regulations in this way because it has, at its core, a millenarian death cult that worships and fears AI as the cause of the end of the world, that feels compelled to create AI, and that at least somewhat wants to destroy AI as a field.
lukewarm707 16d ago
ai alignment and safety is captured by a fringe network of batshit insane rationalists. METR, AISI, MIRI, SBF, FDX, redwood research, yudkowsky, nate soares, anthropic. frankly they are connected in something almost resembling a conspiracy (XD). lower the confidence score of any institute connected on account of their conflict of interest with this religion.
washadjeffmad 14d ago
The author didn't get invited to any parties in high school, and now they're concern-mongering Yud and Buds over some nerd bacchanals. That doesn't scale well. The big question is, who's competing to sell us our salvation this time?

sauwan 16d ago

I think the point is that Yud and the rationalist community have put themselves in an awkward position where their lifestyle now depends on "p(doom) is big" being true. Which means they can't necessarily be trusted to give unbiased updates on the true level of risk we're facing as new data and information comes up. Can "ASI will inevitably kill us all" be falsifiable before we create ASI? If not, then what separates it from religion? Not sure how much I agree with that position, but that's just how I understood it.

Kinrany 16d ago
> Can "ASI will inevitably kill us all" be falsifiable before we create ASI? No, but there are decent proxies, like "Alice who is X standard deviations smarter than Bob will win >50% of the time at game Y even given a disadvantage that would make her chances <5% if she was playing against someone equally smart". I've thought about this analogy for less than five minutes, so it's far from perfect. Point is, ASI being made of silicon isn't very central to the problem. There are lots of analogies that are falsifiable, and "ASI will kill us all" can be decomposed into a simple and bulletproof composition of falsifiable beliefs, even if it is too specific to be itself directly falsifiable.
- mcint 15d ago
  Uh oh, your opinions are merely part of the funnel to the sex cult. Can we really trust them? /s
Viliam1234 16d ago
> their lifestyle now depends on "p(doom) is big" being true How? Suppose that for the following 20 or even 100 years nothing happens. The AIs miraculously stay as they are today, the further progress turns out to be impossible, there will be no more agents hacking various websites, no cyber warfare involving AI agents, nothing. There will be the year 2026, forever. How would any of that impact the lifestyle of anyone in the rationalist community? Please be more specific with your accusation. (Seems like the rationalists are simultaneously the dark lords of TESCREAL, pulling the strings of the world's most powerful billionaires and politicians, but also irrelevant weird nerds who can be ruined at any moment by a smart blogger explaining how their opinions are silly, or even that they know each other.)
- nullc 15d ago
  As an example, in that example who pays for the orgy compound on Telegraph Ave?
rphv 15d ago
> what separates it from religion? The rational methods used to establish a large P(doom). To definitively falsify the claim, humanity would need to live alongside superintelligence forever without it wiping us out. That doesn't mean we can't be pretty darn sure though, in the same way that we can be pretty darn sure that nuclear war could end civilization.
- nullc 15d ago
  Those "rational methods" are essentially just pascal's wager. You stick some ~infinitely bad outcome (nasal daemons torturing simulations of you and everyone you love for all time) and/or ~infinitely good outcome (perfect paradise for as many sentient minds as could fit in the physical universe), then you can get whatever conclusion you want by multiplying any finite probability by these things, and even give yourself bonus asspats for being conservative by adding several order of magnitude fudge factors against your claims. The rationalist doom and sex cult uses convoluted mathematics inspired language to justify their faith-- but at the end of the day it's just faith no matter how they dress it up.
  - hollerith 14d ago
    No, it's not Pascal's wager because Pascal's wager works even when the probabilities are tiny whereas Eliezer's p(doom) is about 99% unless AI "progress" is shut down and kept shut long enough that the current crop of AI researchers is going to have to find something else to do for a career. In other words, it's not Pascal's wager if the very bad outcome is what you naturally expect to happen. And what I just wrote applies to not just Eliezer and his allies: I don't know a single person or a single published argument that has ever used Pascal's-wager-like argumentation to argue that AI "progress" needs to be stopped. Eliezer is the author of a blog post titled "Pascal's mugging", but that blog post and any follow-up discussion are completely silent on the dangers of AI.
- hollerith 14d ago
  I wish you'd used "very large asteroid strike on Earth" rather than "nuclear war" because although nuclear war will probably make thing "uncivilized" for a while, I'm pretty sure human civilization bounces right back after even the largest realistic nuclear war.
joshjob42 15d ago
Why does their lifestyle require p(doom) to be large? Because they work at orgs focused on AI safety? I'm not sure what the claim is here.
- nullc 15d ago
  Because they depend on a supply of anxiety fueled doomers to both fund their lifestyle and provide a supply of willing participants for their control and fetish games.
tripzilch 15d ago
you say "falsifiable" as if it holds some important key, but the concept is really not applicable to predictions that by their very nature, can only ever have one data point (doesn't mean that it's not a religion or it is, but rather I mean it being "falsifiable" or not, doesn't carry any information to that regard)
hollerith 14d ago
There's are many people with high p(human extinction) whose salary does not depend on their continuing to profess a high p(extinction) -- me for example. Also Yud's friends have promised to pay him what he is earning now if the organization he works for ever has to reduce his salary because the organization lacks funding. They did this to neutralize the financial incentive he would otherwise have to hesitate to announce that his p(extinction) had dropped.

xeyownt 16d ago

I tried too, but found no sentence that makes any sense. It's all gibberish from top to bottom. I'm not a native speaker but my only reaction was "wtf this is all about and wtf are you even trying to say???"

pas 16d ago
it's the Ed Zitron style of arguing that something big/expensive/unprecedented/hard is bad, going to fail, because [____________]. and if you put something trivially true into the blank the implication seems logical, and the argument feels powerful. but simply putting a true claim at the end doesn't mean that everything before follows from this claim. (and even if they do the argument is not a total description of all the causal chains that influence the outcomes.) of course it's hard to do a careful survey of the facts and weigh them and then deduce some likelihoods for possibilities. because some aspects are unprecedented (investment ferocity, AI results/progress) while others are very much precedented (collapse of investment bubbles, doomsayers' wrong predictions).
- Yizahi 16d ago
  Just because something was called bad, predicted to fail and actually didn't fail, doesn't make it "definitively not bad, because it didn't fail, I told you so". Case in point - Bitfinex was and is absolutely a criminal enterprise and scam, and it didn't have any backing for years in row. But it didn't fail, because many different criminals in the world had decided that it is quite convenient tool and put a monetary IV line in it. And now ruzzians are buying kamikaze drones with it, selling oil to India with it, drug cartels are shuffling around their money with it and so on. Does it mean it wasn't and isn't bad? Of course no. They simply found a rich sponsor to bail them out. Or the whole 2008 crisis is another popular example.
  - pas 15d ago
    I'm not saying absence of evidence of wrongdoing is evidence of absence of wrongdoing. But note, you have very concrete things to claim about Bitfinex. Criminal enterprise, scam, organized crime funding, facilitating arms dealing and embargo avoidance, and so on. Then you throw in 2008. It's definitely popular but example of what? Bad thing that didn't fail? (Things did fail, hence the crisis? Or you mean the too big to fail attitude?) Policy proposals (eg. "coordinated AI safety R&D should be exempt from antitrust regulations" or "TARP made money on paper, but the financial sector should pay for the total cost of the crisis") cannot be divorced from the provenance of the source (sex cult, goldbugs, anticapitalists), but the same is true of bluesky screeds.

slowmovintarget 16d ago

Read to the end... The front matter is context to the conclusion: > This all most recently came together over the weekend, and they got Dario Amodei and Sam Altman and Elon Musk and Demis Hassabis all onto the same page to say they wanted to WAIVE ANTI-TRUST LAWS so that they could "pace", or slow down, research. > I feel about this roughly the same way I would feel about all the pharmaceutical companies getting together to say they wanted to waive anti-trust laws so they could form a cartel to ban medical research: > these people are the enemies of humanity. > Also: Much of AI research IS medical research. They are very literally proposing to stop a pretty good chunk of medical research in its tracks so that they don't have to fear the paperclipper. > There are many regulations I would be in favor of, but they only ever want to form a cartel or a world government against the paperclipper. > No other industry has ever framed any regulation in terms of "stop", "pause", or "slow down". Those are insane ways to consider any form of policy. Policy should prevent bad outcomes and encourage good ones, it is never about some specific speed. > This industry frames its regulations in this way because it has, at its core, a millenarian death cult that worships and fears AI as the cause of the end of the world, that feels compelled to create AI, and that at least somewhat wants to destroy AI as a field.

lukewarm707 16d ago

ai alignment and safety is captured by a fringe network of batshit insane rationalists. METR, AISI, MIRI, SBF, FDX, redwood research, yudkowsky, nate soares, anthropic. frankly they are connected in something almost resembling a conspiracy (XD). lower the confidence score of any institute connected on account of their conflict of interest with this religion.

washadjeffmad 14d ago

The author didn't get invited to any parties in high school, and now they're concern-mongering Yud and Buds over some nerd bacchanals. That doesn't scale well. The big question is, who's competing to sell us our salvation this time?