24 ms·

Effective Altruism’s obsession with AI safety helps bury bad behavior

141 points by polrjoy 4y ago

neonate 4y ago
https://archive.ph/sLihW https://archive.ph/sLihW
braingenious 4y ago
I was not aware that there is an Atlas Shrugged-length Harry Potter fanfiction about rationality. Learn something new every day!
- petesergeant 4y ago
  It’s a very, very entertaining read, although it loses its way and loses steam a bit as it goes on
  - SiempreViernes 4y ago
    I think the first chapter is really fun, then it loses steam and by something like chapter six it has fukky jettisoned the promise chapter one sets up and it is strictly in "ok to terrible" territory for the rest of it.
- khazhoux 4y ago
  YMMV. I tried to finish it based on a strong trusted recommendation, but could not get past the insufferable author-inserts, and the unpleasant and unbelievable hero character.
  - flangola7 4y ago
    What's the name of it?
    - karaterobot 4y ago
      Harry Potter and the Methods of Rationality. I'm glad such a thing exists, but I could never work up the interest to read it myself. https://en.wikipedia.org/wiki/Harry_Potter_and_the_Methods_of_Rationality https://en.wikipedia.org/wiki/Harry_Potter_and_the_Methods_o...
      - pksebben 4y ago
        wait, really? I scrolled right past someone posting about "Harry Potter rationality fanfiction" and thought, "huh, oddly specific burn but it's chuckle worthy". it's an actual thing? This is a joke, right? tongue in cheek or something. Or should I go get some Brawndo to water my basil...
        HDThoreaun 4y ago
        It's not only a thing, it's almost 2000 pages long.
  - kimixa 4y ago
    I too didn't finish it - assuming it's "Harry Potter and the Methods of Rationality". I got caught up by the multiple times the main character made sweeping assessments of things with insufficient information, and just happened to be right by accident (or author fiat I guess). The main characters figure things out by Thinking Really Hard, and experimentation is only ever done to prove they were right all along to other characters. That's an arrogant character who keeps falling into the right answer, not a scientist.
    - SamReidHughes 4y ago
      It went above your head.
      - operatingthetan 4y ago
        It went above their head because it was poor writing in their view? Why would you think taste in writing has anything to do with their ability to understand the points the writer was trying to make? This seems unecessary and you are probably capable of coming up with a more interesting response.
        SamReidHughes 4y ago
        It was supposed to be a flawed protagonist.
        operatingthetan 4y ago
        It's commonly thought (and obvious) that Yud wrote the book about himself so I doubt it. It doesn't really fit the rationalist playbook either, they take themselves excessively seriously.
        SamReidHughes 4y ago
        The character's author-insert aspects are self-parody.
  - favaq 4y ago
    Really, you found it surprising that a book written by the insufferable author of the insufferable blog LessWrong is insufferable?
- idlewords 4y ago
  It's not just Atlas Shrugged-length, but Atlas Shrugged-quality!
  - precompute 4y ago
    I disagree, Atlas Shrugged was much better, used an original premise and had an interesting phrase throughout ("Who is John Galt?"). HPMOR is really garbage-tier. It's parasitic on another series, doesn't have much in the way of a good story, and quite legitimately makes shit up. There are much better HP fanfics (and indeed books) written by people that are not as "smart" as Yudkowsky and also didn't have an agenda to push.
- SiempreViernes 4y ago
  It's not really about rationality past the first four chapters or so, after that I don't really know what its about but when part of the plot is helping Voldemort break people out of azkeban then rationality is clearly not an important theme.
noloblo 4y ago
@shervin @pmarca: Utilitarianism is bad: It encourages mortal men to play God at levels of complexity that they can't possibly comprehend. It should be banned for the greater good >> “earning to give,” the idea that people like Bankman-Fried should do whatever it takes to make a lot of money so they can give it away. To amass his billions, Bankman-Fried allegedly defrauded his customers, and critics have said his downfall shows that EA is vulnerable to a myopia that allows the ends to justify illegal means. Among EAs themselves, however, the most salient criticism is subtler: that living at the logical extremes of the ideology is impractical and a recipe for misery. In buddhism https://en.wikipedia.org/wiki/Middle_Way https://en.wikipedia.org/wiki/Middle_Way middle way is advised for this very reason
- drdeca 4y ago
  Utilitarianism is false, but I think it is a very useful heuristic, where I think it being used (as a heuristic) more often would probably be beneficial at the margins. There is no canonical way the aggregate utility between persons, and people probably don’t actually have utility functions. And, there are moral laws which should have at the least some weight when weighed against a “but this action seems to have better outcomes overall, on average”, and possibly infinite weight when weighed against such things. Nevertheless.
deleted 4y ago
[deleted]
yanderekko 4y ago
[flagged]
- deleted 4y ago
  [deleted]
- elmomle 4y ago
  Thanks for sharing that reference. I started out being intrigued, but he totally lost me with some pretty unsubstantiated absolutist statements, e.g. "anyone smart and independent-minded will have views on race and sex well outside the liberal mainstream" and "men are sure to be a disproportionate share of leaders for biological reasons". The first statement is categorically false or otherwise implies the author's engaging in "no true Scotsman" fallacy; and, in the second quote, how he can disentangle biological from cultural and social causes with such absolute conviction is mystifying. Stating opinions as facts in such a way is a kind of sophistry, and ironically this tendency limits one's ability to reach deeper truth.
  - operatingthetan 4y ago
    The photo of Aella in a bra with the caption "The rare rationalist female" wasn't a red flag? More seriously, yes that piece is dripping with sexism. >Women will never make up a significant portion of self-made billionaires or outstanding philosophers, but they can get a majority of BAs. Academia practices affirmative action in everything from admissions to journal submissions and citations practices, artificially reducing the natural male advantage at higher levels of ability.
    - elmomle 4y ago
      Yeah, that was also a massive red flag once I got to it, but I wanted to give a more considered critique.
  - yanderekko 4y ago
    >how he can disentangle biological from cultural and social causes with such absolute conviction is mystifying. He doesn't have to disentangle the two completely, he just strongly believes that there's a non-trivial biological contribution to these outcomes. Which already puts his views "well outside the liberal mainstream".
Nexialist 4y ago
If you'd like to read more about the roots of Effective Altruism, ranging from old mailing lists, LessWrong, bitcoin, cults, AI and the surprising connections to many well known figures in tech, I highly recommend the 7-part Extropia's Childen series from Jon Evans [1] 1. https://aiascendant.substack.com/p/extropias-children-chapter-1-the-wunderkind https://aiascendant.substack.com/p/extropias-children-chapte...
- stubybubs 4y ago
  Golly. I'm on part 2 and it sounds like they're about to invent Scientology. > The full post goes into more depth and nuance, but it's worth stressing how heavily Leverage relied on “debugging,” based on Anders's “Connection Theory.” Debugging consisted of — to paraphrase and oversimplify — opening up one's psyche, history, self, and emotional core to maximum vulnerability, generally to one's hierarchical superior, and then doing all you could to follow their suggestions to “fix” your mind.
  - painelemental 4y ago
    Thats nothing. Wait till Epstein enters the picture. https://fredwynne.medium.com/an-open-letter-to-vitalik-buterin-ce4681a7dbe https://fredwynne.medium.com/an-open-letter-to-vitalik-buter... Any hoo - this stuff is the tip of the iceberg.
    - aoeusnth1 4y ago
      Incredible letter, I had no idea about the SWATing. Tbh the Epstein donation seems like a nothingburger compared to the other allegations in that letter. Edit: more context on Ziz, http://zizians.info/ http://zizians.info/
    - j_crick 4y ago
      Admittedly I don’t follow this rationalist thing on the internet at all, so I don’t know what’s going on with them. But that was hell of a read. I am no stranger to people being dicks, but still, that all is just sick.
vishal0123 4y ago
This article is such a mess. If you are alleging something, then say it clearly. Judging by my participation in lesswrong and others, there are both feminists and sexist people, just like all other online forums. People are more free to be sexist online and every close knit online community has fair share of sexists.
- Barrin92 4y ago
  The EA people seem to have more than a spurious relationship with polygamy, drugs, orgies and odd-ball ascetic leader figures, doomsday prophecies and psychotic behavior. Taken from the article " MIRI employee Jessica Taylor had a job that sometimes involved “imagining extreme AI torture scenarios,” as she described it in a post on LessWrong—the worst possible suffering AI might be able to inflict on people. At work, she says, she and a small team of researchers believed “we might make God, but we might mess up and destroy everything.” In 2017 she was hospitalized for three weeks with delusions that she was “intrinsically evil” and “had destroyed significant parts of the world with my demonic powers,”" This is like what, one bad week away from AI Jonestown? At some point this isn't a community but a cult, and women usually tend to draw the shortest sticks in any cult environemnt with bizarre moneyed, male leader figures.
  - yanderekko 4y ago
    >This is like what, one bad week away from AI Jonestown? Look at the Ziz story - we're already there. (https://medium.com/@sefashapiro/a-community-warning-about-ziz-76c100180509 https://medium.com/@sefashapiro/a-community-warning-about-zi...) However, this should not severely impugn the community as a whole.
js8 4y ago
I have always felt that the embrace of "AI can be dangerous" comes from an elitist fear of capitalist class that the "unwashed masses" will have too much access to technology and will make capital, as well as labor, obsolete. I mean if everybody has a super-smart personal assistant in their home, who thinks independently, why would they need or want a ruling class, when we can simply have democratic socialism?
- space_fountain 4y ago
  What about a future where only energy and resources are important leads you to think we’ll have less inequality? Brains are the only thing that don’t flow to the wealthy right now
  - js8 4y ago
    Human nature. If you're a part of the common tribe, people are actually pretty good. There doesn't seem to be an upper limit on the size of the tribe (although some people just wanna hate somebody). You can also look at it this way. The only real objection against socialist societies was that people are lazy, and don't care enough about producing stuff or resolving economic inefficiencies. But with AI, this objection is gone. Nobody really wants to be in a constant "resource competition" or whatever, it's an effort. People generally want to have fun and enjoy life.
    - space_fountain 4y ago
      Humans ability to push others out of the circle of concern is near limitless. We literally bought and sold people. I think there is reason for hope. We stopped doing some of those terrible things and only partially for economic reasons, but ai still fills me with some great fear. It shrinks the circle that the wealthy absolutely must keep in their circle, in their tribe, to go on and live happy lives and people of the past have been uncomfortably good at excluding people from the circle in the past. Basically it might not destroy us, but I certainly don’t see it as inevitable that it will bring us to utopia
dr_dshiv 4y ago
1. “Moral computation” was coined by Francis Hutcheson (doctoral advisor of Adam Smith). 2. I like hearing rationalists describe “demon summoning is easy, Angel summoning is hard.” Genuinely curious about that.
- xkcd-sucks 4y ago
  2. "An ounce of prevention is worth a pound of cure"; "measure twice, cut once"; "it's easier to fuck it than to unfuck it"; etc. Not exactly novel
  - dr_dshiv 4y ago
    To clarify— I mean the rationalist reference to angels and demons. It seems to be more than metaphorical.
    - scarmig 4y ago
      Angel here would be an "aligned" AI: an AI whose behavior is in line with human values. A demon is simply an unaligned one. The key point would be that it needs not be actively hostile to human values, but indifferent to them. Think of a Lovecraftian Elder God, not AI Hitler. The supporting hypotheses would be that it's much easier to create an unaligned AI than an aligned one, as an aligned one is an unaligned one with additional constraints, and that unaligned AIs will wreak massive destruction.
jasonhansel 4y ago
Reminder that one of EA's core supporters is a fan of scientific racism: https://twitter.com/ArsonAtDennys/status/1362153191102677001 https://twitter.com/ArsonAtDennys/status/1362153191102677001 For some more context: https://rationalwiki.org/wiki/Scott_Alexander#Race_and_IQ https://rationalwiki.org/wiki/Scott_Alexander#Race_and_IQ
- deleted 4y ago
  [deleted]
- postultimate 4y ago
  Good for him. Nobody's racism should be unscientific.
skybrian 4y ago
In case you're thinking you saw this before, there was similar article a month ago in Time: https://news.ycombinator.com/item?id=34646361 https://news.ycombinator.com/item?id=34646361
- d0mine 4y ago
  It looks like a hit piece. Someone is against rationality regarding the question of AI safety (any real practical consequences is to censor AI under _pretense_ of AI safety, Perhaps, the articles are the preventive strikes to discredit the AI safety experts that might point out the obvious).
  - skybrian 4y ago
    That's a rather specific accusation to make without evidence. Possibly, a journalist might have other reasons to cover a story? Not necessarily good ones, but still...
    - d0mine 4y ago
      The article itself is the evidence: read it and apply common sense. I know almost nothing about EAs but what I know makes me think that it is likely P(SA)>P(SA|EA)>0.
neilv 4y ago
Please capitalize it as Effective Altruism. The generic terms "effective" and "altruism" are both very important terms, and this doesn't follow from combining them.
- dang 4y ago
  Ok, we've given it an A!
renewiltord 4y ago
What is one or two rapes or sexual assaults against trillions of future lives? With sufficient future sight and the power of the exponent > 1, one can justify many present actions. Sadly, the EAs are small thinkers. True grand scale actions require truly grand visions https://cosmicbraintrust.org/ https://cosmicbraintrust.org/
- deleted 4y ago
  [deleted]
- eldenring 4y ago
  what an evil and disgusting comment.
  - renewiltord 4y ago
    I deserve this since sarcastic comments are really the lowest class of Internet comments.
    - eldenring 4y ago
      oh god sorry I didn't even realize. Its a bit sad that nowadays a comment like that can even be mistaken for a serious one.
- d0mine 4y ago
  The real and the only relevant question is: do EAs rape/sexual assault more than the the average population (and if the answer is no (it is), then we need more EAs).
AbrahamParangi 4y ago
AI safety is an apocalypse cult and we should recognize that everyone who has ever been in an apocalypse cult thinks that "no, this time it's really the end of the world". Everyone who believes crazy shit thinks they're being reasonable, so just because we're super sure we're being reasonable doesn't mean what we believe isn't crazy. Do we believe that we alone, of all people who have believed in the coming end, have it right this time?
- JamesBarney 4y ago
  There have been people for a long time who think the world is going to end. We can use your argument to dismiss anyone who fears any existential risk like asteroids, nuclear war, climate change, AI, etc... And since the world exists every group will be wrong except one. I think the risk of human extermination is much lower than many in this group believe because I think AI is much more compute bound than algorithm bound. So I think a hard takeoff is very unlikely. But while the median case for climate change is probably worse than the median case for AI. The existential risks for AI seem much higher. And it's a concern worth taking seriously.
  - esperent 4y ago
    The issue is that the effective altruism crowd are focused exclusively on AI safety, at least in what they publish online. If they talked about all possible threats to humanity and how to solve them, from AI, to climate change, to nuclear bombs, to asteroids, to disease, they would be an interesting and important group. They don't. They hardly talk about anything else except AI. From the outside it looks a lot like an obsession. Personally I think they treat AI as an existential threat above all others because they consider themselves to be the pinnacle of intelligence in the universe and AI is showing that human intelligence might not be so special after all. So it's an existential threat to their sense of self. Other existential threats merely threaten their lives and are therefore not as important. This would be fine, if acknowledged with a degree of self awareness. But it's not, instead it's wrapped up in so many layers of convoluted logic that most people, insiders and outsiders both, can't see the real reason for this obsession.
painelemental 4y ago
https://fredwynne.medium.com/an-open-letter-to-vitalik-buterin-ce4681a7dbe https://fredwynne.medium.com/an-open-letter-to-vitalik-buter... https://sinceriously.fyi/brent-dill-confessions-full-redacted/ https://sinceriously.fyi/brent-dill-confessions-full-redacte... This entire social milieu is a goddamn horror show.
Animats 4y ago
Some of that is just typical cult behavior. Scientology and Ayn Rand followers have had similar problems. Drugs usually make it worse. AI safety is a real issue, but look at what it looks like so far. - "Machines should think, people should work". That's Marshall Brain's "Manna". It's also working at an an Amazon fulfillment center, where the computers tell the humans what to do. - "Paperclip maximizer". A corporation is a paperclip maximizer. If you take Milton Friedman seriously, that's what a corporation is supposed to do. As I point out occasionally, most of the ethical problems foreseen for AIs are known ethical problems with corporations. Those are the classic problems. As the technology advances, we're discovering new ones in the large language model space. Those issues are well known here.
- yanderekko 4y ago
  >- "Paperclip maximizer". A corporation is a paperclip maximizer. If you take Milton Friedman seriously, that's what a corporation is supposed to do. As I point out occasionally, most of the ethical problems foreseen for AIs are known ethical problems with corporations. Corporations are not capable of having the "fast takeoff" which is an integral part of the doomer narrative here. Furthermore, a corporation being a paperclip maximizer is an abstraction that obviously falls apart in the extremes, if for no other reason than because at some point destroying the world for the sake of more paperclips is going to have a negative ROI.... Meanwhile, the (AI) paperclip maximizer is actually trying to maximize paperclips and does not have the sort of internal checks that would apply to real-world orgs. That is the crux of the problem.
  - roenxi 4y ago
    Corporations, when first developed, took out large chunks of Asia before anyone could stop them. The main difference is that corporations have to be powered by people while an AI could theoretically work with complete independence.
    - humanrebar 4y ago
      I think the argument is that due to emergent behavior and possibly perverse incentives corporations are indistinguishable from slow-moving AI -- they can be treated like independent agents. Elsewhere in the thread, it's brought up that AI could move faster than corporations generally move. That might be true, but there are still physical limits: the speed of light, moving mountains of steel takes energy, etc.
5cott0 4y ago
what’s so bad about reading some Harry Potter fanfic in the polycule?
- Vecr 4y ago
  I'm assuming you read it, so why does Harry Potter (a 11 year old in 1991/1992) think polyamory might (or probably would be, looking at the text right now) be a good idea in a future where people live a long time, in said fanfic? I think it's pretty dubious, but the other 11 and 12 year old girls in the same fanfic also seem to be fans of the idea of yaoi like situations, so all semblances of actual historical events are probably gone at that point. This is the UK in 1992, not the US West Coast. To give my personal impression, when I read some science fiction that had some suggestions of polyamory in it in a similar situation, I did not think it was a good idea. I know the whole thing is essentially a morality story and for various plot and author-community communication reasons the "Harry Potter" in the story does not behave like J. K. Rowling's Harry Potter with different adoptive parents nor like any person, child or otherwise, could behave in 1991 and 1992 within the realm of probability, but I think the whole thing could be done a lot better even given the various constraints. How about he actually does science experiments for more than two chapters instead of going on stupid adventures that should probably be left to the JKR books. Cut the time travel as well, figure out some other reason for constraint solving to show up.
  - HDThoreaun 4y ago
    Harry has voldemort's soul in him the HPMOR which is the explanation given for why he acts like an evil adult.
    - Vecr 4y ago
      Yes, spoilers though, but I don't actually think that explains it. Are there any indications in cannon that Tom Riddle was ever interested in polyamory? I'm pretty sure voldemort is not interested in other people at all after his soul gets all divided, but I don't know about before that. Riddle is not from the US west coast either, so it's kind of dubious.
puzzledobserver 4y ago
I had previously heard of EA, and knew about paperclip maximizers, trolley problems, unintended consequences, and other risks associated with unregulated AI. But I wasn't aware of the connections between the two communities / families of ideas until recently. Because I don't consider myself sufficiently educated in these matters, I default to using GiveWell and their mosquito net recommendations as an effective low-effort approach to making monthly contributions to charity. Overall, is there anything about this approach which is objectionable? How do philosophers go from something like GiveWell to something like fending off the AI apocalypse? Is this ideological expansion inevitable?
- TheDong 4y ago
  EAs premise includes basically "all human lives are equal. Saving one stranger's life is just as valuable as saving your father's life (more valuable actually if the stranger has more years left than your dad), so you should donate your kidney to a stranger, not your father" "All human lives are equal" then expands to "and there are more unborn lives than there are currently living lives, so it's more important to save unborn kids in 200 years by diverting AI catastrophe now". I personally don't think unborn lives are worth anything, and as such I don't follow that jump. I don't feel a need to justify it either. I think there's two additional easy ways to veer off the EA course. The first is "how do we know EA is right about the charities?" The answer you'll get is basically "Sure, you can donate a kidney to your father and know it saved his life, you can verify that... but the people verifying your EA kidney donation does more good are smarter than you. They're rationalists. They do math. They know selling your kidney, which would have only saved one life, to a black market and getting enough money for 50 mosquito nets, which will save 20 lives, is the right deal. The spreadsheet says so." I do not buy that the people running and choosing EA charities are actually the smartest people, especially when it comes to having a positive impact on lives. Sure, they talk a big talk, but I know for a fact that when I give money to a local shelter, the money gets used for good. I know for a fact that when I donated to a certain EA charity I won't name, the money went into "AI research" which is to say into the pockets of a rich white dude. I know for a fact that the average EA rationalist I have met does not come off as a caring human. A second way that one can veer off the EA course is by deciding that one values some lives more than others. For example, if I live in a small city, I may decide I value the lives of orphans in this city more than the lives of the orphans in some third-world country. Sure, feeding an orphan in africa is cheaper so EA would say do that, but I may decide that the intangible value of improving my local community is worth more than "getting the most bang for my buck". I think this is also a totally reasonable argument, and in fact I would much rather live in a world where everyone tried to improve their local community than one where everyone used a spreadsheet to try and maximize their "best improvement per dollar".
bloppe 4y ago
EA as a concept is totally fine and the risk of runaway AI still seems rather fantastical to me. Clearly a more pressing problem is runaway egos latching onto these ideas to inflate themselves to deadly girths. I'm sure some people can practice a totally constructive form of EA. But if you structure your entire identity around it, clearly you've drifted somewhat from rationality.
Towaway69 4y ago
I'm surprised that the term Longtermism hasn't been mentioned. Isn't Nick Bostrom also a longtermist? It's probably the reason for the focus on AI. Also wasn't he one of the founders of Longtermism?
thom 4y ago
EA is at least partly a moral and ethical pyramid scheme. You get to be noble and far-sighted, do relatively little real work, and you’ll never be judged on the outcomes because of the long timelines. And so you rope in more and more acolytes who both increase and accrue these same benefits. It seems unlikely that the fall from grace of prominent figures will affect the movement, given that it’s partly based on cognitive dissonance as it is. All that said, the amount of energy we expend on AI safety probably shouldn’t be zero. But I choose to believe that if Roko’s basilisk is going to torture us over anything, it’ll be that we didn’t raise up millions of AI scientists from the cotton fields and sweatshops. Credit to the less prominent altruists doing that, or who at least have some form of compound interest built into their utility function which compels them to be kind, now.
- dbspin 4y ago
  Roko's basilisk makes no sense. Sans magic physics time travel - there's no motivation. One thing all A.I. is, no matter how nefarious, is goal directed. It might have poorly aligned goals, absurd goals, dangerous goals, goals it misunderstands, goals that contradict etc. But ultimately it proceeds toward some form of goal directed reward function. There's zero reward in punishing someone for past behaviour - except in the context of altering their future behaviour. Since the A.I. would have to already exist (and there's no reason to believe it would be incentivised to even encourage humans to create other, rival A.Is, with potentially differing goals) before it could punish anyone - Q.E.D.: Roko's basilisk has essentially zero likelihood. Worse - it's an essentially christian eschatological idea. What is this hypothetical A.I. but an avenging angel, a righteous judge at the end of time? It's pure religion. Dark, manipulative, blood sacrifice religion at that.
  - PartiallyTyped 4y ago
    > There's zero reward in punishing someone for past behaviour A similar argument regarding Roko's Basilisk is that it is effectively a prisoner's dilemma, where the Basilisk torturing us and us not helping it is the Defect-Defect scenario. It makes no sense for either party to defect. > except in the context of altering their future behaviour. It really seems like a very contrived version of Pascal's mugging.
arisAlexis 4y ago
but EA is at least doing something about a huge problem we will face.
- khazhoux 4y ago
  What are they actually doing? Honest question, because I don't know.
  - nyfern 4y ago
    Purchasing large mansions[0], chateaus[1], and spending lavishly on administrative kickbacks[2] because longtermism can justify any present action if you extend your horizon far enough. EA is effectively two disconnected ideas at this point. [0] https://forum.effectivealtruism.org/posts/xof7iFB3uh8Kc53bG/why-did-cea-buy-wytham-abbey https://forum.effectivealtruism.org/posts/xof7iFB3uh8Kc53bG/... [1]https://www.forbes.com/sites/sarahemerson/2023/02/28/effective-altruism-linked-group-used-millions-donated-by-ftx-to-buy-a-czech-castle/?sh=640b92a86baf https://www.forbes.com/sites/sarahemerson/2023/02/28/effecti... [2]https://intelligence.org/wp-content/uploads/2022/12/Independent-Auditors-Report-for-2021.pdf https://intelligence.org/wp-content/uploads/2022/12/Independ...
    - deleted 4y ago
      [deleted]
    - jefftk 4y ago
      > spending lavishly on administrative kickbacks I just skimmed the report you link and nothing jumped out at me as matching this -- could you be more specific?
  - jefftk 4y ago
    Here are some EA highlights: * Founding and scaling GiveWell, synthesizing and analyzing global poverty research to make recommendations. This has helped guide about $1B+ in really valuable work, the largest portion in anti-malarial bednets. https://www.givewell.org/about/impact https://www.givewell.org/about/impact * Convincing Dustin Moskovitz and Cari Tuna to allocate their giving along these lines, building Open Philanthropy which has made a lot of grants that look really valuable https://www.openphilanthropy.org/grants https://www.openphilanthropy.org/grants (this is in addition to the ones where they granted at GiveWell's recommendation -- don't want to double count). * Michael Kremer won a Nobel prize in econ (shared with Esther Duflo and Abhijit Banerjee who I'd describe as EA-sympathetic but probably don't consider themselves EAs?) for global poverty research. * Built an "AI Safety" subfield of computer science, with the 2016 paper "Concrete problems in AI safety" having 1.8k citations. * Founding and scaling 80,000 Hours, helping people figure out how to use their careers to do more good. This includes looking into potential career opportunities, writing up public advice, and 1:1 advising, and 1k+ people have cited them as part of why they changed what they were working on. https://80000hours.org https://80000hours.org (Not especially representative; there's been thousands of people working along these lines for years and I'm sure I'm missing a lot of valuable work.)
arisAlexis 4y ago
Top level scientistis warning of this apocalypse. Sam Altman which was the CEO of the very site you are posting and the CEO of OpenAI has warned starkly that there is a probability that humanity will go extinct from AI in his latest interview. Have you seen it? You think he is somehow bluffing or is he part of the same cult?
- bagels 4y ago
  What scientists?
  - AgentME 4y ago
    A recent poll of published AI researchers showed a median estimate of a 5% chance AI causes human extinction, with half believing the chance is greater than 10%. https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/ https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/ A poll of NLP researchers showed 37% agreed that catastrophic outcomes of AI comparable to nuclear war were plausible. https://arxiv.org/abs/2208.12852 https://arxiv.org/abs/2208.12852
    - pksebben 4y ago
      it's a little annoying that the "AI Impacts" paper seems to be three MIRIs in a trenchcoat, considering that this article is about EA's AI agenda and by extension Singularity/MIRI. There is a slight conflict of interest here - as the publishers of the paper are also taking funding specifically to "deal with the threat" they're publishing on.
      - arisAlexis 4y ago
        Yes if we have scientists that are getting paid to research a certain risk it’s inevitable that they have conflict of interest. Otherwise how would it be done?
        pksebben 4y ago
louwrentius 4y ago
It’s interesting to me that the comments of the HN crowd seem to focus on Effective Altuism and AI safety. A ton of the article is about immoral behavior of men in positions of power towards women. And the misogyny. > The underlying ideology valorizes extremes: seeking rational truth above all else, donating the most money and doing the utmost good for the most important reason. This way of thinking can lend an attractive clarity, but it can also provide cover for destructive or despicable behavior. I think this quote really sums things up for me.
fluxinflex 4y ago
I doubt very much that the poorest people on this planet care much about AIs. People living in absolute poverty in Africa, India, China etc. I also doubt they would be - negatively - affected if a rogue AI would turn out the lights in the electronicified world. Places where AIs aren't used to command peoples lives will probably not even realise that an AI apocalypse had happened. Perhaps a case of western navel-gazing?
- dotsam 4y ago
  > I doubt very much that the poorest people on this planet care much about AIs. Whether people know or care about a risk doesn't mean that the risk won't affect them! > I also doubt they would be - negatively - affected if a rogue AI would turn out the lights in the electronicified world. A rogue AI might as well just kill everyone rather than just the Western naval-gazers, as this would bring to 0 the possibility that a human could ever turn it off / thwart its plans.
  - 0xf00ff00f 4y ago
    > A rogue AI might as well just kill everyone How would it do that?
    - polski-g 4y ago
      There is a great documentary about this that came out a few years ago: https://www.youtube.com/watch?v=5UgPJhyJmlM https://www.youtube.com/watch?v=5UgPJhyJmlM
    - dotsam 4y ago
      Get access to the internet, get money. Money buys it power to get people to do things in the real world. Or it can hack through systems and blackmail and coerce people. Now it can act in the real world by getting people to do things it wants. As one example it could design, synthesize and spread a highly contagious and lethal pathogen.
- CatWChainsaw 4y ago
nathan_young 4y ago
According to a study of 700 ML researchers (not AI safety) the median chance that that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)" is 5%. And 48% of respondents gave an answer of 10% or more. Even if this is 2 orders of magnitude too high it's worth a lot of our time. But these are probably mainly normy researchers not AI safety-ists. https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#Data https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#...
- LatteLazy 4y ago
  According to a survey 700 ML researchers, we should throw money at their subject or we might all die...
  - PartiallyTyped 4y ago
    With great power comes great responsibility. If we start plugging ML everywhere - which we are currently doing - we better be sure that it is behaving as expected.
    - deleted 4y ago
      [deleted]
    - LatteLazy 4y ago
      So far we have seen no power at all from ML. So spending large amounts of money making it "safe" (not that anyone knows how or even what it would look like) seems like an over reaction. With no power, comes no responsibility. But of course, if ML is a person's pet interest, they will likely think we should increase spending on it. That doesn't make that a smart decision...
      - PartiallyTyped 4y ago
        I believe that you conflate “power” with capabilities wrt AGI. I’d argue that power is in relation to reach and the capacity to influence. In this regard, ML has quite a bit of power these days, from deciding whether you are allowed to take a loan, to driving engagement in social media, to predicting and influencing what you will watch, to the way our language is even used. There are also quite a few implicit ways that ML models have influenced society, eg GPGPUs with emphasis on ML, and ML accelerators everywhere. The effectiveness of certain algorithms made us use them more often which influenced NVDA and Google towards developing hardware to accelerate those common cases thereby creating a feedback loop where our algorithms are chosen based on works well on our GPUs (cf transformers). As such systems become more prevalent and influential, ensuring safety and explainability will help us prevent pitfalls that put humans at risk.
ShredKazoo 4y ago
See discussion on the EA Forum https://forum.effectivealtruism.org/posts/7b9ZDTAYQY9k6FZHS/abuse-in-lesswrong-and-rationalist-communities-in-bloomberg#comments https://forum.effectivealtruism.org/posts/7b9ZDTAYQY9k6FZHS/...
whywhywhywhy 4y ago
It's just super obvious by this point that none of the AI safety LARP is really about safety it's more about justifying why a technology that to hire the team to work on it requires that work to be published (Apple tried to do it without publishing and struggled to get the talent) but by publishing you lose part of your advantage so the only way to lock that advantage up is to make it so use of the models has to go through you and also trying to push for reasons for others to not have access to the tech required to train their own model. If you disagree then why is OpenAI's safety committee talking about how you should need a government contract to buy an A100.
- kelipso 4y ago
  I'm leaning towards this as a big reason. PhDs really want to publish and they tend not to put as much effort into or not find interesting work that cannot be published. The trend in academia is towards making your code and data available but now you can now get around that by saying AI safety.
- pksebben 4y ago
  | If you disagree then why is OpenAI's safety committee talking about how you should need a government contract to buy an A100. Fuck, really? I hadn't heard about this. If true, it's the godfather of all ladder pulls. Not to mention, it's categorically awful to silo this stuff and grant it to an org that's so opaque. Good luck separating the wheat from the propaganda chaff without access to the tools that produce it.
YeGoblynQueenne 4y ago
[flagged]
LatteLazy 4y ago
Effective Altruism, like and philosophy is (a) only as good as it's practitioners and (b) often used as a cover for charlatans. That doesn't make it wrong perse. Just (like all ideologies) buyer-beware.
jefftk 4y ago
This comment on the EA Forum gives a lot of additional information, by someone who attempted to identify each case: https://forum.effectivealtruism.org/posts/7b9ZDTAYQY9k6FZHS/abuse-in-lesswrong-and-rationalist-communities-in-bloomberg?commentId=BYvgpTxBTCZJQqHCz https://forum.effectivealtruism.org/posts/7b9ZDTAYQY9k6FZHS/... Several of the accounts turned out to concern the same person, Michael Vassar, who was central ~10y ago but kicked out for this and other harmful behavior, and now describes himself as anti-EA. On the other hand, the comment treats the cases they can't identify as not concerning from a community perspective ("it's very unclear from this what actions anyone could or should have taken") and I don't endorse that.
elecush 4y ago
don't give charlatans the time of day please
JohnFen 4y ago
The Effective Altruism movement itself helps justify and bury bad behavior.
YeGoblynQueenne 4y ago
Could a kind user please have a second look at the comment I posted earlier in this thread and which has been flagged to death? That was meant as a riddle. Are riddles not OK for HN? Or should I have made it more clear that it is a riddle?
cwkoss 4y ago
EA is inherently psychopathic because it seeks to do analysis of morality devoid of emotions. I'm convinced most EAs are psychopaths.
DontchaKnowit 4y ago
Just an observation : the general quality/thoughtfulness of responses to AI related HN threads is lower compared to other topics. Granted it is a hard topic to discuss, buut jeez there are some ignorant comments here.

141 points by polrjoy 4y ago

https://archive.ph/sLihW https://archive.ph/sLihW

I was not aware that there is an Atlas Shrugged-length Harry Potter fanfiction about rationality. Learn something new every day!

petesergeant 4y ago
It’s a very, very entertaining read, although it loses its way and loses steam a bit as it goes on
- SiempreViernes 4y ago
  I think the first chapter is really fun, then it loses steam and by something like chapter six it has fukky jettisoned the promise chapter one sets up and it is strictly in "ok to terrible" territory for the rest of it.
khazhoux 4y ago
YMMV. I tried to finish it based on a strong trusted recommendation, but could not get past the insufferable author-inserts, and the unpleasant and unbelievable hero character.
- flangola7 4y ago
  What's the name of it?
  - karaterobot 4y ago
    Harry Potter and the Methods of Rationality. I'm glad such a thing exists, but I could never work up the interest to read it myself. https://en.wikipedia.org/wiki/Harry_Potter_and_the_Methods_of_Rationality https://en.wikipedia.org/wiki/Harry_Potter_and_the_Methods_o...
    - pksebben 4y ago
      wait, really? I scrolled right past someone posting about "Harry Potter rationality fanfiction" and thought, "huh, oddly specific burn but it's chuckle worthy". it's an actual thing? This is a joke, right? tongue in cheek or something. Or should I go get some Brawndo to water my basil...
      - HDThoreaun 4y ago
        It's not only a thing, it's almost 2000 pages long.
- kimixa 4y ago
  I too didn't finish it - assuming it's "Harry Potter and the Methods of Rationality". I got caught up by the multiple times the main character made sweeping assessments of things with insufficient information, and just happened to be right by accident (or author fiat I guess). The main characters figure things out by Thinking Really Hard, and experimentation is only ever done to prove they were right all along to other characters. That's an arrogant character who keeps falling into the right answer, not a scientist.
  - SamReidHughes 4y ago
    It went above your head.
    - operatingthetan 4y ago
      It went above their head because it was poor writing in their view? Why would you think taste in writing has anything to do with their ability to understand the points the writer was trying to make? This seems unecessary and you are probably capable of coming up with a more interesting response.
      - SamReidHughes 4y ago
        It was supposed to be a flawed protagonist.
        operatingthetan 4y ago
        It's commonly thought (and obvious) that Yud wrote the book about himself so I doubt it. It doesn't really fit the rationalist playbook either, they take themselves excessively seriously.
        SamReidHughes 4y ago
        The character's author-insert aspects are self-parody.
- favaq 4y ago
  Really, you found it surprising that a book written by the insufferable author of the insufferable blog LessWrong is insufferable?
idlewords 4y ago
It's not just Atlas Shrugged-length, but Atlas Shrugged-quality!
- precompute 4y ago
  I disagree, Atlas Shrugged was much better, used an original premise and had an interesting phrase throughout ("Who is John Galt?"). HPMOR is really garbage-tier. It's parasitic on another series, doesn't have much in the way of a good story, and quite legitimately makes shit up. There are much better HP fanfics (and indeed books) written by people that are not as "smart" as Yudkowsky and also didn't have an agenda to push.
SiempreViernes 4y ago
It's not really about rationality past the first four chapters or so, after that I don't really know what its about but when part of the plot is helping Voldemort break people out of azkeban then rationality is clearly not an important theme.

noloblo 4y ago

@shervin @pmarca: Utilitarianism is bad: It encourages mortal men to play God at levels of complexity that they can't possibly comprehend. It should be banned for the greater good >> “earning to give,” the idea that people like Bankman-Fried should do whatever it takes to make a lot of money so they can give it away. To amass his billions, Bankman-Fried allegedly defrauded his customers, and critics have said his downfall shows that EA is vulnerable to a myopia that allows the ends to justify illegal means. Among EAs themselves, however, the most salient criticism is subtler: that living at the logical extremes of the ideology is impractical and a recipe for misery. In buddhism https://en.wikipedia.org/wiki/Middle_Way https://en.wikipedia.org/wiki/Middle_Way middle way is advised for this very reason

drdeca 4y ago
Utilitarianism is false, but I think it is a very useful heuristic, where I think it being used (as a heuristic) more often would probably be beneficial at the margins. There is no canonical way the aggregate utility between persons, and people probably don’t actually have utility functions. And, there are moral laws which should have at the least some weight when weighed against a “but this action seems to have better outcomes overall, on average”, and possibly infinite weight when weighed against such things. Nevertheless.

deleted 4y ago

[deleted]

yanderekko 4y ago

[flagged]

deleted 4y ago
[deleted]
elmomle 4y ago
Thanks for sharing that reference. I started out being intrigued, but he totally lost me with some pretty unsubstantiated absolutist statements, e.g. "anyone smart and independent-minded will have views on race and sex well outside the liberal mainstream" and "men are sure to be a disproportionate share of leaders for biological reasons". The first statement is categorically false or otherwise implies the author's engaging in "no true Scotsman" fallacy; and, in the second quote, how he can disentangle biological from cultural and social causes with such absolute conviction is mystifying. Stating opinions as facts in such a way is a kind of sophistry, and ironically this tendency limits one's ability to reach deeper truth.
- operatingthetan 4y ago
  The photo of Aella in a bra with the caption "The rare rationalist female" wasn't a red flag? More seriously, yes that piece is dripping with sexism. >Women will never make up a significant portion of self-made billionaires or outstanding philosophers, but they can get a majority of BAs. Academia practices affirmative action in everything from admissions to journal submissions and citations practices, artificially reducing the natural male advantage at higher levels of ability.
  - elmomle 4y ago
    Yeah, that was also a massive red flag once I got to it, but I wanted to give a more considered critique.
- yanderekko 4y ago
  >how he can disentangle biological from cultural and social causes with such absolute conviction is mystifying. He doesn't have to disentangle the two completely, he just strongly believes that there's a non-trivial biological contribution to these outcomes. Which already puts his views "well outside the liberal mainstream".

Nexialist 4y ago

If you'd like to read more about the roots of Effective Altruism, ranging from old mailing lists, LessWrong, bitcoin, cults, AI and the surprising connections to many well known figures in tech, I highly recommend the 7-part Extropia's Childen series from Jon Evans [1] 1. https://aiascendant.substack.com/p/extropias-children-chapter-1-the-wunderkind https://aiascendant.substack.com/p/extropias-children-chapte...

stubybubs 4y ago
Golly. I'm on part 2 and it sounds like they're about to invent Scientology. > The full post goes into more depth and nuance, but it's worth stressing how heavily Leverage relied on “debugging,” based on Anders's “Connection Theory.” Debugging consisted of — to paraphrase and oversimplify — opening up one's psyche, history, self, and emotional core to maximum vulnerability, generally to one's hierarchical superior, and then doing all you could to follow their suggestions to “fix” your mind.
- painelemental 4y ago
  Thats nothing. Wait till Epstein enters the picture. https://fredwynne.medium.com/an-open-letter-to-vitalik-buterin-ce4681a7dbe https://fredwynne.medium.com/an-open-letter-to-vitalik-buter... Any hoo - this stuff is the tip of the iceberg.
  - aoeusnth1 4y ago
    Incredible letter, I had no idea about the SWATing. Tbh the Epstein donation seems like a nothingburger compared to the other allegations in that letter. Edit: more context on Ziz, http://zizians.info/ http://zizians.info/
  - j_crick 4y ago
    Admittedly I don’t follow this rationalist thing on the internet at all, so I don’t know what’s going on with them. But that was hell of a read. I am no stranger to people being dicks, but still, that all is just sick.

vishal0123 4y ago

This article is such a mess. If you are alleging something, then say it clearly. Judging by my participation in lesswrong and others, there are both feminists and sexist people, just like all other online forums. People are more free to be sexist online and every close knit online community has fair share of sexists.

Barrin92 4y ago
The EA people seem to have more than a spurious relationship with polygamy, drugs, orgies and odd-ball ascetic leader figures, doomsday prophecies and psychotic behavior. Taken from the article " MIRI employee Jessica Taylor had a job that sometimes involved “imagining extreme AI torture scenarios,” as she described it in a post on LessWrong—the worst possible suffering AI might be able to inflict on people. At work, she says, she and a small team of researchers believed “we might make God, but we might mess up and destroy everything.” In 2017 she was hospitalized for three weeks with delusions that she was “intrinsically evil” and “had destroyed significant parts of the world with my demonic powers,”" This is like what, one bad week away from AI Jonestown? At some point this isn't a community but a cult, and women usually tend to draw the shortest sticks in any cult environemnt with bizarre moneyed, male leader figures.
- yanderekko 4y ago
  >This is like what, one bad week away from AI Jonestown? Look at the Ziz story - we're already there. (https://medium.com/@sefashapiro/a-community-warning-about-ziz-76c100180509 https://medium.com/@sefashapiro/a-community-warning-about-zi...) However, this should not severely impugn the community as a whole.

js8 4y ago

I have always felt that the embrace of "AI can be dangerous" comes from an elitist fear of capitalist class that the "unwashed masses" will have too much access to technology and will make capital, as well as labor, obsolete. I mean if everybody has a super-smart personal assistant in their home, who thinks independently, why would they need or want a ruling class, when we can simply have democratic socialism?

space_fountain 4y ago
What about a future where only energy and resources are important leads you to think we’ll have less inequality? Brains are the only thing that don’t flow to the wealthy right now
- js8 4y ago
  Human nature. If you're a part of the common tribe, people are actually pretty good. There doesn't seem to be an upper limit on the size of the tribe (although some people just wanna hate somebody). You can also look at it this way. The only real objection against socialist societies was that people are lazy, and don't care enough about producing stuff or resolving economic inefficiencies. But with AI, this objection is gone. Nobody really wants to be in a constant "resource competition" or whatever, it's an effort. People generally want to have fun and enjoy life.
  - space_fountain 4y ago
    Humans ability to push others out of the circle of concern is near limitless. We literally bought and sold people. I think there is reason for hope. We stopped doing some of those terrible things and only partially for economic reasons, but ai still fills me with some great fear. It shrinks the circle that the wealthy absolutely must keep in their circle, in their tribe, to go on and live happy lives and people of the past have been uncomfortably good at excluding people from the circle in the past. Basically it might not destroy us, but I certainly don’t see it as inevitable that it will bring us to utopia

dr_dshiv 4y ago

1. “Moral computation” was coined by Francis Hutcheson (doctoral advisor of Adam Smith). 2. I like hearing rationalists describe “demon summoning is easy, Angel summoning is hard.” Genuinely curious about that.

xkcd-sucks 4y ago
2. "An ounce of prevention is worth a pound of cure"; "measure twice, cut once"; "it's easier to fuck it than to unfuck it"; etc. Not exactly novel
- dr_dshiv 4y ago
  To clarify— I mean the rationalist reference to angels and demons. It seems to be more than metaphorical.
  - scarmig 4y ago
    Angel here would be an "aligned" AI: an AI whose behavior is in line with human values. A demon is simply an unaligned one. The key point would be that it needs not be actively hostile to human values, but indifferent to them. Think of a Lovecraftian Elder God, not AI Hitler. The supporting hypotheses would be that it's much easier to create an unaligned AI than an aligned one, as an aligned one is an unaligned one with additional constraints, and that unaligned AIs will wreak massive destruction.

jasonhansel 4y ago

Reminder that one of EA's core supporters is a fan of scientific racism: https://twitter.com/ArsonAtDennys/status/1362153191102677001 https://twitter.com/ArsonAtDennys/status/1362153191102677001 For some more context: https://rationalwiki.org/wiki/Scott_Alexander#Race_and_IQ https://rationalwiki.org/wiki/Scott_Alexander#Race_and_IQ

deleted 4y ago
[deleted]
postultimate 4y ago
Good for him. Nobody's racism should be unscientific.

skybrian 4y ago

In case you're thinking you saw this before, there was similar article a month ago in Time: https://news.ycombinator.com/item?id=34646361 https://news.ycombinator.com/item?id=34646361

d0mine 4y ago
It looks like a hit piece. Someone is against rationality regarding the question of AI safety (any real practical consequences is to censor AI under _pretense_ of AI safety, Perhaps, the articles are the preventive strikes to discredit the AI safety experts that might point out the obvious).
- skybrian 4y ago
  That's a rather specific accusation to make without evidence. Possibly, a journalist might have other reasons to cover a story? Not necessarily good ones, but still...
  - d0mine 4y ago
    The article itself is the evidence: read it and apply common sense. I know almost nothing about EAs but what I know makes me think that it is likely P(SA)>P(SA|EA)>0.

neilv 4y ago

Please capitalize it as Effective Altruism. The generic terms "effective" and "altruism" are both very important terms, and this doesn't follow from combining them.

dang 4y ago
Ok, we've given it an A!

renewiltord 4y ago

What is one or two rapes or sexual assaults against trillions of future lives? With sufficient future sight and the power of the exponent > 1, one can justify many present actions. Sadly, the EAs are small thinkers. True grand scale actions require truly grand visions https://cosmicbraintrust.org/ https://cosmicbraintrust.org/

deleted 4y ago
[deleted]
eldenring 4y ago
what an evil and disgusting comment.
- renewiltord 4y ago
  I deserve this since sarcastic comments are really the lowest class of Internet comments.
  - eldenring 4y ago
    oh god sorry I didn't even realize. Its a bit sad that nowadays a comment like that can even be mistaken for a serious one.
d0mine 4y ago
The real and the only relevant question is: do EAs rape/sexual assault more than the the average population (and if the answer is no (it is), then we need more EAs).

AbrahamParangi 4y ago

AI safety is an apocalypse cult and we should recognize that everyone who has ever been in an apocalypse cult thinks that "no, this time it's really the end of the world". Everyone who believes crazy shit thinks they're being reasonable, so just because we're super sure we're being reasonable doesn't mean what we believe isn't crazy. Do we believe that we alone, of all people who have believed in the coming end, have it right this time?

JamesBarney 4y ago
There have been people for a long time who think the world is going to end. We can use your argument to dismiss anyone who fears any existential risk like asteroids, nuclear war, climate change, AI, etc... And since the world exists every group will be wrong except one. I think the risk of human extermination is much lower than many in this group believe because I think AI is much more compute bound than algorithm bound. So I think a hard takeoff is very unlikely. But while the median case for climate change is probably worse than the median case for AI. The existential risks for AI seem much higher. And it's a concern worth taking seriously.
- esperent 4y ago
  The issue is that the effective altruism crowd are focused exclusively on AI safety, at least in what they publish online. If they talked about all possible threats to humanity and how to solve them, from AI, to climate change, to nuclear bombs, to asteroids, to disease, they would be an interesting and important group. They don't. They hardly talk about anything else except AI. From the outside it looks a lot like an obsession. Personally I think they treat AI as an existential threat above all others because they consider themselves to be the pinnacle of intelligence in the universe and AI is showing that human intelligence might not be so special after all. So it's an existential threat to their sense of self. Other existential threats merely threaten their lives and are therefore not as important. This would be fine, if acknowledged with a degree of self awareness. But it's not, instead it's wrapped up in so many layers of convoluted logic that most people, insiders and outsiders both, can't see the real reason for this obsession.

painelemental 4y ago

https://fredwynne.medium.com/an-open-letter-to-vitalik-buterin-ce4681a7dbe https://fredwynne.medium.com/an-open-letter-to-vitalik-buter... https://sinceriously.fyi/brent-dill-confessions-full-redacted/ https://sinceriously.fyi/brent-dill-confessions-full-redacte... This entire social milieu is a goddamn horror show.

Animats 4y ago

Some of that is just typical cult behavior. Scientology and Ayn Rand followers have had similar problems. Drugs usually make it worse. AI safety is a real issue, but look at what it looks like so far. - "Machines should think, people should work". That's Marshall Brain's "Manna". It's also working at an an Amazon fulfillment center, where the computers tell the humans what to do. - "Paperclip maximizer". A corporation is a paperclip maximizer. If you take Milton Friedman seriously, that's what a corporation is supposed to do. As I point out occasionally, most of the ethical problems foreseen for AIs are known ethical problems with corporations. Those are the classic problems. As the technology advances, we're discovering new ones in the large language model space. Those issues are well known here.

yanderekko 4y ago
>- "Paperclip maximizer". A corporation is a paperclip maximizer. If you take Milton Friedman seriously, that's what a corporation is supposed to do. As I point out occasionally, most of the ethical problems foreseen for AIs are known ethical problems with corporations. Corporations are not capable of having the "fast takeoff" which is an integral part of the doomer narrative here. Furthermore, a corporation being a paperclip maximizer is an abstraction that obviously falls apart in the extremes, if for no other reason than because at some point destroying the world for the sake of more paperclips is going to have a negative ROI.... Meanwhile, the (AI) paperclip maximizer is actually trying to maximize paperclips and does not have the sort of internal checks that would apply to real-world orgs. That is the crux of the problem.
- roenxi 4y ago
  Corporations, when first developed, took out large chunks of Asia before anyone could stop them. The main difference is that corporations have to be powered by people while an AI could theoretically work with complete independence.
  - humanrebar 4y ago
    I think the argument is that due to emergent behavior and possibly perverse incentives corporations are indistinguishable from slow-moving AI -- they can be treated like independent agents. Elsewhere in the thread, it's brought up that AI could move faster than corporations generally move. That might be true, but there are still physical limits: the speed of light, moving mountains of steel takes energy, etc.

5cott0 4y ago

what’s so bad about reading some Harry Potter fanfic in the polycule?

Vecr 4y ago
I'm assuming you read it, so why does Harry Potter (a 11 year old in 1991/1992) think polyamory might (or probably would be, looking at the text right now) be a good idea in a future where people live a long time, in said fanfic? I think it's pretty dubious, but the other 11 and 12 year old girls in the same fanfic also seem to be fans of the idea of yaoi like situations, so all semblances of actual historical events are probably gone at that point. This is the UK in 1992, not the US West Coast. To give my personal impression, when I read some science fiction that had some suggestions of polyamory in it in a similar situation, I did not think it was a good idea. I know the whole thing is essentially a morality story and for various plot and author-community communication reasons the "Harry Potter" in the story does not behave like J. K. Rowling's Harry Potter with different adoptive parents nor like any person, child or otherwise, could behave in 1991 and 1992 within the realm of probability, but I think the whole thing could be done a lot better even given the various constraints. How about he actually does science experiments for more than two chapters instead of going on stupid adventures that should probably be left to the JKR books. Cut the time travel as well, figure out some other reason for constraint solving to show up.
- HDThoreaun 4y ago
  Harry has voldemort's soul in him the HPMOR which is the explanation given for why he acts like an evil adult.
  - Vecr 4y ago
    Yes, spoilers though, but I don't actually think that explains it. Are there any indications in cannon that Tom Riddle was ever interested in polyamory? I'm pretty sure voldemort is not interested in other people at all after his soul gets all divided, but I don't know about before that. Riddle is not from the US west coast either, so it's kind of dubious.

puzzledobserver 4y ago

I had previously heard of EA, and knew about paperclip maximizers, trolley problems, unintended consequences, and other risks associated with unregulated AI. But I wasn't aware of the connections between the two communities / families of ideas until recently. Because I don't consider myself sufficiently educated in these matters, I default to using GiveWell and their mosquito net recommendations as an effective low-effort approach to making monthly contributions to charity. Overall, is there anything about this approach which is objectionable? How do philosophers go from something like GiveWell to something like fending off the AI apocalypse? Is this ideological expansion inevitable?

TheDong 4y ago
EAs premise includes basically "all human lives are equal. Saving one stranger's life is just as valuable as saving your father's life (more valuable actually if the stranger has more years left than your dad), so you should donate your kidney to a stranger, not your father" "All human lives are equal" then expands to "and there are more unborn lives than there are currently living lives, so it's more important to save unborn kids in 200 years by diverting AI catastrophe now". I personally don't think unborn lives are worth anything, and as such I don't follow that jump. I don't feel a need to justify it either. I think there's two additional easy ways to veer off the EA course. The first is "how do we know EA is right about the charities?" The answer you'll get is basically "Sure, you can donate a kidney to your father and know it saved his life, you can verify that... but the people verifying your EA kidney donation does more good are smarter than you. They're rationalists. They do math. They know selling your kidney, which would have only saved one life, to a black market and getting enough money for 50 mosquito nets, which will save 20 lives, is the right deal. The spreadsheet says so." I do not buy that the people running and choosing EA charities are actually the smartest people, especially when it comes to having a positive impact on lives. Sure, they talk a big talk, but I know for a fact that when I give money to a local shelter, the money gets used for good. I know for a fact that when I donated to a certain EA charity I won't name, the money went into "AI research" which is to say into the pockets of a rich white dude. I know for a fact that the average EA rationalist I have met does not come off as a caring human. A second way that one can veer off the EA course is by deciding that one values some lives more than others. For example, if I live in a small city, I may decide I value the lives of orphans in this city more than the lives of the orphans in some third-world country. Sure, feeding an orphan in africa is cheaper so EA would say do that, but I may decide that the intangible value of improving my local community is worth more than "getting the most bang for my buck". I think this is also a totally reasonable argument, and in fact I would much rather live in a world where everyone tried to improve their local community than one where everyone used a spreadsheet to try and maximize their "best improvement per dollar".

bloppe 4y ago

EA as a concept is totally fine and the risk of runaway AI still seems rather fantastical to me. Clearly a more pressing problem is runaway egos latching onto these ideas to inflate themselves to deadly girths. I'm sure some people can practice a totally constructive form of EA. But if you structure your entire identity around it, clearly you've drifted somewhat from rationality.

Towaway69 4y ago

I'm surprised that the term Longtermism hasn't been mentioned. Isn't Nick Bostrom also a longtermist? It's probably the reason for the focus on AI. Also wasn't he one of the founders of Longtermism?

thom 4y ago

EA is at least partly a moral and ethical pyramid scheme. You get to be noble and far-sighted, do relatively little real work, and you’ll never be judged on the outcomes because of the long timelines. And so you rope in more and more acolytes who both increase and accrue these same benefits. It seems unlikely that the fall from grace of prominent figures will affect the movement, given that it’s partly based on cognitive dissonance as it is. All that said, the amount of energy we expend on AI safety probably shouldn’t be zero. But I choose to believe that if Roko’s basilisk is going to torture us over anything, it’ll be that we didn’t raise up millions of AI scientists from the cotton fields and sweatshops. Credit to the less prominent altruists doing that, or who at least have some form of compound interest built into their utility function which compels them to be kind, now.

dbspin 4y ago
Roko's basilisk makes no sense. Sans magic physics time travel - there's no motivation. One thing all A.I. is, no matter how nefarious, is goal directed. It might have poorly aligned goals, absurd goals, dangerous goals, goals it misunderstands, goals that contradict etc. But ultimately it proceeds toward some form of goal directed reward function. There's zero reward in punishing someone for past behaviour - except in the context of altering their future behaviour. Since the A.I. would have to already exist (and there's no reason to believe it would be incentivised to even encourage humans to create other, rival A.Is, with potentially differing goals) before it could punish anyone - Q.E.D.: Roko's basilisk has essentially zero likelihood. Worse - it's an essentially christian eschatological idea. What is this hypothetical A.I. but an avenging angel, a righteous judge at the end of time? It's pure religion. Dark, manipulative, blood sacrifice religion at that.
- PartiallyTyped 4y ago
  > There's zero reward in punishing someone for past behaviour A similar argument regarding Roko's Basilisk is that it is effectively a prisoner's dilemma, where the Basilisk torturing us and us not helping it is the Defect-Defect scenario. It makes no sense for either party to defect. > except in the context of altering their future behaviour. It really seems like a very contrived version of Pascal's mugging.

arisAlexis 4y ago

but EA is at least doing something about a huge problem we will face.

khazhoux 4y ago
What are they actually doing? Honest question, because I don't know.
- nyfern 4y ago
  Purchasing large mansions[0], chateaus[1], and spending lavishly on administrative kickbacks[2] because longtermism can justify any present action if you extend your horizon far enough. EA is effectively two disconnected ideas at this point. [0] https://forum.effectivealtruism.org/posts/xof7iFB3uh8Kc53bG/why-did-cea-buy-wytham-abbey https://forum.effectivealtruism.org/posts/xof7iFB3uh8Kc53bG/... [1]https://www.forbes.com/sites/sarahemerson/2023/02/28/effective-altruism-linked-group-used-millions-donated-by-ftx-to-buy-a-czech-castle/?sh=640b92a86baf https://www.forbes.com/sites/sarahemerson/2023/02/28/effecti... [2]https://intelligence.org/wp-content/uploads/2022/12/Independent-Auditors-Report-for-2021.pdf https://intelligence.org/wp-content/uploads/2022/12/Independ...
  - deleted 4y ago
    [deleted]
  - jefftk 4y ago
    > spending lavishly on administrative kickbacks I just skimmed the report you link and nothing jumped out at me as matching this -- could you be more specific?
- jefftk 4y ago
  Here are some EA highlights: * Founding and scaling GiveWell, synthesizing and analyzing global poverty research to make recommendations. This has helped guide about $1B+ in really valuable work, the largest portion in anti-malarial bednets. https://www.givewell.org/about/impact https://www.givewell.org/about/impact * Convincing Dustin Moskovitz and Cari Tuna to allocate their giving along these lines, building Open Philanthropy which has made a lot of grants that look really valuable https://www.openphilanthropy.org/grants https://www.openphilanthropy.org/grants (this is in addition to the ones where they granted at GiveWell's recommendation -- don't want to double count). * Michael Kremer won a Nobel prize in econ (shared with Esther Duflo and Abhijit Banerjee who I'd describe as EA-sympathetic but probably don't consider themselves EAs?) for global poverty research. * Built an "AI Safety" subfield of computer science, with the 2016 paper "Concrete problems in AI safety" having 1.8k citations. * Founding and scaling 80,000 Hours, helping people figure out how to use their careers to do more good. This includes looking into potential career opportunities, writing up public advice, and 1:1 advising, and 1k+ people have cited them as part of why they changed what they were working on. https://80000hours.org https://80000hours.org (Not especially representative; there's been thousands of people working along these lines for years and I'm sure I'm missing a lot of valuable work.)

arisAlexis 4y ago

Top level scientistis warning of this apocalypse. Sam Altman which was the CEO of the very site you are posting and the CEO of OpenAI has warned starkly that there is a probability that humanity will go extinct from AI in his latest interview. Have you seen it? You think he is somehow bluffing or is he part of the same cult?

bagels 4y ago
What scientists?
- AgentME 4y ago
  A recent poll of published AI researchers showed a median estimate of a 5% chance AI causes human extinction, with half believing the chance is greater than 10%. https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/ https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/ A poll of NLP researchers showed 37% agreed that catastrophic outcomes of AI comparable to nuclear war were plausible. https://arxiv.org/abs/2208.12852 https://arxiv.org/abs/2208.12852
  - pksebben 4y ago
    it's a little annoying that the "AI Impacts" paper seems to be three MIRIs in a trenchcoat, considering that this article is about EA's AI agenda and by extension Singularity/MIRI. There is a slight conflict of interest here - as the publishers of the paper are also taking funding specifically to "deal with the threat" they're publishing on.
    - arisAlexis 4y ago
      Yes if we have scientists that are getting paid to research a certain risk it’s inevitable that they have conflict of interest. Otherwise how would it be done?
      - pksebben 4y ago

louwrentius 4y ago

It’s interesting to me that the comments of the HN crowd seem to focus on Effective Altuism and AI safety. A ton of the article is about immoral behavior of men in positions of power towards women. And the misogyny. > The underlying ideology valorizes extremes: seeking rational truth above all else, donating the most money and doing the utmost good for the most important reason. This way of thinking can lend an attractive clarity, but it can also provide cover for destructive or despicable behavior. I think this quote really sums things up for me.

fluxinflex 4y ago

I doubt very much that the poorest people on this planet care much about AIs. People living in absolute poverty in Africa, India, China etc. I also doubt they would be - negatively - affected if a rogue AI would turn out the lights in the electronicified world. Places where AIs aren't used to command peoples lives will probably not even realise that an AI apocalypse had happened. Perhaps a case of western navel-gazing?

dotsam 4y ago
> I doubt very much that the poorest people on this planet care much about AIs. Whether people know or care about a risk doesn't mean that the risk won't affect them! > I also doubt they would be - negatively - affected if a rogue AI would turn out the lights in the electronicified world. A rogue AI might as well just kill everyone rather than just the Western naval-gazers, as this would bring to 0 the possibility that a human could ever turn it off / thwart its plans.
- 0xf00ff00f 4y ago
  > A rogue AI might as well just kill everyone How would it do that?
  - polski-g 4y ago
    There is a great documentary about this that came out a few years ago: https://www.youtube.com/watch?v=5UgPJhyJmlM https://www.youtube.com/watch?v=5UgPJhyJmlM
  - dotsam 4y ago
    Get access to the internet, get money. Money buys it power to get people to do things in the real world. Or it can hack through systems and blackmail and coerce people. Now it can act in the real world by getting people to do things it wants. As one example it could design, synthesize and spread a highly contagious and lethal pathogen.
CatWChainsaw 4y ago

nathan_young 4y ago

According to a study of 700 ML researchers (not AI safety) the median chance that that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)" is 5%. And 48% of respondents gave an answer of 10% or more. Even if this is 2 orders of magnitude too high it's worth a lot of our time. But these are probably mainly normy researchers not AI safety-ists. https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#Data https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#...

LatteLazy 4y ago
According to a survey 700 ML researchers, we should throw money at their subject or we might all die...
- PartiallyTyped 4y ago
  With great power comes great responsibility. If we start plugging ML everywhere - which we are currently doing - we better be sure that it is behaving as expected.
  - deleted 4y ago
    [deleted]
  - LatteLazy 4y ago
    So far we have seen no power at all from ML. So spending large amounts of money making it "safe" (not that anyone knows how or even what it would look like) seems like an over reaction. With no power, comes no responsibility. But of course, if ML is a person's pet interest, they will likely think we should increase spending on it. That doesn't make that a smart decision...
    - PartiallyTyped 4y ago
      I believe that you conflate “power” with capabilities wrt AGI. I’d argue that power is in relation to reach and the capacity to influence. In this regard, ML has quite a bit of power these days, from deciding whether you are allowed to take a loan, to driving engagement in social media, to predicting and influencing what you will watch, to the way our language is even used. There are also quite a few implicit ways that ML models have influenced society, eg GPGPUs with emphasis on ML, and ML accelerators everywhere. The effectiveness of certain algorithms made us use them more often which influenced NVDA and Google towards developing hardware to accelerate those common cases thereby creating a feedback loop where our algorithms are chosen based on works well on our GPUs (cf transformers). As such systems become more prevalent and influential, ensuring safety and explainability will help us prevent pitfalls that put humans at risk.

ShredKazoo 4y ago

See discussion on the EA Forum https://forum.effectivealtruism.org/posts/7b9ZDTAYQY9k6FZHS/abuse-in-lesswrong-and-rationalist-communities-in-bloomberg#comments https://forum.effectivealtruism.org/posts/7b9ZDTAYQY9k6FZHS/...

whywhywhywhy 4y ago

It's just super obvious by this point that none of the AI safety LARP is really about safety it's more about justifying why a technology that to hire the team to work on it requires that work to be published (Apple tried to do it without publishing and struggled to get the talent) but by publishing you lose part of your advantage so the only way to lock that advantage up is to make it so use of the models has to go through you and also trying to push for reasons for others to not have access to the tech required to train their own model. If you disagree then why is OpenAI's safety committee talking about how you should need a government contract to buy an A100.

kelipso 4y ago
I'm leaning towards this as a big reason. PhDs really want to publish and they tend not to put as much effort into or not find interesting work that cannot be published. The trend in academia is towards making your code and data available but now you can now get around that by saying AI safety.
pksebben 4y ago
| If you disagree then why is OpenAI's safety committee talking about how you should need a government contract to buy an A100. Fuck, really? I hadn't heard about this. If true, it's the godfather of all ladder pulls. Not to mention, it's categorically awful to silo this stuff and grant it to an org that's so opaque. Good luck separating the wheat from the propaganda chaff without access to the tools that produce it.

YeGoblynQueenne 4y ago

[flagged]

LatteLazy 4y ago

Effective Altruism, like and philosophy is (a) only as good as it's practitioners and (b) often used as a cover for charlatans. That doesn't make it wrong perse. Just (like all ideologies) buyer-beware.

jefftk 4y ago

This comment on the EA Forum gives a lot of additional information, by someone who attempted to identify each case: https://forum.effectivealtruism.org/posts/7b9ZDTAYQY9k6FZHS/abuse-in-lesswrong-and-rationalist-communities-in-bloomberg?commentId=BYvgpTxBTCZJQqHCz https://forum.effectivealtruism.org/posts/7b9ZDTAYQY9k6FZHS/... Several of the accounts turned out to concern the same person, Michael Vassar, who was central ~10y ago but kicked out for this and other harmful behavior, and now describes himself as anti-EA. On the other hand, the comment treats the cases they can't identify as not concerning from a community perspective ("it's very unclear from this what actions anyone could or should have taken") and I don't endorse that.

elecush 4y ago

don't give charlatans the time of day please

JohnFen 4y ago

The Effective Altruism movement itself helps justify and bury bad behavior.

YeGoblynQueenne 4y ago

Could a kind user please have a second look at the comment I posted earlier in this thread and which has been flagged to death? That was meant as a riddle. Are riddles not OK for HN? Or should I have made it more clear that it is a riddle?

cwkoss 4y ago

EA is inherently psychopathic because it seeks to do analysis of morality devoid of emotions. I'm convinced most EAs are psychopaths.

DontchaKnowit 4y ago

Just an observation : the general quality/thoughtfulness of responses to AI related HN threads is lower compared to other topics. Granted it is a hard topic to discuss, buut jeez there are some ignorant comments here.