4 ms·

I do not think that a historical EA movement would have recommended spending marginal dollars distributing copies of The World Set Free, but of course we cannot

by dotsam 4y ago

I do not think that a historical EA movement would have recommended spending marginal dollars distributing copies of The World Set Free, but of course we cannot know that for sure. Apparently the book is positive on the proliferation of the powerful new speculative technology, although I haven't read it. I would imagine EA would have aligned more with an anti-nuclear stance, which is certainly the attitude currently taken towards the possible breakthroughs of AI. On your first point, I can donate towards anti-malarial interventions and towards reducing AI risk: two very diverse causes, but both under the EA umbrella. Diversifying amongst recommended EA causes seems to me a good bet, especially given I expect some of them to require pivoting away from as more research is done. Perhaps we can summarise the last decade of EA into something like: "given the available information (largely based on RCTs, research and an understanding of moral philosophy), we expect that money spent here will do more good than money spent there. More information will change this expectation." We now have information that there is a potentially very grave risk confronting us from AI. For other cause areas we can use evidence like randomised controlled trials to inform our actions. In the case of AI risk, this type of evidence is unavailable, but there are nonetheless other types of information available to us that suggest it is worth taking seriously. This information takes the form of compelling arguments from credible people in the AI industry and the industry's achievements to date. Of course we'd like to have firmer footing, but the nature of this risk doesn't permit it. We can either ignore it, and potentially walk to our doom, or we can take it seriously and try to reduce the uncertainty and the risks. I agree with you that this raises eyebrows, but I don't think it's a gross departure from the last decade. I think it is wrong to say we have "zero evidence" that organisations committed to reducing AI risk can achieve their goals. I think it is more accurate to say it is highly uncertain that they will be able to achieve their goals. My view is that we should try to address AI risk because the stakes are high. I would have sided with Albert Einstein in trying to renounce nuclear weapons, even though we have "zero evidence" for humanity ever coming together to do such a thing in the past. Of course - nuclear weapons really do exist and AI is only a hypothetical threat at the moment. My point is that there was "zero evidence" that the world could come together to voluntarily control a technology, and yet people like Einstein tried to make it happen, and with good reason. Dismissing this is like taking Rutherford's view in the 1930s and saying there's very little point worrying about nuclear weapons because they're not possible, and it's even more pointless thinking about how we might control them. Finally - why might the pitch speak to nerds in particular? I think in part because it is based on Bayesian reasoning, which is both nerdy and incredibly effective in situations of uncertainty.

notahacker 4y ago
I'd have thought it was fairly obvious why a pitch about nerds saving the world from the cyberpunk they grew up reading might appeal to nerds! That's why AI safety organisations are often exempted from the expectation of even basic transparency about what they've done with marginal dollars never mind metrics by the very same people that expect less nerdy causes like feeding people to prove their efficiency and need for the funds in granular detail. The application of the "Bayesian" approach involving setting arbitrarily high "priors" for marginal dollars given to people like them saving humanity where the possibility of others can't even be considered looks like post hoc rationalisation (otherwise pretty much every intervention criticised by GiveWell as inefficient use of funds is subject to the "but what if one of the beneficiaries goes on to directly or indirectly save humanity from AGI/climate change/nuclear war" counter argument...) There's nothing unique to EA in this (people convinced that most aid money is wasted sometimes give to organisations they have some sort of ethnic/political/religious affiliation or people they generally like unconditionally, and sometimes that trust heuristic has even been right!) but it did purport to be the school of altruism that didn't. As for the Wells analogy, the two most notable uses of AI safety funds seem to be (i) attempting to build AIs and (ii) distributing fiction they believe best illustrates the possibilities of AGI, so I'm not sure I'm altogether off the mark in imagining what well-intentioned utilitarian nerds of the time (Wells certainly was one) might have done with more cash. If you subscribe to the view that I do: that "secretly unite to eliminate humanity" is a fairly non-obvious and risky goal for a class of technologies which evolved self awareness from being rewarded for meeting human needs, then popularising the idea of conflict between human and AI could be more likely to have extremely adverse unintended consequences than intended ones. Popularising the idea that it's the self aware general intelligence that's the dangerous bit about AI, not the intermediate step involving humans having access to immense calculating power whose recommendations they don't really understand seems like something of an existential risk blindspot too.
- dotsam 4y ago
  > That's why AI safety organisations are exempted from the expectation of even basic transparency about what they've done with marginal dollars Can you let me know more about that? > If you subscribe to the view that I do: that "secretly unite to eliminate humanity" is a fairly non-obvious and risky goal I don't think AI would have the goal "secretly unite to eliminate humanity", but I do think that most of the goals we can imagine it having can play out in very bad ways that achieve that outcome as a side-effect. There is an incentive for it always be looking for ways to maximise its power (compute, resources, money, influence), and this could happen in ways we really don't want. > Popularising the idea that it's the self aware general intelligence that's the dangerous bit about AI, not the intermediate step involving humans having access to immense calculating power whose recommendations they don't really understand seems like something of an existential risk blindspot too. It sounds like you think AI is risky too - even if only at this intermediate level? I'm curious - do you have a view on how we could go about reducing this risk?

by dotsam 4y ago

notahacker 4y ago

I'd have thought it was fairly obvious why a pitch about nerds saving the world from the cyberpunk they grew up reading might appeal to nerds! That's why AI safety organisations are often exempted from the expectation of even basic transparency about what they've done with marginal dollars never mind metrics by the very same people that expect less nerdy causes like feeding people to prove their efficiency and need for the funds in granular detail. The application of the "Bayesian" approach involving setting arbitrarily high "priors" for marginal dollars given to people like them saving humanity where the possibility of others can't even be considered looks like post hoc rationalisation (otherwise pretty much every intervention criticised by GiveWell as inefficient use of funds is subject to the "but what if one of the beneficiaries goes on to directly or indirectly save humanity from AGI/climate change/nuclear war" counter argument...) There's nothing unique to EA in this (people convinced that most aid money is wasted sometimes give to organisations they have some sort of ethnic/political/religious affiliation or people they generally like unconditionally, and sometimes that trust heuristic has even been right!) but it did purport to be the school of altruism that didn't. As for the Wells analogy, the two most notable uses of AI safety funds seem to be (i) attempting to build AIs and (ii) distributing fiction they believe best illustrates the possibilities of AGI, so I'm not sure I'm altogether off the mark in imagining what well-intentioned utilitarian nerds of the time (Wells certainly was one) might have done with more cash. If you subscribe to the view that I do: that "secretly unite to eliminate humanity" is a fairly non-obvious and risky goal for a class of technologies which evolved self awareness from being rewarded for meeting human needs, then popularising the idea of conflict between human and AI could be more likely to have extremely adverse unintended consequences than intended ones. Popularising the idea that it's the self aware general intelligence that's the dangerous bit about AI, not the intermediate step involving humans having access to immense calculating power whose recommendations they don't really understand seems like something of an existential risk blindspot too.

dotsam 4y ago
> That's why AI safety organisations are exempted from the expectation of even basic transparency about what they've done with marginal dollars Can you let me know more about that? > If you subscribe to the view that I do: that "secretly unite to eliminate humanity" is a fairly non-obvious and risky goal I don't think AI would have the goal "secretly unite to eliminate humanity", but I do think that most of the goals we can imagine it having can play out in very bad ways that achieve that outcome as a side-effect. There is an incentive for it always be looking for ways to maximise its power (compute, resources, money, influence), and this could happen in ways we really don't want. > Popularising the idea that it's the self aware general intelligence that's the dangerous bit about AI, not the intermediate step involving humans having access to immense calculating power whose recommendations they don't really understand seems like something of an existential risk blindspot too. It sounds like you think AI is risky too - even if only at this intermediate level? I'm curious - do you have a view on how we could go about reducing this risk?