3 ms·

I'd have thought it was fairly obvious why a pitch about nerds saving the world from the cyberpunk they grew up reading might appeal to nerds! That's why AI saf

I'd have thought it was fairly obvious why a pitch about nerds saving the world from the cyberpunk they grew up reading might appeal to nerds! That's why AI safety organisations are often exempted from the expectation of even basic transparency about what they've done with marginal dollars never mind metrics by the very same people that expect less nerdy causes like feeding people to prove their efficiency and need for the funds in granular detail. The application of the "Bayesian" approach involving setting arbitrarily high "priors" for marginal dollars given to people like them saving humanity where the possibility of others can't even be considered looks like post hoc rationalisation (otherwise pretty much every intervention criticised by GiveWell as inefficient use of funds is subject to the "but what if one of the beneficiaries goes on to directly or indirectly save humanity from AGI/climate change/nuclear war" counter argument...) There's nothing unique to EA in this (people convinced that most aid money is wasted sometimes give to organisations they have some sort of ethnic/political/religious affiliation or people they generally like unconditionally, and sometimes that trust heuristic has even been right!) but it did purport to be the school of altruism that didn't. As for the Wells analogy, the two most notable uses of AI safety funds seem to be (i) attempting to build AIs and (ii) distributing fiction they believe best illustrates the possibilities of AGI, so I'm not sure I'm altogether off the mark in imagining what well-intentioned utilitarian nerds of the time (Wells certainly was one) might have done with more cash. If you subscribe to the view that I do: that "secretly unite to eliminate humanity" is a fairly non-obvious and risky goal for a class of technologies which evolved self awareness from being rewarded for meeting human needs, then popularising the idea of conflict between human and AI could be more likely to have extremely adverse unintended consequences than intended ones. Popularising the idea that it's the self aware general intelligence that's the dangerous bit about AI, not the intermediate step involving humans having access to immense calculating power whose recommendations they don't really understand seems like something of an existential risk blindspot too.

dotsam 4y ago
> That's why AI safety organisations are exempted from the expectation of even basic transparency about what they've done with marginal dollars Can you let me know more about that? > If you subscribe to the view that I do: that "secretly unite to eliminate humanity" is a fairly non-obvious and risky goal I don't think AI would have the goal "secretly unite to eliminate humanity", but I do think that most of the goals we can imagine it having can play out in very bad ways that achieve that outcome as a side-effect. There is an incentive for it always be looking for ways to maximise its power (compute, resources, money, influence), and this could happen in ways we really don't want. > Popularising the idea that it's the self aware general intelligence that's the dangerous bit about AI, not the intermediate step involving humans having access to immense calculating power whose recommendations they don't really understand seems like something of an existential risk blindspot too. It sounds like you think AI is risky too - even if only at this intermediate level? I'm curious - do you have a view on how we could go about reducing this risk?
- notahacker 4y ago
  > Can you let me know more about that? Compare the level of transparency and need for funds of OpenAI LP with the average organisation criticised by GiveWell for not being transparent about how they spend the money, not providing any metrics or trials, and/or having more money than they need to fulfil their short term goals and plenty of access to more. > I don't think AI would have the goal "secretly unite to eliminate humanity", but I do think that most of the goals we can imagine it having can play out in very bad ways that achieve that outcome as a side-effect. There is an incentive for it always be looking for ways to maximise its power (compute, resources, money, influence), and this could happen in ways we really don't want. Is "people might not recognise how the second order effects of pursuing a goal adversely impact humanity" a problem unique to decisions made by AI? If anything, I would have thought that in addition to actually existing, independent human intelligences' resource demands were already more competitive with human needs than a silicon-based AGI's is particularly likely to be, and demonstrably neglectful of second order effects. With those problems mostly unsolved, it's hard to see tackling the same thorny problem but with hypothetical goals of hypothetical entities would be seen as the more impactful line of research (as interesting as it undoubtedly is). Nothing wrong with intellectually curious altruism, but if you're describing yourselves as the arbiters of effectiveness... > I'm curious - do you have a view on how we could go about reducing this risk? I don't have an easy solution to a range of complex mostly political problems, no, and certainly wouldn't argue for investing in studying them on the grounds of other philanthropy being less effective. Generally attempts to impose bans and developing more, better weapons quicker seem to have been attempted solutions to non-AI based arms races and neither of them have a great track record. Misplaced faith in the output of current techniques we call AI as superior reasoning free from human bias rather than garbage in garbage out mathematical transformation skewed by massive unseen flaws in training sets and prone to non-obvious bugs isn't likely to be reduced by talking about allying with self-updating superintelligences...

dotsam 4y ago

> That's why AI safety organisations are exempted from the expectation of even basic transparency about what they've done with marginal dollars Can you let me know more about that? > If you subscribe to the view that I do: that "secretly unite to eliminate humanity" is a fairly non-obvious and risky goal I don't think AI would have the goal "secretly unite to eliminate humanity", but I do think that most of the goals we can imagine it having can play out in very bad ways that achieve that outcome as a side-effect. There is an incentive for it always be looking for ways to maximise its power (compute, resources, money, influence), and this could happen in ways we really don't want. > Popularising the idea that it's the self aware general intelligence that's the dangerous bit about AI, not the intermediate step involving humans having access to immense calculating power whose recommendations they don't really understand seems like something of an existential risk blindspot too. It sounds like you think AI is risky too - even if only at this intermediate level? I'm curious - do you have a view on how we could go about reducing this risk?

notahacker 4y ago
> Can you let me know more about that? Compare the level of transparency and need for funds of OpenAI LP with the average organisation criticised by GiveWell for not being transparent about how they spend the money, not providing any metrics or trials, and/or having more money than they need to fulfil their short term goals and plenty of access to more. > I don't think AI would have the goal "secretly unite to eliminate humanity", but I do think that most of the goals we can imagine it having can play out in very bad ways that achieve that outcome as a side-effect. There is an incentive for it always be looking for ways to maximise its power (compute, resources, money, influence), and this could happen in ways we really don't want. Is "people might not recognise how the second order effects of pursuing a goal adversely impact humanity" a problem unique to decisions made by AI? If anything, I would have thought that in addition to actually existing, independent human intelligences' resource demands were already more competitive with human needs than a silicon-based AGI's is particularly likely to be, and demonstrably neglectful of second order effects. With those problems mostly unsolved, it's hard to see tackling the same thorny problem but with hypothetical goals of hypothetical entities would be seen as the more impactful line of research (as interesting as it undoubtedly is). Nothing wrong with intellectually curious altruism, but if you're describing yourselves as the arbiters of effectiveness... > I'm curious - do you have a view on how we could go about reducing this risk? I don't have an easy solution to a range of complex mostly political problems, no, and certainly wouldn't argue for investing in studying them on the grounds of other philanthropy being less effective. Generally attempts to impose bans and developing more, better weapons quicker seem to have been attempted solutions to non-AI based arms races and neither of them have a great track record. Misplaced faith in the output of current techniques we call AI as superior reasoning free from human bias rather than garbage in garbage out mathematical transformation skewed by massive unseen flaws in training sets and prone to non-obvious bugs isn't likely to be reduced by talking about allying with self-updating superintelligences...