5 ms·

> That's why AI safety organisations are exempted from the expectation of even basic transparency about what they've done with marginal dollars Can you let me

by dotsam 4y ago

> That's why AI safety organisations are exempted from the expectation of even basic transparency about what they've done with marginal dollars Can you let me know more about that? > If you subscribe to the view that I do: that "secretly unite to eliminate humanity" is a fairly non-obvious and risky goal I don't think AI would have the goal "secretly unite to eliminate humanity", but I do think that most of the goals we can imagine it having can play out in very bad ways that achieve that outcome as a side-effect. There is an incentive for it always be looking for ways to maximise its power (compute, resources, money, influence), and this could happen in ways we really don't want. > Popularising the idea that it's the self aware general intelligence that's the dangerous bit about AI, not the intermediate step involving humans having access to immense calculating power whose recommendations they don't really understand seems like something of an existential risk blindspot too. It sounds like you think AI is risky too - even if only at this intermediate level? I'm curious - do you have a view on how we could go about reducing this risk?

notahacker 4y ago
> Can you let me know more about that? Compare the level of transparency and need for funds of OpenAI LP with the average organisation criticised by GiveWell for not being transparent about how they spend the money, not providing any metrics or trials, and/or having more money than they need to fulfil their short term goals and plenty of access to more. > I don't think AI would have the goal "secretly unite to eliminate humanity", but I do think that most of the goals we can imagine it having can play out in very bad ways that achieve that outcome as a side-effect. There is an incentive for it always be looking for ways to maximise its power (compute, resources, money, influence), and this could happen in ways we really don't want. Is "people might not recognise how the second order effects of pursuing a goal adversely impact humanity" a problem unique to decisions made by AI? If anything, I would have thought that in addition to actually existing, independent human intelligences' resource demands were already more competitive with human needs than a silicon-based AGI's is particularly likely to be, and demonstrably neglectful of second order effects. With those problems mostly unsolved, it's hard to see tackling the same thorny problem but with hypothetical goals of hypothetical entities would be seen as the more impactful line of research (as interesting as it undoubtedly is). Nothing wrong with intellectually curious altruism, but if you're describing yourselves as the arbiters of effectiveness... > I'm curious - do you have a view on how we could go about reducing this risk? I don't have an easy solution to a range of complex mostly political problems, no, and certainly wouldn't argue for investing in studying them on the grounds of other philanthropy being less effective. Generally attempts to impose bans and developing more, better weapons quicker seem to have been attempted solutions to non-AI based arms races and neither of them have a great track record. Misplaced faith in the output of current techniques we call AI as superior reasoning free from human bias rather than garbage in garbage out mathematical transformation skewed by massive unseen flaws in training sets and prone to non-obvious bugs isn't likely to be reduced by talking about allying with self-updating superintelligences...

by dotsam 4y ago

notahacker 4y ago

> Can you let me know more about that? Compare the level of transparency and need for funds of OpenAI LP with the average organisation criticised by GiveWell for not being transparent about how they spend the money, not providing any metrics or trials, and/or having more money than they need to fulfil their short term goals and plenty of access to more. > I don't think AI would have the goal "secretly unite to eliminate humanity", but I do think that most of the goals we can imagine it having can play out in very bad ways that achieve that outcome as a side-effect. There is an incentive for it always be looking for ways to maximise its power (compute, resources, money, influence), and this could happen in ways we really don't want. Is "people might not recognise how the second order effects of pursuing a goal adversely impact humanity" a problem unique to decisions made by AI? If anything, I would have thought that in addition to actually existing, independent human intelligences' resource demands were already more competitive with human needs than a silicon-based AGI's is particularly likely to be, and demonstrably neglectful of second order effects. With those problems mostly unsolved, it's hard to see tackling the same thorny problem but with hypothetical goals of hypothetical entities would be seen as the more impactful line of research (as interesting as it undoubtedly is). Nothing wrong with intellectually curious altruism, but if you're describing yourselves as the arbiters of effectiveness... > I'm curious - do you have a view on how we could go about reducing this risk? I don't have an easy solution to a range of complex mostly political problems, no, and certainly wouldn't argue for investing in studying them on the grounds of other philanthropy being less effective. Generally attempts to impose bans and developing more, better weapons quicker seem to have been attempted solutions to non-AI based arms races and neither of them have a great track record. Misplaced faith in the output of current techniques we call AI as superior reasoning free from human bias rather than garbage in garbage out mathematical transformation skewed by massive unseen flaws in training sets and prone to non-obvious bugs isn't likely to be reduced by talking about allying with self-updating superintelligences...