5 ms·

Can you expand on why you think it's nonsense? You seem to have very strong opinions on this and I'm just wondering why.

ajross 4y ago
It's exasperation more than anything else. Again, these are people who came so close to thinking about genuine issues of welfare, put their money behind their blog posts, and then... walked back into the introvert weeds of worrying about "stuff that looks more fun to worry about". Now all those billions of dollars are going to do, what, fund speculative sci-fi about AI? I mean, I loved The Terminator too, but this isn't what we were promised.
- atlasunshrugged 4y ago
  I mean, I also agree that I wish more people cared about global development but I'm pretty sure tremendous amounts of funding still go to those areas from the EA community (e.g. Givewell now directs hundreds of millions in funding that almost all go to more traditional development). I can't find the post but there was one recently breaking this down on the EA forum.
- kalkin 4y ago
  How seriously have you thought about the arguments for AI risk being more than speculative scifi? My take as recently as a year ago was pretty much exactly yours, but I've had time to read and think a bit more recently and I am much less sanguine than I was.
  - kalkin 4y ago
    A bit more specifically, "misalignment" is super common in existing ML, and models seem to be gaining capability (and knocking down targets that until recently seemed well beyond our reach) much more rapidly than we are gaining the ability to understand their internals or make guarantees of any kind about their worst-case or out-of-distribution behavior.
- jbay808 4y ago
  It's possible that they aren't worrying because it's fun to worry about, but because they're actually worried about it, and not having any fun? I mean, I'm worried about climate change, but not because it's fun to worry about. And I put a lot of my time and energy into thinking about how to do something about it effectively, but not so much because it's fun. Sometimes I'll hit on a fun idea in the process, but I'd be much happier if I didn't have to worry about it at all, but that's not really a choice I have. I'm not sure how much I should worry about AI, but I'm certain that for the people who are worried about it, it's because they're convinced it's a threat, and they aren't having fun.
  - s17n 4y ago
    > I'm not sure how much I should worry about AI, but I'm certain that for the people who are worried about it, it's because they're convinced it's a threat, and they aren't having fun. I've met a lot of these people and regardless of how you feel about the sincerity of their convictions, they are definitely having fun.
    - jbay808 4y ago
      I guess we've met different people!
    - MichaelZuo 4y ago
      Why do you believe you met a representative cross section of their community?
- dotsam 4y ago
  Firstly, something can be more fun to worry about (for some people), sound like speculative sci-fi (to some people) and nevertheless still be a real problem. Consider the dawn of the nuclear age: > On the morning of September 12, 1933, on a miserable, wet, quintessentially English autumn day, at the intersection where Russell Square meets Southampton Row, Leó Szilárd waited irritably at a traffic light waiting for it to change from red to green. He had just attended a lecture by the great English physicist Ernest Rutherford. Rutherford, known to many as the father of nuclear physics, was discussing the newly prophesied release of energy from atoms, most notably by science-fiction pioneer H G Wells in his book The World Set Free. In his baritone voice, Rutherford, acknowledged master of the atomic domain, dismissed this fanciful idea as nonsense. Any thought of releasing the energy locked in atoms, he said, was "moonshine". ... > What Szilard realised as he stepped off that curb was that if we found an element that when bombarded by one neutron would release two neutrons, it could lead to a chain reaction that could possibly release vast amounts of energy https://blogs.scientificamerican.com/the-curious-wavefunction/leo-szilard-a-traffic-light-and-a-slice-of-nuclear-history/ https://blogs.scientificamerican.com/the-curious-wavefunctio... Secondly, you can have a diverse portfolio of causes you care about, and Effective Altruism has recommendations for areas including global health and animal welfare. Taking AI risk seriously doesn't mean you can't give time and money to other causes with more tangible results.
  - notahacker 4y ago
    > Taking AI risk seriously doesn't mean you can't give time and money to other causes with more tangible results. But if you've spent the past decade arguing precisely the opposite - that instead of spreading philanthropic dollars over broad portfolios because true efficacy is unmeasurable, they should be directed to causes which provide hard evidence that they're among the most efficient use of a marginal dollar, and that charities that aren't conducting RCTs to prove how much better off people are after being fed aren't really evidence-based enough to recommend - it's going to raise eyebrows when you pivot to recommending people also give their money to opaque organisations with vague goals and zero evidence they can achieve them, whose only thing in common with the highly evidence-focused aid agencies is that their pitch is nerdier than the average nonprofit. I mean, there's nothing particularly wrong with "I give to causes because their pitch speaks to me" as a heuristic for philanthropy - it's how most altruists think - but it's not how EA pitched itself as being different from that and caring about efficacy and evidence. The Wells analogy is awkward for the AI safety crowd for another reason too. Wells' central thesis that an atomic super weapon would guarantee a peaceful, utopian world turned out to be horribly wrong. We got actually used bombs, arms races, proxy wars and very nearly used bombs instead, and that's without Nazi atomic scientists getting a heads up. To the extent his writings on the subject of atomic weapon safety had any effect at all on atomic weapons, they made them more imminent and more likely to be used. If marginal dollars spent on promoting the distribution of The World Set Free would have had any effect at all on the probability of armageddon, they would probably have increased it.
- arinlen 4y ago
  > Again, these are people who came so close to thinking about genuine issues of welfare, put their money behind their blog posts, and then... walked back into the introvert weeds of worrying about (...) So many words just to say "they don't value the exact same stuff I value". You're free to put your money where your heart is, just like them. It's not right to hit back at those who are doing what's right by their own account.
  - ajross 4y ago
    > You're free to put your money where your heart is, just like them. This argument is appearing again and again in this subthread. "Putting your money where your heart is" is the exact opposite of what EA claims to believe. EA is about investing effort and money into areas with the highest marginal value to humanity. So if they're spending their time talking about the robot invasion, we can only conclude that's where they think that effort should be spent. Obviously they're free to do whatever they want with their own time and money. But if they spend that time pretending that SkyNet or WOPR is the greatest threat to humanity that the rest of us should be spending our marginal effort on, it's on the rest of us to call them out on that nosense.
    - arinlen 4y ago
      > EA is about investing effort and money into areas with the highest marginal value to humanity. And it's the individual who determines that? Or does your own personal opinion override everyone else's world views?
      - ajross 4y ago
        If I think you're wrong in your advocacy for some policy initiative, it's absolutely my responsibility[1] to tell people you're wrong. (It's also my right to tell people how exasperating I find your wrong priorities to be given the principles you claim are behind them, FWIW.) [1] "Responsibitility", i.e. not just my "right" in a liberty sense! Again, the theory behind EA is that we should collectively decide on the most effective altruistic actions based on rationalist principles. Those principles absolutely include "you're wrong about the robot overlords thing".
- arisAlexis 4y ago
  You seem to be very misinformed about the potential AI impact. Some reading wouldn't be bad. I urge you to read Bostroms "Superintelligence".
kiliantics 4y ago
I have a slightly different take to the parent. I think it's nonsense because the ostensible core purpose of this movement is about taking competent, motivated people and deploying them in such a way that they can make a meaningful change in how the world works. However, the main effect of their recommendation is to go out and participate in the system in a pretty standard way. How will meaningful change happen in a system in which you aren't acting in any way differently to the standard operations of the system? Sure, you donated a significant fraction of your income to some non-profit. But that effect pales in comparison to the economic contribution of your labor in big tech/finance/whatever which are the dominant institutions structuring our society. And most of the non-development problems such as AI risk or nuclear risk come from the same kinds of structure dynamics of society that caused the wealth inequality and environmental degradation. So I believe, for a group supposedly run by very smart people with very noble ideals, they kind of fall pretty short of where they could be and don't go deep enough. They are proposing band-aids to problems rather than true cures. To me it just becomes another part of the non-profit industrial complex, a neoliberal-shaped industry that effectively perpetuates the problems it purports to solve since it relies on the existence of those problems to survive. We've known for a long time how charity and aid can do more harm than good and plenty of strategies formerly championed by 80000 hours have now been found to be more extractive/exploitative than helpful, such as microlending. Fundamentally, they just can't seem to make the necessary leap to criticising capitalism and strategising against it, instead promoting it as the preferred tool to use. But how will capitalist tools solve the problems caused by capitalism? If instead they got all their smart followers to use their privileged positions and training not to reinforce the most powerful institutions of capital but instead build alternatives, I would think they are on the right track, even if it included AI risk initiatives.
- jefftk 4y ago
  > plenty of strategies formerly championed by 80000 hours have now been found to be more extractive/exploitative than helpful, such as microlending. I don't think 80,000 Hours has ever promoted microlending. They were critical of it ten years ago: https://80000hours.org/2012/11/is-microcredit-mostly-hype/ https://80000hours.org/2012/11/is-microcredit-mostly-hype/

ajross 4y ago

It's exasperation more than anything else. Again, these are people who came so close to thinking about genuine issues of welfare, put their money behind their blog posts, and then... walked back into the introvert weeds of worrying about "stuff that looks more fun to worry about". Now all those billions of dollars are going to do, what, fund speculative sci-fi about AI? I mean, I loved The Terminator too, but this isn't what we were promised.

atlasunshrugged 4y ago
I mean, I also agree that I wish more people cared about global development but I'm pretty sure tremendous amounts of funding still go to those areas from the EA community (e.g. Givewell now directs hundreds of millions in funding that almost all go to more traditional development). I can't find the post but there was one recently breaking this down on the EA forum.
kalkin 4y ago
How seriously have you thought about the arguments for AI risk being more than speculative scifi? My take as recently as a year ago was pretty much exactly yours, but I've had time to read and think a bit more recently and I am much less sanguine than I was.
- kalkin 4y ago
  A bit more specifically, "misalignment" is super common in existing ML, and models seem to be gaining capability (and knocking down targets that until recently seemed well beyond our reach) much more rapidly than we are gaining the ability to understand their internals or make guarantees of any kind about their worst-case or out-of-distribution behavior.
jbay808 4y ago
It's possible that they aren't worrying because it's fun to worry about, but because they're actually worried about it, and not having any fun? I mean, I'm worried about climate change, but not because it's fun to worry about. And I put a lot of my time and energy into thinking about how to do something about it effectively, but not so much because it's fun. Sometimes I'll hit on a fun idea in the process, but I'd be much happier if I didn't have to worry about it at all, but that's not really a choice I have. I'm not sure how much I should worry about AI, but I'm certain that for the people who are worried about it, it's because they're convinced it's a threat, and they aren't having fun.
- s17n 4y ago
  > I'm not sure how much I should worry about AI, but I'm certain that for the people who are worried about it, it's because they're convinced it's a threat, and they aren't having fun. I've met a lot of these people and regardless of how you feel about the sincerity of their convictions, they are definitely having fun.
  - jbay808 4y ago
    I guess we've met different people!
  - MichaelZuo 4y ago
    Why do you believe you met a representative cross section of their community?
dotsam 4y ago
Firstly, something can be more fun to worry about (for some people), sound like speculative sci-fi (to some people) and nevertheless still be a real problem. Consider the dawn of the nuclear age: > On the morning of September 12, 1933, on a miserable, wet, quintessentially English autumn day, at the intersection where Russell Square meets Southampton Row, Leó Szilárd waited irritably at a traffic light waiting for it to change from red to green. He had just attended a lecture by the great English physicist Ernest Rutherford. Rutherford, known to many as the father of nuclear physics, was discussing the newly prophesied release of energy from atoms, most notably by science-fiction pioneer H G Wells in his book The World Set Free. In his baritone voice, Rutherford, acknowledged master of the atomic domain, dismissed this fanciful idea as nonsense. Any thought of releasing the energy locked in atoms, he said, was "moonshine". ... > What Szilard realised as he stepped off that curb was that if we found an element that when bombarded by one neutron would release two neutrons, it could lead to a chain reaction that could possibly release vast amounts of energy https://blogs.scientificamerican.com/the-curious-wavefunction/leo-szilard-a-traffic-light-and-a-slice-of-nuclear-history/ https://blogs.scientificamerican.com/the-curious-wavefunctio... Secondly, you can have a diverse portfolio of causes you care about, and Effective Altruism has recommendations for areas including global health and animal welfare. Taking AI risk seriously doesn't mean you can't give time and money to other causes with more tangible results.
- notahacker 4y ago
  > Taking AI risk seriously doesn't mean you can't give time and money to other causes with more tangible results. But if you've spent the past decade arguing precisely the opposite - that instead of spreading philanthropic dollars over broad portfolios because true efficacy is unmeasurable, they should be directed to causes which provide hard evidence that they're among the most efficient use of a marginal dollar, and that charities that aren't conducting RCTs to prove how much better off people are after being fed aren't really evidence-based enough to recommend - it's going to raise eyebrows when you pivot to recommending people also give their money to opaque organisations with vague goals and zero evidence they can achieve them, whose only thing in common with the highly evidence-focused aid agencies is that their pitch is nerdier than the average nonprofit. I mean, there's nothing particularly wrong with "I give to causes because their pitch speaks to me" as a heuristic for philanthropy - it's how most altruists think - but it's not how EA pitched itself as being different from that and caring about efficacy and evidence. The Wells analogy is awkward for the AI safety crowd for another reason too. Wells' central thesis that an atomic super weapon would guarantee a peaceful, utopian world turned out to be horribly wrong. We got actually used bombs, arms races, proxy wars and very nearly used bombs instead, and that's without Nazi atomic scientists getting a heads up. To the extent his writings on the subject of atomic weapon safety had any effect at all on atomic weapons, they made them more imminent and more likely to be used. If marginal dollars spent on promoting the distribution of The World Set Free would have had any effect at all on the probability of armageddon, they would probably have increased it.
arinlen 4y ago
> Again, these are people who came so close to thinking about genuine issues of welfare, put their money behind their blog posts, and then... walked back into the introvert weeds of worrying about (...) So many words just to say "they don't value the exact same stuff I value". You're free to put your money where your heart is, just like them. It's not right to hit back at those who are doing what's right by their own account.
- ajross 4y ago
  > You're free to put your money where your heart is, just like them. This argument is appearing again and again in this subthread. "Putting your money where your heart is" is the exact opposite of what EA claims to believe. EA is about investing effort and money into areas with the highest marginal value to humanity. So if they're spending their time talking about the robot invasion, we can only conclude that's where they think that effort should be spent. Obviously they're free to do whatever they want with their own time and money. But if they spend that time pretending that SkyNet or WOPR is the greatest threat to humanity that the rest of us should be spending our marginal effort on, it's on the rest of us to call them out on that nosense.
  - arinlen 4y ago
    > EA is about investing effort and money into areas with the highest marginal value to humanity. And it's the individual who determines that? Or does your own personal opinion override everyone else's world views?
    - ajross 4y ago
      If I think you're wrong in your advocacy for some policy initiative, it's absolutely my responsibility[1] to tell people you're wrong. (It's also my right to tell people how exasperating I find your wrong priorities to be given the principles you claim are behind them, FWIW.) [1] "Responsibitility", i.e. not just my "right" in a liberty sense! Again, the theory behind EA is that we should collectively decide on the most effective altruistic actions based on rationalist principles. Those principles absolutely include "you're wrong about the robot overlords thing".
arisAlexis 4y ago
You seem to be very misinformed about the potential AI impact. Some reading wouldn't be bad. I urge you to read Bostroms "Superintelligence".

kiliantics 4y ago

I have a slightly different take to the parent. I think it's nonsense because the ostensible core purpose of this movement is about taking competent, motivated people and deploying them in such a way that they can make a meaningful change in how the world works. However, the main effect of their recommendation is to go out and participate in the system in a pretty standard way. How will meaningful change happen in a system in which you aren't acting in any way differently to the standard operations of the system? Sure, you donated a significant fraction of your income to some non-profit. But that effect pales in comparison to the economic contribution of your labor in big tech/finance/whatever which are the dominant institutions structuring our society. And most of the non-development problems such as AI risk or nuclear risk come from the same kinds of structure dynamics of society that caused the wealth inequality and environmental degradation. So I believe, for a group supposedly run by very smart people with very noble ideals, they kind of fall pretty short of where they could be and don't go deep enough. They are proposing band-aids to problems rather than true cures. To me it just becomes another part of the non-profit industrial complex, a neoliberal-shaped industry that effectively perpetuates the problems it purports to solve since it relies on the existence of those problems to survive. We've known for a long time how charity and aid can do more harm than good and plenty of strategies formerly championed by 80000 hours have now been found to be more extractive/exploitative than helpful, such as microlending. Fundamentally, they just can't seem to make the necessary leap to criticising capitalism and strategising against it, instead promoting it as the preferred tool to use. But how will capitalist tools solve the problems caused by capitalism? If instead they got all their smart followers to use their privileged positions and training not to reinforce the most powerful institutions of capital but instead build alternatives, I would think they are on the right track, even if it included AI risk initiatives.

jefftk 4y ago
> plenty of strategies formerly championed by 80000 hours have now been found to be more extractive/exploitative than helpful, such as microlending. I don't think 80,000 Hours has ever promoted microlending. They were critical of it ten years ago: https://80000hours.org/2012/11/is-microcredit-mostly-hype/ https://80000hours.org/2012/11/is-microcredit-mostly-hype/