4 ms·

"Maybe it's some kind of horrible drug interaction with Rationalism, or maybe EA is fundamentally Rationalist?" I have always associated EA with the rationalis

"Maybe it's some kind of horrible drug interaction with Rationalism, or maybe EA is fundamentally Rationalist?" I have always associated EA with the rationalist movement, since I came across it via reading rationalist stuff. If you define EA as "figuring out how to deploy a given set of resources so as to do the most good," your conclusions are going to depend on your value system: how do you define and measure good? So when a bunch of rationalists who believe in consequentialism do that, they're going to maximize their value system and end up looking really weird to everyone else who doesn't share their values. You'd have the same outcome in a different flavor if a bunch of Christians did it. I disagree with a bunch of the underlying values in the EA movement but I really admire the mindset that actually tries to solve the problem of "how can we be most effective with our giving?" instead of what I personally do, which is get overwhelmed by guilt and confusion, stop thinking about helping other people, and go back to my default of living my life mostly thinking about how to maximize my own comfort and preferences.

UncleMeat 4y ago
One of the problems I observe in forum posts from these communities is that it is fun and cool to think differently than others. "What if fighting poverty now is actually bad and preventing the AI apocalypse later is actually better?" Then you get to feel superior to everybody, even those goody-two-shoes people down the street that donate a lot to the food bank. You know the secrets. It is the same kind of thinking that makes people think that Roko's Basilisk is cool or frightening or meaningful. It makes you feel like you have secret knowledge that others do not. But... because we tend to interrogate our second option less than our first option people end up throwing their EA money at fucking ridiculous things while there are people out there dying in childbirth because their local hospital doesn't have consistent electricity. "Yeah, I just have a different value function than everybody else" does not absolve people from the criticism that their value function is stupid.
- heavyset_go 4y ago
  If you aren't aware already: https://rationalwiki.org/wiki/Essay:Second-option_bias https://rationalwiki.org/wiki/Essay:Second-option_bias
- wolverine876 4y ago
  Like much tech wealth wisdom, it's rationalization of their megalomania, their self-serving desire for power. Humans naturally desire power to varying degrees, but we have other desires and we have other things that limit us (e.g., our consciences, or Adam Smith's 'impartial spectator'). With enough real power, you can ignore these things and others will not only fail to check you, but will encourage you. Power corrupts.
jerf 4y ago
"If you define EA as "figuring out how to deploy a given set of resources so as to do the most good," your conclusions are going to depend on your value system: how do you define and measure good?" I think more subtly it also depends on your calculation methodology. Rationalists talk a lot about the value of numbers for their Bayesian analyses and their "shut up and calculate" motto, but to my mind they overestimate the accuracy they have in their calculations by a lot. Of course I want to go with the 99% likely-to-help choice over the 25%-likely-to-help choice, but along with the fact my value system is indeed much different than theirs (in some important ways, infinitely so), I think all their Bayesian calculations need error bars in them. I think some rationalists would claim their probabilities have that error built into them, and I disagree. I think 75% likely is very different from 75% +/- 20% [1] when you start feeding that number through a chain of other calculations. A chain of calculations that nominally results in a 99.99% probability is a lot less impressive if a more honest calculation that propagates error bars correctly results in "somewhere between 2% and 99.9999%". It becomes more clear that the 99.99% is a very suspect number, even if it is nominally the middle of the distribution, especially given the number of times out in the real world the "highly improbable" ends up happening when it turns out that the assessment of the outcome being "highly improbable" was wrong. The probability distributions in the real world are highly pathological much more often than a naive analysis would suggest. When analyzing the virtues of stealing billions to donate it to an effective charity, you better not just think that the value of the charity is higher than the cost of the loss of the money, you better be really sure your calculation is correct. I think the rationalists could argue amoungst themselves endlessly on the first point, but I do not believe they have any way whatsoever to establish their second point to my satisfaction, which renders the entire conversation moot, when there is no way they could ever convince me that their calculation is correct regardless of what it is. Even when my value system does in fact line up with theirs, and there is non-trivial overlap, I disagree with their calculation methodologies quite substantially as I think they consistently act with much more confidence then is justified. Against that, I'm comfortable with the societally-evolved (evolution is important too, right, rationalists?) standard that in addition to the first-order effects of theft, one much consider the second-order effects of the dangers of normalizing outright theft because of the fact that someone else thinks they could use the resources better. Even if that is locally true, if everyone globally acts like that, global utility will plummet because everybody always thinks everybody else could spend their money better and the resulting chaos will not be helpful. (The Kantian imperative is completely inadequate as a theoretical foundation of philosophy but I do find it a useful practical tool, as long as one is aware to not place too much confidence in an extended analysis using it... the theme of this post comes up yet again.) The world is not improved by turning everyone into a central authority over all resource allocation in the world. [1]: I know real error bars in probability space would never look like this, but mathematical accuracy would distract from my point here.
tptacek 4y ago
"How can we be most effective with our giving" is a good thought. It's what I thought EA was, like, two weeks ago. "The movement can literally compute the optimal pattern of giving, such that to be most effective with our giving means donating money to people to ride around on bicycles thinking about how to prevent future AGIs from killing us" is not a good thought, but it turns out to be a description of what EA is like; some of this is on their own website (another thing I've learned: beware any movement that calls its documentation "sequences"). "The moral urgency of giving is such that a reasonable person should seek to maximize their income" is a bad thought. It leads to dark places. "Earning to give" is a reasonable idea: it is probably more effective for people to excel at their jobs and donate the proceeds than for them to phone bank or canvass. But Rationalists can't seem to help falling into these asymptote traps, so that "donating a portion of ones income" becomes "pursue billions in wealth, and clear a path for other billionaires, to enable concentrated giving". That's a demented idea. The problems here aren't abstract. They just lead to Scott Aaronson withholding moral judgement on Sam Bankman-Fried, strictly because of the logic of the preceding paragraph.
- FormerBandmate 4y ago
  Rationalists brought us Roko’s Basilisk after all. A lot of the leaders don’t seem to have basic sanity checks in their brains that stop their ideas from getting wildly out of control, which can lead to stupid shit like the Basilisk on a small scale but end up justifying fraud on a much larger scale. It’s like fundamentalist religion
  - Viliam1234 4y ago
    > Rationalists brought us Roko’s Basilisk after all. This reminds me of the Zen story about two monks and a woman (one helped her cross a river, and the other couldn't stop thinking about her for the whole day). The "Roko's Basilisk" was an edgy comment that was written on a rationalist web forum, and then deleted by the admin. A decade later, some people outside the rationalist community still keep discussing it, which is the only reason why anyone cares. > A lot of the leaders don’t seem to have basic sanity checks in their brains that stop their ideas from getting wildly out of control Only in this story, the admin was the one who deleted that comment, not the one who wrote it. Still gets blamed for it.
Centigonal 4y ago
> If you define EA as "figuring out how to deploy a given set of resources so as to do the most good," your conclusions are going to depend on your value system: how do you define and measure good? This is exactly it. Because "good" is subjective to the individual, EA without some kind of consensus building reduces to "maximize your agency as an individual in the world, and do what you think is good," which gets uncomfortably close to Ayn Rand territory for me. That's not just speculative either - if you read some of Caroline Ellison (Ex-CEO of Alameda Capital and an EA disciple) purported tumblr posts, it paints an image of someone who's passionate about ranking the world's issues in descending order of priority, investigating genetic differences between humans from different ethnic backgrounds, and "optimizing the world." We're not there yet, but it's also not a stretch to see EA turn into eugenics for someone with these interests and values.
- PebblesRox 4y ago
  > EA without some kind of consensus building reduces to "maximize your agency as an individual in the world.... I agree and would say that to the extent that there is a consensus, it becomes “maximize our agency as a coordinated group of people” – hence the focus on promoting and expanding EA as a movement
wolverine876 4y ago
> your conclusions are going to depend on your value system That's a logical extreme, not reality; it's a philosophy class debate, used to rationalize everything and not applicable to life, where survival, freedom, prosperity, and happiness depend on our abilty to judge and discern - especially between values.

UncleMeat 4y ago

One of the problems I observe in forum posts from these communities is that it is fun and cool to think differently than others. "What if fighting poverty now is actually bad and preventing the AI apocalypse later is actually better?" Then you get to feel superior to everybody, even those goody-two-shoes people down the street that donate a lot to the food bank. You know the secrets. It is the same kind of thinking that makes people think that Roko's Basilisk is cool or frightening or meaningful. It makes you feel like you have secret knowledge that others do not. But... because we tend to interrogate our second option less than our first option people end up throwing their EA money at fucking ridiculous things while there are people out there dying in childbirth because their local hospital doesn't have consistent electricity. "Yeah, I just have a different value function than everybody else" does not absolve people from the criticism that their value function is stupid.

heavyset_go 4y ago
If you aren't aware already: https://rationalwiki.org/wiki/Essay:Second-option_bias https://rationalwiki.org/wiki/Essay:Second-option_bias
wolverine876 4y ago
Like much tech wealth wisdom, it's rationalization of their megalomania, their self-serving desire for power. Humans naturally desire power to varying degrees, but we have other desires and we have other things that limit us (e.g., our consciences, or Adam Smith's 'impartial spectator'). With enough real power, you can ignore these things and others will not only fail to check you, but will encourage you. Power corrupts.

jerf 4y ago

"If you define EA as "figuring out how to deploy a given set of resources so as to do the most good," your conclusions are going to depend on your value system: how do you define and measure good?" I think more subtly it also depends on your calculation methodology. Rationalists talk a lot about the value of numbers for their Bayesian analyses and their "shut up and calculate" motto, but to my mind they overestimate the accuracy they have in their calculations by a lot. Of course I want to go with the 99% likely-to-help choice over the 25%-likely-to-help choice, but along with the fact my value system is indeed much different than theirs (in some important ways, infinitely so), I think all their Bayesian calculations need error bars in them. I think some rationalists would claim their probabilities have that error built into them, and I disagree. I think 75% likely is very different from 75% +/- 20% [1] when you start feeding that number through a chain of other calculations. A chain of calculations that nominally results in a 99.99% probability is a lot less impressive if a more honest calculation that propagates error bars correctly results in "somewhere between 2% and 99.9999%". It becomes more clear that the 99.99% is a very suspect number, even if it is nominally the middle of the distribution, especially given the number of times out in the real world the "highly improbable" ends up happening when it turns out that the assessment of the outcome being "highly improbable" was wrong. The probability distributions in the real world are highly pathological much more often than a naive analysis would suggest. When analyzing the virtues of stealing billions to donate it to an effective charity, you better not just think that the value of the charity is higher than the cost of the loss of the money, you better be really sure your calculation is correct. I think the rationalists could argue amoungst themselves endlessly on the first point, but I do not believe they have any way whatsoever to establish their second point to my satisfaction, which renders the entire conversation moot, when there is no way they could ever convince me that their calculation is correct regardless of what it is. Even when my value system does in fact line up with theirs, and there is non-trivial overlap, I disagree with their calculation methodologies quite substantially as I think they consistently act with much more confidence then is justified. Against that, I'm comfortable with the societally-evolved (evolution is important too, right, rationalists?) standard that in addition to the first-order effects of theft, one much consider the second-order effects of the dangers of normalizing outright theft because of the fact that someone else thinks they could use the resources better. Even if that is locally true, if everyone globally acts like that, global utility will plummet because everybody always thinks everybody else could spend their money better and the resulting chaos will not be helpful. (The Kantian imperative is completely inadequate as a theoretical foundation of philosophy but I do find it a useful practical tool, as long as one is aware to not place too much confidence in an extended analysis using it... the theme of this post comes up yet again.) The world is not improved by turning everyone into a central authority over all resource allocation in the world. [1]: I know real error bars in probability space would never look like this, but mathematical accuracy would distract from my point here.

tptacek 4y ago

"How can we be most effective with our giving" is a good thought. It's what I thought EA was, like, two weeks ago. "The movement can literally compute the optimal pattern of giving, such that to be most effective with our giving means donating money to people to ride around on bicycles thinking about how to prevent future AGIs from killing us" is not a good thought, but it turns out to be a description of what EA is like; some of this is on their own website (another thing I've learned: beware any movement that calls its documentation "sequences"). "The moral urgency of giving is such that a reasonable person should seek to maximize their income" is a bad thought. It leads to dark places. "Earning to give" is a reasonable idea: it is probably more effective for people to excel at their jobs and donate the proceeds than for them to phone bank or canvass. But Rationalists can't seem to help falling into these asymptote traps, so that "donating a portion of ones income" becomes "pursue billions in wealth, and clear a path for other billionaires, to enable concentrated giving". That's a demented idea. The problems here aren't abstract. They just lead to Scott Aaronson withholding moral judgement on Sam Bankman-Fried, strictly because of the logic of the preceding paragraph.

FormerBandmate 4y ago
Rationalists brought us Roko’s Basilisk after all. A lot of the leaders don’t seem to have basic sanity checks in their brains that stop their ideas from getting wildly out of control, which can lead to stupid shit like the Basilisk on a small scale but end up justifying fraud on a much larger scale. It’s like fundamentalist religion
- Viliam1234 4y ago
  > Rationalists brought us Roko’s Basilisk after all. This reminds me of the Zen story about two monks and a woman (one helped her cross a river, and the other couldn't stop thinking about her for the whole day). The "Roko's Basilisk" was an edgy comment that was written on a rationalist web forum, and then deleted by the admin. A decade later, some people outside the rationalist community still keep discussing it, which is the only reason why anyone cares. > A lot of the leaders don’t seem to have basic sanity checks in their brains that stop their ideas from getting wildly out of control Only in this story, the admin was the one who deleted that comment, not the one who wrote it. Still gets blamed for it.

Centigonal 4y ago

> If you define EA as "figuring out how to deploy a given set of resources so as to do the most good," your conclusions are going to depend on your value system: how do you define and measure good? This is exactly it. Because "good" is subjective to the individual, EA without some kind of consensus building reduces to "maximize your agency as an individual in the world, and do what you think is good," which gets uncomfortably close to Ayn Rand territory for me. That's not just speculative either - if you read some of Caroline Ellison (Ex-CEO of Alameda Capital and an EA disciple) purported tumblr posts, it paints an image of someone who's passionate about ranking the world's issues in descending order of priority, investigating genetic differences between humans from different ethnic backgrounds, and "optimizing the world." We're not there yet, but it's also not a stretch to see EA turn into eugenics for someone with these interests and values.

PebblesRox 4y ago
> EA without some kind of consensus building reduces to "maximize your agency as an individual in the world.... I agree and would say that to the extent that there is a consensus, it becomes “maximize our agency as a coordinated group of people” – hence the focus on promoting and expanding EA as a movement

wolverine876 4y ago

> your conclusions are going to depend on your value system That's a logical extreme, not reality; it's a philosophy class debate, used to rationalize everything and not applicable to life, where survival, freedom, prosperity, and happiness depend on our abilty to judge and discern - especially between values.