4 ms·

> People’s terminal preferences are the root of who they are. There is no goal that exists beyond one’s terminal preferences. You can prefer to obey some moral

by zepto 5y ago

> People’s terminal preferences are the root of who they are. There is no goal that exists beyond one’s terminal preferences. You can prefer to obey some moral distinction, but you prefer it because that increases your happiness/life satisfaction. If it didn’t, why would your brain ever bother to spit out a “you should do it” answer? This is circular reasoning. You have baked in the assumption that people operate by maximizing their individual self-interest. A simple answer to why one might do it, even if it didn’t maximize their own happiness/satisfaction, is that they are genuinely focused on a less individualistic aim. Self-sacrifice in service of higher value doesn’t necessarily make one happy, but it can be part of one’s nature to recognize that one’s own sacrifice benefits others in some way.

derefr 5y ago
You misunderstand what I mean by “happiness”, I think. (Probably because I should have said “predicted utility of all agents similar to oneself”, but I didn’t, because that’s pretty dang jargon-y, and assumes you’ve walked through the relevant thought experiments about self-sacrifice to help other copies of yourself, etc.) If someone martyrs themselves, then in the moment they do that, they’re thinking that that is a good thing to do — i.e. the reward calculation in their brain comes out in favour of doing that thing instead of anything else they could be doing instead. They expect that good things will happen due to this—things that cohere with their preferences. This is still true even if they don’t live to see those good things happen, because brains are adaptation executors, not fitness maximizers: we prefer the kinds of things that made our lineage successful, not the things that make us successful. But those are still preferences! “Being of use to others” is a preference. You do it because that’s the thing your brain releases reward chemicals when you do. This is what you are terming something being “part of one’s nature”; “your nature” is equivalent to “your preference function.” And, if you can assume that to be true for a second, then I can restate my point in terms of it: many of the things people have terms for in their preference functions are based on adhering to an abstract role or identity, something like “an honorable person”, their understanding of which was inculcated by local, culturally-contextual societal mores. For most people, doing something “because it’s honorable” isn’t a post-hoc rationalization of some spiritual connection to the concept of honor that informs their life. Instead, it’s an instrumental preference — a thing you do in order to work toward some higher-level preference you have — where the higher-level preference is “cohere more closely with my own understanding of what it means to be honorable.” In these cases, people don’t like X and dislike Y because their brain is directly telling them that X is good and Y isn’t (where they then notice a class of such preferential distinctions they’re making, and then call that class of distinctions “honor” because it seems to be the same as a class of distinctions that other people call “honor.”) Instead, people like X and dislike Y because they learned a concept like “honor” from their societal context, and they learned that their society considers X to be honorable and Y to be dishonorable; and so they then prefer to do X and not do Y because they want to live in the image of what they believe honor to be, as learned from their society. For these people, if they experience other cultures’ conceptions of these abstract concepts, and through that learn more about what these abstract concepts can mean to different people, this is usually enough to shift how they understand the abstract concept itself; and due to that, they will usually end up with different instrumental preferences—because now what they have learned “is honorable”, is a different set of things. (Compare and contrast: etiquette. “What is etiquette” can feel objective if you never leave the culture you were born into. But you’ll have a very different idea of things being or not being “polite” if you experience a variety of different cultures that all have developed different customs around etiquette.)
- zepto 5y ago
  > the reward calculation in their brain comes out in favour of doing that thing instead of anything else they could be doing instead Nope. That’s you baking in the assumption that they are making a calculation about reward. That’s an axiom of your way of analyzing the situation. Nothing more.

by zepto 5y ago

derefr 5y ago

You misunderstand what I mean by “happiness”, I think. (Probably because I should have said “predicted utility of all agents similar to oneself”, but I didn’t, because that’s pretty dang jargon-y, and assumes you’ve walked through the relevant thought experiments about self-sacrifice to help other copies of yourself, etc.) If someone martyrs themselves, then in the moment they do that, they’re thinking that that is a good thing to do — i.e. the reward calculation in their brain comes out in favour of doing that thing instead of anything else they could be doing instead. They expect that good things will happen due to this—things that cohere with their preferences. This is still true even if they don’t live to see those good things happen, because brains are adaptation executors, not fitness maximizers: we prefer the kinds of things that made our lineage successful, not the things that make us successful. But those are still preferences! “Being of use to others” is a preference. You do it because that’s the thing your brain releases reward chemicals when you do. This is what you are terming something being “part of one’s nature”; “your nature” is equivalent to “your preference function.” And, if you can assume that to be true for a second, then I can restate my point in terms of it: many of the things people have terms for in their preference functions are based on adhering to an abstract role or identity, something like “an honorable person”, their understanding of which was inculcated by local, culturally-contextual societal mores. For most people, doing something “because it’s honorable” isn’t a post-hoc rationalization of some spiritual connection to the concept of honor that informs their life. Instead, it’s an instrumental preference — a thing you do in order to work toward some higher-level preference you have — where the higher-level preference is “cohere more closely with my own understanding of what it means to be honorable.” In these cases, people don’t like X and dislike Y because their brain is directly telling them that X is good and Y isn’t (where they then notice a class of such preferential distinctions they’re making, and then call that class of distinctions “honor” because it seems to be the same as a class of distinctions that other people call “honor.”) Instead, people like X and dislike Y because they learned a concept like “honor” from their societal context, and they learned that their society considers X to be honorable and Y to be dishonorable; and so they then prefer to do X and not do Y because they want to live in the image of what they believe honor to be, as learned from their society. For these people, if they experience other cultures’ conceptions of these abstract concepts, and through that learn more about what these abstract concepts can mean to different people, this is usually enough to shift how they understand the abstract concept itself; and due to that, they will usually end up with different instrumental preferences—because now what they have learned “is honorable”, is a different set of things. (Compare and contrast: etiquette. “What is etiquette” can feel objective if you never leave the culture you were born into. But you’ll have a very different idea of things being or not being “polite” if you experience a variety of different cultures that all have developed different customs around etiquette.)

zepto 5y ago
> the reward calculation in their brain comes out in favour of doing that thing instead of anything else they could be doing instead Nope. That’s you baking in the assumption that they are making a calculation about reward. That’s an axiom of your way of analyzing the situation. Nothing more.