5 ms·

It's Not the Incentives (2018)

48 points by godelski 2y ago

Sniffnoy 2y ago
(2018)
JohnFen 2y ago
Very well said. It's not just scientists, I've seen the same thing happen across the board -- including here on HN. Yes, incentives exist that can make doing the wrong thing easy and even personally advantageous. That doesn't excuse or forgive doing the wrong thing -- that thing is still wrong and people doing it are responsible for their own behavior regardless of "incentives".
- smallmancontrov 2y ago
  Blame the system not the person => person skates => bad but not catastrophic. Blame the person not the system => system skates => catastrophic. If you want to punish wrongdoers on your own time, great, we have no quarrel. Ideally we would blame both, but 80% of the time when someone is advocating blaming the person I find that they have a conflict of interest and secretly want to preserve the system. "Small town morality" sounds good but does not scale and this combination of facts is easily exploited to divert attention away from important system maintenance. There was a time when I felt obliged to extend the benefit of the doubt on this matter but after having said benefit exploited very intentionally on two different occasions I now consider it a bad policy, so: first we worry about fixing the system. That is not negotiable.
  - skybrian 2y ago
    I think the article is about a different scenario: Blame the system => person skates and the system doesn’t change That is, blaming the system often isn’t about changing it. You need the power to actually fix things and a plan to fix them.
  - deleted 2y ago
    [deleted]
kubb 2y ago
This talks mainly about research. But if you work at a megacorp, like Facebook, and lobotomize people for money, is it your fault, and are you a bad person for doing it?
- Centigonal 2y ago
  yes, and maybe? I think it's important to hold the tension that you are doing something that is both good and harmful (like both building a really nice chat app you give away for free and using the network effects of that app to collect everyone's data and influence them, or protecting people from fraud but also improving the bottom line of Wall Street banks that are contributing to the increasing financialization of everything). I think acknowledging incentive structures is alright, but ending the process there and not leaning into understanding and manipulating them is the real issue.
- tejohnso 2y ago
  The "I'm just doing my job" excuse goes a long way with people. Every once in a while you'll hear about someone who just couldn't stomach it anymore. Working in megacorp finance for example. Or insurance, where there are plenty of unkind incentives. Doesn't happen very often though. For the most part people have a job to do and they do it.
- __MatrixMan__ 2y ago
  Yes and yes.
jgeada 2y ago
It is the incentives. Maybe you're moral and keep to the straight and narrow. However, the system hires all types and the ones that just follow the incentives will do better. They get promoted, have more power, hire more people like them. Eventually the moral types will just be the exception and no longer affect the average. Incentives exist because they change the behavior of the whole; they work as intended. Just that what is intended isn't always desirable or even a good idea.
- treetalker 2y ago
  I agree with you. It's the law of large numbers; individuals make free choices, yet in the aggregate the incentives make the likelihood of decisions fairly predictable. This is the essence of "nudging" and decision architecture (and dark patterns, etc.).
- beepbopboopp 2y ago
  Its funny, the whole presenting ones self as an objective beacon of morality is a response to an incentive system in itself. For whatever reason; parents, school the author was convinced that this type of morality would yield a better life or piece of mind or some optimal outcome.
  - codetrotter 2y ago
    > convinced that this type of morality would yield a better life or piece of mind or some optimal outcome Yeah. For example a society where people work together for the benefit of all, instead of having some people exploit the others.
    - leereeves 2y ago
      It's interesting that this concept of ethics evolved at all. "Nature is red in tooth and claw." It's a brutal, heartless competition for resources and survival. And yet through the vicious process of evolution, we developed empathy and a sense of fairness. Those instincts must have some cold, rational benefit for the survival of the group.
deleted 2y ago
[deleted]
bitwize 2y ago
Perverse incentives + prisoner's dilemma = it gets REALLY hard to not follow the edict of the incentives.
- parrit 2y ago
  The world is sinful on many levels there is a lot to change. It is hard to exist in a way that doesn't fuck someone over. If you live in the global west and participate in modern life then you are in some area definitely following a perverse inventive. For example do you live in a building? Do you heat it? There you go. You should be in a tent. Climate change. Have an SP500 index fund? Investing in weapons and other dystopia. Generally support globalization, regulatory capture, lobbying, bad work conditions and tax evasion over supporting small business. Vote for one of the 2 main parties? Well done both support genocide. Protest vote? Oh then you voted for Trump in this system well done! The only moral thing is to change the constitution (fwiw) for proportional representation. Send your kids to a good school? The list doesn't end.
  - rickdicker 2y ago
    If enough people protest-vote then it isn't necessarily a vote for the winner - if the number of protest-voters is large enough to have changed the outcome had they all voted for the loser, then the losing party (if they are doing their job) has to take notice and listen to why this group isn't happy. If the protest-voters become a massive group of people (say 1/3 of the vote), they will probably become the focus of a lot of political debates and tv-news stories, and elected officials will have to start catering to them. But it all only works if lots of people can have the integrity to not blame "the system" for why they voted for the-crook-who-was-better-than-the-other-crook. (FYI I'm aware this has very little to do with the point you were making. I mostly agree with that)
kelseyfrog 2y ago
It's the incentives, or more accurately it's the fact that we can't change incentives because when that conversation starts, a hundred people come out of the walls and say, "But changing the incentives will make it worse!", "Don't you see this edge case that someone will exploit?", and "You can't complain about fixing the incentives unless you have an air-tight plan that has no flaws!" It demoralizes everyone until we all just give up and continue to blame the incentives. The incentives will continue until the incentive to change grow stronger than not changing and for the folks with the opinions above, the incentive to change will have to grow very large indeed.
- rickdicker 2y ago
  How do we fix that? Perhaps it can't be done democratically, and you just need someone to come into a high position of power who is willing to be dramatic and disruptive?
  - justin_oaks 2y ago
    Generally it isn't done because the people in power benefit from the status quo. To fix it, you need collective action. And now you have a collective action problem: https://en.wikipedia.org/wiki/Collective_action_problem https://en.wikipedia.org/wiki/Collective_action_problem
  - lelandbatey 2y ago
    One way to address it: change very slowly. You can strangle the bad incentives by reducing their perceived utility by introducing alternatives. The bad incentive exists because it's trying to do something. Ideally, you set up a companion incentive that gets the system a little more in the direction everyone wants, you systemically allow folks to effectively "choose their incentive", and eventually you phase out the old incentive. Problems with this are: it's glacial, and it doesn't solve everything instantaneously. Folks don't like either of those things, so you probably need to use other tools of political change to make those things more palatable. In general, it's a lot of work, it's all in the details, and it takes forever. So lots of folks will fall back on the "why not just give the small dictator(s) all the power?"
bitmasher9 2y ago
> You’re (probably) not going to “change things from the inside” I’ve had this exact conversation with so many students across so many industries. A good chunk of them end up middle managers keeping their head down never feeling empowered enough to make the changes they want to see. Those that end up actually making impactful changes were disrupting from the beginning.
parrit 2y ago
If you don't follow the incentives are you out of a career? And if so, what is the pragmatic thing to do to change things. Because getting a job in tech instead just means you swap to another set of bad incentives. I wonder if unionizing would help? Otherwise how does the idealist eat?
deleted 2y ago
[deleted]
ttoinou 2y ago
A random bystander who happened to eavesdrop on a conversation between a group of scientists kvetching about The Incentives could be forgiven for thinking that maybe, just maybe, a bunch of very industrious people who generally pride themselves on their creativity, persistence, and intelligence could find some way to work around, or through, the problem. Thats exactly what they did. They understood the incentives better than others and stayed away from The Officially Approved Science. Reducing talent quality is a good way way to help solve this problem : your system sucks = smart useful people are going somewhere else
atian 2y ago
> Show me the incentive, I’ll show you the outcome.
- readthenotes1 2y ago
  I enjoyed reading the book "punished by rewards" that warned us that we often confuse rewards with activities if we are rewarded for doing the activity. Think about sports. People play to win, when we really should be playing to play. It may be "me", but I have to remind my self to play as well as I can and not think about win/loss
__MatrixMan__ 2y ago
Yes, let's all take this to heart. But once we get to the corollary: > It's not the incentives, it's them What are we to do? Shame them, sure, but until a critical mass of us is prepared to interfere, to inject ourselves without consent into the business of others, then it comes back to fixing the incentives.
- justin_oaks 2y ago
  Avoiding shame is an incentive. It boils down to: "Why are people violating these unenforced rules? Sure it benefits them, but don't they feel bad?"
deleted 2y ago
[deleted]
rgyams 2y ago
I totally understand the frustration with the common excuse of: oh it's the incentives, but I believe it oversimplifies the issue. While it's true that perverse incentives exist, we still have the agency to choose how we respond to them. If we can't resist cutting corners in small ways, we shouldn't be surprised when others do so in far more damaging ways.
taylorallred 2y ago
Question from genuine ignorance: are scientists not incentivized to do good work even if it doesn't result in cool/useful/desired outcomes? I would think that even "the data from the experiment was inconclusive" should still be considered a valid contribution to the field as long as it was done correctly.
- sashank_1509 2y ago
  Yes, to a gross approximation, scientists cannot and do not advance their career by giving a negative result. That is just how it is, anyone who’s tried being a scientist for a day realizes this. In ML for example, if you try some weird idea, and it does not beat baseline methods on any specific benchmark, then your paper 9.99/10 will not get accepted. In fact I don’t think I’ve ever seen a negative result paper ever get accepted into a good ML conference. At the very least, the authors make up their own benchmark and claim their method is best in their own benchmark, and the reviewers then quibble about whether such a benchmark is relevant, and then after back and forth, they come to a decision and decide to accept the paper or not.
paulorlando 2y ago
From Munger in The Psychology of Human Misjudgment: "And human nature, with its version of what I call incentive-caused bias, causes this terrible abuse. And many of the people who are doing it you would be glad to have married into your family compared to what you’re otherwise going to get. "Now there are huge implications from the fact that the human mind is put together this way, and that is that people who create things like cash registers, which make most behavior hard, are some of the effective saints of our civilization. And the cash register was a great moral instrument when it was created. And Patterson knew that, by the way. He had a little store, and the people were stealing him blind and never made any money, and people sold him a couple of cash registers and it went to profit immediately."
- huijzer 2y ago
  Yes exactly Charlie has been saying again and again to not underestimate the power of incentives. I’m a big fan of Tal Yarkoni but I think this article is probably not correct.
stared 2y ago
"It's worth being honest, even though it doesn't always pay off. Being dishonest pays off, but it's not worth it." - Władysław Bartoszewski
justin_oaks 2y ago
The author doesn't seem to consider someone's desire to behave morally as an incentive. How odd.
erelong 2y ago
I do think it's about the incentives, but I've had conversations elsewhere between a perceived distinction between "morals" and, for lack of a better word, "ethics" Morals you might not want to violate under nearly any circumstances: theft, violent crime, and so on, regardless if there are "good incentives" for you to do so. "Ethics" might be more of things that produce bad outcomes but aren't a "sin". You might choose to uphold a personal ethic for example against participating in some of these practices which seem to make society worse over time, but which might personally benefit you. But you might not be under a "moral obligation" to do so. Or, for example, otherwise violence might be a bad thing, but violence used in self-defense is generally accepted as being "ok". You might out of "self-defense" do some of these things that otherwise seem to produce "bad outcomes for society", out of coercion by the mass of people making "bad" decisions. But you might also "virtuously abstain", for whatever reason, as well.

48 points by godelski 2y ago

(2018)

Very well said. It's not just scientists, I've seen the same thing happen across the board -- including here on HN. Yes, incentives exist that can make doing the wrong thing easy and even personally advantageous. That doesn't excuse or forgive doing the wrong thing -- that thing is still wrong and people doing it are responsible for their own behavior regardless of "incentives".

smallmancontrov 2y ago
Blame the system not the person => person skates => bad but not catastrophic. Blame the person not the system => system skates => catastrophic. If you want to punish wrongdoers on your own time, great, we have no quarrel. Ideally we would blame both, but 80% of the time when someone is advocating blaming the person I find that they have a conflict of interest and secretly want to preserve the system. "Small town morality" sounds good but does not scale and this combination of facts is easily exploited to divert attention away from important system maintenance. There was a time when I felt obliged to extend the benefit of the doubt on this matter but after having said benefit exploited very intentionally on two different occasions I now consider it a bad policy, so: first we worry about fixing the system. That is not negotiable.
- skybrian 2y ago
  I think the article is about a different scenario: Blame the system => person skates and the system doesn’t change That is, blaming the system often isn’t about changing it. You need the power to actually fix things and a plan to fix them.
- deleted 2y ago
  [deleted]

kubb 2y ago

This talks mainly about research. But if you work at a megacorp, like Facebook, and lobotomize people for money, is it your fault, and are you a bad person for doing it?

Centigonal 2y ago
yes, and maybe? I think it's important to hold the tension that you are doing something that is both good and harmful (like both building a really nice chat app you give away for free and using the network effects of that app to collect everyone's data and influence them, or protecting people from fraud but also improving the bottom line of Wall Street banks that are contributing to the increasing financialization of everything). I think acknowledging incentive structures is alright, but ending the process there and not leaning into understanding and manipulating them is the real issue.
tejohnso 2y ago
The "I'm just doing my job" excuse goes a long way with people. Every once in a while you'll hear about someone who just couldn't stomach it anymore. Working in megacorp finance for example. Or insurance, where there are plenty of unkind incentives. Doesn't happen very often though. For the most part people have a job to do and they do it.
__MatrixMan__ 2y ago
Yes and yes.

jgeada 2y ago

It is the incentives. Maybe you're moral and keep to the straight and narrow. However, the system hires all types and the ones that just follow the incentives will do better. They get promoted, have more power, hire more people like them. Eventually the moral types will just be the exception and no longer affect the average. Incentives exist because they change the behavior of the whole; they work as intended. Just that what is intended isn't always desirable or even a good idea.

treetalker 2y ago
I agree with you. It's the law of large numbers; individuals make free choices, yet in the aggregate the incentives make the likelihood of decisions fairly predictable. This is the essence of "nudging" and decision architecture (and dark patterns, etc.).
beepbopboopp 2y ago
Its funny, the whole presenting ones self as an objective beacon of morality is a response to an incentive system in itself. For whatever reason; parents, school the author was convinced that this type of morality would yield a better life or piece of mind or some optimal outcome.
- codetrotter 2y ago
  > convinced that this type of morality would yield a better life or piece of mind or some optimal outcome Yeah. For example a society where people work together for the benefit of all, instead of having some people exploit the others.
  - leereeves 2y ago
    It's interesting that this concept of ethics evolved at all. "Nature is red in tooth and claw." It's a brutal, heartless competition for resources and survival. And yet through the vicious process of evolution, we developed empathy and a sense of fairness. Those instincts must have some cold, rational benefit for the survival of the group.

deleted 2y ago

[deleted]

bitwize 2y ago

Perverse incentives + prisoner's dilemma = it gets REALLY hard to not follow the edict of the incentives.

parrit 2y ago
The world is sinful on many levels there is a lot to change. It is hard to exist in a way that doesn't fuck someone over. If you live in the global west and participate in modern life then you are in some area definitely following a perverse inventive. For example do you live in a building? Do you heat it? There you go. You should be in a tent. Climate change. Have an SP500 index fund? Investing in weapons and other dystopia. Generally support globalization, regulatory capture, lobbying, bad work conditions and tax evasion over supporting small business. Vote for one of the 2 main parties? Well done both support genocide. Protest vote? Oh then you voted for Trump in this system well done! The only moral thing is to change the constitution (fwiw) for proportional representation. Send your kids to a good school? The list doesn't end.
- rickdicker 2y ago
  If enough people protest-vote then it isn't necessarily a vote for the winner - if the number of protest-voters is large enough to have changed the outcome had they all voted for the loser, then the losing party (if they are doing their job) has to take notice and listen to why this group isn't happy. If the protest-voters become a massive group of people (say 1/3 of the vote), they will probably become the focus of a lot of political debates and tv-news stories, and elected officials will have to start catering to them. But it all only works if lots of people can have the integrity to not blame "the system" for why they voted for the-crook-who-was-better-than-the-other-crook. (FYI I'm aware this has very little to do with the point you were making. I mostly agree with that)

kelseyfrog 2y ago

It's the incentives, or more accurately it's the fact that we can't change incentives because when that conversation starts, a hundred people come out of the walls and say, "But changing the incentives will make it worse!", "Don't you see this edge case that someone will exploit?", and "You can't complain about fixing the incentives unless you have an air-tight plan that has no flaws!" It demoralizes everyone until we all just give up and continue to blame the incentives. The incentives will continue until the incentive to change grow stronger than not changing and for the folks with the opinions above, the incentive to change will have to grow very large indeed.

rickdicker 2y ago
How do we fix that? Perhaps it can't be done democratically, and you just need someone to come into a high position of power who is willing to be dramatic and disruptive?
- justin_oaks 2y ago
  Generally it isn't done because the people in power benefit from the status quo. To fix it, you need collective action. And now you have a collective action problem: https://en.wikipedia.org/wiki/Collective_action_problem https://en.wikipedia.org/wiki/Collective_action_problem
- lelandbatey 2y ago
  One way to address it: change very slowly. You can strangle the bad incentives by reducing their perceived utility by introducing alternatives. The bad incentive exists because it's trying to do something. Ideally, you set up a companion incentive that gets the system a little more in the direction everyone wants, you systemically allow folks to effectively "choose their incentive", and eventually you phase out the old incentive. Problems with this are: it's glacial, and it doesn't solve everything instantaneously. Folks don't like either of those things, so you probably need to use other tools of political change to make those things more palatable. In general, it's a lot of work, it's all in the details, and it takes forever. So lots of folks will fall back on the "why not just give the small dictator(s) all the power?"

bitmasher9 2y ago

> You’re (probably) not going to “change things from the inside” I’ve had this exact conversation with so many students across so many industries. A good chunk of them end up middle managers keeping their head down never feeling empowered enough to make the changes they want to see. Those that end up actually making impactful changes were disrupting from the beginning.

parrit 2y ago

If you don't follow the incentives are you out of a career? And if so, what is the pragmatic thing to do to change things. Because getting a job in tech instead just means you swap to another set of bad incentives. I wonder if unionizing would help? Otherwise how does the idealist eat?

deleted 2y ago

[deleted]

ttoinou 2y ago

A random bystander who happened to eavesdrop on a conversation between a group of scientists kvetching about The Incentives could be forgiven for thinking that maybe, just maybe, a bunch of very industrious people who generally pride themselves on their creativity, persistence, and intelligence could find some way to work around, or through, the problem. Thats exactly what they did. They understood the incentives better than others and stayed away from The Officially Approved Science. Reducing talent quality is a good way way to help solve this problem : your system sucks = smart useful people are going somewhere else

atian 2y ago

> Show me the incentive, I’ll show you the outcome.

readthenotes1 2y ago
I enjoyed reading the book "punished by rewards" that warned us that we often confuse rewards with activities if we are rewarded for doing the activity. Think about sports. People play to win, when we really should be playing to play. It may be "me", but I have to remind my self to play as well as I can and not think about win/loss

__MatrixMan__ 2y ago

Yes, let's all take this to heart. But once we get to the corollary: > It's not the incentives, it's them What are we to do? Shame them, sure, but until a critical mass of us is prepared to interfere, to inject ourselves without consent into the business of others, then it comes back to fixing the incentives.

justin_oaks 2y ago
Avoiding shame is an incentive. It boils down to: "Why are people violating these unenforced rules? Sure it benefits them, but don't they feel bad?"

deleted 2y ago

[deleted]

rgyams 2y ago

I totally understand the frustration with the common excuse of: oh it's the incentives, but I believe it oversimplifies the issue. While it's true that perverse incentives exist, we still have the agency to choose how we respond to them. If we can't resist cutting corners in small ways, we shouldn't be surprised when others do so in far more damaging ways.

taylorallred 2y ago

Question from genuine ignorance: are scientists not incentivized to do good work even if it doesn't result in cool/useful/desired outcomes? I would think that even "the data from the experiment was inconclusive" should still be considered a valid contribution to the field as long as it was done correctly.

sashank_1509 2y ago
Yes, to a gross approximation, scientists cannot and do not advance their career by giving a negative result. That is just how it is, anyone who’s tried being a scientist for a day realizes this. In ML for example, if you try some weird idea, and it does not beat baseline methods on any specific benchmark, then your paper 9.99/10 will not get accepted. In fact I don’t think I’ve ever seen a negative result paper ever get accepted into a good ML conference. At the very least, the authors make up their own benchmark and claim their method is best in their own benchmark, and the reviewers then quibble about whether such a benchmark is relevant, and then after back and forth, they come to a decision and decide to accept the paper or not.

paulorlando 2y ago

From Munger in The Psychology of Human Misjudgment: "And human nature, with its version of what I call incentive-caused bias, causes this terrible abuse. And many of the people who are doing it you would be glad to have married into your family compared to what you’re otherwise going to get. "Now there are huge implications from the fact that the human mind is put together this way, and that is that people who create things like cash registers, which make most behavior hard, are some of the effective saints of our civilization. And the cash register was a great moral instrument when it was created. And Patterson knew that, by the way. He had a little store, and the people were stealing him blind and never made any money, and people sold him a couple of cash registers and it went to profit immediately."

huijzer 2y ago
Yes exactly Charlie has been saying again and again to not underestimate the power of incentives. I’m a big fan of Tal Yarkoni but I think this article is probably not correct.

stared 2y ago

"It's worth being honest, even though it doesn't always pay off. Being dishonest pays off, but it's not worth it." - Władysław Bartoszewski

justin_oaks 2y ago

The author doesn't seem to consider someone's desire to behave morally as an incentive. How odd.

erelong 2y ago

I do think it's about the incentives, but I've had conversations elsewhere between a perceived distinction between "morals" and, for lack of a better word, "ethics" Morals you might not want to violate under nearly any circumstances: theft, violent crime, and so on, regardless if there are "good incentives" for you to do so. "Ethics" might be more of things that produce bad outcomes but aren't a "sin". You might choose to uphold a personal ethic for example against participating in some of these practices which seem to make society worse over time, but which might personally benefit you. But you might not be under a "moral obligation" to do so. Or, for example, otherwise violence might be a bad thing, but violence used in self-defense is generally accepted as being "ok". You might out of "self-defense" do some of these things that otherwise seem to produce "bad outcomes for society", out of coercion by the mass of people making "bad" decisions. But you might also "virtuously abstain", for whatever reason, as well.