6 ms·

I agree with other commenters that it's unlikely nudges never have an impact. We should also be wary of high-profile debunkings, now that they're increasingly

I agree with other commenters that it's unlikely nudges never have an impact. We should also be wary of high-profile debunkings, now that they're increasingly in fashion due to the replication crisis and the general dour mood. It's easy to p-hack a result into significance, but you can just as easily hack results into insignificance. These days, both findings and debunkings need a skeptical eye.

themitigating 4y ago
Isn't that always the case?
- pessimizer 4y ago
  I think some people careen from fully trusting one thing to fully trusting the opposite thing. If you're not one of those people, you'll never understand dismissing things on the basis of "you should be critical of this, because not everything that people say is true." I do feel like that, even though being critical is something we should always do, that in cases where 1) the only reason you started paying attention to something was an intuitive hunch that it could matter, and 2) the only reason you started treating that hunch as established science is because you did experiments that had significant results, then 3) later you found that significance could be entirely accounted for by the file-drawer effect, you need to adjust your expectation that there actually is an effect to lower than your expectation was at step 1). It isn't that the theory hasn't been tested (although you can argue it hasn't been tested for ingeniously enough yet), it's that it has been tested and no effect has been shown. If you allow the existence of interest in a theory (represented in amount of ink spilled and number of experiments done) to raise your expectation that the theory is true, despite experimental indications to the contrary, you're not really doing science, you're just throwing good money after bad, probably motivated by a desire to protect the researchers and institutions that are heavily committed to the truth of the theory and/or the desire to protect other theories that depend on the one that hasn't shown results.
  - civilized 4y ago
    There's a lot of junk behavioral science out there, but things like "people often go with a default or recommended option so they can move on with their day" seem so obvious to me that I become suspicious of this debunking for debunking too much.
    - pessimizer 4y ago
      That's just refusing to be convinced by evidence, though. It's good to have hunches, but it's good to let them go after you've done the experiments. Come up with a new hunch and a new experiment that shows why the expected effects weren't seen, and you're right back in there.
      - civilized 4y ago
        "Refusing to be convinced by evidence" is a simplistic false dichotomy. The evidence is interesting but I have several reasons not to immediately take it as definitive. Are you really certain that a big debunking in PNAS, surfing a wave of other celebrated debunkings, should be taken as definitive, when a good deal of the research being debunked was published to similar fanfare in PNAS back when a different kind of research was fashionable? I take neither the original research nor the debunking as particularly credible. Without technical expertise, I'm left to educated guess. It's just my guess.
        pessimizer 4y ago
        > I have several reasons not to immediately take it as definitive. The one that you expressed is that it "seems so obvious to you that you become suspicious." I'm just taking you at your word.
        civilized 4y ago
        The idea that we should always be convinced by evidence regardless of context is a vast overgeneralization, impossible ("the evidence" overall rarely points only one way, even if the latest chunk of new evidence does), and in contradiction with Bayesian epistemology.
nabla9 4y ago
There is no evidence of impact. That's different from the existence of the phenomenon. Same thing happened to Kahneman, Daniel (2011) and his book of Thinking, Fast and Slow. He acknowledges that several pieces of evidence he presents in the book has disappeared and can't be replicated. He still thinks he is right, he just admits that he does not have strong evidence anymore. What is left is a theory with less and less evidence supporting it.
- civilized 4y ago
  I just think it's implausible that there's been no good research showing solid evidence of choice architecture mattering. Could be, I'm not an expert, but I'd like to see where things stand after a couple more years of research and debate.
  - nabla9 4y ago
    >I just think it's implausible that there's b The job of scientist is to find and present that evidence. If your assumption is correct, and Kahneman failed to find good evidence that was there, that makes him a incompetent scientist. I don't think he is.
    - civilized 4y ago
      I respect Kahneman too but rigorous behavioral science can be extremely demanding. Read the rat story here: http://people.cs.uchicago.edu/~ravenben/cargocult.html http://people.cs.uchicago.edu/~ravenben/cargocult.html Feynman taught us to be scornful of cargo cult science, but fewer have internalized how difficult real science can be in comparison. If this is the case, though, maybe I shouldn't be surprised that maybe no quality research has been done.
- BeetleB 4y ago
  > He acknowledges that several pieces of evidence he presents in the book has disappeared and can't be replicated. He mostly says that about just one chapter. A significant portion of the book is fallacies of basic statistics and logic.
JusticeJuice 4y ago
I'm a UI designer, and my experience of implementing 'nudges' is that sometimes they work, and sometimes they don't. The reality is that the way people make decisions is stupidly complex, because people have stupidly complex lives. Some tweak will work great for one project, and do nothing on the next one. It's hard to even say if it was the nudge that worked the first time. I really view nudge theory as one of many ideas of things you can try, a tool in a toolkit. But the only tool I really feel confident works is the design-test-iterate loop.
- solarkraft 4y ago
  I'd love to read more about this. Do you have examples?
  - JusticeJuice 4y ago
    I was working on a fintech project (gonna be vague as it's not yet released). The legal team told us we couldn't use default choices anywhere, as it could count as giving financial advice. Fair enough. So we designed the onboarding, and there was this choice the user had to make before we could create their account. During testing, we found people were getting really stuck on this choice, to the point of giving up. The choice actually had quite low impact, but it was really technical - a lot of people just didn't understand it. Which makes sense our users weren't financial experts, which was our target user. This choice was a new concept for the market, so we couldn't relate it to other products they might know. The options inside also had quite a lot of detail when you started digging into them, detail we had to provide if somebody went looking for it. Our testers would hit this choice, get stuck, feel the urge to research this decision, get overwhelmed, give up. We spent so long trying to reframe this choice, explaining it better in a nice succinct way, we even tried to get this feature removed entirely - but nothing stuck. Eventually after lots of discussion with legal we were allowed to have a 'base' choice, which the user could optionally change. We tested the new design, and it made a significant difference in conversion rates. Huzzar for nudge theory! Right? Well, maybe. I think it's a bit more complicated. - The new design was faster. There was less screens with simpler choices. It went from a 'pick one of 5' to a 'heres the default, would you like to change it?'. Was it just the speed that made a difference? - The user was not a financial expert, and the company behind the product was. In some sense was the user just thinking 'these guys probably know more than me I'll leave it at that'. Imagine trying to implement this exact change on something the user is an expert in - say like your meal choice in an airplane. I imagine most people would think "How rude choosing for me! I'm an expert in what I feel like eating I want to see all the options". - It had less of a cognitive load. Like the whole onboarding flow was already really complicated, just reducing the overall mental strain to make an account may have just improved the whole experience. E.g. if we had removed decisions earlier in the flow, would this one still have been as big of an issue? We never had time to test it, so I can't say for sure. - Lack of confusion == confidence. For the users who didn't look at the options and took the default, did they just feel more in control and confident because they weren't exposed with unfamiliar terms and choices? They never experienced the urge to research. Like at the surface level this new design worked great, so job done. But it's hard to say definitively it was because of nudge theory. I don't think you can really blindly say "oh yeah defaults == always good" and slap them on every problem - which is why the design-test-iterate loop is so important.
t_mann 4y ago
The paper actually explicitly addresses this: > However, all intervention categories and domains apart from “finance” show evidence for heterogeneity, which implies that some nudges might be effective, even when there is evidence against the mean effect.

themitigating 4y ago

Isn't that always the case?

pessimizer 4y ago
I think some people careen from fully trusting one thing to fully trusting the opposite thing. If you're not one of those people, you'll never understand dismissing things on the basis of "you should be critical of this, because not everything that people say is true." I do feel like that, even though being critical is something we should always do, that in cases where 1) the only reason you started paying attention to something was an intuitive hunch that it could matter, and 2) the only reason you started treating that hunch as established science is because you did experiments that had significant results, then 3) later you found that significance could be entirely accounted for by the file-drawer effect, you need to adjust your expectation that there actually is an effect to lower than your expectation was at step 1). It isn't that the theory hasn't been tested (although you can argue it hasn't been tested for ingeniously enough yet), it's that it has been tested and no effect has been shown. If you allow the existence of interest in a theory (represented in amount of ink spilled and number of experiments done) to raise your expectation that the theory is true, despite experimental indications to the contrary, you're not really doing science, you're just throwing good money after bad, probably motivated by a desire to protect the researchers and institutions that are heavily committed to the truth of the theory and/or the desire to protect other theories that depend on the one that hasn't shown results.
- civilized 4y ago
  There's a lot of junk behavioral science out there, but things like "people often go with a default or recommended option so they can move on with their day" seem so obvious to me that I become suspicious of this debunking for debunking too much.
  - pessimizer 4y ago
    That's just refusing to be convinced by evidence, though. It's good to have hunches, but it's good to let them go after you've done the experiments. Come up with a new hunch and a new experiment that shows why the expected effects weren't seen, and you're right back in there.
    - civilized 4y ago
      "Refusing to be convinced by evidence" is a simplistic false dichotomy. The evidence is interesting but I have several reasons not to immediately take it as definitive. Are you really certain that a big debunking in PNAS, surfing a wave of other celebrated debunkings, should be taken as definitive, when a good deal of the research being debunked was published to similar fanfare in PNAS back when a different kind of research was fashionable? I take neither the original research nor the debunking as particularly credible. Without technical expertise, I'm left to educated guess. It's just my guess.
      - pessimizer 4y ago
        > I have several reasons not to immediately take it as definitive. The one that you expressed is that it "seems so obvious to you that you become suspicious." I'm just taking you at your word.
        civilized 4y ago
        The idea that we should always be convinced by evidence regardless of context is a vast overgeneralization, impossible ("the evidence" overall rarely points only one way, even if the latest chunk of new evidence does), and in contradiction with Bayesian epistemology.

nabla9 4y ago

There is no evidence of impact. That's different from the existence of the phenomenon. Same thing happened to Kahneman, Daniel (2011) and his book of Thinking, Fast and Slow. He acknowledges that several pieces of evidence he presents in the book has disappeared and can't be replicated. He still thinks he is right, he just admits that he does not have strong evidence anymore. What is left is a theory with less and less evidence supporting it.

civilized 4y ago
I just think it's implausible that there's been no good research showing solid evidence of choice architecture mattering. Could be, I'm not an expert, but I'd like to see where things stand after a couple more years of research and debate.
- nabla9 4y ago
  >I just think it's implausible that there's b The job of scientist is to find and present that evidence. If your assumption is correct, and Kahneman failed to find good evidence that was there, that makes him a incompetent scientist. I don't think he is.
  - civilized 4y ago
    I respect Kahneman too but rigorous behavioral science can be extremely demanding. Read the rat story here: http://people.cs.uchicago.edu/~ravenben/cargocult.html http://people.cs.uchicago.edu/~ravenben/cargocult.html Feynman taught us to be scornful of cargo cult science, but fewer have internalized how difficult real science can be in comparison. If this is the case, though, maybe I shouldn't be surprised that maybe no quality research has been done.
BeetleB 4y ago
> He acknowledges that several pieces of evidence he presents in the book has disappeared and can't be replicated. He mostly says that about just one chapter. A significant portion of the book is fallacies of basic statistics and logic.

JusticeJuice 4y ago

I'm a UI designer, and my experience of implementing 'nudges' is that sometimes they work, and sometimes they don't. The reality is that the way people make decisions is stupidly complex, because people have stupidly complex lives. Some tweak will work great for one project, and do nothing on the next one. It's hard to even say if it was the nudge that worked the first time. I really view nudge theory as one of many ideas of things you can try, a tool in a toolkit. But the only tool I really feel confident works is the design-test-iterate loop.

solarkraft 4y ago
I'd love to read more about this. Do you have examples?
- JusticeJuice 4y ago
  I was working on a fintech project (gonna be vague as it's not yet released). The legal team told us we couldn't use default choices anywhere, as it could count as giving financial advice. Fair enough. So we designed the onboarding, and there was this choice the user had to make before we could create their account. During testing, we found people were getting really stuck on this choice, to the point of giving up. The choice actually had quite low impact, but it was really technical - a lot of people just didn't understand it. Which makes sense our users weren't financial experts, which was our target user. This choice was a new concept for the market, so we couldn't relate it to other products they might know. The options inside also had quite a lot of detail when you started digging into them, detail we had to provide if somebody went looking for it. Our testers would hit this choice, get stuck, feel the urge to research this decision, get overwhelmed, give up. We spent so long trying to reframe this choice, explaining it better in a nice succinct way, we even tried to get this feature removed entirely - but nothing stuck. Eventually after lots of discussion with legal we were allowed to have a 'base' choice, which the user could optionally change. We tested the new design, and it made a significant difference in conversion rates. Huzzar for nudge theory! Right? Well, maybe. I think it's a bit more complicated. - The new design was faster. There was less screens with simpler choices. It went from a 'pick one of 5' to a 'heres the default, would you like to change it?'. Was it just the speed that made a difference? - The user was not a financial expert, and the company behind the product was. In some sense was the user just thinking 'these guys probably know more than me I'll leave it at that'. Imagine trying to implement this exact change on something the user is an expert in - say like your meal choice in an airplane. I imagine most people would think "How rude choosing for me! I'm an expert in what I feel like eating I want to see all the options". - It had less of a cognitive load. Like the whole onboarding flow was already really complicated, just reducing the overall mental strain to make an account may have just improved the whole experience. E.g. if we had removed decisions earlier in the flow, would this one still have been as big of an issue? We never had time to test it, so I can't say for sure. - Lack of confusion == confidence. For the users who didn't look at the options and took the default, did they just feel more in control and confident because they weren't exposed with unfamiliar terms and choices? They never experienced the urge to research. Like at the surface level this new design worked great, so job done. But it's hard to say definitively it was because of nudge theory. I don't think you can really blindly say "oh yeah defaults == always good" and slap them on every problem - which is why the design-test-iterate loop is so important.

t_mann 4y ago

The paper actually explicitly addresses this: > However, all intervention categories and domains apart from “finance” show evidence for heterogeneity, which implies that some nudges might be effective, even when there is evidence against the mean effect.