5 ms·

>That's pretty close to saying it doesn't work. No it's really not. To say things a different way, I don't think this study will change anything for people ac

by akyu 4y ago

>That's pretty close to saying it doesn't work. No it's really not. To say things a different way, I don't think this study will change anything for people actually doing choice architecture in applied settings. They have results that speak for themselves.

DangitBobby 4y ago
They would be the people who published, in this scenario.
Beldin 4y ago
I think gp and you probably see eye to eye, but gp has a problem with your phrasing. If the effect does not live up to scientific rigour, that (more or less) implies that the effect is roughly indistinguishable from randomness. If folks have results that speak for themselves, then the effect more than likely is scientifically rigorously testable. It may already have been - by those very results.
mcswell 4y ago
"I don't think this study will change anything for people actually doing choice architecture in applied settings." Probably true, but then evidence that horoscopes etc. don't work, doesn't prevent people from drawing horoscopes, or other people from relying on their horoscope to plan out their day. "They have results that speak for themselves." Let me put my point differently. Suppose that nudges don't have any effect at all (null hypothesis). More concretely--and just to take a random number--suppose that 50% of the time when a nudge is used, the nudgees happen to behave in the direction that the nudge was intended to move them, and 50% of the time they don't move, or they move in the opposite direction. And suppose there are a number of nudgers, maybe 100. Then some nudgers will get better than random results, while others will get no result, or negative results. The former nudgers will have results that appear to speak for themselves, even if the nudges actually have no effect whatsoever. This is the same as asking if a fair coin is tossed ten times, what is the probability that you'll get at least 7 heads. The probability of such a number of heads in a single run is ~17%. So 17% of those nudgers could be getting apparently significant results, even if their results are actually random.
pessimizer 4y ago
> results that speak for themselves. This is exactly how a midwife explained to me why she uses magic crystals. She told me that there's science, and there's results, and that she's seen the crystals work.
- rsanek 4y ago
  I mean, yeah, if she has solid RCT data on thousands to millions of childbirths and has found a statistically significant impact from using the magic crystals, I would support their use. A/B as well as scientific research uses the same basis. The issue is that in fact the midwife will not have such data. The comparison being made is that A/B testing, if run competently, is pretty close to scientific research, in particular for research related to nudging.
  - pessimizer 4y ago
    But if run rigorously, A/B testing is identical to scientific research, and the scientific research fails to show an effect.
  - asdff 4y ago
    I wonder how many engineers crack open a statistics book to find the correct test versus just plotting box plots and saying "see looks pretty different"
    - saalweachter 4y ago
      To be fair, the more profound a result the less math you need to convince anyone it is the case.
      - asdff 4y ago
        Maybe if you are parroting the result in front of investors instead of statisticians that's the case
  - sacrosancty 4y ago

by akyu 4y ago

DangitBobby 4y ago

They would be the people who published, in this scenario.

Beldin 4y ago

I think gp and you probably see eye to eye, but gp has a problem with your phrasing. If the effect does not live up to scientific rigour, that (more or less) implies that the effect is roughly indistinguishable from randomness. If folks have results that speak for themselves, then the effect more than likely is scientifically rigorously testable. It may already have been - by those very results.

mcswell 4y ago

"I don't think this study will change anything for people actually doing choice architecture in applied settings." Probably true, but then evidence that horoscopes etc. don't work, doesn't prevent people from drawing horoscopes, or other people from relying on their horoscope to plan out their day. "They have results that speak for themselves." Let me put my point differently. Suppose that nudges don't have any effect at all (null hypothesis). More concretely--and just to take a random number--suppose that 50% of the time when a nudge is used, the nudgees happen to behave in the direction that the nudge was intended to move them, and 50% of the time they don't move, or they move in the opposite direction. And suppose there are a number of nudgers, maybe 100. Then some nudgers will get better than random results, while others will get no result, or negative results. The former nudgers will have results that appear to speak for themselves, even if the nudges actually have no effect whatsoever. This is the same as asking if a fair coin is tossed ten times, what is the probability that you'll get at least 7 heads. The probability of such a number of heads in a single run is ~17%. So 17% of those nudgers could be getting apparently significant results, even if their results are actually random.

pessimizer 4y ago

> results that speak for themselves. This is exactly how a midwife explained to me why she uses magic crystals. She told me that there's science, and there's results, and that she's seen the crystals work.

rsanek 4y ago
I mean, yeah, if she has solid RCT data on thousands to millions of childbirths and has found a statistically significant impact from using the magic crystals, I would support their use. A/B as well as scientific research uses the same basis. The issue is that in fact the midwife will not have such data. The comparison being made is that A/B testing, if run competently, is pretty close to scientific research, in particular for research related to nudging.
- pessimizer 4y ago
  But if run rigorously, A/B testing is identical to scientific research, and the scientific research fails to show an effect.
- asdff 4y ago
  I wonder how many engineers crack open a statistics book to find the correct test versus just plotting box plots and saying "see looks pretty different"
  - saalweachter 4y ago
    To be fair, the more profound a result the less math you need to convince anyone it is the case.
    - asdff 4y ago
      Maybe if you are parroting the result in front of investors instead of statisticians that's the case
- sacrosancty 4y ago