20 ms·

No evidence for nudging after adjusting for publication bias

246 points by ororm 4y ago

fbartos 4y ago
Hi all, I'm one of the authors of the reply and it was very interesting reading so many diverse thoughts and comments. I would love to respond to all of them, but it would take ages. Luckily, Stuart Ritchie (@StuartJRitchie) wrote an awesome post on his substack (https://stuartritchie.substack.com/p/nudge-meta https://stuartritchie.substack.com/p/nudge-meta) that goes much deeper and adresses many questions and the fair critique raised here. Also, note that there is only a limited amount of information and nuance you can comprise into a strict 500 words reply limit in PNAS, which is the reason we focus only on one aspect of the original meta-analysis -- publication bias. Cheers, Frantisek
mikkergp 4y ago
If publication bias is the exclusion of publishing results that doesn’t support your hypothesis, how are they taking that into account? If I’m interpreting this correctly(and I by no means am sure that I am), I infer that they are saying in a fair publishing environment you’d expect to see more results that show less decisive results, therefor the current set of results is likely biased. Couldn’t this bias also happen in the other direction? It sounds like they’re saying the results are too good and don’t match other scientific patterns of publishing results.
- mcswell 4y ago
  "Couldn’t this bias also happen in the other direction?" The general notion is that positive results ("we tried nudging and succeeded in getting people to behave more in X way") are more publishable than negative results ("we tried nudging, and nothing much happened"). It is a common and well-attested problem in many areas of science, but probably particularly in behavioral sciences; I have not heard of cases where publication of negative results is more likely than publication of positive results, although there are obviously heated debates over some results. Whether publication bias is the explanation in this example, I don't pretend to know.
  - mikkergp 4y ago
    I guess the question would be are positive results missing for another reason, like they are harder to test for, therefor the data looks better than it should because in the aggregate they are better, but yeah this is probably unlikely.
  - ghaff 4y ago
    There's also an implicit assumption with nudges/defaults that you're nudging people towards a reasonable place for some combination of policy and preference reasons. But imagine that a company would just as soon not pay out more 401k matching than it has to, so it makes the default zero. (Which of course is often the norm for different reasons.) That's as valid a nudge as anything but we shouldn't be surprised if a lot of people don't go with the default. We probably also shouldn't be surprised if a lot of people maybe wouldn't go with a maxed out default. Defaults wouldn't be nearly so powerful if they weren't typically chosen to be fairly reasonable for the average person in the target audience.
    - mikkergp 4y ago
      But I don’t think the goal is “most people can be nudged” so much as “a nudge is a cheap way to increase a behavior by 5-10% which is probably quite significant in policy circles. Low single digit percentage increase in people who pay their taxes would be huge.
- notafraudster 4y ago
  The easiest to understand diagnostic used to measure publication bias is the funnel plot. Suppose the true effect of interest is theta = 0.2. Then the observed effects in studies should be centered at 0.2; some will be higher, some will be lower. Assuming no systematic error, the degree to which study results vary around 0.2 should be proportional to the precision of the study (think sampling error given a sampling design). A hypothetical study of an infinitely large meta-population would produce the effect estimate of exactly 0.2, infinitely precisely. A series of very small studies will likely show quite divergent results, just on the basis of precision. A funnel plot plots effect sizes on the x axis and precision on the y axis. The most precise studies should be tightly grouped around the meta-analytic average effect; the least precise studies should be spread more widely. This forms a triangular, funnel shape. If no publication bias exists, the spread of studies below the magnitude of the average effect should be comparable to the spread of studies above the magnitude of the average effect. If there is publication bias, then the points that would form the left (without loss of generality; right if negative effect size) portion of the funnel will not be observable. There are issues with funnel plots and there are other diagnostics but I hope this provides insight into one of the tools used. Notably, as a diagnostic, funnel plots work whether the true effect is positive, negative, or null; they assume only that the underlying assumptions of meta-analysis are true (that studies represent a sample of the same, true underlying effect -- other diagnostics and corrections exist when this is violated)
  - lmeyerov 4y ago
    Interesting -- the problem may be a misapplication of funnel plots for metaanalysis. I'm not sure what theta is representing and only skimmed the paper, but especially in social scenarios and across social papers, seems unlikely to assume the same distributions and parameters across tasks & populations. Sometimes comes down to 'is there any effect??' and sometimes a precise notion of effect size in a lucky/clever specific scenario. Likewise, social science is one of the hardest fields to setup a good experiment, and few publications accepts negative results, so mostly only 'good' p-value ranges getting published seems normal. The Wikipedia page on funnel plots shows, afaict, the same criticism of the technique. Whether about the effect size or how it is reported, funnel plots seem an inappropriate choice for debunking something as general as 'nudges' across heterogeneous studies. Skimming made the metaanalysis feel rather lazy (lack of cross validation, interpretation, ...). Not my field, but I would have had to do some digging before accepting this metaanalysis in review, and by default, would be 'not ready'.
picardo 4y ago
I'm not very familiar with how "nudging" is defined in the behavioral economics, and perhaps someone can enlighten me, but personally I find it hard to believe that it can be disproven that the way a choice is presented doesn't play any role in one's decision. The Goldilock principle is well-known. Most people instinctively choose the middle option, when given a three things to choose from. Does this study imply that choice architecture plays no role in our decisions? Or am I mis-understanding it?
- hackerlight 4y ago
  You are misunderstanding. They're saying that there's an absence of evidence, not evidence of absence.
  - tgv 4y ago
    OTOH, people have been looking for evidence of nudging, and didn't find it. Since the a priori probability of a more than marginal effect of nudging is unlikely, we can conclude that it's much more likely that nudging doesn't "work" than that it does.
  - IshKebab 4y ago
    Absence of evidence is evidence of absence. It's just not proof of absence. For example, if I search your entire house for drugs, using drug sniffing dogs and so on and I don't find any at all, that's pretty good evidence that you don't have any. It's not proof though - you might have just stashed them really well. Similarly, if people have been looking for nudge effects for ages, doing loads of studies on it for years, and none of them have found any effect, then that's pretty good evidence that the effects don't exist. It's not proof though; they might just have not been very good experiments.
    - hackerlight 4y ago
      Well said. I'd just change your first sentence to "Absence of evidence can be (but isn't necessarily) evidence of absence.", which is more in line with the rest of your post.
bilsbie 4y ago
How do we know this study wasn’t “nudged”?
- mcswell 4y ago
  because it's a meta-study?
- svnt 4y ago
  That’s not the nudge you’re looking for.
rayiner 4y ago
Remember, Obama appointed a regulatory czar based on this science: https://www.nytimes.com/2010/05/16/magazine/16Sunstein-t.html https://www.nytimes.com/2010/05/16/magazine/16Sunstein-t.htm...
- pessimizer 4y ago
  Really one of the creepiest people connected to government.
- tptacek 4y ago
  Obama would have appointed Sunstein no matter what book he wrote, because he was one of UChicago's superstar professors (and was at the same time being aggressively courted by schools around the country; he ended up, of course, at Harvard). His appointment was notable not so much for "nudges" as for the fact that Sunstein was probably one of the most conservative appointments Obama made.
  - rayiner 4y ago
    Possibly. I recall nudges being a major talking point during the appointment: https://www.politico.com/agenda/story/2015/10/obamas-effort-to-nudge-america-000276 https://www.politico.com/agenda/story/2015/10/obamas-effort-....
    - tptacek 4y ago
      Your comment makes it seem as if the Nudge work had qualified him for OIRA, when in fact he was probably one of the country's most obvious lay-up candidates for that role. As I think you know, given his own background, it would have been weirder if Obama hadn't found a role in the government for Sunstein.
      - rayiner 4y ago
        He released a book called “Nudge” the year before his appointment. I’m not saying he wasn’t qualified for OIRA without it but “making society better through subconscious manipulation” was definitely pitched as part of his shtick at the time.
gumby 4y ago
And a million advertisers and designers cry out. We should spread this widely in the hope that the pop ups and banners die off.
- findalex 4y ago
  This is why I think nudging must work on some level. What are trends, advertising, fads, etc. if not nudges? Is the existence of an intelligent nudger required or can the hive nudge itself?
  - gumby 4y ago
    > This is why I think nudging must work on some level. Or just that people want to believe it does?
oxfordmale 4y ago
Reading this article, it is good that the UK COVID policy wasn't based on behavioural nudging /sarcasm. The UK COVID policy heavily relied on this and one of the unwanted side effects was scaring a certain section of the population into submission. Although that may have been effective during COVID, it made it a lot harder for that segment of society to return back to normal.
- themitigating 4y ago
  For what purpose?
- gsatic 4y ago
  If they can be nugdged one way why not the other?
- renewiltord 4y ago
  Wait, you're arguing that it does have an effect but the effect direction is unpredictable. As an example to differentiate: drinking a homeopathic solution for health has no effect; driving a radium solution for health hurts.
- lozenge 4y ago
  Isn't locking down the obvious departure from a "nudge policy"? I've seen the claim a lot but it all goes back to documents like this one which discusses strategies for communicating to increase compliance with lockdowns. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/882722/25-options-for-increasing-adherence-to-social-distancing-measures-22032020.pdf https://assets.publishing.service.gov.uk/government/uploads/... Yes, the UK government received such shocking insights as "Messaging needs to emphasise and explain the duty to protect others", and "Messaging about actions need to be framed positively in terms of protecting oneself and the community, and increase confidence that they will be effective". Of course, the government did pick and choose what to follow, so it would be absurd to say the entire COVID policy was "based on behavioural nudging". The UK's adherence to isolating after positive tests was thought to be one of the lowest of any country. When SPI-B pointed out that financial support would increase adherence to isolating, no reaction from the government. https://www.instituteforgovernment.org.uk/blog/government-support-self-isolate https://www.instituteforgovernment.org.uk/blog/government-su...
lazyant 4y ago
What's exactly "nudging" here?. For example it has been shown that for organ donation, if the default is affirmative (opt-in) and you have to opt-out, then organ donors double https://www.science.org/doi/10.1126/science.1091721 https://www.science.org/doi/10.1126/science.1091721 , I think this is one of the "nudge" example in pop-science books.
- endominus 4y ago
  I think that's more narrowly defined as "status quo bias" - people tend to take the lowest energy path, so generally accept default choices. The definition of nudging that I could determine from the original book's Wikipedia page includes that, as well as other forms of nudges. I wonder if separating out these nudges by type would result in different results in this metaanalysis. But that is also analogous to p-hacking, isn't it?
- adammarples 4y ago
  I think Newton covered that in his first law. Nobody is actually being nudged, which implies a behavior change, at all.
- tgv 4y ago
  > A nudge, according to Thaler and Sunstein is any form of choice architecture that alters people's behaviour in a predictable way without restricting options or significantly changing their economic incentives. To count as a mere nudge, the intervention must require minimal intervention and must be cheap. Thaler and Sunstein wrote the book on nudges, quite literally. So their definition counts, and it's the one from the article. The opt-in/out decision you mention isn't a nudge in this sense. You're not asked what you prefer, you have to be aware that you can opt-in/out and then actively pursue that option.
  - svnt 4y ago
    A nudge is naive if not circularly defined then as it presumes at least two permanently distinct classes of humans: informed humans who can architect nudges and learn about them and other humans who must respond in the same way every time and cannot do this meta-learning.
akyu 4y ago
No evidence for nudging =/= nudging doesn't exist. I'm fairly sure anyone who has done A/B testing at scale has plenty of evidence that nudging works. Perhaps not up to the standard of science, but there are literally people who manipulate choice architecture for a living and I'm fairly convinced a lot of that stuff actually works.
- omginternets 4y ago
  What exactly makes you convinced that it works? To be specific: why wouldn’t there be bias in the A/B testing results, too? There are literally people who give astrological analyses for a living.
  - mcswell 4y ago
    Great minds! I was writing more or less the same thing, you beat me to publication by three minutes.
  - lIl-IIIl 4y ago
    We are talking about publication bias, where the decision whether to publish something is biased by the outcome of the experiment. I think this doesn't really apply to A/B testing, because people are incentivized pay as much attention to negative results as to positive ones.
    - zeroonetwothree 4y ago
      From what I’ve seen there is even more incentive to focus on positive A/B tests. It’s the way you get credit for your work at a company. A negative test is counted as barely anything. So your incentive is to run tons of tests, then cherry pick only the positive ones and announce them widely. Another strategy is to track multiple metrics for each test and not adjust for that when computing p values. But then at the end you only report the one metric that was positive.
    - aaaaaaaaaaab 4y ago
      People are incentivized to pay attention to the result that increases their mid-year bonus the most.
light_hue_1 4y ago
There's a lot of confusion here about what this article is talking about. Nudges aren't just defaults. We've known for over a century that people are influenced by defaults. Nudges also aren't anchoring, where choices influence one another. Kahneman & Tversky won a Nobel prize for that and other behavioral economics ideas a decade before the idea of nudges. Nudges are a bigger idea that many small changes lead to huge behavioral changes. Like providing a social reference point (see the average electricity use of your neighbors), surfacing hidden information (a red light to remind you to change your A/C filter), change the financial effort involved in something (deposit your drinking money into an account that you lose when you drink again; health plans that pay to stay healthy), change the physical effort of making bad choices (a motorcycle license for people who don't want to wear helmets that is much harder to get), change the consequences of options (pay a teenager $1/day to not get pregnant), provide reminders (check if an email is rude and have someone confirm they want to send it), public commitments (say you are doing X makes you more likely to do X), etc. There are various examples of each of these working to some extent in specific circumstances. But we have a lot of other tools for changing people's behavior. We have education campaigns. We have fines. We have taxes. We have tax breaks. The idea behind nudges is that they're an easy replacement for many of these other tools. But the meta-analysis shows that nudges aren't a general-purpose tool that leads to significant changes in people's behavior. The behavioral changes are small, the same as we get from a fine, a tax, or an education campaign. Aside from specific circumstances, nudges don't work better (and may be much worse) compared to our usual tools for getting people to behave.
- vishnugupta 4y ago
  I'm not a social scientist, so please help me understand this. The way you have defined nudge it seems like a very broad category. Some passive nudge (defaults) and some active (red lights) and a whole lot others. If a category has such a broad number of phenomenon then shouldn't we be analysing individual phenomenon instead of the category as a whole? For example; defaults may work and red-light thing may not work. Why place them both under the same bucket at all? Why not study them in isolation?
  - light_hue_1 4y ago
civilized 4y ago
I agree with other commenters that it's unlikely nudges never have an impact. We should also be wary of high-profile debunkings, now that they're increasingly in fashion due to the replication crisis and the general dour mood. It's easy to p-hack a result into significance, but you can just as easily hack results into insignificance. These days, both findings and debunkings need a skeptical eye.
- themitigating 4y ago
  Isn't that always the case?
  - pessimizer 4y ago
    I think some people careen from fully trusting one thing to fully trusting the opposite thing. If you're not one of those people, you'll never understand dismissing things on the basis of "you should be critical of this, because not everything that people say is true." I do feel like that, even though being critical is something we should always do, that in cases where 1) the only reason you started paying attention to something was an intuitive hunch that it could matter, and 2) the only reason you started treating that hunch as established science is because you did experiments that had significant results, then 3) later you found that significance could be entirely accounted for by the file-drawer effect, you need to adjust your expectation that there actually is an effect to lower than your expectation was at step 1). It isn't that the theory hasn't been tested (although you can argue it hasn't been tested for ingeniously enough yet), it's that it has been tested and no effect has been shown. If you allow the existence of interest in a theory (represented in amount of ink spilled and number of experiments done) to raise your expectation that the theory is true, despite experimental indications to the contrary, you're not really doing science, you're just throwing good money after bad, probably motivated by a desire to protect the researchers and institutions that are heavily committed to the truth of the theory and/or the desire to protect other theories that depend on the one that hasn't shown results.
    - civilized 4y ago
      There's a lot of junk behavioral science out there, but things like "people often go with a default or recommended option so they can move on with their day" seem so obvious to me that I become suspicious of this debunking for debunking too much.
walterbell 4y ago
Nudging is like SEO, both nudges and perception filters are constantly evolving. Yesterday's nudge is today's fodder for memes.
a_simm 4y ago
Wow. As a former social scientist with an axe to grind this hits hard. I like to provide that HN community with some context as to what this means. There are some 300 “research” departments in each of the major social sciences: psychology, sociology, economics and anthropology. If you believe what they say, about half of their mission is teaching and the other half is research. That’s a lot, tens of billions of dollars. The nudge findings were among the few to not only reach the level of public knowledge but, more importantly, directly influence on public policy. To use the one I most familiar with: the so called default for defined contribution retirement plans, eg 401k. These government regs assumed, for good reason, that maximizing contributions was in the public interest. Based on the nudge findings, after much debate and effort, they were updated to dictate that the max options forms was pre selected in the brief it would cause more individuals would opt for that as opposed to contributing zero. So far so good, right? In fact nudge has become a canonical example in introductory public policy courses as to how their research can in some sense make things better. This meta-analytic finding turns on the authors’ method for measuring publication bias. Because I accept that, I must believe that this entire body of research, probably the signal behavioral economics work, is essentially worthless! Thus, all that effort has not only been wasted but the credibility of social science in general is damaged. Adding this to the well/known gamesmanship in peer review, debate over tenure and etc. means it’s past time to reform a large chunk of academia.
- xyzzy123 4y ago
  In the example you cited I don't think that's a nudge? Or is it? I ask because I am sure that changing defaults DEFINITELY works, especially if the user does not have a strong existing preference. You're not really changing user behaviour most of the time, you're changing the outcome of what they're trying to do, which is to reduce their cognitive load by ignoring as much as they possibly can.
  - JusticeJuice 4y ago
    Nudges are often imagined as just how choices are presented, but yes the default option is considered part of nudge theory. As also is social proof ("Your friends picked this choice"). https://en.wikipedia.org/wiki/Nudge_theory#Types_of_nudges https://en.wikipedia.org/wiki/Nudge_theory#Types_of_nudges
SeanLuke 4y ago
Links to references are entirely broken in Firefox on the PNAS website. Does nobody test for these things?
h2odragon 4y ago
Awww, someone ran over the Dogma again.
hkon 4y ago
Sometimes the implementation of "nudges" is so obvious it has the opposite of the desired effect.
icegreentea2 4y ago
The linked study (and the Merten's study it's built ontop of) classifies defaults as "structural" interventions. In the linked meta-analysis, after stripping out estimates of publication basis, structural interventions have the most "robust" evidence left (95% CI of 0.00-0.43), and as their paper text says "whereas the evidence is undecided for “structure” interventions". Other structural interventions include, making it easier to select the desired outcome (or make it harder to switch away from desired out come), changing the range of options to facilitate evaluation, or trying to compensate for biases and loss aversion in choice structure. As you can see, this is a broad range of interventions. A little bit further they say "However, all intervention categories and domains apart from “finance” show evidence for heterogeneity, which implies that some nudges might be effective, even when there is evidence against the mean effect", which makes sense. People understand stakes generally, and will likely apply different care/effort in different context, modifying the context specific effect of any given intervention. I think the paper makes a reasonable argument: 1. There is significant publication bias in nudging studies 2. The effect of providing additional information at time of selection, or providing reminders/affirmations for self control is basically non-existent 3. The effect of modifying choice structure is indecisive. Likely we'll find that some structural modifications have strong effects in some context, but others have little or no effect is other context.
wawjgreen 4y ago
There's a little-known theory (whose name eludes me) that states: any outcome in behaviour is highly dependent on the immediate and unpredictable interplay of various environment variables and their real and perceived effect on the person. this alone cancels the efficacy of any nudges (but some variables may be in the same direction as the nudge--hence the original but misguided nudge research: they were fooled by randomness). I have seen married women who were loyal to their husbands (and who had no idea they would fall for a guy who practiced "seduction" on them) become bewildered and surprised by their own behaviour even though the behavior went against their firmly-held opinions about themselves (that is: I love my husband and I am loyal to him.). The environmental variables used by the person (who is a marketer) were too strong for their opinions to hold out against. As an example, he would invite them to his studio, which he had decorated (and clean) and made it so homely and snug and comfy that the first lines of defense were broken before they had a chance to realize what trap they were in. The marketer also tried to brainwash me (but failed) because I knew the power of variables and this knowledge alone saved me--even though he seemed irresistibly charming in the capacity of a father figure I never had.
- asdff 4y ago
  Wait until you start reading about genotype by environment interactions and realizing the implications that has on just about everything in biology and society
  - wawjgreen 4y ago
- enviclash 4y ago
  Environment variables are mostly underplayed in social science, but why?
  - wawjgreen 4y ago
rjmunro 4y ago
Reading this, I don't get how you can take all "nudging" and declare "No evidence". Surely "nudging" encompasses a whole range of different actual actions. Some nudges work, some don't. You can't just average across all of them. I'm probably totally misunderstanding, but it sounds similar to saying "there is no evidence for medicine" because you've averaged all the papers describing medical interventions that work and those that don't. I thought the point of "nudges" is that they are so cheap to implement you can easily afford to try many. Most won't work, some will.
- 13415 4y ago
  I have the same concern. Nudging is an umbrella term for a vast number of very different activities. For example, nudging is a term used for motivating more carefully designed road markings. I find it hard to believe that some of these newer designs don't "work" better than the old ones, some of them are quite ingenious. At the same time "nudging" is also used for all sorts of public policy framing issues that are more questionable and have probably harder to measure effects. As you say, each "nudge" needs to be evaluated individually.
  - picardo 4y ago
    Agree with you. Nudging is a type of user experience design. UX designers nudge with every design decision they make, and the effectiveness of those decisions is quantifiable. So it's hard to argue that all nudges are ineffective, just like one can't argue that all UX design is ineffective.
- mcswell 4y ago
  There are a number of posts that address this issue, as well as issues raised by responses to this post. I posted somewhere "no true Scotsman"--I'm not claiming my post was enlightening in itself, but the post I was responded to (and the entire thread) was, IMO, enlightening. As for averaging, yes, you can: if a nudge is ineffective, then its result will be random, and a bunch of ineffective nudges will average zero effectiveness. The effective nudges will then push the overall average above zero. We don't see that. (The same would be true for medical interventions, unless some cause harm.) As for being able to try lots of them: in some circumstances, maybe. But when a government is trying to nudge people towards some desired behavior (vaccination, say, to take a random example), they don't try sending out a bunch to different groups of people, then polling each of those groups to see which groups--and therefore which nudges--moved. And it's not always practical, anyway (and the vaccination example is a case where it's almost certainly not practical). See also threads that mention A/B testing.
turns0ut 4y ago
I can’t help but see social science as humans attempting to modernize memory of imperialism and religious belief embedded by prior experience. Think of how popular it became as a field in the last 50-100 years as the populace became less religious. The US adult population recently crossed a threshold where <50% believe in higher power now. No science gives social scientists higher powers of forecasting human future, yet we took the ideas and applied them with the same conviction some believe in gods, in the same way; a network of randos spreading their gossip, wrapping it in technical jargon biased by past ignorance. Consider how much of this work was being leveraged against an ignorant public with no opt out button, via print and TV. How is that informed consent? Social media comes along, upends those forms of media, creates a new meta awareness we lived in a society policed by high minded but normal people. That awareness means we can opt out of being influenced by intentional nudges, same as we opt out of believing in intentional nudges to abide higher powers. Social science “worked” when the masses were unaware it was happening to them. As the public has become more aware of how it works, it’s all Soylent green; just people.
mikkergp 4y ago
it seems like what they are saying is here is some statistics that suggests that maybe there’s a bunch of missing data that would Show that nudge doesn’t have an effect. Like most science it seems like the conclusion should be, go do more research to see if we’re right, not, you should conclude that nudging doesn’t work.
- mcswell 4y ago
  They're implying that some such research has already been done, but wasn't published because of publication bias against negative results. So the data exists, but is missing from publication. (I suspect it's also possible that more skeptical researchers have been dissuaded from even doing the research, for fear they'd be wasting their time and research budget because after they did the research, it wouldn't get published.)
  - mikkergp 4y ago
    Right but they’re implying that statistically, they don’t have a pile of said research on their desk that they’re pointing too.
darkerside 4y ago
So many hot takes about a paper that says, > A newly proposed bias correction technique, robust Bayesian metaanalysis (RoBMA) (6), avoids an all-or-none debate over whether or not publication bias is “severe.” Absence of evidence doesn't mean it's not true. It doesn't even imply it.
karpierz 4y ago
The issue with this kind of meta analysis is that, as the author, you get to decide what the groupings are. No two studies will be identical, so you can invent bucketing strategies until you find that some buckets have the results you want and focus on those. In addition, they don't seem to have shown that the technique they're applying actually works for modelling the distributions that they're analyzing.
arpinum 4y ago
Not a great journal if you are trying to publish something with potentially large import. Its reasonable to guess that something is seriously wrong with the study to not get it in a good journal. This publication does not move my opinion on the matter.
lifeisstillgood 4y ago
Their summary csv listing all the studies reviewed is at https://osf.io/ubt9a https://osf.io/ubt9a And boy am I struggling - I am amazed it's even possible to group all of these studies under the same umbrella unless that is "misc". Claiming that how people choose to treat their cancer, portion sizes at restaurants, rural Kenyan maternal health and Dutch childrens vegetable choices are even in the same field seems - incredible. Maybe I am agreeing with the study in a roundabout way. If all of these things are under the heading "nudge" then it is too broad a heading. It's probably impossible to say one way or another that nudging works because you can never unpick all the confounding factors. Did the Dutch children have a popular TV show about vegetables while the Kenya media ran months long articles about unsanitary hospital conditions? With my cynical hat on Nudge is a way for politicians to try something even when the real fix is intractable. I don't oppose "do something positive" - Injust oppose abusing power, violence for political gain, and all the other reasons why we can agree on a nudge in the right direction but cannot agree on a structural fix in the right direction. I guess if they worked then they would solve the problems without structural chnage and so would defeat the forces that benefit from the status quo. so yeah. it does not work. looks like we will have to go back to the old politics and revolt.
Angostura 4y ago
I'm just in the process of reading 'Nudge - the final edition' - definitely worth a read - it's thought-provoking, funny, insightful and enjoyable. It would be a shame if it all turned out to be bullshit as the examples given in the book seem straightforward and plausibly effective.
verisimi 4y ago
Whether 'nudging' works or not, the concept is unacceptable to me. First, the term 'nudging' is a misnomer. Let's call it what it is - manipulation. Manipulating the options or defaults to some other set in order to achieve a better outcome for someone... Well, who is that someone? The government? Who says that their values align with mine? I wouldn't have responded as the government did to the pandemic, but their nudge units went into overdrive nudging people into vaccinations, etc. Is preventing access to bank accounts for protesting government actions (as in Canada) a 'nudge'? Can I challenge the promoted values? If the state apparatus has its own values and agenda, how do I get to state mine - where is the values/ethics discussion being had, and how do I get my say? I find the promoted values Orwellian, communistic, overly progressive - one for all, but not all for one... is that opinion fair to hold? Or must I be nudged over the cliff? Aren't we really just talking about soft-sell authoritarianism here? Weren't we just meant to vote for people, not have a perpetual nanny state guiding us?
- sacrosancty 4y ago
  Your vote says their values align with yours, or at least most people's. Governments have to do things all the time that step on a few toes for the overall good. What about all other public health campaigns? Drink driving, cancer screening, anti-smoking? Not everyone will want what they're pushing but we mostly agree that's a good thing for people, and that's why we let the government promote those things.
iambateman 4y ago
Keep in mind that the people who publish a correction like this have a lot to gain from getting a lot of attention. They succeed when lots of people say “what?! This broadly accepted idea is wrong?” It’s the equivalent of a Buzzfeed headline, even if backed by thoughtful research. The new research may be correct in invalidating the prior experiment’s evidence but the reality is that we all know the “nudge” idea is useful at a practical level. If I ask my two-year old son if he wants milk, the odds that he is drinking milk 5 minutes later skyrocket. The same principle applies to people making all kinds of choices - from buying insurance to picking a University. Choice architecture matters and we all know it.
- gwd 4y ago
  > Keep in mind that the people who publish a correction like this have a lot to gain from getting a lot of attention. So you're saying that we need to account for publication bias when reading a paper proposing a new way to evaluate publication bias?
malisper 4y ago
Reading through the article, it seems the actual claim is much, much weaker than the title: > However, all intervention categories and domains apart from “finance” show evidence for heterogeneity, which implies that some nudges might be effective, even when there is evidence against the mean effect So the article is saying when you look at studies of all "nudges" as whole, adjust for publication bias[0], there isn't evidence for nudges as a whole. Of course individual nudges could still have a positive impact. Maybe I'm misinterpreting what it's saying, but as an analogy, that would be like doing a study of all "diets", determining that when you combine data for all studies on diets there isn't a positive effect, then writing an article with the claim "no evidence for diets". There's no way you could reasonably make the claim that no diets work. [0] If you look at the studies on "nudges", studies with smaller sample sizes detected a larger positive effect size. This is because smaller studies that get a positive effect are more likely to be published than smaller studies with a negative effect. The article uses this to analyze just how strong the publication bias is and adjust for it.
- rahimnathwani 4y ago
  > There's no way you could reasonably make the claim that no diets work. Even if some studies results suggest that diets work, that does not, by itself, mean we should reject the null hypothesis (that diets don't work). Coincidences exist: https://xkcd.com/882/ https://xkcd.com/882/
- whimsicalism 4y ago
  You are misinterpreting what it is saying. It's like multi comparison bias - if you do a bunch of comparisons and find one significant at 0.05, you can go ahead and say that is significant if you had looked at it alone. But if you consider all of the experiments you are doing, you would expect to find one as significant just by random chance even if it isn't significant.
exac 4y ago
Anyone who has ever run an installer, that has an annoying setting hidden in the default-toggled-off-advanced section, knows that nudges do work. Perhaps nudges do not work on you, but when you see your parent's computer with all the desktop icons and browser toolbars in days gone bye, you will see the evidence of them working.
irjustin 4y ago
Genuine Question: has anyone tried to reproduce the study of the toilet fly? I tried looking but all I can get are articles citing the original Thaler claims/studies.
aaron695 4y ago

246 points by ororm 4y ago

Hi all, I'm one of the authors of the reply and it was very interesting reading so many diverse thoughts and comments. I would love to respond to all of them, but it would take ages. Luckily, Stuart Ritchie (@StuartJRitchie) wrote an awesome post on his substack (https://stuartritchie.substack.com/p/nudge-meta https://stuartritchie.substack.com/p/nudge-meta) that goes much deeper and adresses many questions and the fair critique raised here. Also, note that there is only a limited amount of information and nuance you can comprise into a strict 500 words reply limit in PNAS, which is the reason we focus only on one aspect of the original meta-analysis -- publication bias. Cheers, Frantisek

mikkergp 4y ago

If publication bias is the exclusion of publishing results that doesn’t support your hypothesis, how are they taking that into account? If I’m interpreting this correctly(and I by no means am sure that I am), I infer that they are saying in a fair publishing environment you’d expect to see more results that show less decisive results, therefor the current set of results is likely biased. Couldn’t this bias also happen in the other direction? It sounds like they’re saying the results are too good and don’t match other scientific patterns of publishing results.

mcswell 4y ago
"Couldn’t this bias also happen in the other direction?" The general notion is that positive results ("we tried nudging and succeeded in getting people to behave more in X way") are more publishable than negative results ("we tried nudging, and nothing much happened"). It is a common and well-attested problem in many areas of science, but probably particularly in behavioral sciences; I have not heard of cases where publication of negative results is more likely than publication of positive results, although there are obviously heated debates over some results. Whether publication bias is the explanation in this example, I don't pretend to know.
- mikkergp 4y ago
  I guess the question would be are positive results missing for another reason, like they are harder to test for, therefor the data looks better than it should because in the aggregate they are better, but yeah this is probably unlikely.
- ghaff 4y ago
  There's also an implicit assumption with nudges/defaults that you're nudging people towards a reasonable place for some combination of policy and preference reasons. But imagine that a company would just as soon not pay out more 401k matching than it has to, so it makes the default zero. (Which of course is often the norm for different reasons.) That's as valid a nudge as anything but we shouldn't be surprised if a lot of people don't go with the default. We probably also shouldn't be surprised if a lot of people maybe wouldn't go with a maxed out default. Defaults wouldn't be nearly so powerful if they weren't typically chosen to be fairly reasonable for the average person in the target audience.
  - mikkergp 4y ago
    But I don’t think the goal is “most people can be nudged” so much as “a nudge is a cheap way to increase a behavior by 5-10% which is probably quite significant in policy circles. Low single digit percentage increase in people who pay their taxes would be huge.
notafraudster 4y ago
The easiest to understand diagnostic used to measure publication bias is the funnel plot. Suppose the true effect of interest is theta = 0.2. Then the observed effects in studies should be centered at 0.2; some will be higher, some will be lower. Assuming no systematic error, the degree to which study results vary around 0.2 should be proportional to the precision of the study (think sampling error given a sampling design). A hypothetical study of an infinitely large meta-population would produce the effect estimate of exactly 0.2, infinitely precisely. A series of very small studies will likely show quite divergent results, just on the basis of precision. A funnel plot plots effect sizes on the x axis and precision on the y axis. The most precise studies should be tightly grouped around the meta-analytic average effect; the least precise studies should be spread more widely. This forms a triangular, funnel shape. If no publication bias exists, the spread of studies below the magnitude of the average effect should be comparable to the spread of studies above the magnitude of the average effect. If there is publication bias, then the points that would form the left (without loss of generality; right if negative effect size) portion of the funnel will not be observable. There are issues with funnel plots and there are other diagnostics but I hope this provides insight into one of the tools used. Notably, as a diagnostic, funnel plots work whether the true effect is positive, negative, or null; they assume only that the underlying assumptions of meta-analysis are true (that studies represent a sample of the same, true underlying effect -- other diagnostics and corrections exist when this is violated)
- lmeyerov 4y ago
  Interesting -- the problem may be a misapplication of funnel plots for metaanalysis. I'm not sure what theta is representing and only skimmed the paper, but especially in social scenarios and across social papers, seems unlikely to assume the same distributions and parameters across tasks & populations. Sometimes comes down to 'is there any effect??' and sometimes a precise notion of effect size in a lucky/clever specific scenario. Likewise, social science is one of the hardest fields to setup a good experiment, and few publications accepts negative results, so mostly only 'good' p-value ranges getting published seems normal. The Wikipedia page on funnel plots shows, afaict, the same criticism of the technique. Whether about the effect size or how it is reported, funnel plots seem an inappropriate choice for debunking something as general as 'nudges' across heterogeneous studies. Skimming made the metaanalysis feel rather lazy (lack of cross validation, interpretation, ...). Not my field, but I would have had to do some digging before accepting this metaanalysis in review, and by default, would be 'not ready'.

picardo 4y ago

I'm not very familiar with how "nudging" is defined in the behavioral economics, and perhaps someone can enlighten me, but personally I find it hard to believe that it can be disproven that the way a choice is presented doesn't play any role in one's decision. The Goldilock principle is well-known. Most people instinctively choose the middle option, when given a three things to choose from. Does this study imply that choice architecture plays no role in our decisions? Or am I mis-understanding it?

hackerlight 4y ago
You are misunderstanding. They're saying that there's an absence of evidence, not evidence of absence.
- tgv 4y ago
  OTOH, people have been looking for evidence of nudging, and didn't find it. Since the a priori probability of a more than marginal effect of nudging is unlikely, we can conclude that it's much more likely that nudging doesn't "work" than that it does.
- IshKebab 4y ago
  Absence of evidence is evidence of absence. It's just not proof of absence. For example, if I search your entire house for drugs, using drug sniffing dogs and so on and I don't find any at all, that's pretty good evidence that you don't have any. It's not proof though - you might have just stashed them really well. Similarly, if people have been looking for nudge effects for ages, doing loads of studies on it for years, and none of them have found any effect, then that's pretty good evidence that the effects don't exist. It's not proof though; they might just have not been very good experiments.
  - hackerlight 4y ago
    Well said. I'd just change your first sentence to "Absence of evidence can be (but isn't necessarily) evidence of absence.", which is more in line with the rest of your post.

bilsbie 4y ago

How do we know this study wasn’t “nudged”?

mcswell 4y ago
because it's a meta-study?
svnt 4y ago
That’s not the nudge you’re looking for.

rayiner 4y ago

Remember, Obama appointed a regulatory czar based on this science: https://www.nytimes.com/2010/05/16/magazine/16Sunstein-t.html https://www.nytimes.com/2010/05/16/magazine/16Sunstein-t.htm...

pessimizer 4y ago
Really one of the creepiest people connected to government.
tptacek 4y ago
Obama would have appointed Sunstein no matter what book he wrote, because he was one of UChicago's superstar professors (and was at the same time being aggressively courted by schools around the country; he ended up, of course, at Harvard). His appointment was notable not so much for "nudges" as for the fact that Sunstein was probably one of the most conservative appointments Obama made.
- rayiner 4y ago
  Possibly. I recall nudges being a major talking point during the appointment: https://www.politico.com/agenda/story/2015/10/obamas-effort-to-nudge-america-000276 https://www.politico.com/agenda/story/2015/10/obamas-effort-....
  - tptacek 4y ago
    Your comment makes it seem as if the Nudge work had qualified him for OIRA, when in fact he was probably one of the country's most obvious lay-up candidates for that role. As I think you know, given his own background, it would have been weirder if Obama hadn't found a role in the government for Sunstein.
    - rayiner 4y ago
      He released a book called “Nudge” the year before his appointment. I’m not saying he wasn’t qualified for OIRA without it but “making society better through subconscious manipulation” was definitely pitched as part of his shtick at the time.

gumby 4y ago

And a million advertisers and designers cry out. We should spread this widely in the hope that the pop ups and banners die off.

findalex 4y ago
This is why I think nudging must work on some level. What are trends, advertising, fads, etc. if not nudges? Is the existence of an intelligent nudger required or can the hive nudge itself?
- gumby 4y ago
  > This is why I think nudging must work on some level. Or just that people want to believe it does?

oxfordmale 4y ago

Reading this article, it is good that the UK COVID policy wasn't based on behavioural nudging /sarcasm. The UK COVID policy heavily relied on this and one of the unwanted side effects was scaring a certain section of the population into submission. Although that may have been effective during COVID, it made it a lot harder for that segment of society to return back to normal.

themitigating 4y ago
For what purpose?
gsatic 4y ago
If they can be nugdged one way why not the other?
renewiltord 4y ago
Wait, you're arguing that it does have an effect but the effect direction is unpredictable. As an example to differentiate: drinking a homeopathic solution for health has no effect; driving a radium solution for health hurts.
lozenge 4y ago
Isn't locking down the obvious departure from a "nudge policy"? I've seen the claim a lot but it all goes back to documents like this one which discusses strategies for communicating to increase compliance with lockdowns. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/882722/25-options-for-increasing-adherence-to-social-distancing-measures-22032020.pdf https://assets.publishing.service.gov.uk/government/uploads/... Yes, the UK government received such shocking insights as "Messaging needs to emphasise and explain the duty to protect others", and "Messaging about actions need to be framed positively in terms of protecting oneself and the community, and increase confidence that they will be effective". Of course, the government did pick and choose what to follow, so it would be absurd to say the entire COVID policy was "based on behavioural nudging". The UK's adherence to isolating after positive tests was thought to be one of the lowest of any country. When SPI-B pointed out that financial support would increase adherence to isolating, no reaction from the government. https://www.instituteforgovernment.org.uk/blog/government-support-self-isolate https://www.instituteforgovernment.org.uk/blog/government-su...

lazyant 4y ago

What's exactly "nudging" here?. For example it has been shown that for organ donation, if the default is affirmative (opt-in) and you have to opt-out, then organ donors double https://www.science.org/doi/10.1126/science.1091721 https://www.science.org/doi/10.1126/science.1091721 , I think this is one of the "nudge" example in pop-science books.

endominus 4y ago
I think that's more narrowly defined as "status quo bias" - people tend to take the lowest energy path, so generally accept default choices. The definition of nudging that I could determine from the original book's Wikipedia page includes that, as well as other forms of nudges. I wonder if separating out these nudges by type would result in different results in this metaanalysis. But that is also analogous to p-hacking, isn't it?
adammarples 4y ago
I think Newton covered that in his first law. Nobody is actually being nudged, which implies a behavior change, at all.
tgv 4y ago
> A nudge, according to Thaler and Sunstein is any form of choice architecture that alters people's behaviour in a predictable way without restricting options or significantly changing their economic incentives. To count as a mere nudge, the intervention must require minimal intervention and must be cheap. Thaler and Sunstein wrote the book on nudges, quite literally. So their definition counts, and it's the one from the article. The opt-in/out decision you mention isn't a nudge in this sense. You're not asked what you prefer, you have to be aware that you can opt-in/out and then actively pursue that option.
- svnt 4y ago
  A nudge is naive if not circularly defined then as it presumes at least two permanently distinct classes of humans: informed humans who can architect nudges and learn about them and other humans who must respond in the same way every time and cannot do this meta-learning.

akyu 4y ago

No evidence for nudging =/= nudging doesn't exist. I'm fairly sure anyone who has done A/B testing at scale has plenty of evidence that nudging works. Perhaps not up to the standard of science, but there are literally people who manipulate choice architecture for a living and I'm fairly convinced a lot of that stuff actually works.

omginternets 4y ago
What exactly makes you convinced that it works? To be specific: why wouldn’t there be bias in the A/B testing results, too? There are literally people who give astrological analyses for a living.
- mcswell 4y ago
  Great minds! I was writing more or less the same thing, you beat me to publication by three minutes.
- lIl-IIIl 4y ago
  We are talking about publication bias, where the decision whether to publish something is biased by the outcome of the experiment. I think this doesn't really apply to A/B testing, because people are incentivized pay as much attention to negative results as to positive ones.
  - zeroonetwothree 4y ago
    From what I’ve seen there is even more incentive to focus on positive A/B tests. It’s the way you get credit for your work at a company. A negative test is counted as barely anything. So your incentive is to run tons of tests, then cherry pick only the positive ones and announce them widely. Another strategy is to track multiple metrics for each test and not adjust for that when computing p values. But then at the end you only report the one metric that was positive.
  - aaaaaaaaaaab 4y ago
    People are incentivized to pay attention to the result that increases their mid-year bonus the most.

light_hue_1 4y ago

There's a lot of confusion here about what this article is talking about. Nudges aren't just defaults. We've known for over a century that people are influenced by defaults. Nudges also aren't anchoring, where choices influence one another. Kahneman & Tversky won a Nobel prize for that and other behavioral economics ideas a decade before the idea of nudges. Nudges are a bigger idea that many small changes lead to huge behavioral changes. Like providing a social reference point (see the average electricity use of your neighbors), surfacing hidden information (a red light to remind you to change your A/C filter), change the financial effort involved in something (deposit your drinking money into an account that you lose when you drink again; health plans that pay to stay healthy), change the physical effort of making bad choices (a motorcycle license for people who don't want to wear helmets that is much harder to get), change the consequences of options (pay a teenager $1/day to not get pregnant), provide reminders (check if an email is rude and have someone confirm they want to send it), public commitments (say you are doing X makes you more likely to do X), etc. There are various examples of each of these working to some extent in specific circumstances. But we have a lot of other tools for changing people's behavior. We have education campaigns. We have fines. We have taxes. We have tax breaks. The idea behind nudges is that they're an easy replacement for many of these other tools. But the meta-analysis shows that nudges aren't a general-purpose tool that leads to significant changes in people's behavior. The behavioral changes are small, the same as we get from a fine, a tax, or an education campaign. Aside from specific circumstances, nudges don't work better (and may be much worse) compared to our usual tools for getting people to behave.

vishnugupta 4y ago
I'm not a social scientist, so please help me understand this. The way you have defined nudge it seems like a very broad category. Some passive nudge (defaults) and some active (red lights) and a whole lot others. If a category has such a broad number of phenomenon then shouldn't we be analysing individual phenomenon instead of the category as a whole? For example; defaults may work and red-light thing may not work. Why place them both under the same bucket at all? Why not study them in isolation?
- light_hue_1 4y ago

civilized 4y ago

I agree with other commenters that it's unlikely nudges never have an impact. We should also be wary of high-profile debunkings, now that they're increasingly in fashion due to the replication crisis and the general dour mood. It's easy to p-hack a result into significance, but you can just as easily hack results into insignificance. These days, both findings and debunkings need a skeptical eye.

themitigating 4y ago
Isn't that always the case?
- pessimizer 4y ago
  I think some people careen from fully trusting one thing to fully trusting the opposite thing. If you're not one of those people, you'll never understand dismissing things on the basis of "you should be critical of this, because not everything that people say is true." I do feel like that, even though being critical is something we should always do, that in cases where 1) the only reason you started paying attention to something was an intuitive hunch that it could matter, and 2) the only reason you started treating that hunch as established science is because you did experiments that had significant results, then 3) later you found that significance could be entirely accounted for by the file-drawer effect, you need to adjust your expectation that there actually is an effect to lower than your expectation was at step 1). It isn't that the theory hasn't been tested (although you can argue it hasn't been tested for ingeniously enough yet), it's that it has been tested and no effect has been shown. If you allow the existence of interest in a theory (represented in amount of ink spilled and number of experiments done) to raise your expectation that the theory is true, despite experimental indications to the contrary, you're not really doing science, you're just throwing good money after bad, probably motivated by a desire to protect the researchers and institutions that are heavily committed to the truth of the theory and/or the desire to protect other theories that depend on the one that hasn't shown results.
  - civilized 4y ago
    There's a lot of junk behavioral science out there, but things like "people often go with a default or recommended option so they can move on with their day" seem so obvious to me that I become suspicious of this debunking for debunking too much.

walterbell 4y ago

Nudging is like SEO, both nudges and perception filters are constantly evolving. Yesterday's nudge is today's fodder for memes.

a_simm 4y ago

Wow. As a former social scientist with an axe to grind this hits hard. I like to provide that HN community with some context as to what this means. There are some 300 “research” departments in each of the major social sciences: psychology, sociology, economics and anthropology. If you believe what they say, about half of their mission is teaching and the other half is research. That’s a lot, tens of billions of dollars. The nudge findings were among the few to not only reach the level of public knowledge but, more importantly, directly influence on public policy. To use the one I most familiar with: the so called default for defined contribution retirement plans, eg 401k. These government regs assumed, for good reason, that maximizing contributions was in the public interest. Based on the nudge findings, after much debate and effort, they were updated to dictate that the max options forms was pre selected in the brief it would cause more individuals would opt for that as opposed to contributing zero. So far so good, right? In fact nudge has become a canonical example in introductory public policy courses as to how their research can in some sense make things better. This meta-analytic finding turns on the authors’ method for measuring publication bias. Because I accept that, I must believe that this entire body of research, probably the signal behavioral economics work, is essentially worthless! Thus, all that effort has not only been wasted but the credibility of social science in general is damaged. Adding this to the well/known gamesmanship in peer review, debate over tenure and etc. means it’s past time to reform a large chunk of academia.

xyzzy123 4y ago
In the example you cited I don't think that's a nudge? Or is it? I ask because I am sure that changing defaults DEFINITELY works, especially if the user does not have a strong existing preference. You're not really changing user behaviour most of the time, you're changing the outcome of what they're trying to do, which is to reduce their cognitive load by ignoring as much as they possibly can.
- JusticeJuice 4y ago
  Nudges are often imagined as just how choices are presented, but yes the default option is considered part of nudge theory. As also is social proof ("Your friends picked this choice"). https://en.wikipedia.org/wiki/Nudge_theory#Types_of_nudges https://en.wikipedia.org/wiki/Nudge_theory#Types_of_nudges

SeanLuke 4y ago

Links to references are entirely broken in Firefox on the PNAS website. Does nobody test for these things?

h2odragon 4y ago

Awww, someone ran over the Dogma again.

hkon 4y ago

Sometimes the implementation of "nudges" is so obvious it has the opposite of the desired effect.

icegreentea2 4y ago

The linked study (and the Merten's study it's built ontop of) classifies defaults as "structural" interventions. In the linked meta-analysis, after stripping out estimates of publication basis, structural interventions have the most "robust" evidence left (95% CI of 0.00-0.43), and as their paper text says "whereas the evidence is undecided for “structure” interventions". Other structural interventions include, making it easier to select the desired outcome (or make it harder to switch away from desired out come), changing the range of options to facilitate evaluation, or trying to compensate for biases and loss aversion in choice structure. As you can see, this is a broad range of interventions. A little bit further they say "However, all intervention categories and domains apart from “finance” show evidence for heterogeneity, which implies that some nudges might be effective, even when there is evidence against the mean effect", which makes sense. People understand stakes generally, and will likely apply different care/effort in different context, modifying the context specific effect of any given intervention. I think the paper makes a reasonable argument: 1. There is significant publication bias in nudging studies 2. The effect of providing additional information at time of selection, or providing reminders/affirmations for self control is basically non-existent 3. The effect of modifying choice structure is indecisive. Likely we'll find that some structural modifications have strong effects in some context, but others have little or no effect is other context.

wawjgreen 4y ago

There's a little-known theory (whose name eludes me) that states: any outcome in behaviour is highly dependent on the immediate and unpredictable interplay of various environment variables and their real and perceived effect on the person. this alone cancels the efficacy of any nudges (but some variables may be in the same direction as the nudge--hence the original but misguided nudge research: they were fooled by randomness). I have seen married women who were loyal to their husbands (and who had no idea they would fall for a guy who practiced "seduction" on them) become bewildered and surprised by their own behaviour even though the behavior went against their firmly-held opinions about themselves (that is: I love my husband and I am loyal to him.). The environmental variables used by the person (who is a marketer) were too strong for their opinions to hold out against. As an example, he would invite them to his studio, which he had decorated (and clean) and made it so homely and snug and comfy that the first lines of defense were broken before they had a chance to realize what trap they were in. The marketer also tried to brainwash me (but failed) because I knew the power of variables and this knowledge alone saved me--even though he seemed irresistibly charming in the capacity of a father figure I never had.

asdff 4y ago
Wait until you start reading about genotype by environment interactions and realizing the implications that has on just about everything in biology and society
- wawjgreen 4y ago
enviclash 4y ago
Environment variables are mostly underplayed in social science, but why?
- wawjgreen 4y ago

rjmunro 4y ago

Reading this, I don't get how you can take all "nudging" and declare "No evidence". Surely "nudging" encompasses a whole range of different actual actions. Some nudges work, some don't. You can't just average across all of them. I'm probably totally misunderstanding, but it sounds similar to saying "there is no evidence for medicine" because you've averaged all the papers describing medical interventions that work and those that don't. I thought the point of "nudges" is that they are so cheap to implement you can easily afford to try many. Most won't work, some will.

13415 4y ago
I have the same concern. Nudging is an umbrella term for a vast number of very different activities. For example, nudging is a term used for motivating more carefully designed road markings. I find it hard to believe that some of these newer designs don't "work" better than the old ones, some of them are quite ingenious. At the same time "nudging" is also used for all sorts of public policy framing issues that are more questionable and have probably harder to measure effects. As you say, each "nudge" needs to be evaluated individually.
- picardo 4y ago
  Agree with you. Nudging is a type of user experience design. UX designers nudge with every design decision they make, and the effectiveness of those decisions is quantifiable. So it's hard to argue that all nudges are ineffective, just like one can't argue that all UX design is ineffective.
mcswell 4y ago
There are a number of posts that address this issue, as well as issues raised by responses to this post. I posted somewhere "no true Scotsman"--I'm not claiming my post was enlightening in itself, but the post I was responded to (and the entire thread) was, IMO, enlightening. As for averaging, yes, you can: if a nudge is ineffective, then its result will be random, and a bunch of ineffective nudges will average zero effectiveness. The effective nudges will then push the overall average above zero. We don't see that. (The same would be true for medical interventions, unless some cause harm.) As for being able to try lots of them: in some circumstances, maybe. But when a government is trying to nudge people towards some desired behavior (vaccination, say, to take a random example), they don't try sending out a bunch to different groups of people, then polling each of those groups to see which groups--and therefore which nudges--moved. And it's not always practical, anyway (and the vaccination example is a case where it's almost certainly not practical). See also threads that mention A/B testing.

turns0ut 4y ago

I can’t help but see social science as humans attempting to modernize memory of imperialism and religious belief embedded by prior experience. Think of how popular it became as a field in the last 50-100 years as the populace became less religious. The US adult population recently crossed a threshold where <50% believe in higher power now. No science gives social scientists higher powers of forecasting human future, yet we took the ideas and applied them with the same conviction some believe in gods, in the same way; a network of randos spreading their gossip, wrapping it in technical jargon biased by past ignorance. Consider how much of this work was being leveraged against an ignorant public with no opt out button, via print and TV. How is that informed consent? Social media comes along, upends those forms of media, creates a new meta awareness we lived in a society policed by high minded but normal people. That awareness means we can opt out of being influenced by intentional nudges, same as we opt out of believing in intentional nudges to abide higher powers. Social science “worked” when the masses were unaware it was happening to them. As the public has become more aware of how it works, it’s all Soylent green; just people.

mikkergp 4y ago

it seems like what they are saying is here is some statistics that suggests that maybe there’s a bunch of missing data that would Show that nudge doesn’t have an effect. Like most science it seems like the conclusion should be, go do more research to see if we’re right, not, you should conclude that nudging doesn’t work.

mcswell 4y ago
They're implying that some such research has already been done, but wasn't published because of publication bias against negative results. So the data exists, but is missing from publication. (I suspect it's also possible that more skeptical researchers have been dissuaded from even doing the research, for fear they'd be wasting their time and research budget because after they did the research, it wouldn't get published.)
- mikkergp 4y ago
  Right but they’re implying that statistically, they don’t have a pile of said research on their desk that they’re pointing too.

darkerside 4y ago

So many hot takes about a paper that says, > A newly proposed bias correction technique, robust Bayesian metaanalysis (RoBMA) (6), avoids an all-or-none debate over whether or not publication bias is “severe.” Absence of evidence doesn't mean it's not true. It doesn't even imply it.

karpierz 4y ago

The issue with this kind of meta analysis is that, as the author, you get to decide what the groupings are. No two studies will be identical, so you can invent bucketing strategies until you find that some buckets have the results you want and focus on those. In addition, they don't seem to have shown that the technique they're applying actually works for modelling the distributions that they're analyzing.

arpinum 4y ago

Not a great journal if you are trying to publish something with potentially large import. Its reasonable to guess that something is seriously wrong with the study to not get it in a good journal. This publication does not move my opinion on the matter.

lifeisstillgood 4y ago

Their summary csv listing all the studies reviewed is at https://osf.io/ubt9a https://osf.io/ubt9a And boy am I struggling - I am amazed it's even possible to group all of these studies under the same umbrella unless that is "misc". Claiming that how people choose to treat their cancer, portion sizes at restaurants, rural Kenyan maternal health and Dutch childrens vegetable choices are even in the same field seems - incredible. Maybe I am agreeing with the study in a roundabout way. If all of these things are under the heading "nudge" then it is too broad a heading. It's probably impossible to say one way or another that nudging works because you can never unpick all the confounding factors. Did the Dutch children have a popular TV show about vegetables while the Kenya media ran months long articles about unsanitary hospital conditions? With my cynical hat on Nudge is a way for politicians to try something even when the real fix is intractable. I don't oppose "do something positive" - Injust oppose abusing power, violence for political gain, and all the other reasons why we can agree on a nudge in the right direction but cannot agree on a structural fix in the right direction. I guess if they worked then they would solve the problems without structural chnage and so would defeat the forces that benefit from the status quo. so yeah. it does not work. looks like we will have to go back to the old politics and revolt.

Angostura 4y ago

I'm just in the process of reading 'Nudge - the final edition' - definitely worth a read - it's thought-provoking, funny, insightful and enjoyable. It would be a shame if it all turned out to be bullshit as the examples given in the book seem straightforward and plausibly effective.

verisimi 4y ago

Whether 'nudging' works or not, the concept is unacceptable to me. First, the term 'nudging' is a misnomer. Let's call it what it is - manipulation. Manipulating the options or defaults to some other set in order to achieve a better outcome for someone... Well, who is that someone? The government? Who says that their values align with mine? I wouldn't have responded as the government did to the pandemic, but their nudge units went into overdrive nudging people into vaccinations, etc. Is preventing access to bank accounts for protesting government actions (as in Canada) a 'nudge'? Can I challenge the promoted values? If the state apparatus has its own values and agenda, how do I get to state mine - where is the values/ethics discussion being had, and how do I get my say? I find the promoted values Orwellian, communistic, overly progressive - one for all, but not all for one... is that opinion fair to hold? Or must I be nudged over the cliff? Aren't we really just talking about soft-sell authoritarianism here? Weren't we just meant to vote for people, not have a perpetual nanny state guiding us?

sacrosancty 4y ago
Your vote says their values align with yours, or at least most people's. Governments have to do things all the time that step on a few toes for the overall good. What about all other public health campaigns? Drink driving, cancer screening, anti-smoking? Not everyone will want what they're pushing but we mostly agree that's a good thing for people, and that's why we let the government promote those things.

iambateman 4y ago

Keep in mind that the people who publish a correction like this have a lot to gain from getting a lot of attention. They succeed when lots of people say “what?! This broadly accepted idea is wrong?” It’s the equivalent of a Buzzfeed headline, even if backed by thoughtful research. The new research may be correct in invalidating the prior experiment’s evidence but the reality is that we all know the “nudge” idea is useful at a practical level. If I ask my two-year old son if he wants milk, the odds that he is drinking milk 5 minutes later skyrocket. The same principle applies to people making all kinds of choices - from buying insurance to picking a University. Choice architecture matters and we all know it.

gwd 4y ago
> Keep in mind that the people who publish a correction like this have a lot to gain from getting a lot of attention. So you're saying that we need to account for publication bias when reading a paper proposing a new way to evaluate publication bias?

malisper 4y ago

Reading through the article, it seems the actual claim is much, much weaker than the title: > However, all intervention categories and domains apart from “finance” show evidence for heterogeneity, which implies that some nudges might be effective, even when there is evidence against the mean effect So the article is saying when you look at studies of all "nudges" as whole, adjust for publication bias[0], there isn't evidence for nudges as a whole. Of course individual nudges could still have a positive impact. Maybe I'm misinterpreting what it's saying, but as an analogy, that would be like doing a study of all "diets", determining that when you combine data for all studies on diets there isn't a positive effect, then writing an article with the claim "no evidence for diets". There's no way you could reasonably make the claim that no diets work. [0] If you look at the studies on "nudges", studies with smaller sample sizes detected a larger positive effect size. This is because smaller studies that get a positive effect are more likely to be published than smaller studies with a negative effect. The article uses this to analyze just how strong the publication bias is and adjust for it.

rahimnathwani 4y ago
> There's no way you could reasonably make the claim that no diets work. Even if some studies results suggest that diets work, that does not, by itself, mean we should reject the null hypothesis (that diets don't work). Coincidences exist: https://xkcd.com/882/ https://xkcd.com/882/
whimsicalism 4y ago
You are misinterpreting what it is saying. It's like multi comparison bias - if you do a bunch of comparisons and find one significant at 0.05, you can go ahead and say that is significant if you had looked at it alone. But if you consider all of the experiments you are doing, you would expect to find one as significant just by random chance even if it isn't significant.

exac 4y ago

Anyone who has ever run an installer, that has an annoying setting hidden in the default-toggled-off-advanced section, knows that nudges do work. Perhaps nudges do not work on you, but when you see your parent's computer with all the desktop icons and browser toolbars in days gone bye, you will see the evidence of them working.

irjustin 4y ago

Genuine Question: has anyone tried to reproduce the study of the toilet fly? I tried looking but all I can get are articles citing the original Thaler claims/studies.

aaron695 4y ago