7 ms·

The ego depletion saga demonstrates the importance of effect sizes

64 points by luu 7y ago

rmbryan 7y ago
I found this helpful: https://en.wikipedia.org/wiki/Ego_depletion https://en.wikipedia.org/wiki/Ego_depletion
- klingonopera 7y ago
  Me too. When I started reading, I realized that my cluelessness is what computer illiterate people probably feel when reading about IT stuff. But on the whole, I felt that the article was rather trying to state that effect sizes matter, using the ego depletion theory as an example for that. The implications at the end of the article are particularly interesting, or at least, if I were to be more brazen, don't really deny commonly held prejudices on soft sciences...
  - larnmar 7y ago
    It seems to me that if a psychological effect is so small that you can meta-analyse hundreds of studies and still not be sure whether it exists or not, then it doesn’t matter if it does or not. This isn’t physics, where the existence of a tiny effect such as time dilation might nonetheless have significance beyond the observable regime. If a psychological effect is barely measurable then it’s of no use to anybody. In particular you don’t need to go around taking it into effect in your everyday life. Compare to the original analogy of a muscle. An actual muscle, if you work it close to its limit, will be weaker one minute later... but if you work it to its limit three times a week for a month it will get stronger. These effect sizes are large, they’re consistently observed in ~100% of healthy adults, and you can actually apply them in your life.
    - heracles 7y ago
      I am so sick and tired of the muscle analogy (not criticizing you, but the general usage)! Almost always when I see it used, it is to describe a situation where we know almost _nothing_ about the fundamental phenomena! So any so-called similarity is 100 % speculative. I'd consider "it is like a muscle" a red flag when reading news/science.
      - AstralStorm 7y ago
        It's doubly funny, because we actually have not enough of an idea how muscles do what they do... Even so far as immune system may be involved in their growth and strength adaptation.
        eru 7y ago
        So it's an apt analogy after all.
        heracles 7y ago
        I don't think so, no. Here is why: the basic behavior of muscles are decently well understood. For example why and how they contract and relax, and why they tire. We know of several reasons why muscles might malfunction. What is not well established is how and why muscles become stronger over time. Compare this to for example ego depletion, and I would say that the analogy is not apt.
        eru 7y ago
        Yes, true.
      - lidHanteyk 7y ago
        If one pumps iron, then one gets big and strong. What is the if-then for ego depletion? This entire fuzzy informal idea of common-sense causality is what is being wagered by these flimsy psychological concepts.
        feanaro 7y ago
        If one exercises his willpower a lot†, then one has a progressively harder time doing so again shortly after‡. This is so because willpower is a finite, expendable resource which has to be renewed (through not exercising it). † - for instance, by denying himself bad, tempting things ‡ - gives in to temptation It's not flimsy at all, it just appears to be wrong.
    - o09rdk 7y ago
      The ego depletion effect story is interesting to me for numerous reasons, some of which are statistical/inferential, some of which are more theoretical, and some of which are kind of personal or anecdotal (I know the author of the blog post in question and our research overlaps in important ways, although not with regard to the ego depletion effect in particular; I don't research it). I kind of think Brent could have gone further with some of the implications of the ED effect size being so small. To me, it's interesting that it is basically zero (I don't think it's even .08, based on Bayesian prior choice effects, which is another interesting angle on this story). Why? Because I think it's a fairly intuitive idea that gets applied by people in everyday life over and over again, and has broader implications. For example, you are in a situation where you're tempted by something, have to resist temptation, and experience it as exhausting. You go home from work and are tired and need to relax, ostensibly because you're tired by being on your best behavior all day. There's recent very cogently argued public health literature arguing that diets don't work largely because it's too hard (read: too fatiguing, too effortful) to resist our favorite foods, either in the short term or over the long term. In short, I think there are a lot of people who would be surprised that the ego depletion effect is so close to zero. Maybe not everyone, but a lot of people. It also raises the question of why it's difficult to resist temptations over a sustained period, and how to model that process more accurately (my reading of the ego depletion literature is that a lot of the effect can be explained by sustained negative emotional state or heightened attention, rather than resource depletion, but that's a different issue; another explanation is with reward prediction models, but I think those are predicting a different aspect of behavior). I guess I'm agreeing with you, but I think the magnitude of the effect size being zero or close to it is important for a lot of reasons aside from meta-scientific issues pertaining to replicability and trustworthiness of results.
code_biologist 7y ago
Great article. This hits close to home as a product developer. I'm certain this phenomena happens in "data driven" product decisions. Top of the funnel, we have the numbers to get statistically sound outcomes on A/B tests quickly. I work on a B2B product (high value, low volume sales cycle) and bottom of the funnel or internal sales process experimentation we try our best at is sketchy and hard to do. Experimental design is hard trying to get adequate power. Any HNers have tips for working in low-data regimes (N=low hundreds)? Is there some magic Bayesian angle I don't know about? We haven't systemically looked at effect sizes and this article is a good push for me to dig into that.
- jack_pp 7y ago
  More data = better prediction. If you have a million users you can get few data per user but lots of data in total. If you have 100 users you can get a lot of data on each one by direct contact
  - AstralStorm 7y ago
    That always depends on the goal. Locking yourself into 100 early adopters may or may not be prudent. Main thing to answer is how representative your sample of target users is. The deeper the questions asked, less likely it is.
- eru 7y ago
  There's money on the line for companies. So directly or indirectly developer are going to be paranoid about fooling themselves. Like you demonstrate here! Alas, some of the social sciences didn't use to be so honest. See this infamous piece about 'methodological terrorism': http://datacolada.org/wp-content/uploads/2016/09/Fiske-presidential-guest-column_APS-Observer_copy-edited.pdf http://datacolada.org/wp-content/uploads/2016/09/Fiske-presi... (or https://www.thecut.com/2016/10/inside-psychologys-methodological-terrorism-debate.html https://www.thecut.com/2016/10/inside-psychologys-methodolog... for an overview).
seibelj 7y ago
Sure fire way to make it to the top of psychology / sociology academia: 1) Go to Ivy school, get your phd, make all the key connections 2) Find something that makes intuitive sense to a highly educated person, and create a test that you claim measures it. Even better if it’s really fuzzy, like “ego depletion”. For example, call it “mood enhancement”. This is if your mood is elevated. The test would ask 20 questions about your current mood, like “Do I feel happy” and “I feel above-normal positive emotions”. These 20 questions will use 1 to 5 scale and provide a 100 point outcome. 3) Use your connections to get people in the academic industry to start promoting it and using it. Publish a paper showing what baseline is. 4) Now is your time to strike. Start making experiments that you anticipate would modify someone’s mood, like giving them a glass of wine before the test, or having them call their mother. Do control and experimental and see the boost to “mood enhancement”. 5) Publish paper after paper on your amazing discoveries of how to boost “mood enhancement”. Get others to cite your publications and make a cottage industry of “mood enhancement” research experts. More people, more papers, more funding - and you started it all with the key genesis “mood enhancement” paper! 6) Write books, go on NPR as a guest speaker, give speeches. You have quantified mood and determined how to boost it. “My research shows that a 7.2 percent increase in mood occurs when you call your mother before a big company presentation. It’s science!” You are now a public figure and can get tenure and big corporate speech bucks. The world has been forever changed by your mood enhancing research!
- dang 7y ago
  Despite its length, this comment is a shallow dismissal in the sense that the site guidelines ask you not to post here: "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." Elongating an extra cynical rant does not count as teaching us something, so please don't post like this to HN. If what you're writing is based on knowledge or experience, please share your knowledge or experience directly. If it isn't, please post elsewhere. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - seibelj 7y ago
    I am indeed throwing stones at the entire academic discipline of psychology and sociology. I see your point but I am curious in the counterpoints in replies, given the massive edifice that has been constructed on taxpayer dollars and has been shown to have absolutely no basis in reality with further pre-registered studies. I can link to many articles about this crisis if you are unaware
lalaland1125 7y ago
The real news is that if current trends continue, ego depletion will soon change sign https://statmodeling.stat.columbia.edu/2019/11/19/break-out-the-marshmallows-friends-ego-depletion-is-due-to-change-sign https://statmodeling.stat.columbia.edu/2019/11/19/break-out-.... This will bring about the ego singularity as the positive feedback loop quickly spirals out of control.
- toolslive 7y ago
  that's a great link!
Angostura 7y ago
For those who may nbot be familiar with the term: From Wiki: Ego depletion refers to the idea that self-control or willpower draws upon a limited pool of mental resources that can be used up. When the energy for mental activity is low, self-control is typically impaired, which would be considered a state of ego depletion.
- yebyen 7y ago
  I've been following this through the lens of Beeminder (and "Quantified Self") -- taking away the idea that if you know your own willpower better, you can do magic tricks like "only promise what you can deliver on" and "incrementally ramping up your own willpower" After looking up this concept of Ego Depletion, which seemed like a new idea I haven't heard of before, when I clicked this discussion today... I can see it's basically the same idea. But what I hadn't considered is that the same ideas are able to be used by, eg. advertisers! While my goal of study in this field is making the most of my own efforts, there is someone out there looking at the same charts, trying to figure out exactly where is the right place to put their advertisement to maximize the effect it has on me, even going as far as to get me to buy something that I wouldn't have bought otherwise. That's just fascinating. (Alternatively, that's a horrifying idea...)
twic 7y ago
What is an "experimental theoretician"?

64 points by luu 7y ago

rmbryan 7y ago

I found this helpful: https://en.wikipedia.org/wiki/Ego_depletion https://en.wikipedia.org/wiki/Ego_depletion

klingonopera 7y ago
Me too. When I started reading, I realized that my cluelessness is what computer illiterate people probably feel when reading about IT stuff. But on the whole, I felt that the article was rather trying to state that effect sizes matter, using the ego depletion theory as an example for that. The implications at the end of the article are particularly interesting, or at least, if I were to be more brazen, don't really deny commonly held prejudices on soft sciences...
- larnmar 7y ago
  It seems to me that if a psychological effect is so small that you can meta-analyse hundreds of studies and still not be sure whether it exists or not, then it doesn’t matter if it does or not. This isn’t physics, where the existence of a tiny effect such as time dilation might nonetheless have significance beyond the observable regime. If a psychological effect is barely measurable then it’s of no use to anybody. In particular you don’t need to go around taking it into effect in your everyday life. Compare to the original analogy of a muscle. An actual muscle, if you work it close to its limit, will be weaker one minute later... but if you work it to its limit three times a week for a month it will get stronger. These effect sizes are large, they’re consistently observed in ~100% of healthy adults, and you can actually apply them in your life.
  - heracles 7y ago
    I am so sick and tired of the muscle analogy (not criticizing you, but the general usage)! Almost always when I see it used, it is to describe a situation where we know almost _nothing_ about the fundamental phenomena! So any so-called similarity is 100 % speculative. I'd consider "it is like a muscle" a red flag when reading news/science.
    - AstralStorm 7y ago
      It's doubly funny, because we actually have not enough of an idea how muscles do what they do... Even so far as immune system may be involved in their growth and strength adaptation.
      - eru 7y ago
        So it's an apt analogy after all.
        heracles 7y ago
        I don't think so, no. Here is why: the basic behavior of muscles are decently well understood. For example why and how they contract and relax, and why they tire. We know of several reasons why muscles might malfunction. What is not well established is how and why muscles become stronger over time. Compare this to for example ego depletion, and I would say that the analogy is not apt.
        eru 7y ago
        Yes, true.
    - lidHanteyk 7y ago
      If one pumps iron, then one gets big and strong. What is the if-then for ego depletion? This entire fuzzy informal idea of common-sense causality is what is being wagered by these flimsy psychological concepts.
      - feanaro 7y ago
        If one exercises his willpower a lot†, then one has a progressively harder time doing so again shortly after‡. This is so because willpower is a finite, expendable resource which has to be renewed (through not exercising it). † - for instance, by denying himself bad, tempting things ‡ - gives in to temptation It's not flimsy at all, it just appears to be wrong.
  - o09rdk 7y ago
    The ego depletion effect story is interesting to me for numerous reasons, some of which are statistical/inferential, some of which are more theoretical, and some of which are kind of personal or anecdotal (I know the author of the blog post in question and our research overlaps in important ways, although not with regard to the ego depletion effect in particular; I don't research it). I kind of think Brent could have gone further with some of the implications of the ED effect size being so small. To me, it's interesting that it is basically zero (I don't think it's even .08, based on Bayesian prior choice effects, which is another interesting angle on this story). Why? Because I think it's a fairly intuitive idea that gets applied by people in everyday life over and over again, and has broader implications. For example, you are in a situation where you're tempted by something, have to resist temptation, and experience it as exhausting. You go home from work and are tired and need to relax, ostensibly because you're tired by being on your best behavior all day. There's recent very cogently argued public health literature arguing that diets don't work largely because it's too hard (read: too fatiguing, too effortful) to resist our favorite foods, either in the short term or over the long term. In short, I think there are a lot of people who would be surprised that the ego depletion effect is so close to zero. Maybe not everyone, but a lot of people. It also raises the question of why it's difficult to resist temptations over a sustained period, and how to model that process more accurately (my reading of the ego depletion literature is that a lot of the effect can be explained by sustained negative emotional state or heightened attention, rather than resource depletion, but that's a different issue; another explanation is with reward prediction models, but I think those are predicting a different aspect of behavior). I guess I'm agreeing with you, but I think the magnitude of the effect size being zero or close to it is important for a lot of reasons aside from meta-scientific issues pertaining to replicability and trustworthiness of results.

code_biologist 7y ago

Great article. This hits close to home as a product developer. I'm certain this phenomena happens in "data driven" product decisions. Top of the funnel, we have the numbers to get statistically sound outcomes on A/B tests quickly. I work on a B2B product (high value, low volume sales cycle) and bottom of the funnel or internal sales process experimentation we try our best at is sketchy and hard to do. Experimental design is hard trying to get adequate power. Any HNers have tips for working in low-data regimes (N=low hundreds)? Is there some magic Bayesian angle I don't know about? We haven't systemically looked at effect sizes and this article is a good push for me to dig into that.

jack_pp 7y ago
More data = better prediction. If you have a million users you can get few data per user but lots of data in total. If you have 100 users you can get a lot of data on each one by direct contact
- AstralStorm 7y ago
  That always depends on the goal. Locking yourself into 100 early adopters may or may not be prudent. Main thing to answer is how representative your sample of target users is. The deeper the questions asked, less likely it is.
eru 7y ago
There's money on the line for companies. So directly or indirectly developer are going to be paranoid about fooling themselves. Like you demonstrate here! Alas, some of the social sciences didn't use to be so honest. See this infamous piece about 'methodological terrorism': http://datacolada.org/wp-content/uploads/2016/09/Fiske-presidential-guest-column_APS-Observer_copy-edited.pdf http://datacolada.org/wp-content/uploads/2016/09/Fiske-presi... (or https://www.thecut.com/2016/10/inside-psychologys-methodological-terrorism-debate.html https://www.thecut.com/2016/10/inside-psychologys-methodolog... for an overview).

seibelj 7y ago

Sure fire way to make it to the top of psychology / sociology academia: 1) Go to Ivy school, get your phd, make all the key connections 2) Find something that makes intuitive sense to a highly educated person, and create a test that you claim measures it. Even better if it’s really fuzzy, like “ego depletion”. For example, call it “mood enhancement”. This is if your mood is elevated. The test would ask 20 questions about your current mood, like “Do I feel happy” and “I feel above-normal positive emotions”. These 20 questions will use 1 to 5 scale and provide a 100 point outcome. 3) Use your connections to get people in the academic industry to start promoting it and using it. Publish a paper showing what baseline is. 4) Now is your time to strike. Start making experiments that you anticipate would modify someone’s mood, like giving them a glass of wine before the test, or having them call their mother. Do control and experimental and see the boost to “mood enhancement”. 5) Publish paper after paper on your amazing discoveries of how to boost “mood enhancement”. Get others to cite your publications and make a cottage industry of “mood enhancement” research experts. More people, more papers, more funding - and you started it all with the key genesis “mood enhancement” paper! 6) Write books, go on NPR as a guest speaker, give speeches. You have quantified mood and determined how to boost it. “My research shows that a 7.2 percent increase in mood occurs when you call your mother before a big company presentation. It’s science!” You are now a public figure and can get tenure and big corporate speech bucks. The world has been forever changed by your mood enhancing research!

dang 7y ago
Despite its length, this comment is a shallow dismissal in the sense that the site guidelines ask you not to post here: "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." Elongating an extra cynical rant does not count as teaching us something, so please don't post like this to HN. If what you're writing is based on knowledge or experience, please share your knowledge or experience directly. If it isn't, please post elsewhere. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- seibelj 7y ago
  I am indeed throwing stones at the entire academic discipline of psychology and sociology. I see your point but I am curious in the counterpoints in replies, given the massive edifice that has been constructed on taxpayer dollars and has been shown to have absolutely no basis in reality with further pre-registered studies. I can link to many articles about this crisis if you are unaware

lalaland1125 7y ago

The real news is that if current trends continue, ego depletion will soon change sign https://statmodeling.stat.columbia.edu/2019/11/19/break-out-the-marshmallows-friends-ego-depletion-is-due-to-change-sign https://statmodeling.stat.columbia.edu/2019/11/19/break-out-.... This will bring about the ego singularity as the positive feedback loop quickly spirals out of control.

toolslive 7y ago
that's a great link!

Angostura 7y ago

For those who may nbot be familiar with the term: From Wiki: Ego depletion refers to the idea that self-control or willpower draws upon a limited pool of mental resources that can be used up. When the energy for mental activity is low, self-control is typically impaired, which would be considered a state of ego depletion.

yebyen 7y ago
I've been following this through the lens of Beeminder (and "Quantified Self") -- taking away the idea that if you know your own willpower better, you can do magic tricks like "only promise what you can deliver on" and "incrementally ramping up your own willpower" After looking up this concept of Ego Depletion, which seemed like a new idea I haven't heard of before, when I clicked this discussion today... I can see it's basically the same idea. But what I hadn't considered is that the same ideas are able to be used by, eg. advertisers! While my goal of study in this field is making the most of my own efforts, there is someone out there looking at the same charts, trying to figure out exactly where is the right place to put their advertisement to maximize the effect it has on me, even going as far as to get me to buy something that I wouldn't have bought otherwise. That's just fascinating. (Alternatively, that's a horrifying idea...)

twic 7y ago

What is an "experimental theoretician"?