6 ms·

The gloves are off, the pants are on

46 points by luu 5y ago

breckenedge 5y ago
When I started getting into reading pop psychology in my mid-20s, I was very lucky to have a dad who reminded me to be skeptical of this stuff, despite it all sounding very confident. Doubly lucky to have a brother-in-law who eagerly criticized Gladwell’s “Outliers.”
- smitty1e 5y ago
  I was very appreciative of the Scott Adams podcast[1] when he pointed out that any story or study that is too "on the nose", that is, hits our confirmation biases too perfectly, needs to be approached with extra caution. [1] https://www.scottadamssays.com/ https://www.scottadamssays.com/
  - perl4ever 5y ago
    That idea didn't originate with him. I can't recall enough of a quote to find it right now, but someone said something like "any story that's good enough to be repeated should be regarded with skepticism'. It's the sort of thing that might be attributed to Mark Twain or Robert Heinlein, whether or not they said it.
    - ksaj 5y ago
      That's a great quote. I tried and failed to find the source, but if it comes to mind some time between now and before you've forgotten you ever mentioned it, please consider adding it to the thread.
    - TrevorJ 5y ago
      I find it wonderfully self-referential that the quote in question can't really be attributed reliably. It's perfect.
  - hobs 5y ago
    Advice he should have probably taken when referring to 4d chess.
xcdfgvd 5y ago
Finished up at the proctologist's office?
tyingq 5y ago
Similarly fun to have to spend days at work doing "Myers Briggs", "Clifton Strengths" or variations thereof. Then have it woven into unrelated conversations for months until the novelty wears off.
- jessriedel 5y ago
  Fwiw, the Big 5 (OCEAN) personality inventory is one of the more robust findings in psychology (much more so than all the examples from the blog post) https://en.wikipedia.org/wiki/Big_Five_personality_traits https://en.wikipedia.org/wiki/Big_Five_personality_traits and the four Meyers Briggs "types" are quite correlated with four of the Big 5. https://en.wikipedia.org/wiki/Myers%E2%80%93Briggs_Type_Indicator#Big_Five https://en.wikipedia.org/wiki/Myers%E2%80%93Briggs_Type_Indi... Biggest issue with Meyers Briggs is that it gives people the impression that types are discrete (i.e., traits have a bimodal distribution) when in fact the traits are normally distributed. And of course it's just out of date since MB was developed decades ago and is entrenched in the business world.
  - YeGoblynQueenne 5y ago
    >> Fwiw, the Big 5 (OCEAN) personality inventory is one of the more robust findings in psychology (much more so than all the examples from the blog post) That depends on who you ask: Many investigations into the structure of individual differences theorize in terms of latent variables, but rely on Principal Components Analyses (PCA) when it comes to the analysis of empirical data. However, the extraction of a principal components structure, by itself, will not ordinarily shed much light on the correspondence with a putative latent variable structure. The reason is that PCA is not a latent variablemodel but a data reduction technique (e.g., Bartholomew, 2004). This is no problem as long as one does not go beyond the obvious interpretation of a principal component, which is that it is a conveniently weighted sumscore. Unfortunately, however, this is not the preferred interpretation among the enthusiastic users of principal components analysis. Consider, for instance, the personality literature,where people have discovered that executing a PCA of large numbers of personality subtest scores, and selecting components by the usual selection criteria, often returns five principal components. What is the interpretation of these components? They are “biologically based psychological tendencies,” and as such are endowed with causal forces (McCrae et al., 2000, p. 173). This interpretation cannot be justified solely on the basis of a PCA, if only because PCA is a formative model and not a reflective one (Bollen & Lennox, 1991; Borsboom, Mellenbergh, & Van Heerden, 2003). As such, it conceptualizes constructs as causally determined by the observations, rather than the other way around (Edwards & Bagozzi, 2000). In the case of PCA, the causal relation is moreover rather uninteresting; principal component scores are “caused” by their indicators inmuch the same way that sumscores are “caused” by item scores. Clearly, there is no conceivable way in which the Big Five could cause subtest scores on personality tests (or anything else, for that matter), unless they were in fact not principal components, but belonged to a more interesting species of theoretical entities; for instance, latent variables. Testing the hypothesis that the personality traits in question are causal determinants of personality test scores thus, at a minimum, requires the specification of a reflective latent variable model (Edwards & Bagozzi, 2000). A good example would be a Confirmatory Factor Analysis (CFA) model. Now it turns out that, with respect to the Big Five, CFA gives Big Problems. For instance, McCrae, Zonderman, Costa, Bond, & Paunonen (1996) found that a five factor model is not supported by the data, even though the tests involved in the analysis were specifically designed on the basis of the PCA solution. What does one conclude from this? Well, obviously, because the Big Five exist, but CFA cannot find them, CFA is wrong. “In actual analyses of personality data [. . .] structures that are known to be reliable [from principal components analyses] showed poor fits when evaluated by CFA techniques. We believe this points to serious problems with CFA itself when used to examine personality structure” (McCrae et al., 1996, p. 563). Denny Borsboom, The attack of the psychometricians, Psychometrika. 2006 Sep;71(3):425-440 https://pubmed.ncbi.nlm.nih.gov/19946599/ https://pubmed.ncbi.nlm.nih.gov/19946599/ In other words: bad statistics, badly applied.
AlbertCory 5y ago
On typing: "other, intangible process factors, e.g., the preference of certain personality types for functional, static and strongly typed languages" stands out. As he says, you may be able to show a slight benefit, but I would bet that if you controlled for that "personality type", the strong and replicable result would be "people who program in a language they like do a better job than those forced to use one they don't like."
derbOac 5y ago
FWIW, I don't think the timescales are correct with the longitudinal twin-exercise studies. I don't think anyone would predict that exercise today would have anything (or rather, very little) causal to do with mood 2 years from now. Sustained exercise over the course of a couple of weeks or more might for that period or for maybe a month or two afterward, but I doubt anything more than that. I also am not entirely sure that twin studies are the best way to get at causality in terms of exercise and mood, because the exercise is nonrandom. I think someone might object in the sense that that's the point, but my perspective is that it's still not the same as randomly assigning exercise. There's a lot buried in their acknowledgment that a third factor could still account for the results, as that means a lot given the design. FWIW, there's also evidence in the opposite direction using similar but different logic without twins, but using molecular genetic risk scores: https://www.nature.com/articles/s41398-021-01306-w https://www.nature.com/articles/s41398-021-01306-w "The risk of depression was 22% higher among those at high genetic risk compared with those at low genetic risk (HR = 1.22, 95% CI: 1.14–1.30). Participants with high genetic risk and unfavorable lifestyle had a more than two-fold risk of incident depression compared with low genetic risk and favorable lifestyle (HR = 2.18, 95% CI: 1.84–2.58). There was no significant interaction between genetic risk and lifestyle factors (P for interaction = 0.69). Among participants at high genetic risk, a favorable lifestyle was associated with nearly 50% lower relative risk of depression than an unfavorable lifestyle (HR = 0.51, 95% CI: 0.43–0.60). We concluded that genetic and lifestyle factors were independently associated with risk of incident depression. Adherence to healthy lifestyles may lower the risk of depression regardless of genetic risk." I guess the broader point is to be careful about making claims of BS (or anything else) based on a single study.
Joeri 5y ago
I’m sort of skeptical about these “turns out” stories. Generally when you follow the trail of turns out the original statement is kind of right but for the wrong reasons. It’s sort of like high school physics, kind of wrong, but right enough to be useful. Take dunning-kruger for example. Yes, it was not shown that people of low ability rank themselves higher than those of high ability. But it was shown that people of low ability rank themselves higher than they should, and that makes dunning-kruger’s popsci version right enough in many circumstances where it is used to make a point. Or take the findings on happiness. Yes, money can make you happier, contrary to prevailing opinion. But other factors affect happiness as well if not more, so if you have to choose between money and other things, often times the other things will make you happier. Again, the popsci version of “money doesn’t make you happy” is right enough to be useful. Same thing with climate change skepticism. When you first start digging into the better skeptic arguments they seem to be onto something, but as you dig down into the details and peel back the layers of the onion, the popsci consensus view of climate change turns out to be right enough to be useful. I’ve come to the conclusion that these false trueisms hang around because they are “good enough”. It turns out the turns outs are much less insightful than they seem.
- bussierem 5y ago
  Like every other popular thought/meme/idea though, it can be equally true that they stick around, not because they are "good enough" but rather because they support the opinion the person (or people) already held. "money doesn't make you happy" could just as easily be commonly believed because it helps people stay happy when they don't have much money, and to explain away not achieving more because "those rich folk aren't REALLY happy, but I am". Same with the statically typing -- we all know how opinionated people are about code, this just helps in the echo chambers too. But also, just because something is "true enough" isn't the issue -- the issue is that these are touted as ways SCIENCE proved someone right, when in fact they are absolutely not "facts" like many believe, and so that can erode belief in science.
cjlm 5y ago
The background to a lot of this skepticism is outlined well in Rutger Bregman's Humankind [0]. I just finished it [1] and enjoyed it. [0] https://www.littlebrown.com/titles/rutger-bregman/humankind/9780316418553/ https://www.littlebrown.com/titles/rutger-bregman/humankind/... [1] https://cjlm.ca/notes/humankind/ https://cjlm.ca/notes/humankind/
twobitshifter 5y ago
Radio stations near me seem to always be playing John Tesh “intelligence for your life” I’ve often wondered a) how many people take that advice and apply it and b) if you tried to apply all the advice what would become of you. I’d wager that you would have to start getting ready for bed immediately upon waking to just handle all the things that are recommended to do “before bed”
draw_down 5y ago
A very recent example: a made-up story about a hospital being overrun with patients who had overdosed on Ivermectin, such that they were unable to treat gunshot victims. This ran in Rolling Stone, the Guardian and other pubs. Your usual run of smart, skeptical urbane types accepted it uncritically. Be careful when something you would like to be true comes along. It’s also worth examining what happens after something like this is shown to be untrue. Some will own up to their mistake, most will not. But will continue banging on about “misinformation” and so forth.
vorpalhex 5y ago
The happiness/income debunking here is flawed. The use of logarithm for the graph axis to give the perspective of a linear relationship is flawed - it's also responding to a mis-summary of the research. Also note that the graph axis for income tops out at around.. about that 75k/year figure. Whoops. It should also be noted that 75k figure is a median, it changes based on cost of living.
- srinivgp 5y ago
  It is not flawed. The subject is under study, continually. The debunking comes from https://www.pnas.org/content/118/4/e2016976118 https://www.pnas.org/content/118/4/e2016976118 where yes, many measures of happiness seem to increase linearly as log(income) increases. That means diminishing marginal utility of money, but it does not mean there's a cap - the relationship seems to continue at all income levels. The main reason to believe the famous previous research was in good faith and could still miss this result is because the previous research used a measure of happiness which itself capped out, and so _could not detect_ changes in happiness after a certain level. If your instruments can't register any changes above some amount, it is no wonder your results level off and stop at that amount. This new research could be wrong, of course. But there's no mis-summary here.
  - niko001 5y ago
    That's super interesting! I can see their study included asking to what extent the participants are experiencing certain negative/positive feelings (which may give some more context), but isn't their core measure of happiness also capped? They use a continuous response scale, but in terms of being capped, it shouldn't matter if it's measured on a 0-100 or 1-5 scale, no? The participants can't be happier than "extremely happy" in their study design as far as I can tell.
- datavirtue 5y ago
  At $75k you stop sucking water with every breath. $150k is a whole different life.
jpfed 5y ago
>One person argues that the Prison Experiment was a case of subjects behaving as their experimenters clearly wanted them to. Is that to say that the Prison Experiment is actually explained by the Milgram obedience experiments? :-)

46 points by luu 5y ago

When I started getting into reading pop psychology in my mid-20s, I was very lucky to have a dad who reminded me to be skeptical of this stuff, despite it all sounding very confident. Doubly lucky to have a brother-in-law who eagerly criticized Gladwell’s “Outliers.”

smitty1e 5y ago
I was very appreciative of the Scott Adams podcast[1] when he pointed out that any story or study that is too "on the nose", that is, hits our confirmation biases too perfectly, needs to be approached with extra caution. [1] https://www.scottadamssays.com/ https://www.scottadamssays.com/
- perl4ever 5y ago
  That idea didn't originate with him. I can't recall enough of a quote to find it right now, but someone said something like "any story that's good enough to be repeated should be regarded with skepticism'. It's the sort of thing that might be attributed to Mark Twain or Robert Heinlein, whether or not they said it.
  - ksaj 5y ago
    That's a great quote. I tried and failed to find the source, but if it comes to mind some time between now and before you've forgotten you ever mentioned it, please consider adding it to the thread.
  - TrevorJ 5y ago
    I find it wonderfully self-referential that the quote in question can't really be attributed reliably. It's perfect.
- hobs 5y ago
  Advice he should have probably taken when referring to 4d chess.

xcdfgvd 5y ago

Finished up at the proctologist's office?

tyingq 5y ago

Similarly fun to have to spend days at work doing "Myers Briggs", "Clifton Strengths" or variations thereof. Then have it woven into unrelated conversations for months until the novelty wears off.

jessriedel 5y ago
Fwiw, the Big 5 (OCEAN) personality inventory is one of the more robust findings in psychology (much more so than all the examples from the blog post) https://en.wikipedia.org/wiki/Big_Five_personality_traits https://en.wikipedia.org/wiki/Big_Five_personality_traits and the four Meyers Briggs "types" are quite correlated with four of the Big 5. https://en.wikipedia.org/wiki/Myers%E2%80%93Briggs_Type_Indicator#Big_Five https://en.wikipedia.org/wiki/Myers%E2%80%93Briggs_Type_Indi... Biggest issue with Meyers Briggs is that it gives people the impression that types are discrete (i.e., traits have a bimodal distribution) when in fact the traits are normally distributed. And of course it's just out of date since MB was developed decades ago and is entrenched in the business world.
- YeGoblynQueenne 5y ago
  >> Fwiw, the Big 5 (OCEAN) personality inventory is one of the more robust findings in psychology (much more so than all the examples from the blog post) That depends on who you ask: Many investigations into the structure of individual differences theorize in terms of latent variables, but rely on Principal Components Analyses (PCA) when it comes to the analysis of empirical data. However, the extraction of a principal components structure, by itself, will not ordinarily shed much light on the correspondence with a putative latent variable structure. The reason is that PCA is not a latent variablemodel but a data reduction technique (e.g., Bartholomew, 2004). This is no problem as long as one does not go beyond the obvious interpretation of a principal component, which is that it is a conveniently weighted sumscore. Unfortunately, however, this is not the preferred interpretation among the enthusiastic users of principal components analysis. Consider, for instance, the personality literature,where people have discovered that executing a PCA of large numbers of personality subtest scores, and selecting components by the usual selection criteria, often returns five principal components. What is the interpretation of these components? They are “biologically based psychological tendencies,” and as such are endowed with causal forces (McCrae et al., 2000, p. 173). This interpretation cannot be justified solely on the basis of a PCA, if only because PCA is a formative model and not a reflective one (Bollen & Lennox, 1991; Borsboom, Mellenbergh, & Van Heerden, 2003). As such, it conceptualizes constructs as causally determined by the observations, rather than the other way around (Edwards & Bagozzi, 2000). In the case of PCA, the causal relation is moreover rather uninteresting; principal component scores are “caused” by their indicators inmuch the same way that sumscores are “caused” by item scores. Clearly, there is no conceivable way in which the Big Five could cause subtest scores on personality tests (or anything else, for that matter), unless they were in fact not principal components, but belonged to a more interesting species of theoretical entities; for instance, latent variables. Testing the hypothesis that the personality traits in question are causal determinants of personality test scores thus, at a minimum, requires the specification of a reflective latent variable model (Edwards & Bagozzi, 2000). A good example would be a Confirmatory Factor Analysis (CFA) model. Now it turns out that, with respect to the Big Five, CFA gives Big Problems. For instance, McCrae, Zonderman, Costa, Bond, & Paunonen (1996) found that a five factor model is not supported by the data, even though the tests involved in the analysis were specifically designed on the basis of the PCA solution. What does one conclude from this? Well, obviously, because the Big Five exist, but CFA cannot find them, CFA is wrong. “In actual analyses of personality data [. . .] structures that are known to be reliable [from principal components analyses] showed poor fits when evaluated by CFA techniques. We believe this points to serious problems with CFA itself when used to examine personality structure” (McCrae et al., 1996, p. 563). Denny Borsboom, The attack of the psychometricians, Psychometrika. 2006 Sep;71(3):425-440 https://pubmed.ncbi.nlm.nih.gov/19946599/ https://pubmed.ncbi.nlm.nih.gov/19946599/ In other words: bad statistics, badly applied.

AlbertCory 5y ago

On typing: "other, intangible process factors, e.g., the preference of certain personality types for functional, static and strongly typed languages" stands out. As he says, you may be able to show a slight benefit, but I would bet that if you controlled for that "personality type", the strong and replicable result would be "people who program in a language they like do a better job than those forced to use one they don't like."

derbOac 5y ago

FWIW, I don't think the timescales are correct with the longitudinal twin-exercise studies. I don't think anyone would predict that exercise today would have anything (or rather, very little) causal to do with mood 2 years from now. Sustained exercise over the course of a couple of weeks or more might for that period or for maybe a month or two afterward, but I doubt anything more than that. I also am not entirely sure that twin studies are the best way to get at causality in terms of exercise and mood, because the exercise is nonrandom. I think someone might object in the sense that that's the point, but my perspective is that it's still not the same as randomly assigning exercise. There's a lot buried in their acknowledgment that a third factor could still account for the results, as that means a lot given the design. FWIW, there's also evidence in the opposite direction using similar but different logic without twins, but using molecular genetic risk scores: https://www.nature.com/articles/s41398-021-01306-w https://www.nature.com/articles/s41398-021-01306-w "The risk of depression was 22% higher among those at high genetic risk compared with those at low genetic risk (HR = 1.22, 95% CI: 1.14–1.30). Participants with high genetic risk and unfavorable lifestyle had a more than two-fold risk of incident depression compared with low genetic risk and favorable lifestyle (HR = 2.18, 95% CI: 1.84–2.58). There was no significant interaction between genetic risk and lifestyle factors (P for interaction = 0.69). Among participants at high genetic risk, a favorable lifestyle was associated with nearly 50% lower relative risk of depression than an unfavorable lifestyle (HR = 0.51, 95% CI: 0.43–0.60). We concluded that genetic and lifestyle factors were independently associated with risk of incident depression. Adherence to healthy lifestyles may lower the risk of depression regardless of genetic risk." I guess the broader point is to be careful about making claims of BS (or anything else) based on a single study.

Joeri 5y ago

I’m sort of skeptical about these “turns out” stories. Generally when you follow the trail of turns out the original statement is kind of right but for the wrong reasons. It’s sort of like high school physics, kind of wrong, but right enough to be useful. Take dunning-kruger for example. Yes, it was not shown that people of low ability rank themselves higher than those of high ability. But it was shown that people of low ability rank themselves higher than they should, and that makes dunning-kruger’s popsci version right enough in many circumstances where it is used to make a point. Or take the findings on happiness. Yes, money can make you happier, contrary to prevailing opinion. But other factors affect happiness as well if not more, so if you have to choose between money and other things, often times the other things will make you happier. Again, the popsci version of “money doesn’t make you happy” is right enough to be useful. Same thing with climate change skepticism. When you first start digging into the better skeptic arguments they seem to be onto something, but as you dig down into the details and peel back the layers of the onion, the popsci consensus view of climate change turns out to be right enough to be useful. I’ve come to the conclusion that these false trueisms hang around because they are “good enough”. It turns out the turns outs are much less insightful than they seem.

bussierem 5y ago
Like every other popular thought/meme/idea though, it can be equally true that they stick around, not because they are "good enough" but rather because they support the opinion the person (or people) already held. "money doesn't make you happy" could just as easily be commonly believed because it helps people stay happy when they don't have much money, and to explain away not achieving more because "those rich folk aren't REALLY happy, but I am". Same with the statically typing -- we all know how opinionated people are about code, this just helps in the echo chambers too. But also, just because something is "true enough" isn't the issue -- the issue is that these are touted as ways SCIENCE proved someone right, when in fact they are absolutely not "facts" like many believe, and so that can erode belief in science.

cjlm 5y ago

The background to a lot of this skepticism is outlined well in Rutger Bregman's Humankind [0]. I just finished it [1] and enjoyed it. [0] https://www.littlebrown.com/titles/rutger-bregman/humankind/9780316418553/ https://www.littlebrown.com/titles/rutger-bregman/humankind/... [1] https://cjlm.ca/notes/humankind/ https://cjlm.ca/notes/humankind/

twobitshifter 5y ago

Radio stations near me seem to always be playing John Tesh “intelligence for your life” I’ve often wondered a) how many people take that advice and apply it and b) if you tried to apply all the advice what would become of you. I’d wager that you would have to start getting ready for bed immediately upon waking to just handle all the things that are recommended to do “before bed”

draw_down 5y ago

A very recent example: a made-up story about a hospital being overrun with patients who had overdosed on Ivermectin, such that they were unable to treat gunshot victims. This ran in Rolling Stone, the Guardian and other pubs. Your usual run of smart, skeptical urbane types accepted it uncritically. Be careful when something you would like to be true comes along. It’s also worth examining what happens after something like this is shown to be untrue. Some will own up to their mistake, most will not. But will continue banging on about “misinformation” and so forth.

vorpalhex 5y ago

The happiness/income debunking here is flawed. The use of logarithm for the graph axis to give the perspective of a linear relationship is flawed - it's also responding to a mis-summary of the research. Also note that the graph axis for income tops out at around.. about that 75k/year figure. Whoops. It should also be noted that 75k figure is a median, it changes based on cost of living.

srinivgp 5y ago
It is not flawed. The subject is under study, continually. The debunking comes from https://www.pnas.org/content/118/4/e2016976118 https://www.pnas.org/content/118/4/e2016976118 where yes, many measures of happiness seem to increase linearly as log(income) increases. That means diminishing marginal utility of money, but it does not mean there's a cap - the relationship seems to continue at all income levels. The main reason to believe the famous previous research was in good faith and could still miss this result is because the previous research used a measure of happiness which itself capped out, and so _could not detect_ changes in happiness after a certain level. If your instruments can't register any changes above some amount, it is no wonder your results level off and stop at that amount. This new research could be wrong, of course. But there's no mis-summary here.
- niko001 5y ago
  That's super interesting! I can see their study included asking to what extent the participants are experiencing certain negative/positive feelings (which may give some more context), but isn't their core measure of happiness also capped? They use a continuous response scale, but in terms of being capped, it shouldn't matter if it's measured on a 0-100 or 1-5 scale, no? The participants can't be happier than "extremely happy" in their study design as far as I can tell.
datavirtue 5y ago
At $75k you stop sucking water with every breath. $150k is a whole different life.

jpfed 5y ago

>One person argues that the Prison Experiment was a case of subjects behaving as their experimenters clearly wanted them to. Is that to say that the Prison Experiment is actually explained by the Milgram obedience experiments? :-)