5 ms·

I do research in psychology and also a little in machine learning (classification using text samples). I've actually been working with the Many Labs data (the p

I do research in psychology and also a little in machine learning (classification using text samples). I've actually been working with the Many Labs data (the project that's the focus of the article). My impressions of machine learning match up with yours. There's just so many parameters that there's so many opportunities to capitalize on chance, combined with [necessarily] huge datasets that preclude replication, that it's hard to avoid. I have also been surprised at how much tweaking there is of parameters; I'm used to work in traditional statistics where more is derived from theoretical principles, as opposed to trying a bunch of values to see what "works." This all lends itself to overfitting. I strongly believe that a lot of the adversarial input work is basically capitalizing on this overfitting. To be fair, I don't think people are necessarily being nefarious, I think people across all fields just don't appreciate the dangers of overfitting. The one upside of this Many Labs work, mentioned in the paper, is that it tends to show that the common criticism of "how can you generalize from this sample of undergrads" is not so much of a problem. Not that it's not an issue at all, but if you're studying some basic cognitive process it's probably not going to matter that much if you use undergrads versus some perfectly representative sample of the population. People have shown this in different ways before but it's useful to know more about. Obviously with some things sociogeographic variation will matter more though. One thing that might not be totally obvious from this article and others is that although some effects clearly replicate, and others do not, there are some effects that seem to be in a grey area, where the effects are probably real but much smaller than originally reported. The distribution of effect size estimate distributions is more continuous, with shades of grey, than these news reports would have you think. Whether or not it matters that an effect is tiny versus zero might not practically matter, but at some level it is important to be mindful of.

forapurpose 8y ago
> One thing that might not be totally obvious from this article and others is that although some effects clearly replicate, and others do not, there are some effects that seem to be in a grey area, where the effects are probably real but much smaller than originally reported. The two prior replication studies that I know of - the first was several years ago and raised the issue in the public mind afaik, then second maybe in the last year - both replicated almost all of the effects. The problems were that some effects were weaker than in the original studies. Are you saying that this most recent attempt at replication shows a large number of studies with no significant effects at all?
- your-nanny 8y ago
  real effects might be weak enough that they only make it through the first time (non replicative) publication process when chance makes the effect seem stronger than it actually is.
  - forapurpose 8y ago
    > real effects might be weak enough that they only make it through the first time (non replicative) publication process when chance makes the effect seem stronger than it actually is. That's hypothetical. What actually happened was what I wrote in the GP: The effects were replicated almost every time, except sometimes they were weaker.
    - your-nanny 8y ago
      you miss my point. the strength of a novel main finding must be sufficient to warrant the opportunity cost of trying to get it published. a weak but real effect will be more or less strongly indicated in an experiment, just by the vagaries of the sample, but weak results typically will not get published. it is thus not surprising to me that findings might frequently be replicable but weaker than initially was found.
SubiculumCode 8y ago
In my mind, 50% replication is not that horrible. Even decent seeming sample sizes might have an 80% chance of replicating at a certain effect size, and the effect size originally reported could be (by chance or other reasons) over-stated, leading to insufficient power. More certainty ends up meaning vast increases in sample size). Another issue that should be tested (have not read paper but doubt it) is not just whether the result is statistically significant, but whether it statistically differs from the original published result. If the effect is weaker, but not statistically different from the published result, I would hesitate to say the effect is not real. Then there is that some of the psychology findings investigated were rather dubious to begin with, and the ones that even non-specialists can identify as being likely to be true replicated.
- GreenToad5 8y ago
  So, a result that is statistically no more accurate than flipping a coin is acceptable to you? Large cultural shifts have been pushed on a large scale based on studies that may have no more probability of truth than flipping a coin and this is OK to build a society on? I couldn't disagree more.
  - SubiculumCode 8y ago
    Your characterization of what I said is overly simplistic and ..wrong. I am saying that even with a good sample, replication studies only have about at most an 80% chance to replicate a true effect for studies that had reported a "moderately-sized" effect size. A lot of other things can account for the extra 30%, including picking the more dubious of findings out there (which seem to be mainly social psychology not cognitive psychology studies). Looking at your other comments on HN, you seem to love throwing bombs and trolling.
    - GreenToad5 8y ago
      That isn't actually even close to what you said (very contrasting statements). 50% is quite horrible (I was not the one who down voted you, but I see others agree). BTW, mentioning your opinion of someone's "other comments" doesn't help your case either. It makes you appear desperate to cause a distraction or detraction.
whatshisface 8y ago
Why don't the peer reviewers demand higher standards?
- _Nat_ 8y ago
  Inability. One of the foundational problems with the soft/social sciences is that many of its practitioners have such weak analytical thinking skills (Note below) and the educational requirements for Psychology majors typically just include a few semesters of statistics courses designed to be understandable for them. Peer reviewers in Psychology generally don't expect authors to go much beyond the basic statistical tricks students are taught in undergrad. From prior discussions with professors in these fields, it's my impression that most are unable to understand just how flimsy such methods are. Personally, my prediction is that this won't get better until we have more AI-guided analytical analysis software packages that make more robust analysis accessible to those in soft-science fields. This is, something to replace reliance on stuff like R^2 and p- values. --- Note: Just to provide some numerical evidence for the assertion that Psychology practitioners tend to be weak with their analytical skills, here's [a PDF of GRE scores](https://www.ets.org/s/gre/pdf/gre_guide_table4.pdf https://www.ets.org/s/gre/pdf/gre_guide_table4.pdf). GRE test-takers planning to study "Psychology" had quantitative reasoning (QR) mean of 149. This is pretty bottom-of-the-barrel; even people planning to go for grad school to do "Arts - Performance & Studio" did a little better, with an average of 151.
  - celrod 8y ago
    "One of the foundational problems with the soft/social sciences is that many of its practitioners have such weak analytical thinking skills (Note below)" Andrew Gelman argued that a fundamental problem with peer review is that it is done by peers. Peers whose background and skills are similar to those of the authors, and thus not likely to catch things that were missed. [When does peer review make no damn sense?](https://andrewgelman.com/2016/02/01/peer-review-make-no-damn-sense/ https://andrewgelman.com/2016/02/01/peer-review-make-no-damn...) I'm sure something could be gained via education. I don't think you necessarily have to be good at math to understand the concepts. But a lot of undergrad courses focus on hypothesis testing and p-values (despite those methods being condemned by the American Statistical Association), and encourage memorization of steps to do simple math, over understanding what any of it means. I think the ad-hoc approach of many machine-learning intros would do better. Maybe programming isn't any easier for a psychology student, but simply hammering the ideas, with short demos of pitfalls, may help. For "small n" problems like in psychology, psych students are much better off with a statistics background than machine learning. So what I'm advocating is a change in how these classes are taught. Of course, I've known plenty of students who program via copy and pasting code, and modifying it until it runs as necessary. The equivalent of memorizing steps math steps. So it will take more to solve the problem.

forapurpose 8y ago

> One thing that might not be totally obvious from this article and others is that although some effects clearly replicate, and others do not, there are some effects that seem to be in a grey area, where the effects are probably real but much smaller than originally reported. The two prior replication studies that I know of - the first was several years ago and raised the issue in the public mind afaik, then second maybe in the last year - both replicated almost all of the effects. The problems were that some effects were weaker than in the original studies. Are you saying that this most recent attempt at replication shows a large number of studies with no significant effects at all?

your-nanny 8y ago
real effects might be weak enough that they only make it through the first time (non replicative) publication process when chance makes the effect seem stronger than it actually is.
- forapurpose 8y ago
  > real effects might be weak enough that they only make it through the first time (non replicative) publication process when chance makes the effect seem stronger than it actually is. That's hypothetical. What actually happened was what I wrote in the GP: The effects were replicated almost every time, except sometimes they were weaker.
  - your-nanny 8y ago
    you miss my point. the strength of a novel main finding must be sufficient to warrant the opportunity cost of trying to get it published. a weak but real effect will be more or less strongly indicated in an experiment, just by the vagaries of the sample, but weak results typically will not get published. it is thus not surprising to me that findings might frequently be replicable but weaker than initially was found.

SubiculumCode 8y ago

In my mind, 50% replication is not that horrible. Even decent seeming sample sizes might have an 80% chance of replicating at a certain effect size, and the effect size originally reported could be (by chance or other reasons) over-stated, leading to insufficient power. More certainty ends up meaning vast increases in sample size). Another issue that should be tested (have not read paper but doubt it) is not just whether the result is statistically significant, but whether it statistically differs from the original published result. If the effect is weaker, but not statistically different from the published result, I would hesitate to say the effect is not real. Then there is that some of the psychology findings investigated were rather dubious to begin with, and the ones that even non-specialists can identify as being likely to be true replicated.

GreenToad5 8y ago
So, a result that is statistically no more accurate than flipping a coin is acceptable to you? Large cultural shifts have been pushed on a large scale based on studies that may have no more probability of truth than flipping a coin and this is OK to build a society on? I couldn't disagree more.
- SubiculumCode 8y ago
  Your characterization of what I said is overly simplistic and ..wrong. I am saying that even with a good sample, replication studies only have about at most an 80% chance to replicate a true effect for studies that had reported a "moderately-sized" effect size. A lot of other things can account for the extra 30%, including picking the more dubious of findings out there (which seem to be mainly social psychology not cognitive psychology studies). Looking at your other comments on HN, you seem to love throwing bombs and trolling.
  - GreenToad5 8y ago
    That isn't actually even close to what you said (very contrasting statements). 50% is quite horrible (I was not the one who down voted you, but I see others agree). BTW, mentioning your opinion of someone's "other comments" doesn't help your case either. It makes you appear desperate to cause a distraction or detraction.

whatshisface 8y ago

Why don't the peer reviewers demand higher standards?

_Nat_ 8y ago
Inability. One of the foundational problems with the soft/social sciences is that many of its practitioners have such weak analytical thinking skills (Note below) and the educational requirements for Psychology majors typically just include a few semesters of statistics courses designed to be understandable for them. Peer reviewers in Psychology generally don't expect authors to go much beyond the basic statistical tricks students are taught in undergrad. From prior discussions with professors in these fields, it's my impression that most are unable to understand just how flimsy such methods are. Personally, my prediction is that this won't get better until we have more AI-guided analytical analysis software packages that make more robust analysis accessible to those in soft-science fields. This is, something to replace reliance on stuff like R^2 and p- values. --- Note: Just to provide some numerical evidence for the assertion that Psychology practitioners tend to be weak with their analytical skills, here's [a PDF of GRE scores](https://www.ets.org/s/gre/pdf/gre_guide_table4.pdf https://www.ets.org/s/gre/pdf/gre_guide_table4.pdf). GRE test-takers planning to study "Psychology" had quantitative reasoning (QR) mean of 149. This is pretty bottom-of-the-barrel; even people planning to go for grad school to do "Arts - Performance & Studio" did a little better, with an average of 151.
- celrod 8y ago
  "One of the foundational problems with the soft/social sciences is that many of its practitioners have such weak analytical thinking skills (Note below)" Andrew Gelman argued that a fundamental problem with peer review is that it is done by peers. Peers whose background and skills are similar to those of the authors, and thus not likely to catch things that were missed. [When does peer review make no damn sense?](https://andrewgelman.com/2016/02/01/peer-review-make-no-damn-sense/ https://andrewgelman.com/2016/02/01/peer-review-make-no-damn...) I'm sure something could be gained via education. I don't think you necessarily have to be good at math to understand the concepts. But a lot of undergrad courses focus on hypothesis testing and p-values (despite those methods being condemned by the American Statistical Association), and encourage memorization of steps to do simple math, over understanding what any of it means. I think the ad-hoc approach of many machine-learning intros would do better. Maybe programming isn't any easier for a psychology student, but simply hammering the ideas, with short demos of pitfalls, may help. For "small n" problems like in psychology, psych students are much better off with a statistics background than machine learning. So what I'm advocating is a change in how these classes are taught. Of course, I've known plenty of students who program via copy and pasting code, and modifying it until it runs as necessary. The equivalent of memorizing steps math steps. So it will take more to solve the problem.