7 ms·

Should this paper in psychological science be retracted? (2016)

48 points by luu 4y ago

gus_massa 4y ago
Extending a trend beyond the interval it is measured is a big mistake. There are a lot of curves that fit the data in the low talen interval and the extend very differently to the high talent interval. If you don't want to get too fancy, a cubic polynomial will probably predict a fake huge advantage for teams with very high talent ratio. I expect something like f(t)=C-expt(t/K) that looks like a quadratic for the low values but it's very flat near the high values. I found many times good fits with f(t)=A log(t/B)+C. In any case, the solution is to have a good model and amore data. Both are very complicated here, and extending the trend is a mistake, even as a unsupported conjeture. I'm not sure if is should be retracted, but if you make a graphic like this in the physics lab, the T.A. will make you redo the whole presentation.
- hgsgm 4y ago
  This was a big issue in Covid case forecasting: changing the model polynomial/formula and then fitting it to recent case count data points, gives different forecasts for when and how high the peak will be.
  - ARandomerDude 4y ago
    [flagged]
    - dang 4y ago
      Please don't cross into unsubstantive comments or flamebait. "Comments should get more thoughtful and substantive, not less, as a topic gets more divisive." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - gus_massa 4y ago
    For a short time prediction a quadratic/cubic is usually not that bad. I guess for covid-19 cases a week is fine. More than a week is probably too much, because the peaks last like a month from beginning to end, so it's too weird for a simple model. I guess you need a SIR model for that https://en.wikipedia.org/wiki/Compartmental_models_in_epidemiology https://en.wikipedia.org/wiki/Compartmental_models_in_epidem... Also, the grow phase is quite exponential, so even for a polynomial fit I'd prefer to use it after changing the vertical axis. Otherwise the polynomial must fit too many almost zeros and then jump, and you will get weird and negative results. There was also a strong weekly variation, cases on weekends were written down on Monday (or Tuesday), so I think it's better to look at the 7-day running average. The problem is that in this paper they are extending the "forecast" too much. It's somewhat the equivalent of 6 months. I thin no serious source tried to make very long term forecasts. (Anyway, some trends and general forecasts are possible, like with the flu.)
- apwheele 4y ago
  I will have to give these formulations a try. IMO pretty common to have asymptote like theoretical behavior in the tails with social science data, https://andrewpwheeler.com/2022/05/02/fitting-a-plateau-effects-model-in-scipy/ https://andrewpwheeler.com/2022/05/02/fitting-a-plateau-effe.... (Formulation is exactly quadratic for lower values and flattens at higher, and can re-parameterize to make it easier to fit.)
monero-xmr 4y ago
I would take it even farther and claim that very little in experimental psychology / sociology has any value whatsoever. Google the top papers of all time in psychology and its either been formally debunked (Stanford prison experiment) or it has no applicable impact on society. Using computers to parse statistical data is interesting and can reveal truths about society. Trying to correlate the delay between eating a marshmellow as a child and income as an adult is ridiculous, yet collectively we have spent untold money and time on tenured professors, conferences, papers, and pop psychology books about such things. Another example is Power Posing. How Amy Cuddy still gets employment is a mystery.
- spekcular 4y ago
  The marshmallow effect replicated (IIRC), and is probably correct, unlike the power posing stuff. That behavior in a single trial in childhood moderately predicts success in adulthood seems like an interesting fact about humans.
  - monero-xmr 4y ago
    What's the applicable result to this? Put children who eat marshmellows too quickly in special "delayed gratification" classes? Is the result such that all children who eat marshmellows too quickly should give up, as their success is predetermined, or is it the impact is barely statistically significant? If I were the parent of a child who ate the marshmellow too quickly, I would do nothing different, as the whole thing is meaningless and doesn't change anything. It's more like, "Oh, OK, my son is 4% less likely to be in the top 20% of the income distribution (according to this barely coherent theory)." Why do we spend so much money, time, and effort discussing and researching this?
    - hgsgm 4y ago
      What? If you found your child was lacking in a valuable life skill, you wouldn't try to teach that skill?
      - 4y ago
Galanwe 4y ago
I'm not sure the methodology as a whole makes any sense, even with the prediction overfit aside. The Y axis is the mean score of the team, but that score is not independent from the opposing team. It means that it does not refute the hypothesis that sparse top talents in a team happen at low level games, where the top talents make a huge difference. When reaching higher level games by acquiring new top talents, the playing field is more leveled, and the scores will even out. To remove that effect, you would need to select games where a team score does not depend on the opposing team score. Or add a new dimension to the regression which is the opposing teams top talent %. TLDR: if you put Michael Jordan in an amateur team, he will overperform and crank up the team score. If you then add 5 other pro players to the team, you'll enter the NBA. Your team will perform better, even though your number of points may decrease, as other teams are closer to your elite teams level.
- dvt 4y ago
  > The Y axis is the mean score of the team, but that score is not independent from the opposing team. The paper is garbage, but this is not true. All Y-axes are how they define "team performance" -- in the case of the FIFA study, it's qualification rankings. In the case of the NBA/MLB study it's end-of-season team win percentages.
  - Galanwe 4y ago
    Indeed, my bad. The argument still holds though. Overall team disparity should be much higher at low level than high level. A team composed of "Michael Jordan + random players" would crush any amateur league with 100% win rate. The same team with "Michael Jordan + Magic Johnson + random players" would probably get 75% win rate in an NBA match. So the metric "win rate" alone, or any other metric non independent of the opposing teams level, is naturally useless in this study.
evolve2k 4y ago
Anyone know of any browser extensions / projects that highlight when an article or PDF had been debunked / retracted? Seems like without this meta data over the original article that were destined to keep consuming even retracted knowledge.
- invig 4y ago
  Is there a standard or something for how this kind of meta data could be kept on articles / papers?
  - kuhewa 4y ago
    The database with the most coverage to my knowledge is Crossref but I'm not sure if a retraction would be associated with the original doi.
- imadethis 4y ago
  retractionwatch.com and .org monitor retracted papers, and they publish plugins for Zotero and similar tools.
- kuhewa 4y ago
  PubPeer
mannerheim 4y ago
The ever-relevant Richard Feynman: 'I think the educational and psychological studies I mentioned are examples of what I would like to call Cargo Cult Science. In the South Seas there is a Cargo Cult of people. During the war they saw airplanes land with lots of good materials, and they want the same thing to happen now. So they’ve arranged to make things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head like headphones and bars of bamboo sticking out like antennas—he’s the controller—and they wait for the airplanes to land. They’re doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn’t work. No airplanes land. So I call these things Cargo Cult Science, because they follow all the apparent precepts and forms of scientific investigation, but they’re missing something essential, because the planes don’t land.' https://calteches.library.caltech.edu/51/2/CargoCult.htm https://calteches.library.caltech.edu/51/2/CargoCult.htm
- deleted 4y ago
  [deleted]
- rahimnathwani 4y ago
  I like the bit later in the essay where he describes what's often missing: "It’s a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty—a kind of leaning over backwards. For example, if you’re doing an experiment, you should report everything that you think might make it invalid—not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you’ve eliminated by some other experiment, and how they worked—to make sure the other fellow can tell they have been eliminated."
  - bongoman37 4y ago
    [dead]

48 points by luu 4y ago

Extending a trend beyond the interval it is measured is a big mistake. There are a lot of curves that fit the data in the low talen interval and the extend very differently to the high talent interval. If you don't want to get too fancy, a cubic polynomial will probably predict a fake huge advantage for teams with very high talent ratio. I expect something like f(t)=C-expt(t/K) that looks like a quadratic for the low values but it's very flat near the high values. I found many times good fits with f(t)=A log(t/B)+C. In any case, the solution is to have a good model and amore data. Both are very complicated here, and extending the trend is a mistake, even as a unsupported conjeture. I'm not sure if is should be retracted, but if you make a graphic like this in the physics lab, the T.A. will make you redo the whole presentation.

hgsgm 4y ago
This was a big issue in Covid case forecasting: changing the model polynomial/formula and then fitting it to recent case count data points, gives different forecasts for when and how high the peak will be.
- ARandomerDude 4y ago
  [flagged]
  - dang 4y ago
    Please don't cross into unsubstantive comments or flamebait. "Comments should get more thoughtful and substantive, not less, as a topic gets more divisive." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- gus_massa 4y ago
  For a short time prediction a quadratic/cubic is usually not that bad. I guess for covid-19 cases a week is fine. More than a week is probably too much, because the peaks last like a month from beginning to end, so it's too weird for a simple model. I guess you need a SIR model for that https://en.wikipedia.org/wiki/Compartmental_models_in_epidemiology https://en.wikipedia.org/wiki/Compartmental_models_in_epidem... Also, the grow phase is quite exponential, so even for a polynomial fit I'd prefer to use it after changing the vertical axis. Otherwise the polynomial must fit too many almost zeros and then jump, and you will get weird and negative results. There was also a strong weekly variation, cases on weekends were written down on Monday (or Tuesday), so I think it's better to look at the 7-day running average. The problem is that in this paper they are extending the "forecast" too much. It's somewhat the equivalent of 6 months. I thin no serious source tried to make very long term forecasts. (Anyway, some trends and general forecasts are possible, like with the flu.)
apwheele 4y ago
I will have to give these formulations a try. IMO pretty common to have asymptote like theoretical behavior in the tails with social science data, https://andrewpwheeler.com/2022/05/02/fitting-a-plateau-effects-model-in-scipy/ https://andrewpwheeler.com/2022/05/02/fitting-a-plateau-effe.... (Formulation is exactly quadratic for lower values and flattens at higher, and can re-parameterize to make it easier to fit.)

monero-xmr 4y ago

I would take it even farther and claim that very little in experimental psychology / sociology has any value whatsoever. Google the top papers of all time in psychology and its either been formally debunked (Stanford prison experiment) or it has no applicable impact on society. Using computers to parse statistical data is interesting and can reveal truths about society. Trying to correlate the delay between eating a marshmellow as a child and income as an adult is ridiculous, yet collectively we have spent untold money and time on tenured professors, conferences, papers, and pop psychology books about such things. Another example is Power Posing. How Amy Cuddy still gets employment is a mystery.

spekcular 4y ago
The marshmallow effect replicated (IIRC), and is probably correct, unlike the power posing stuff. That behavior in a single trial in childhood moderately predicts success in adulthood seems like an interesting fact about humans.
- monero-xmr 4y ago
  What's the applicable result to this? Put children who eat marshmellows too quickly in special "delayed gratification" classes? Is the result such that all children who eat marshmellows too quickly should give up, as their success is predetermined, or is it the impact is barely statistically significant? If I were the parent of a child who ate the marshmellow too quickly, I would do nothing different, as the whole thing is meaningless and doesn't change anything. It's more like, "Oh, OK, my son is 4% less likely to be in the top 20% of the income distribution (according to this barely coherent theory)." Why do we spend so much money, time, and effort discussing and researching this?
  - hgsgm 4y ago
    What? If you found your child was lacking in a valuable life skill, you wouldn't try to teach that skill?
    - 4y ago

Galanwe 4y ago

I'm not sure the methodology as a whole makes any sense, even with the prediction overfit aside. The Y axis is the mean score of the team, but that score is not independent from the opposing team. It means that it does not refute the hypothesis that sparse top talents in a team happen at low level games, where the top talents make a huge difference. When reaching higher level games by acquiring new top talents, the playing field is more leveled, and the scores will even out. To remove that effect, you would need to select games where a team score does not depend on the opposing team score. Or add a new dimension to the regression which is the opposing teams top talent %. TLDR: if you put Michael Jordan in an amateur team, he will overperform and crank up the team score. If you then add 5 other pro players to the team, you'll enter the NBA. Your team will perform better, even though your number of points may decrease, as other teams are closer to your elite teams level.

dvt 4y ago
> The Y axis is the mean score of the team, but that score is not independent from the opposing team. The paper is garbage, but this is not true. All Y-axes are how they define "team performance" -- in the case of the FIFA study, it's qualification rankings. In the case of the NBA/MLB study it's end-of-season team win percentages.
- Galanwe 4y ago
  Indeed, my bad. The argument still holds though. Overall team disparity should be much higher at low level than high level. A team composed of "Michael Jordan + random players" would crush any amateur league with 100% win rate. The same team with "Michael Jordan + Magic Johnson + random players" would probably get 75% win rate in an NBA match. So the metric "win rate" alone, or any other metric non independent of the opposing teams level, is naturally useless in this study.

evolve2k 4y ago

Anyone know of any browser extensions / projects that highlight when an article or PDF had been debunked / retracted? Seems like without this meta data over the original article that were destined to keep consuming even retracted knowledge.

invig 4y ago
Is there a standard or something for how this kind of meta data could be kept on articles / papers?
- kuhewa 4y ago
  The database with the most coverage to my knowledge is Crossref but I'm not sure if a retraction would be associated with the original doi.
imadethis 4y ago
retractionwatch.com and .org monitor retracted papers, and they publish plugins for Zotero and similar tools.
kuhewa 4y ago
PubPeer

mannerheim 4y ago

The ever-relevant Richard Feynman: 'I think the educational and psychological studies I mentioned are examples of what I would like to call Cargo Cult Science. In the South Seas there is a Cargo Cult of people. During the war they saw airplanes land with lots of good materials, and they want the same thing to happen now. So they’ve arranged to make things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head like headphones and bars of bamboo sticking out like antennas—he’s the controller—and they wait for the airplanes to land. They’re doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn’t work. No airplanes land. So I call these things Cargo Cult Science, because they follow all the apparent precepts and forms of scientific investigation, but they’re missing something essential, because the planes don’t land.' https://calteches.library.caltech.edu/51/2/CargoCult.htm https://calteches.library.caltech.edu/51/2/CargoCult.htm

deleted 4y ago
[deleted]
rahimnathwani 4y ago
I like the bit later in the essay where he describes what's often missing: "It’s a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty—a kind of leaning over backwards. For example, if you’re doing an experiment, you should report everything that you think might make it invalid—not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you’ve eliminated by some other experiment, and how they worked—to make sure the other fellow can tell they have been eliminated."
- bongoman37 4y ago
  [dead]