7 ms·

About those baby brainwaves

58 points by _dain_ 5y ago

mc32 5y ago
I’m not surprised. What’s surprising is the impact on policy low quality have. Government policies Kees and the news, like KQED and others routinely refer to these kinds of studies when they espouse something and politicians and some of the twitterverse et all latch on to these things have very little validity if any.
BryantD 5y ago
Good critique of that specific article. I would have found it more useful if, instead of saying "and most other studies share these issues," the author had presented guidelines to detecting studies that share those issues. There's something ironic about flagging issues with small sample sizes while extrapolating from a single study to "mostly a fraud."
pphysch 5y ago
I would assume by default that any "policy-relevant" science is poisoned by some CoI, unless proven otherwise, with few exceptions.
- karpierz 5y ago
  All science is poisoned by conflicts of interest, as the authors are always incentivized to have people believe the results of the paper, regardless of the validity of it. Whether that be because they want to get tenure, enact change in the world, build prestige for their company to improve hiring/stock prices, or get a book deal.
  - pphysch 5y ago
    You're omitting two of the main historical motivations for science: 1) Solving concrete problems (desalination, energy storage, flying like a bird, getting to the moon, etc) 2) Discovery for its own sake, sense of power over nature Both of these can be pursued safely without a toxic CoI.
    - karpierz 5y ago
      Yes, if the people researching it aren't subject to pressures that humans are normally subject to. If have any desires beyond 1/2, then there's a conflict of interest. For example, if you need money to eat. Or, if you want to solve a concrete problem/discover things, and believe that if you got funding you could do so. Or if getting tenure would mean you get to solve more concrete problems, and fudging one paper would let you do that. It's not possible to avoid conflicts of interest categorically. It is possible to manage the impact of them. For example, the peer review process makes it harder (but not impossible) for a lone researcher to publish inaccurate findings. The question isn't whether there's a conflict of interest in the research, the question is whether the system publishes solid research despite the conflicts of interest.
skybrian 5y ago
The headline is very general, but the article is mostly a thorough takedown of one bad study and how it was reported in the media. (I expect it's part of a larger pattern, but the article doesn't really show this.)
- xyzzyz 5y ago
  This is very much a part of a larger pattern. See, for example, Andrew Gelman's blog: https://statmodeling.stat.columbia.edu/ https://statmodeling.stat.columbia.edu/. He is as close to an expert in bad studies as there exists. Just scroll down, there will be many examples every week. Here's for example a second most recent post: The journal Public Choice goes the way of Lancet and publishes a paper that is “riddled with errors” but comes to a political conclusion that they want to support. https://statmodeling.stat.columbia.edu/2022/04/03/the-journal-public-choice-goes-the-way-of-lancet-and-publishes-a-paper-that-is-riddled-with-errors-but-comes-to-a-political-conclusion-that-they-want-to-support/ https://statmodeling.stat.columbia.edu/2022/04/03/the-journa... Replication crisis in social science is a very real thing, and it's not a problem with a few odd studies: at this point, at least double digit percentage, if not outright majority of studies in social sciences, are simply untrue. Here's, for example, a quiz https://80000hours.org/psychology-replication-quiz/ https://80000hours.org/psychology-replication-quiz/, based on a study published in Nature ( https://www.nature.com/collections/nfkchhxllx https://www.nature.com/collections/nfkchhxllx ), which tried to replicate some 21 experimental studies published in Nature or Science in a few previous years. These were not some crappy papers published in pay-to-publish journals: Nature and Science are the top journals, as high quality as you can get. Still, only barely over a half of the studies replicated! In less renowned journals, the ratio must be even worse.
isx726552 5y ago
This paper appears to be criticizing just one specific study and there’s no support for the broader conclusion in the headline that such studies are “mostly” fraudulent. Not to mention this paper notes in its conclusion that the study in question was already roundly criticized at the time (see the paragraph beginning with “[t]he public arena is at least somewhat self-correcting”), so it’s not clear what is being added to the discourse, other than unsupported generalizations.
- brimble 5y ago
  I've not heard of this place before, but it seems like a budding or wannabe "think tank". Getting things wrong (drawing too broad a conclusion, using misleading framing) even when you're right (this particular paper under scrutiny is bad) is kinda the whole thing such institutions do.
  - m-watson 5y ago
    They were formed in 2020 [https://www.influencewatch.org/non-profit/center-for-study-of-partisanship-and-ideology-cspi/ https://www.influencewatch.org/non-profit/center-for-study-o...] and seem to have a few reports doing the same thing. They start with an actual specific analysis and then expand dramatically.
- dang 5y ago
  Ok, we'll use the more specific part of the title above. Submitted title was '“Policy-Relevant” Social Science Is Mostly a Fraud', which wasn't so bad - it's indeed what the article title says - but maybe it's misleading on the article's own part.
deleted 5y ago
[deleted]
temp8964 5y ago
Is this clickbait? There are lots of efforts put in the improvement of quantitative methods in social science studies. There are good studies out there. There are also trash out there. Even policy relevant social science research can produce good studies in highly contested fields, because of the mechanisms of peer review and open debate.
- iratewizard 5y ago
  Social sciences have the highest number of people who self-identified as Marxist back in 06. The rate was 17.6 as Marxist, 24.0 as radicals [1]. They've also been the worst field for the three decades of reproducibility crisis. You would have trouble convincing me that they aren't a fake, compromised "science" that ignores the scientific method to produce whitepaper propaganda. 1 - http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.147.6141&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.147...
  - temp8964 5y ago
    I know there are lots of Marxist/radical nuts in "social sciences", but social sciences as a broad concept includes too many fields. If the survey limits to the target we are talking about here (quantitative researchers with heavy training in statistics but never took a course in critical theory etc.), the number of these kind of nuts would be close to zero. The reproducibility crisis is more related to social psychologists, which include lots of nuts. They are not what I was talking about.
  - LordDragonfang 5y ago
    While it's definitely well-documented that social sciences are among the hardest hit by the replication crises, putting that point second after the political beliefs of those in the field is a weird axe to grind.
    - iratewizard 5y ago
      It specifically illustrates my last point. The field is filled with ideological zealots who put our bogus propaganda to further their ideology. More in common with a cult than a science.
cscurmudgeon 5y ago
> Despite its inconclusiveness, the study was portrayed as relevant to child tax credit policy, vigorously promoted in media outlets like Vox and The New York Times, and championed by thinktanks and policymakers. Here is the real tragedy: None of those outlets will report on this analysis and their proponents will beat us over the heads with "Trust the Science!"
- iratewizard 5y ago
  What's a good solution? A steep fine everytime they fail to vet a source properly? A yearly report that has to be displayed prominently on their website and lists false claims, changes and settled lawsuits against paper? The one I've found that works well is ignoring the deceitful liars.
  - unmole 5y ago
    > The one I've found that works well is ignoring the deceitful liars. That doesn't work if a large number of your fellow citizens believe them.
TameAntelope 5y ago
What does it mean for CSPI if the thing they primarily study is, "mostly a fraud"?
- slibhb 5y ago
  They aren't saying it has to be this way, they're just saying it is this way. I think we can conclude that they mean "besides us". Personally, when someone makes reference to "policy relevant social science," I always ask myself: if the study concluded the opposite, would this person's opinion be different? The answer I come to is almost always no.
- notahacker 5y ago
  It means they can rant about all the other academics, whilst cherry picking their own conclusions in the opposite direction. This[1] is an example of it which borders on Swiftian satire... [1] https://cspicenter.org/reports/the-accuracy-of-stereotypes-data-and-implications/ https://cspicenter.org/reports/the-accuracy-of-stereotypes-d...
- cato_the_elder 5y ago
  If it's true, it's means that they probably can do much better than others by simply not being a fraud?
ZeroGravitas 5y ago
The author feels like he's on the opposite political side to me, and so I'm ironically sceptical of his own 'policy relevant' opinions. Having said that, using a study done in 2022 (which we're only just out of the first quarter of) to justify basically anything, seems kind of ridiculous, and I'd like to live in a world where people just laughed that off in the same way they would if someone said an astrologer predicted their policy would be successful or if Joe Rogan was offering medical advice. And that's before we get to the brainwave part, which seems to push this into the ridiculous. If we really have no better research to suggest how to improve society in general or child welfare in particular, then that alone seems like a sad state of affairs. We have a whole globe full of children, more than 2 Billion of them in various different locations and societies, have we really not got any way to identify what is helpful. Are we claiming nothing at all that happens to any of those 2 Billion children makes any difference? That seems very unlikely. Or that we already do everything we know to do? This feels only marginally more plausible.
- unrealp 5y ago
  "Are we claiming nothing at all that happens to any of those 2 Billion children makes any difference?" >> B12 supplementation would certainly help some and is quite cheap on a mass scale. I think it boils down to what people want to read from these kind of studies.
- Georgelemental 5y ago
  > "using a study done in 2022 (which we're only just out of the first quarter of) to justify basically anything, seems kind of ridiculous" In this case, and most others, yours is the correct reaction. But sometimes you don't have a choice but to be hasty, as with rapidly-developing situations (e.g. COVID).
- rayiner 5y ago
  The last bullet point is something folks on both sides of the aisle should appreciate: > Thus, we ought to be skeptical of social scientists’ ability to reliably inform public policy, and of policymakers’ ability to evaluate social-scientific research objectively, particularly when it is “policy relevant.” Think of how much we’re debating what’s causing the current inflation we’re seeing. That’s one number in a field that’s quite amenable to quantitative analysis. I think we have pretty much zero credible scientific evidence to justify most policies involving public welfare. There’s a handful of exceptions—e.g. the tremendous work done studying the benefits of the clean air act, for example. But that was rigorous research done by the EPA over literally decades. I think we should give poor families with kids money because that seems like the morally correct thing to do. But this sort of thing is just scientism: https://calteches.library.caltech.edu/51/2/CargoCult.htm https://calteches.library.caltech.edu/51/2/CargoCult.htm
michaelmrose 5y ago
This org is as full of shit as the study is. Number two on their areas of interest >---------------------------------------- 2. The Great Awokening In the last decade, white liberals in America have shifted far to the left on issues of race, gender, and sexuality. Ideas once seen as radical, like defunding the police and gender being on a spectrum, are embraced by more people than ever before. Social scientists know little about the social and psychological factors underlying this “Great Awokening.” Is wokeness a status competition among elites? An extension of WEIRD (Western, educated, industrial, rich, democratic) morality? Or something new altogether? CSPI is interested in funding scholars studying woke attitudes, beliefs, and behaviors. We are also interested in work examining the impact of woke policies and practices on organizational culture and performance. >------------------------------------------------------------- It seems trivially true that supporting parents especially at the low end of the spectrum provides them with less stress, less home insecurity, less food insecurity, more likelihood that one or both parents can spend less time at work and ergo more time with the child. If the actual question is does giving money to parents improve childhood trajectories the answer appears to be much more likely to be yes. Attacking a study that tries to go about proving this in a probably wrongheaded and silly way appears to be attacking sidelong a strong opponent via a weak front even if the stated aim is to pick on bad studies that indeed that studies peers already debunked as if it was representative of the entire field. Here is a fun exercise do a search for site:https://cspicenter.org https://cspicenter.org covid some fun results https://cspicenter.org/blog/waronscience/why-covid-19-is-here-to-stay-and-why-you-shouldnt-worry-about-it/ https://cspicenter.org/blog/waronscience/why-covid-19-is-her... Almost a year ago they said SARS-CoV-2 is going to become mostly harmless Then they used Sweden to prove that lock downs were pointless ignoring how much worse they did per capita than Norway and Finland. It's like fox news went to school and got an education.
- deleted 5y ago
  [deleted]

58 points by _dain_ 5y ago

mc32 5y ago

I’m not surprised. What’s surprising is the impact on policy low quality have. Government policies Kees and the news, like KQED and others routinely refer to these kinds of studies when they espouse something and politicians and some of the twitterverse et all latch on to these things have very little validity if any.

BryantD 5y ago

Good critique of that specific article. I would have found it more useful if, instead of saying "and most other studies share these issues," the author had presented guidelines to detecting studies that share those issues. There's something ironic about flagging issues with small sample sizes while extrapolating from a single study to "mostly a fraud."

pphysch 5y ago

I would assume by default that any "policy-relevant" science is poisoned by some CoI, unless proven otherwise, with few exceptions.

karpierz 5y ago
All science is poisoned by conflicts of interest, as the authors are always incentivized to have people believe the results of the paper, regardless of the validity of it. Whether that be because they want to get tenure, enact change in the world, build prestige for their company to improve hiring/stock prices, or get a book deal.
- pphysch 5y ago
  You're omitting two of the main historical motivations for science: 1) Solving concrete problems (desalination, energy storage, flying like a bird, getting to the moon, etc) 2) Discovery for its own sake, sense of power over nature Both of these can be pursued safely without a toxic CoI.
  - karpierz 5y ago
    Yes, if the people researching it aren't subject to pressures that humans are normally subject to. If have any desires beyond 1/2, then there's a conflict of interest. For example, if you need money to eat. Or, if you want to solve a concrete problem/discover things, and believe that if you got funding you could do so. Or if getting tenure would mean you get to solve more concrete problems, and fudging one paper would let you do that. It's not possible to avoid conflicts of interest categorically. It is possible to manage the impact of them. For example, the peer review process makes it harder (but not impossible) for a lone researcher to publish inaccurate findings. The question isn't whether there's a conflict of interest in the research, the question is whether the system publishes solid research despite the conflicts of interest.

skybrian 5y ago

The headline is very general, but the article is mostly a thorough takedown of one bad study and how it was reported in the media. (I expect it's part of a larger pattern, but the article doesn't really show this.)

xyzzyz 5y ago
This is very much a part of a larger pattern. See, for example, Andrew Gelman's blog: https://statmodeling.stat.columbia.edu/ https://statmodeling.stat.columbia.edu/. He is as close to an expert in bad studies as there exists. Just scroll down, there will be many examples every week. Here's for example a second most recent post: The journal Public Choice goes the way of Lancet and publishes a paper that is “riddled with errors” but comes to a political conclusion that they want to support. https://statmodeling.stat.columbia.edu/2022/04/03/the-journal-public-choice-goes-the-way-of-lancet-and-publishes-a-paper-that-is-riddled-with-errors-but-comes-to-a-political-conclusion-that-they-want-to-support/ https://statmodeling.stat.columbia.edu/2022/04/03/the-journa... Replication crisis in social science is a very real thing, and it's not a problem with a few odd studies: at this point, at least double digit percentage, if not outright majority of studies in social sciences, are simply untrue. Here's, for example, a quiz https://80000hours.org/psychology-replication-quiz/ https://80000hours.org/psychology-replication-quiz/, based on a study published in Nature ( https://www.nature.com/collections/nfkchhxllx https://www.nature.com/collections/nfkchhxllx ), which tried to replicate some 21 experimental studies published in Nature or Science in a few previous years. These were not some crappy papers published in pay-to-publish journals: Nature and Science are the top journals, as high quality as you can get. Still, only barely over a half of the studies replicated! In less renowned journals, the ratio must be even worse.

isx726552 5y ago

This paper appears to be criticizing just one specific study and there’s no support for the broader conclusion in the headline that such studies are “mostly” fraudulent. Not to mention this paper notes in its conclusion that the study in question was already roundly criticized at the time (see the paragraph beginning with “[t]he public arena is at least somewhat self-correcting”), so it’s not clear what is being added to the discourse, other than unsupported generalizations.

brimble 5y ago
I've not heard of this place before, but it seems like a budding or wannabe "think tank". Getting things wrong (drawing too broad a conclusion, using misleading framing) even when you're right (this particular paper under scrutiny is bad) is kinda the whole thing such institutions do.
- m-watson 5y ago
  They were formed in 2020 [https://www.influencewatch.org/non-profit/center-for-study-of-partisanship-and-ideology-cspi/ https://www.influencewatch.org/non-profit/center-for-study-o...] and seem to have a few reports doing the same thing. They start with an actual specific analysis and then expand dramatically.
dang 5y ago
Ok, we'll use the more specific part of the title above. Submitted title was '“Policy-Relevant” Social Science Is Mostly a Fraud', which wasn't so bad - it's indeed what the article title says - but maybe it's misleading on the article's own part.

deleted 5y ago

[deleted]

temp8964 5y ago

Is this clickbait? There are lots of efforts put in the improvement of quantitative methods in social science studies. There are good studies out there. There are also trash out there. Even policy relevant social science research can produce good studies in highly contested fields, because of the mechanisms of peer review and open debate.

iratewizard 5y ago
Social sciences have the highest number of people who self-identified as Marxist back in 06. The rate was 17.6 as Marxist, 24.0 as radicals [1]. They've also been the worst field for the three decades of reproducibility crisis. You would have trouble convincing me that they aren't a fake, compromised "science" that ignores the scientific method to produce whitepaper propaganda. 1 - http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.147.6141&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.147...
- temp8964 5y ago
  I know there are lots of Marxist/radical nuts in "social sciences", but social sciences as a broad concept includes too many fields. If the survey limits to the target we are talking about here (quantitative researchers with heavy training in statistics but never took a course in critical theory etc.), the number of these kind of nuts would be close to zero. The reproducibility crisis is more related to social psychologists, which include lots of nuts. They are not what I was talking about.
- LordDragonfang 5y ago
  While it's definitely well-documented that social sciences are among the hardest hit by the replication crises, putting that point second after the political beliefs of those in the field is a weird axe to grind.
  - iratewizard 5y ago
    It specifically illustrates my last point. The field is filled with ideological zealots who put our bogus propaganda to further their ideology. More in common with a cult than a science.

cscurmudgeon 5y ago

> Despite its inconclusiveness, the study was portrayed as relevant to child tax credit policy, vigorously promoted in media outlets like Vox and The New York Times, and championed by thinktanks and policymakers. Here is the real tragedy: None of those outlets will report on this analysis and their proponents will beat us over the heads with "Trust the Science!"

iratewizard 5y ago
What's a good solution? A steep fine everytime they fail to vet a source properly? A yearly report that has to be displayed prominently on their website and lists false claims, changes and settled lawsuits against paper? The one I've found that works well is ignoring the deceitful liars.
- unmole 5y ago
  > The one I've found that works well is ignoring the deceitful liars. That doesn't work if a large number of your fellow citizens believe them.

TameAntelope 5y ago

What does it mean for CSPI if the thing they primarily study is, "mostly a fraud"?

slibhb 5y ago
They aren't saying it has to be this way, they're just saying it is this way. I think we can conclude that they mean "besides us". Personally, when someone makes reference to "policy relevant social science," I always ask myself: if the study concluded the opposite, would this person's opinion be different? The answer I come to is almost always no.
notahacker 5y ago
It means they can rant about all the other academics, whilst cherry picking their own conclusions in the opposite direction. This[1] is an example of it which borders on Swiftian satire... [1] https://cspicenter.org/reports/the-accuracy-of-stereotypes-data-and-implications/ https://cspicenter.org/reports/the-accuracy-of-stereotypes-d...
cato_the_elder 5y ago
If it's true, it's means that they probably can do much better than others by simply not being a fraud?

ZeroGravitas 5y ago

The author feels like he's on the opposite political side to me, and so I'm ironically sceptical of his own 'policy relevant' opinions. Having said that, using a study done in 2022 (which we're only just out of the first quarter of) to justify basically anything, seems kind of ridiculous, and I'd like to live in a world where people just laughed that off in the same way they would if someone said an astrologer predicted their policy would be successful or if Joe Rogan was offering medical advice. And that's before we get to the brainwave part, which seems to push this into the ridiculous. If we really have no better research to suggest how to improve society in general or child welfare in particular, then that alone seems like a sad state of affairs. We have a whole globe full of children, more than 2 Billion of them in various different locations and societies, have we really not got any way to identify what is helpful. Are we claiming nothing at all that happens to any of those 2 Billion children makes any difference? That seems very unlikely. Or that we already do everything we know to do? This feels only marginally more plausible.

unrealp 5y ago
"Are we claiming nothing at all that happens to any of those 2 Billion children makes any difference?" >> B12 supplementation would certainly help some and is quite cheap on a mass scale. I think it boils down to what people want to read from these kind of studies.
Georgelemental 5y ago
> "using a study done in 2022 (which we're only just out of the first quarter of) to justify basically anything, seems kind of ridiculous" In this case, and most others, yours is the correct reaction. But sometimes you don't have a choice but to be hasty, as with rapidly-developing situations (e.g. COVID).
rayiner 5y ago
The last bullet point is something folks on both sides of the aisle should appreciate: > Thus, we ought to be skeptical of social scientists’ ability to reliably inform public policy, and of policymakers’ ability to evaluate social-scientific research objectively, particularly when it is “policy relevant.” Think of how much we’re debating what’s causing the current inflation we’re seeing. That’s one number in a field that’s quite amenable to quantitative analysis. I think we have pretty much zero credible scientific evidence to justify most policies involving public welfare. There’s a handful of exceptions—e.g. the tremendous work done studying the benefits of the clean air act, for example. But that was rigorous research done by the EPA over literally decades. I think we should give poor families with kids money because that seems like the morally correct thing to do. But this sort of thing is just scientism: https://calteches.library.caltech.edu/51/2/CargoCult.htm https://calteches.library.caltech.edu/51/2/CargoCult.htm

michaelmrose 5y ago

This org is as full of shit as the study is. Number two on their areas of interest >---------------------------------------- 2. The Great Awokening In the last decade, white liberals in America have shifted far to the left on issues of race, gender, and sexuality. Ideas once seen as radical, like defunding the police and gender being on a spectrum, are embraced by more people than ever before. Social scientists know little about the social and psychological factors underlying this “Great Awokening.” Is wokeness a status competition among elites? An extension of WEIRD (Western, educated, industrial, rich, democratic) morality? Or something new altogether? CSPI is interested in funding scholars studying woke attitudes, beliefs, and behaviors. We are also interested in work examining the impact of woke policies and practices on organizational culture and performance. >------------------------------------------------------------- It seems trivially true that supporting parents especially at the low end of the spectrum provides them with less stress, less home insecurity, less food insecurity, more likelihood that one or both parents can spend less time at work and ergo more time with the child. If the actual question is does giving money to parents improve childhood trajectories the answer appears to be much more likely to be yes. Attacking a study that tries to go about proving this in a probably wrongheaded and silly way appears to be attacking sidelong a strong opponent via a weak front even if the stated aim is to pick on bad studies that indeed that studies peers already debunked as if it was representative of the entire field. Here is a fun exercise do a search for site:https://cspicenter.org https://cspicenter.org covid some fun results https://cspicenter.org/blog/waronscience/why-covid-19-is-here-to-stay-and-why-you-shouldnt-worry-about-it/ https://cspicenter.org/blog/waronscience/why-covid-19-is-her... Almost a year ago they said SARS-CoV-2 is going to become mostly harmless Then they used Sweden to prove that lock downs were pointless ignoring how much worse they did per capita than Norway and Finland. It's like fox news went to school and got an education.

deleted 5y ago
[deleted]