7 ms·

Great article. This part especially rings true to me: "If researchers have an ideological bent, a meta-analytic null may just be an expression of the typical se

by poomer 3y ago

Great article. This part especially rings true to me: "If researchers have an ideological bent, a meta-analytic null may just be an expression of the typical sentiments of researchers". When I was in academia, it was increasingly the case that my peers thought of research less as a way to determine the truth, but just as a method to influence policy and public opinion. If we thought something was 80% likely to be true, there was pressure to "close ranks" and pretend as though it was 100% true, and to avoid publishing anything that contradicted it. It is also well known that papers that support certain "sides" tend to be easier to publish (and in higher ranked journals), plus can yield more media attention. See for example, this fraud in sociology - https://en.wikipedia.org/wiki/When_contact_changes_minds https://en.wikipedia.org/wiki/When_contact_changes_minds. This may be better in the natural sciences, but in social science you should not trust any paper unless you read through and fully understand the methodology. Any non experimental results has so much wiggle room in the modeling methodology that it's easy to generate any result you want. The actual percentage of papers with credible results is very low, much lower than laypeople think.

getoffmycase 3y ago
It is not better in natural sciences. Alzheimer’s disease treatment based on the amyloid plaque hypothesis being the prime example. Basically researchers closed ranks and doled out grant money disproportionately to the amyloid hypothesis researchers.
- nathan_compton 3y ago
  This is sort of like adding momentum to a gradient descent algorithm, though. It is rational congeal support around a plausible hypothesis to see if it pans out rather than pursue all hypotheses (whatever that would even mean) in a desultory fashion. In the case of amyloid plaques there may have been some academic misconduct involved as well. Sometimes I feel like non-scientists have this idea that scientists should be super-rational actors and never be bamboozled or be wrong, but frankly, that isn't realistic. Scientists are fallible, have finite time, and are subject to social trends and pressures like the rest of us. Expecting that they always get it right is unrealistic and not good for science.
  - getoffmycase 3y ago
    Scientists should not shut out other ideas for two decades even though there were serious challenges to the hypothesis the whole time. Part of being a good scientist is admitting your chosen hypothesis isn’t correct and that it’s time to create a new theory based on the experimental evidence available. Otherwise, it’s just a research based cult.
    - nathan_compton 3y ago
      This almost never happens, to my knowledge. Most fields have people pursuing alternative hypotheses even when one thing is in vogue. For example, in the case of Alzheimer's Disease there were and are a lot of other threads of research and I would be that your typical AD scientist would have had a good grasp of many of them regardless of the research they were focusing on. https://biology.stackexchange.com/questions/3223/what-alternatives-are-there-to-the-amyloid-hypothesis https://biology.stackexchange.com/questions/3223/what-altern...
  - naasking 3y ago
    > It is rational congeal support around a plausible hypothesis to see if it pans out rather than pursue all hypotheses Is it though?
    - nathan_compton 3y ago
      Yes. At least there is a plausible argument for it in many contexts. Imagine a suite of ten hypothesis, one of which is correct but each of which requires 6 effort units to prove or disprove or discover or whatever. If we only have a 10 units of effort to devote to research, we won't get anywhere if we divide them equally but we can eliminate each hypothesis if we work serially. In reality I would expect the effect to be even more pronounced because progress on a hypothesis is almost certainly non-linearly related to the number of people working on it (with some point of diminishing return, of course). It is totally reasonable for a community to more or less work on proving or eliminating the few most reasonable hypotheses at a time rather than to spread themselves over the (frankly enormous) space of hypotheses relevant to a particular area. I'd hardly argue that our system of allocating scientific research effort is perfect, but how could it be?
piuantiderp 3y ago
People relying on a salary are rarely free to pursuit the truth
brightball 3y ago
Once, years ago, I got into a debate with somebody online who just kept dropping links to studies after making a statement as if it proved them correct. One day, I got bored and decided to actually read every single study. The studies said nothing close to what the debater suggested and seriously opened my eyes to just how terrible some studies can be in terms of quality. Now, I just assume that anybody who starts link bombing in a conversation has no idea what they're talking about and can't engage on a logical discussion.
- hospitalJail 3y ago
  If you ever want to reject a study, just read it. You will find something to nit pick.
  - ben_w 3y ago
    I'm not saying you're wrong — I've got my name on exactly one paper so what would I know — but that being true would suggest that peer review is fundamentally broken.
    - jvanderbot 3y ago
      No, it would suggest that every paper has something in it that can be considered a fatal flaw by a person biased against it. Peer review is precisely supposed to admit this and minimize flaws in conclusions given inevitable flaws in methodology. Sort of a research paper equivalent of: "If you give me six lines written by the hand of the most honest of men, I will find something in them which will hang him." by Cardinal Richelieu
    - matusp 3y ago
      Peer review is not a certificate of truthfulness, peer review just says that a grad student reading the paper did not found anything too suspicious there and it looks like a typical paper in that field.
      - some_random 3y ago
deleted 3y ago
[deleted]
jfengel 3y ago
Sociology lives in an uncomfortable place at the intersection of "important" and "difficult". Forming a rigid experiment on human beings is vastly more difficult than a simple thing like an atom or a mineral, and so much worse with groups of human beings. STEM people like to dismiss it because it doesn't produce the same kind of rigid results and is therefore useless. But sociology is nonetheless important. Like it or not, we have to make decisions about how the world will run, and the fact that we don't have perfect information doesn't let us opt out of that. Combine that with all of the usual human failings -- pettiness, meanness, closed-mindedness, greed, etc -- it sounds impossible. But it does, slowly, gather data and formulate theories. This isn't made better by the fact that most STEM people imagine they can read primary source material and understand it -- a mistake they wouldn't make for a "hard science" in a different discipline. Like every field, the frontiers are based on huge amounts of background material, which is even more vast for a field that's more complicated than the nice, neat laws of physics or chemistry. None of that excuses those human failings of sociologists. They need to do better, and hold each other to account (and not just to foment their own failings in their place). But we do need to recognize that this work is important. The world is complex and difficult and we'll make better choices if we try to understand, rather than dismiss it as unknowable.
- DontchaKnowit 3y ago
  While I agree that there is some work in the field that is important, there is an absolute deluge of ideologically driven garbage. There is also a lot of garbage of the "yeah, duh, of-fucking-course" variety. studies like "negative interactions with community reduce feelings of belonging" ... uh yeah, no shit sherlock. And then even the more important stuff cannot be investigated in the same rigorous manner as other scientific disciplines. I think we just need to stop calling sociology papers "scientific". Fundamentally, they are not, and should almost always be taken with a massive grain of salt, and they damn sure shouldn't be influencing public policy decisions to the degree that they currently do.
  - arp242 3y ago
    > There is also a lot of garbage of the "yeah, duh, of-fucking-course" variety. studies like "negative interactions with community reduce feelings of belonging" ... uh yeah, no shit sherlock. I think these studies are useful, first of all to test whether it's actually true because sometimes "well du'h" turns out to be wrong, but also to quantify the exact effects. Is it a large effect or a small effect? How large exactly? Which factors exactly contribute to this effect? What exactly is the breakdown of the effects? It might be possible that 20% of the people are effected by it and 80% of people are not; or perhaps everyone is effected by it. There's often all sorts of non-obvious nuance that's possible, which can be very significant.
pessimizer 3y ago
[flagged]
- anonymouskimmer 3y ago
  I'm not sure if that particular study is a good example of this as everything and its aunt was getting published about COVID back then. It is disappointing that other studies were shut down on the assumption that the mortality results from such a limited (and fatally flawed) analysis were correct, especially given previous knowledge of the side effects and dosing considerations of hydroxychloroquine.
rqtwteye 3y ago
“ thought of research less as a way to determine the truth, but just as a method to influence policy and public opinion. If we thought something was 80% likely to be true, there was pressure to "close ranks" and pretend as though it was 100% true, and to avoid publishing anything that contradicted it. ” That seems to be the general spirit of this time. People feel the need to take sides and then make sure their side wins. Same happens in journalism. Most journalism these days seems to be about supporting a viewpoint and less about conveying neutral or complete information. Sad that scientists also feel that they should be activists.
whyenot 3y ago
If you are a physical anthropologist there are many questions you can not ask, and many findings that people do not want to hear. Tread carefully, because one wrong step can end your career.

by poomer 3y ago

getoffmycase 3y ago

It is not better in natural sciences. Alzheimer’s disease treatment based on the amyloid plaque hypothesis being the prime example. Basically researchers closed ranks and doled out grant money disproportionately to the amyloid hypothesis researchers.

nathan_compton 3y ago
This is sort of like adding momentum to a gradient descent algorithm, though. It is rational congeal support around a plausible hypothesis to see if it pans out rather than pursue all hypotheses (whatever that would even mean) in a desultory fashion. In the case of amyloid plaques there may have been some academic misconduct involved as well. Sometimes I feel like non-scientists have this idea that scientists should be super-rational actors and never be bamboozled or be wrong, but frankly, that isn't realistic. Scientists are fallible, have finite time, and are subject to social trends and pressures like the rest of us. Expecting that they always get it right is unrealistic and not good for science.
- getoffmycase 3y ago
  Scientists should not shut out other ideas for two decades even though there were serious challenges to the hypothesis the whole time. Part of being a good scientist is admitting your chosen hypothesis isn’t correct and that it’s time to create a new theory based on the experimental evidence available. Otherwise, it’s just a research based cult.
  - nathan_compton 3y ago
    This almost never happens, to my knowledge. Most fields have people pursuing alternative hypotheses even when one thing is in vogue. For example, in the case of Alzheimer's Disease there were and are a lot of other threads of research and I would be that your typical AD scientist would have had a good grasp of many of them regardless of the research they were focusing on. https://biology.stackexchange.com/questions/3223/what-alternatives-are-there-to-the-amyloid-hypothesis https://biology.stackexchange.com/questions/3223/what-altern...
- naasking 3y ago
  > It is rational congeal support around a plausible hypothesis to see if it pans out rather than pursue all hypotheses Is it though?
  - nathan_compton 3y ago
    Yes. At least there is a plausible argument for it in many contexts. Imagine a suite of ten hypothesis, one of which is correct but each of which requires 6 effort units to prove or disprove or discover or whatever. If we only have a 10 units of effort to devote to research, we won't get anywhere if we divide them equally but we can eliminate each hypothesis if we work serially. In reality I would expect the effect to be even more pronounced because progress on a hypothesis is almost certainly non-linearly related to the number of people working on it (with some point of diminishing return, of course). It is totally reasonable for a community to more or less work on proving or eliminating the few most reasonable hypotheses at a time rather than to spread themselves over the (frankly enormous) space of hypotheses relevant to a particular area. I'd hardly argue that our system of allocating scientific research effort is perfect, but how could it be?

piuantiderp 3y ago

People relying on a salary are rarely free to pursuit the truth

brightball 3y ago

Once, years ago, I got into a debate with somebody online who just kept dropping links to studies after making a statement as if it proved them correct. One day, I got bored and decided to actually read every single study. The studies said nothing close to what the debater suggested and seriously opened my eyes to just how terrible some studies can be in terms of quality. Now, I just assume that anybody who starts link bombing in a conversation has no idea what they're talking about and can't engage on a logical discussion.

hospitalJail 3y ago
If you ever want to reject a study, just read it. You will find something to nit pick.
- ben_w 3y ago
  I'm not saying you're wrong — I've got my name on exactly one paper so what would I know — but that being true would suggest that peer review is fundamentally broken.
  - jvanderbot 3y ago
    No, it would suggest that every paper has something in it that can be considered a fatal flaw by a person biased against it. Peer review is precisely supposed to admit this and minimize flaws in conclusions given inevitable flaws in methodology. Sort of a research paper equivalent of: "If you give me six lines written by the hand of the most honest of men, I will find something in them which will hang him." by Cardinal Richelieu
  - matusp 3y ago
    Peer review is not a certificate of truthfulness, peer review just says that a grad student reading the paper did not found anything too suspicious there and it looks like a typical paper in that field.
    - some_random 3y ago

deleted 3y ago

[deleted]

jfengel 3y ago

Sociology lives in an uncomfortable place at the intersection of "important" and "difficult". Forming a rigid experiment on human beings is vastly more difficult than a simple thing like an atom or a mineral, and so much worse with groups of human beings. STEM people like to dismiss it because it doesn't produce the same kind of rigid results and is therefore useless. But sociology is nonetheless important. Like it or not, we have to make decisions about how the world will run, and the fact that we don't have perfect information doesn't let us opt out of that. Combine that with all of the usual human failings -- pettiness, meanness, closed-mindedness, greed, etc -- it sounds impossible. But it does, slowly, gather data and formulate theories. This isn't made better by the fact that most STEM people imagine they can read primary source material and understand it -- a mistake they wouldn't make for a "hard science" in a different discipline. Like every field, the frontiers are based on huge amounts of background material, which is even more vast for a field that's more complicated than the nice, neat laws of physics or chemistry. None of that excuses those human failings of sociologists. They need to do better, and hold each other to account (and not just to foment their own failings in their place). But we do need to recognize that this work is important. The world is complex and difficult and we'll make better choices if we try to understand, rather than dismiss it as unknowable.

DontchaKnowit 3y ago
While I agree that there is some work in the field that is important, there is an absolute deluge of ideologically driven garbage. There is also a lot of garbage of the "yeah, duh, of-fucking-course" variety. studies like "negative interactions with community reduce feelings of belonging" ... uh yeah, no shit sherlock. And then even the more important stuff cannot be investigated in the same rigorous manner as other scientific disciplines. I think we just need to stop calling sociology papers "scientific". Fundamentally, they are not, and should almost always be taken with a massive grain of salt, and they damn sure shouldn't be influencing public policy decisions to the degree that they currently do.
- arp242 3y ago
  > There is also a lot of garbage of the "yeah, duh, of-fucking-course" variety. studies like "negative interactions with community reduce feelings of belonging" ... uh yeah, no shit sherlock. I think these studies are useful, first of all to test whether it's actually true because sometimes "well du'h" turns out to be wrong, but also to quantify the exact effects. Is it a large effect or a small effect? How large exactly? Which factors exactly contribute to this effect? What exactly is the breakdown of the effects? It might be possible that 20% of the people are effected by it and 80% of people are not; or perhaps everyone is effected by it. There's often all sorts of non-obvious nuance that's possible, which can be very significant.

pessimizer 3y ago

[flagged]

anonymouskimmer 3y ago
I'm not sure if that particular study is a good example of this as everything and its aunt was getting published about COVID back then. It is disappointing that other studies were shut down on the assumption that the mortality results from such a limited (and fatally flawed) analysis were correct, especially given previous knowledge of the side effects and dosing considerations of hydroxychloroquine.

rqtwteye 3y ago

“ thought of research less as a way to determine the truth, but just as a method to influence policy and public opinion. If we thought something was 80% likely to be true, there was pressure to "close ranks" and pretend as though it was 100% true, and to avoid publishing anything that contradicted it. ” That seems to be the general spirit of this time. People feel the need to take sides and then make sure their side wins. Same happens in journalism. Most journalism these days seems to be about supporting a viewpoint and less about conveying neutral or complete information. Sad that scientists also feel that they should be activists.

whyenot 3y ago

If you are a physical anthropologist there are many questions you can not ask, and many findings that people do not want to hear. Tread carefully, because one wrong step can end your career.