6 ms·

> I believe 80% of science wrong AND not reproducible Maybe that's the case in your area of research. In mine (math, physics) it definitely isn't. So I would b

> I believe 80% of science wrong AND not reproducible Maybe that's the case in your area of research. In mine (math, physics) it definitely isn't. So I would be a bit more careful about the wording here. ("80% of science" – what science?)

OrangeMonkey 4y ago
In healthcare and life sciences research, I absolutely believe at least 80% is wrong and/or not reproducible. Not due to bad actors but, more likely, dataset bias or accidents upon patient matching algorithms or confounders. P < 0.05 is outdated at this point, for anything truly ground breaking p < 0.005 or < 0.0005 is probably a better choice, and even then I would ask "Where did you get your dataset from and did you combine (!!!) datasets from multiple orgs."
- nradov 4y ago
  Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. https://doi.org/10.1371/journal.pmed.0020124 https://doi.org/10.1371/journal.pmed.0020124
  - epistasis 4y ago
    This paper would be more relevant if most science came down to evaluating a single p-value. Perhaps there are some fields of science where that is true, but I have not encountered them. I mostly look at molecular biology and infectious disease. Clinical trials in medicine come down to a single p-value by design, but don't fall prey to what he's talking about.
- themitigating 4y ago
  What is the source for your belief? Something unscientific perhaps?
  - OrangeMonkey 4y ago
    The source for my beliefs is that I perform life sciences research to either generate new studies that are for publication or validate our studies against studies ran in the past. Its actually my job.
    - epistasis 4y ago
      Any study in life sciences will have lots of discussion of claims and modes for the future. But at the core will be the data, which is really the only thing that any researcher actually should trust from a paper. What are you designing new experiments for? Are you taking results in one cell like and condition and trying to transfer them to new cell lines and conditions? Are you trying to reproduce the same data in the same cel lines/organisms? There's such a huge wide array of what your could be taking about, but based on my experience in life sciences what you say does not seem the least bit realistic, in my first interpretation of what you are saying.
      - OrangeMonkey 4y ago
        The problem with getting more and more specific about my experiences and expertise is you get closer and closer to doxxing yourself. Honestly, it may be time to retire this account again. Also its worth noting that I am not claiming my experience is the end-all be-all. I am stating that, from my experience, I have incredible distrust for many studies that are published with 'amazing' results until peer reviewed preferably on disparate datasets. My experience is based around outcome based studies of the effect of drugs/treatment/regimens in oncology and oncology adjacent fields. This includes drugs treated alongside traditional cancer regimens to assist with managing adverse events and toxicities. Other fields may not have this reproducibility problem. Mine does. Even if the study design if perfect, and I can't imagine most are, the data itself can be questionable. Consider - what dataset would you use to identify if patients taking keytruda had a higher incidence of high blood pressure? You can use data from an EHR, licensed for deidentified studies, but EHR data is a burnt down trailer park of questionableness and its use in studies has been laughed at in many conferences. You can use data from individual enrolled patients (for a clinical study) but then the cost is extremely high vs a non interventional observational study using other data methods. The value of the data is likely to be higher, but since it costs more to collect maybe you are only in a few regions that may have a higher prevalence or incidence of this anyway. Troublesome. What about insurance data? You can get it cheaply, if you have high blood pressure GOOD doctors are likely to medicate you with a drug meant to treat it, and you can get it across the country. Seems good right? And it is, as you can generally draw an implication of high-blood-pressure->treatment-with-drug-x. So for a yes/no study it can help, but what if the base condition causes high blood pressure and we want to tell if the drug causes a HIGHER amount of high blood pressure than others. Insurance data by itself may not be enough to tell this data. So what do you do? You are stuck with no great answers.... and this is assuming your study design is perfect. So you buy multiple datasets in some third party health marketplace, and someone gets the great idea to combine the datasets to increase the n value. Well, too bad those datasets have a high overlap. So you have attributed a higher power to the study than is relevent. I hope this explains more about the concerns I have. Though, I suppose, it may mean that this account is now dead. I will have to think further. Anyway, hope you have a great day.
- hackerlight 4y ago
  P less than 0.0005 isn't much better because if that's the criteria then you still have to accept that psychic phenomena have been demonstrated in the lab. The methodology is deeply flawed, given that you haven't stopped accidental p-hacking (publication bias) and deliberate p-hacking (researcher data mining, e.g adding endpoints after looking at results), which compounds to create fake results with astronomically tiny p-values. You can only trust single, extremely large, canonical RCTs which were announced in advance, and in which you are confident there is no survivorship bias in terms of the possibility that this RCT would have been cancelled had the results been thought to be negative halfway through. Epidemiology (victim of researcher p-hacking and impossible to deal with confounders) or meta-analysis of RCTs or single small RCTs or RCTs that weren't announced in advance (victim of publication bias) should be taken with a grain of salt. If you accept conclusions drawn from these things at face value, be prepared to accept anything, because the methodology you've accepted is proven to be easily capable of demonstrating any fake phenomenon as true.
  - OrangeMonkey 4y ago
    I think you hit it exactly right. There are so many ways that are studies can be flawed either accidentally or purposefully. I truly assume most studies are not purposefully fraudulent - people, by and large, are honest... but I do believe there are enough problems with our current methods that most studies are not truly accurate. Having a publish clinical trial regimen - created before your study starts - helps with a lot of this. It answers how you segment patients, match patients, handle patient dropout, and specifies what you are trying to compare for outcomes. Is it the end-all be-all? obviously not. I don't know what the true answer is.
- ta988 4y ago
  This is assuming your experiment was designed to test your hypothesis and that you model your null hypothesis properly. A p value means nothing anyway until you have both of these. Add to that the fact that many wrongly consider that a high p value disproves the original hypothesis.
  - OrangeMonkey 4y ago
    I think you got it - there are many ways for studies to fail and its hard, at this time, to validate that a study succeeded except multiple studies confirming the original finding. Even then, I would worry that the results may be caused by some confounder in the original dataset/design instead of something you can trust.
nemo44x 4y ago
Math is by its nature falsifiable. Much of what is considered “science” today is not. This is often the stuff that’s used as a political weapon via smug “trust the science” justifications for the application of power.
- throwoutway 4y ago
  I agree but It’s weird that now Math is being politicized in California then. No limits to activists I guess
  - lupire 4y ago
    Math is not politicized in California. Math education, and the social applications of math, are politicized.
  - lokeg 4y ago
    I am out of the loop, but surely this would be politicized math education? Which is absolutely different from mathematics research and not related to reproducibility.
    - shreyshnaccount 4y ago
      how does that happen? idk much about what's going on in the US, but the what now? how do you make math political?
      - denton-scratch 4y ago
        > how do you make math political? By proposing a math curriculum that requires teaching all students the same material, regardless of their ability, with the aim of increasing social equality. I'm OK with that aim; but I know from my own experience that trying to teach calculus to someone that's not ready for it isn't just a waste of effort, it's disastrously counter-productive (I totally fell out of love with maths when I was taught integration, failed to "get" it, and my well-regarded teacher didn't get why I didn't get it). My understanding is that nowadays in UK state schools, maths is largely student-paced, using worksheets; they've given up on trying to get a whole class of students to all understand the same stuff. That's partly because a set of worksheets is much easier to come by than a good maths teacher, of course. I'm not a maths teacher, and I don't know enough about the California curriculum arguments to have a view.
t6jvcereio 4y ago
I'm also in physics and I agree with this. There's no chance that 80% is wrong. One factor contributing to this. In natural sciences, you take other people's papers as truth and build on that. In theoretical physics on the other hand the _first step_ is you reproduce their results.
- stevenjgarner 4y ago
  Why do you think that is? (Serious question). My understanding in life science research is that as a rule, papers do not get reproduced, as there is limited benefit to the researcher. Why is the same not true in physics? What is so seriously flawed about life science research that is not flawed about physics?
  - martincmartin 4y ago
    My guess: In theoretical physics, you reproduce by spending an afternoon manipulating equations. Maybe a few days at most. But you reproduce in life science you need to get test subjects and treatments, etc. For example, how would you reproduce that a diet high in olive oil reduces the chance of heart attack?
    - t6jvcereio 4y ago
      It's usually a lot more than "a few days at most". In my field it was often between a week to a couple of months. But you got the right idea.
  - jltsiren 4y ago
    It depends on the type of research. If your research is mathematical or involves a lot of engineering, you are building on earlier results. In many cases, your own results will depend on the correctness of earlier results in a measurable way. You end up replicating others' research without even trying. Empirical research is harder. You cite earlier results, but there is no clear connection between their correctness and your results. Especially if effect size is small. Earlier research has more effect on the framework you use to interpret your results than on the results. Academic researchers rarely replicate others' results, because it's expensive and not particularly interesting. People typically come to the academia because they want to work on something they personally find interesting. If you want to have experienced scientists working on something administrators tell them to do, you better pay industry-level salaries.
turns0ut 4y ago
That’s the last 10% they left on the table. That last 10% makes up 100% of legit science.
sosull 4y ago
“Oh, people can come up with statistics to prove anything, Kent. Forty percent of all people know that.”
ajsnigrutin 4y ago
Well, math is one of those things where (in most cases) anyone (...with the right knowledge in the field...) can verify the results (mostly) relatively easily and cheaply. For physics, it depends.. is it just "applied math" (so, verified easily), is it a CERN-type (LHC,...) experiment (hard to replicate, unless well.. you work at CERN), where many many people process the data, or is it something that is done only in your "lab", and hard for others to replicate. On the other hand, finding thousands of patients and running a study is practically always hard and expensive.
1-6 4y ago
Wow, sounds like the Pareto Principle is alive and well in the arena of Science as well.
User23 4y ago
Math isn’t immune to error and more commonly ambiguity. That said mathematicians tend to have an admirable diligence and sky high intelligence, so yeah there’s a ton of quality work. With rigorous formality and computer checking it’s conceivable that math as a field can become virtually flawless. As for physics, well it depends. Laboratory physics has produced the finest predictions in any science by several orders of magnitude. Quantum Electrodynamics is freakishly accurate. On the other hand it’s hard for me to see cosmology, to gently pick on an easy target, as more than extremely well researched and plausible science fiction. Then you have particle physics which has excellent laboratory equipment and produced fantastic results, but which has, in the opinion of at least one elite particle physicist personally known to me, perhaps painted itself into a corner. The Standard Model is good enough for government work, but nobody believes it’s the best possible theory.

OrangeMonkey 4y ago

In healthcare and life sciences research, I absolutely believe at least 80% is wrong and/or not reproducible. Not due to bad actors but, more likely, dataset bias or accidents upon patient matching algorithms or confounders. P < 0.05 is outdated at this point, for anything truly ground breaking p < 0.005 or < 0.0005 is probably a better choice, and even then I would ask "Where did you get your dataset from and did you combine (!!!) datasets from multiple orgs."

nradov 4y ago
Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. https://doi.org/10.1371/journal.pmed.0020124 https://doi.org/10.1371/journal.pmed.0020124
- epistasis 4y ago
  This paper would be more relevant if most science came down to evaluating a single p-value. Perhaps there are some fields of science where that is true, but I have not encountered them. I mostly look at molecular biology and infectious disease. Clinical trials in medicine come down to a single p-value by design, but don't fall prey to what he's talking about.
themitigating 4y ago
What is the source for your belief? Something unscientific perhaps?
- OrangeMonkey 4y ago
  The source for my beliefs is that I perform life sciences research to either generate new studies that are for publication or validate our studies against studies ran in the past. Its actually my job.
  - epistasis 4y ago
    Any study in life sciences will have lots of discussion of claims and modes for the future. But at the core will be the data, which is really the only thing that any researcher actually should trust from a paper. What are you designing new experiments for? Are you taking results in one cell like and condition and trying to transfer them to new cell lines and conditions? Are you trying to reproduce the same data in the same cel lines/organisms? There's such a huge wide array of what your could be taking about, but based on my experience in life sciences what you say does not seem the least bit realistic, in my first interpretation of what you are saying.
    - OrangeMonkey 4y ago
      The problem with getting more and more specific about my experiences and expertise is you get closer and closer to doxxing yourself. Honestly, it may be time to retire this account again. Also its worth noting that I am not claiming my experience is the end-all be-all. I am stating that, from my experience, I have incredible distrust for many studies that are published with 'amazing' results until peer reviewed preferably on disparate datasets. My experience is based around outcome based studies of the effect of drugs/treatment/regimens in oncology and oncology adjacent fields. This includes drugs treated alongside traditional cancer regimens to assist with managing adverse events and toxicities. Other fields may not have this reproducibility problem. Mine does. Even if the study design if perfect, and I can't imagine most are, the data itself can be questionable. Consider - what dataset would you use to identify if patients taking keytruda had a higher incidence of high blood pressure? You can use data from an EHR, licensed for deidentified studies, but EHR data is a burnt down trailer park of questionableness and its use in studies has been laughed at in many conferences. You can use data from individual enrolled patients (for a clinical study) but then the cost is extremely high vs a non interventional observational study using other data methods. The value of the data is likely to be higher, but since it costs more to collect maybe you are only in a few regions that may have a higher prevalence or incidence of this anyway. Troublesome. What about insurance data? You can get it cheaply, if you have high blood pressure GOOD doctors are likely to medicate you with a drug meant to treat it, and you can get it across the country. Seems good right? And it is, as you can generally draw an implication of high-blood-pressure->treatment-with-drug-x. So for a yes/no study it can help, but what if the base condition causes high blood pressure and we want to tell if the drug causes a HIGHER amount of high blood pressure than others. Insurance data by itself may not be enough to tell this data. So what do you do? You are stuck with no great answers.... and this is assuming your study design is perfect. So you buy multiple datasets in some third party health marketplace, and someone gets the great idea to combine the datasets to increase the n value. Well, too bad those datasets have a high overlap. So you have attributed a higher power to the study than is relevent. I hope this explains more about the concerns I have. Though, I suppose, it may mean that this account is now dead. I will have to think further. Anyway, hope you have a great day.
hackerlight 4y ago
P less than 0.0005 isn't much better because if that's the criteria then you still have to accept that psychic phenomena have been demonstrated in the lab. The methodology is deeply flawed, given that you haven't stopped accidental p-hacking (publication bias) and deliberate p-hacking (researcher data mining, e.g adding endpoints after looking at results), which compounds to create fake results with astronomically tiny p-values. You can only trust single, extremely large, canonical RCTs which were announced in advance, and in which you are confident there is no survivorship bias in terms of the possibility that this RCT would have been cancelled had the results been thought to be negative halfway through. Epidemiology (victim of researcher p-hacking and impossible to deal with confounders) or meta-analysis of RCTs or single small RCTs or RCTs that weren't announced in advance (victim of publication bias) should be taken with a grain of salt. If you accept conclusions drawn from these things at face value, be prepared to accept anything, because the methodology you've accepted is proven to be easily capable of demonstrating any fake phenomenon as true.
- OrangeMonkey 4y ago
  I think you hit it exactly right. There are so many ways that are studies can be flawed either accidentally or purposefully. I truly assume most studies are not purposefully fraudulent - people, by and large, are honest... but I do believe there are enough problems with our current methods that most studies are not truly accurate. Having a publish clinical trial regimen - created before your study starts - helps with a lot of this. It answers how you segment patients, match patients, handle patient dropout, and specifies what you are trying to compare for outcomes. Is it the end-all be-all? obviously not. I don't know what the true answer is.
ta988 4y ago
This is assuming your experiment was designed to test your hypothesis and that you model your null hypothesis properly. A p value means nothing anyway until you have both of these. Add to that the fact that many wrongly consider that a high p value disproves the original hypothesis.
- OrangeMonkey 4y ago
  I think you got it - there are many ways for studies to fail and its hard, at this time, to validate that a study succeeded except multiple studies confirming the original finding. Even then, I would worry that the results may be caused by some confounder in the original dataset/design instead of something you can trust.

nemo44x 4y ago

Math is by its nature falsifiable. Much of what is considered “science” today is not. This is often the stuff that’s used as a political weapon via smug “trust the science” justifications for the application of power.

throwoutway 4y ago
I agree but It’s weird that now Math is being politicized in California then. No limits to activists I guess
- lupire 4y ago
  Math is not politicized in California. Math education, and the social applications of math, are politicized.
- lokeg 4y ago
  I am out of the loop, but surely this would be politicized math education? Which is absolutely different from mathematics research and not related to reproducibility.
  - shreyshnaccount 4y ago
    how does that happen? idk much about what's going on in the US, but the what now? how do you make math political?
    - denton-scratch 4y ago
      > how do you make math political? By proposing a math curriculum that requires teaching all students the same material, regardless of their ability, with the aim of increasing social equality. I'm OK with that aim; but I know from my own experience that trying to teach calculus to someone that's not ready for it isn't just a waste of effort, it's disastrously counter-productive (I totally fell out of love with maths when I was taught integration, failed to "get" it, and my well-regarded teacher didn't get why I didn't get it). My understanding is that nowadays in UK state schools, maths is largely student-paced, using worksheets; they've given up on trying to get a whole class of students to all understand the same stuff. That's partly because a set of worksheets is much easier to come by than a good maths teacher, of course. I'm not a maths teacher, and I don't know enough about the California curriculum arguments to have a view.

t6jvcereio 4y ago

I'm also in physics and I agree with this. There's no chance that 80% is wrong. One factor contributing to this. In natural sciences, you take other people's papers as truth and build on that. In theoretical physics on the other hand the _first step_ is you reproduce their results.

stevenjgarner 4y ago
Why do you think that is? (Serious question). My understanding in life science research is that as a rule, papers do not get reproduced, as there is limited benefit to the researcher. Why is the same not true in physics? What is so seriously flawed about life science research that is not flawed about physics?
- martincmartin 4y ago
  My guess: In theoretical physics, you reproduce by spending an afternoon manipulating equations. Maybe a few days at most. But you reproduce in life science you need to get test subjects and treatments, etc. For example, how would you reproduce that a diet high in olive oil reduces the chance of heart attack?
  - t6jvcereio 4y ago
    It's usually a lot more than "a few days at most". In my field it was often between a week to a couple of months. But you got the right idea.
- jltsiren 4y ago
  It depends on the type of research. If your research is mathematical or involves a lot of engineering, you are building on earlier results. In many cases, your own results will depend on the correctness of earlier results in a measurable way. You end up replicating others' research without even trying. Empirical research is harder. You cite earlier results, but there is no clear connection between their correctness and your results. Especially if effect size is small. Earlier research has more effect on the framework you use to interpret your results than on the results. Academic researchers rarely replicate others' results, because it's expensive and not particularly interesting. People typically come to the academia because they want to work on something they personally find interesting. If you want to have experienced scientists working on something administrators tell them to do, you better pay industry-level salaries.

turns0ut 4y ago

That’s the last 10% they left on the table. That last 10% makes up 100% of legit science.

sosull 4y ago

“Oh, people can come up with statistics to prove anything, Kent. Forty percent of all people know that.”

ajsnigrutin 4y ago

Well, math is one of those things where (in most cases) anyone (...with the right knowledge in the field...) can verify the results (mostly) relatively easily and cheaply. For physics, it depends.. is it just "applied math" (so, verified easily), is it a CERN-type (LHC,...) experiment (hard to replicate, unless well.. you work at CERN), where many many people process the data, or is it something that is done only in your "lab", and hard for others to replicate. On the other hand, finding thousands of patients and running a study is practically always hard and expensive.

1-6 4y ago

Wow, sounds like the Pareto Principle is alive and well in the arena of Science as well.

User23 4y ago

Math isn’t immune to error and more commonly ambiguity. That said mathematicians tend to have an admirable diligence and sky high intelligence, so yeah there’s a ton of quality work. With rigorous formality and computer checking it’s conceivable that math as a field can become virtually flawless. As for physics, well it depends. Laboratory physics has produced the finest predictions in any science by several orders of magnitude. Quantum Electrodynamics is freakishly accurate. On the other hand it’s hard for me to see cosmology, to gently pick on an easy target, as more than extremely well researched and plausible science fiction. Then you have particle physics which has excellent laboratory equipment and produced fantastic results, but which has, in the opinion of at least one elite particle physicist personally known to me, perhaps painted itself into a corner. The Standard Model is good enough for government work, but nobody believes it’s the best possible theory.