3 ms·

This is what you always hear from biomed/psych/etc. It simply is not important to get details right. Misinterpret the results of your analysis (p-values), use t

by nonbel 9y ago

This is what you always hear from biomed/psych/etc. It simply is not important to get details right. Misinterpret the results of your analysis (p-values), use the wrong cell lines, measure the wrong thing, etc. It rarely seems to affect the conclusions. If anyone who points out errors is being pedantic, it makes you wonder about the point of doing all that. Just come up with an idea A, say "heads = conclusion A", "tails = conclusion not A", and flip a coin.

Fomite 9y ago
I think one of the things that is actually happening is that for many examples, while the quantitative answer changes, the qualitative conclusion isn't altered by the errors. In graduate school, as part of a class on survival analysis, we subjected the same data set to increasingly sophisticated analysis techniques to account for all kinds of things. Each time, the effect estimate changed. At the end, while discussing them, the professor asked a very simple question: "Do any of these suggest that HAART is a bad idea?" Similarly, during the Ebola epidemic, when everyone was fussing about the various forecasts missing the mark, etc., what they were actually predicting was "This is a serious crisis that needs international intervention".
- nonbel 9y ago
  I'm sure that HAART is a good idea sometimes and a bad idea other times. Also, this will change as time goes on (HIV mutates, demographics change, other treatments are available, etc) I'm also sure that there are many things going on at any given moment that should count as "serious crisis that needs international intervention". The real question is whether it is a more serious crisis than other things currently happening. To deal with both these issues you are going to need deeper understanding than "good idea vs bad idea" or "is crisis vs not crisis". But the real problem with being unable to quantify your understanding is that you are left unable to make precise predictions, thus you can never perform any stringent tests. If all you can predict is something vague (eg, HAART will increase 5 year survival), there will be many ways to misinterpret the data in support of your explanation even if it is totally wrong.
  - Fomite 9y ago
    The point is not that it will change over time - it was that in this setting with this information different methods can give you different answers, but that may not actually matter. If you're HIV+ in this country right now, you want HAART, whether I did some fancy marginal structural modeling or not. > To deal with both these issues you are going to need deeper understanding than "good idea vs bad idea" or "is crisis vs not crisis". This is not, actually, how the response to Ebola worked. It was very much "crisis vs. not crisis". And a huge amount of clinical decision making is "good idea vs. bad idea". The suggestion was not that you're not able to quantify your understanding. The suggestion was there are errors that are possible to make that, while changing your effect estimate, do not change whether or not you do a thing. To return to the HAART example, imagine you're an HIV+ patient and I've told you that a major study failed to control for time-varying confounding, and that upon re-analysis, instead of doubling your 5 year survival, it only increases it by 87%. Or, for a form of "is this repeatable?" that I particularly despise, that it still doubles your chances of survival, but the p-value has gone from 0.047 to 0.062. Do you want to stop taking the drug?
    - nonbel 9y ago
      >"To return to the HAART example, imagine you're an HIV+ patient and I've told you that a major study failed to control for time-varying confounding, and that upon re-analysis, instead of doubling your 5 year survival, it only increases it by 87%. Or, for a form of "is this repeatable?" that I particularly despise, that it still doubles your chances of survival, but the p-value has gone from 0.047 to 0.062. Do you want to stop taking the drug?" This obviously depends on the relative costs. Cost of side effects, buying the drugs, time going to treatment, etc. That all requires accurate quantification. But cost-benefit isn't even what I meant. To begin with, if they can't nail down a quantifiable effect that is stable from study to study, who knows what is going on? Why would you have confidence in their estimates of effectiveness if they are inconsistent with one another?
      - Fomite 9y ago
        "This obviously depends on the relative costs. Cost of side effects, buying the drugs, time going to treatment, etc." I'm going to suggest if you're facing death from an AIDS-related illness, a relative risk of 2.00 vs. 1.87 will feel very, very similar to you. "To begin with, if they can't nail down a quantifiable effect that is stable from study to study, who knows what is going on? Why would you have confidence in their estimates of effectiveness if they are inconsistent with one another?" Define "stable" - because even if the effect of something is fixed in the same way a physical constant it, it's invariable sampled with error. Again, if I give you five studies that suggest that HAART improves survival by: 100%, 102%, 87%, 94% and 96% are you really going to suggest that we don't know that HAART improves survival?
Gatsky 9y ago
Immortalised cell lines are a bad model, for many reasons. Anybody that has done cell culture knows this. Anyone that has tried to translate in vitro work to in vivo or into actual clinicsl trials in people knows this even more. The article is acting like the fact that cell lines are an even worse model than we thought is a massive deal, which it really isn't, in my opinion.

by nonbel 9y ago

Fomite 9y ago

I think one of the things that is actually happening is that for many examples, while the quantitative answer changes, the qualitative conclusion isn't altered by the errors. In graduate school, as part of a class on survival analysis, we subjected the same data set to increasingly sophisticated analysis techniques to account for all kinds of things. Each time, the effect estimate changed. At the end, while discussing them, the professor asked a very simple question: "Do any of these suggest that HAART is a bad idea?" Similarly, during the Ebola epidemic, when everyone was fussing about the various forecasts missing the mark, etc., what they were actually predicting was "This is a serious crisis that needs international intervention".

nonbel 9y ago
I'm sure that HAART is a good idea sometimes and a bad idea other times. Also, this will change as time goes on (HIV mutates, demographics change, other treatments are available, etc) I'm also sure that there are many things going on at any given moment that should count as "serious crisis that needs international intervention". The real question is whether it is a more serious crisis than other things currently happening. To deal with both these issues you are going to need deeper understanding than "good idea vs bad idea" or "is crisis vs not crisis". But the real problem with being unable to quantify your understanding is that you are left unable to make precise predictions, thus you can never perform any stringent tests. If all you can predict is something vague (eg, HAART will increase 5 year survival), there will be many ways to misinterpret the data in support of your explanation even if it is totally wrong.
- Fomite 9y ago
  The point is not that it will change over time - it was that in this setting with this information different methods can give you different answers, but that may not actually matter. If you're HIV+ in this country right now, you want HAART, whether I did some fancy marginal structural modeling or not. > To deal with both these issues you are going to need deeper understanding than "good idea vs bad idea" or "is crisis vs not crisis". This is not, actually, how the response to Ebola worked. It was very much "crisis vs. not crisis". And a huge amount of clinical decision making is "good idea vs. bad idea". The suggestion was not that you're not able to quantify your understanding. The suggestion was there are errors that are possible to make that, while changing your effect estimate, do not change whether or not you do a thing. To return to the HAART example, imagine you're an HIV+ patient and I've told you that a major study failed to control for time-varying confounding, and that upon re-analysis, instead of doubling your 5 year survival, it only increases it by 87%. Or, for a form of "is this repeatable?" that I particularly despise, that it still doubles your chances of survival, but the p-value has gone from 0.047 to 0.062. Do you want to stop taking the drug?
  - nonbel 9y ago
    >"To return to the HAART example, imagine you're an HIV+ patient and I've told you that a major study failed to control for time-varying confounding, and that upon re-analysis, instead of doubling your 5 year survival, it only increases it by 87%. Or, for a form of "is this repeatable?" that I particularly despise, that it still doubles your chances of survival, but the p-value has gone from 0.047 to 0.062. Do you want to stop taking the drug?" This obviously depends on the relative costs. Cost of side effects, buying the drugs, time going to treatment, etc. That all requires accurate quantification. But cost-benefit isn't even what I meant. To begin with, if they can't nail down a quantifiable effect that is stable from study to study, who knows what is going on? Why would you have confidence in their estimates of effectiveness if they are inconsistent with one another?
    - Fomite 9y ago
      "This obviously depends on the relative costs. Cost of side effects, buying the drugs, time going to treatment, etc." I'm going to suggest if you're facing death from an AIDS-related illness, a relative risk of 2.00 vs. 1.87 will feel very, very similar to you. "To begin with, if they can't nail down a quantifiable effect that is stable from study to study, who knows what is going on? Why would you have confidence in their estimates of effectiveness if they are inconsistent with one another?" Define "stable" - because even if the effect of something is fixed in the same way a physical constant it, it's invariable sampled with error. Again, if I give you five studies that suggest that HAART improves survival by: 100%, 102%, 87%, 94% and 96% are you really going to suggest that we don't know that HAART improves survival?

Gatsky 9y ago

Immortalised cell lines are a bad model, for many reasons. Anybody that has done cell culture knows this. Anyone that has tried to translate in vitro work to in vivo or into actual clinicsl trials in people knows this even more. The article is acting like the fact that cell lines are an even worse model than we thought is a massive deal, which it really isn't, in my opinion.