4 ms·

I'm sure that HAART is a good idea sometimes and a bad idea other times. Also, this will change as time goes on (HIV mutates, demographics change, other treatme

by nonbel 9y ago

I'm sure that HAART is a good idea sometimes and a bad idea other times. Also, this will change as time goes on (HIV mutates, demographics change, other treatments are available, etc) I'm also sure that there are many things going on at any given moment that should count as "serious crisis that needs international intervention". The real question is whether it is a more serious crisis than other things currently happening. To deal with both these issues you are going to need deeper understanding than "good idea vs bad idea" or "is crisis vs not crisis". But the real problem with being unable to quantify your understanding is that you are left unable to make precise predictions, thus you can never perform any stringent tests. If all you can predict is something vague (eg, HAART will increase 5 year survival), there will be many ways to misinterpret the data in support of your explanation even if it is totally wrong.

Fomite 9y ago
The point is not that it will change over time - it was that in this setting with this information different methods can give you different answers, but that may not actually matter. If you're HIV+ in this country right now, you want HAART, whether I did some fancy marginal structural modeling or not. > To deal with both these issues you are going to need deeper understanding than "good idea vs bad idea" or "is crisis vs not crisis". This is not, actually, how the response to Ebola worked. It was very much "crisis vs. not crisis". And a huge amount of clinical decision making is "good idea vs. bad idea". The suggestion was not that you're not able to quantify your understanding. The suggestion was there are errors that are possible to make that, while changing your effect estimate, do not change whether or not you do a thing. To return to the HAART example, imagine you're an HIV+ patient and I've told you that a major study failed to control for time-varying confounding, and that upon re-analysis, instead of doubling your 5 year survival, it only increases it by 87%. Or, for a form of "is this repeatable?" that I particularly despise, that it still doubles your chances of survival, but the p-value has gone from 0.047 to 0.062. Do you want to stop taking the drug?
- nonbel 9y ago
  >"To return to the HAART example, imagine you're an HIV+ patient and I've told you that a major study failed to control for time-varying confounding, and that upon re-analysis, instead of doubling your 5 year survival, it only increases it by 87%. Or, for a form of "is this repeatable?" that I particularly despise, that it still doubles your chances of survival, but the p-value has gone from 0.047 to 0.062. Do you want to stop taking the drug?" This obviously depends on the relative costs. Cost of side effects, buying the drugs, time going to treatment, etc. That all requires accurate quantification. But cost-benefit isn't even what I meant. To begin with, if they can't nail down a quantifiable effect that is stable from study to study, who knows what is going on? Why would you have confidence in their estimates of effectiveness if they are inconsistent with one another?
  - Fomite 9y ago
    "This obviously depends on the relative costs. Cost of side effects, buying the drugs, time going to treatment, etc." I'm going to suggest if you're facing death from an AIDS-related illness, a relative risk of 2.00 vs. 1.87 will feel very, very similar to you. "To begin with, if they can't nail down a quantifiable effect that is stable from study to study, who knows what is going on? Why would you have confidence in their estimates of effectiveness if they are inconsistent with one another?" Define "stable" - because even if the effect of something is fixed in the same way a physical constant it, it's invariable sampled with error. Again, if I give you five studies that suggest that HAART improves survival by: 100%, 102%, 87%, 94% and 96% are you really going to suggest that we don't know that HAART improves survival?
    - nonbel 9y ago
      >"I'm going to suggest if you're facing death from an AIDS-related illness, a relative risk of 2.00 vs. 1.87 will feel very, very similar to you." I at first thought we both understood there would be some uncertainty about these values no matter what but we were leaving that out out for simplicity's sake. Roughly, I was assuming it is at least +/- 10-20%. The reason these values "feel very, very similar" is that I would not expect medical data to be able to distinguish between them. >"Define "stable" - because even if the effect of something is fixed in the same way a physical constant it, it's invariable sampled with error." Ok... so you are implicitly considering uncertainty. >"Again, if I give you five studies that suggest that HAART improves survival by: 100%, 102%, 87%, 94% and 96% are you really going to suggest that we don't know that HAART improves survival?" Not enough info. - Where did the uncertainty go? - What methods were used to generate these values? Even basic strategies like blinding the people collecting/processing/analyzing data is often still missing. - What population/frame do those numbers refer to, and how much error can we expect from extrapolating for other situations in the future? You need a reliable quantification, doing this hand wavy "things are better/worse, significant/insignificant" is an awful idea.

by nonbel 9y ago

Fomite 9y ago

The point is not that it will change over time - it was that in this setting with this information different methods can give you different answers, but that may not actually matter. If you're HIV+ in this country right now, you want HAART, whether I did some fancy marginal structural modeling or not. > To deal with both these issues you are going to need deeper understanding than "good idea vs bad idea" or "is crisis vs not crisis". This is not, actually, how the response to Ebola worked. It was very much "crisis vs. not crisis". And a huge amount of clinical decision making is "good idea vs. bad idea". The suggestion was not that you're not able to quantify your understanding. The suggestion was there are errors that are possible to make that, while changing your effect estimate, do not change whether or not you do a thing. To return to the HAART example, imagine you're an HIV+ patient and I've told you that a major study failed to control for time-varying confounding, and that upon re-analysis, instead of doubling your 5 year survival, it only increases it by 87%. Or, for a form of "is this repeatable?" that I particularly despise, that it still doubles your chances of survival, but the p-value has gone from 0.047 to 0.062. Do you want to stop taking the drug?

nonbel 9y ago
>"To return to the HAART example, imagine you're an HIV+ patient and I've told you that a major study failed to control for time-varying confounding, and that upon re-analysis, instead of doubling your 5 year survival, it only increases it by 87%. Or, for a form of "is this repeatable?" that I particularly despise, that it still doubles your chances of survival, but the p-value has gone from 0.047 to 0.062. Do you want to stop taking the drug?" This obviously depends on the relative costs. Cost of side effects, buying the drugs, time going to treatment, etc. That all requires accurate quantification. But cost-benefit isn't even what I meant. To begin with, if they can't nail down a quantifiable effect that is stable from study to study, who knows what is going on? Why would you have confidence in their estimates of effectiveness if they are inconsistent with one another?
- Fomite 9y ago
  "This obviously depends on the relative costs. Cost of side effects, buying the drugs, time going to treatment, etc." I'm going to suggest if you're facing death from an AIDS-related illness, a relative risk of 2.00 vs. 1.87 will feel very, very similar to you. "To begin with, if they can't nail down a quantifiable effect that is stable from study to study, who knows what is going on? Why would you have confidence in their estimates of effectiveness if they are inconsistent with one another?" Define "stable" - because even if the effect of something is fixed in the same way a physical constant it, it's invariable sampled with error. Again, if I give you five studies that suggest that HAART improves survival by: 100%, 102%, 87%, 94% and 96% are you really going to suggest that we don't know that HAART improves survival?
  - nonbel 9y ago
    >"I'm going to suggest if you're facing death from an AIDS-related illness, a relative risk of 2.00 vs. 1.87 will feel very, very similar to you." I at first thought we both understood there would be some uncertainty about these values no matter what but we were leaving that out out for simplicity's sake. Roughly, I was assuming it is at least +/- 10-20%. The reason these values "feel very, very similar" is that I would not expect medical data to be able to distinguish between them. >"Define "stable" - because even if the effect of something is fixed in the same way a physical constant it, it's invariable sampled with error." Ok... so you are implicitly considering uncertainty. >"Again, if I give you five studies that suggest that HAART improves survival by: 100%, 102%, 87%, 94% and 96% are you really going to suggest that we don't know that HAART improves survival?" Not enough info. - Where did the uncertainty go? - What methods were used to generate these values? Even basic strategies like blinding the people collecting/processing/analyzing data is often still missing. - What population/frame do those numbers refer to, and how much error can we expect from extrapolating for other situations in the future? You need a reliable quantification, doing this hand wavy "things are better/worse, significant/insignificant" is an awful idea.