4 ms·

It's only testable in the data if these decisions are being made based on factors that are adequately represented in your statistics (no unobserved confounding)

It's only testable in the data if these decisions are being made based on factors that are adequately represented in your statistics (no unobserved confounding). Maybe there's patient A, a smoker who says they're a non-smoker but seems like a liar to the doctor, and an otherwise identical patient B who is actually a non-smoker. Patient A is probably higher risk, so there's a difference between doing A during conference week then B later, versus B during conference week than A later. Even supposing that the doctor wrote that they think A might be lying and actually maybe a smoker, it's going to be hard to control for those notes in a t-test. It's a shitty example, but hopefully demonstrates the idea. It's always super hard to make the argument that there's no unobserved confounding.

deleted 6y ago
[deleted]
Retric 6y ago
Conference dates are independent of patient heath, so you can just look at the numbers directly across many healthcare systems and several decades. Vastly more data generally trumps better analysis.
- 6gvONxR4sf7o 6y ago
  > Vastly more data generally trumps better analysis. If you mean more data in the sense of more covariates, sure. If you mean more data in terms of just more observations, no way. Not when it comes to causal inference.
  - lowdose 6y ago
    Could you elaborate on this point "no way", for more observations?
    - 6gvONxR4sf7o 6y ago
      Think about getting a helicopter to the hospital. It's probably pretty serious, right? You're more likely to die if you get a helicopter evac than if you don't. If all you have recorded is whether they got to the hospital via helicopter or not, and whether they died or not, then you're going to see that helicopters are associated with death. If you could control for what's wrong with the person, you might see that helicopters save lives. If you don't have that data, then just having more observations of (helicopter y/n, died y/n) will just look like helivac is murder. No matter what you do with that data, you just can't control for what you'd need to, because what you need was an unobserved confounder. When you're trying to establish causation, you have to rule out unobserved confounders, which is tricky. There's always something where someone might say "well what if it was X?" and the data doesn't contain that answer. In this particular case, that commenter said that conference dates are independent enough from patient health to say that aside from the important things, everything else is equal. As in, the only difference between the treatments is who is doing the treatment. I disagree with that assertion, but that's the kind of argument you need for causation: Once we've dealt with/controlled for XYZ, the only difference left is the one we're interested in. It's very difficult to demonstrate that the only difference left is what you care about when you can't even see certain variables. Someone says "what if people in group A are more likely to be left handed and it's patient handedness instead of doctor quality that's causing death here" but you didn't measure left handedness. More and more observations of group A without measuring handedness can't rule out that maybe they were lefties. So you either measure handedness, or argue that your setup will even it out (via randomization for example), or argue that handedness just doesn't matter here. And not just for handedness, but for everything imaginable.
      - Retric 6y ago
        While intuitively that reasoning seems valid, having more data makes signals more clear. If your sample sizes is say 99.8% of all doctors that’s going to be representative in ways a carefully crafted representative sample of 1,000 doctors simply isn’t. Sampling always loses out to having the entire population. But you say missing data makes the interpretation incorrect. That’s always possible but only in ways that also fit the original data. Thus, you can then preform a study that improves upon the understanding of the correlation, but it’s not going to disprove the original correlation. PS: And even here it’s easy to test with more data, just compare total deaths for the month around conferences.
- smallnamespace 6y ago
  Is that true? A pandemic affects both patient health and conference dates.
  - Retric 6y ago
    Attendance may be linked, but the proposed dates should only link to minor seasonal variations not specific days.

deleted 6y ago

[deleted]

Retric 6y ago

Conference dates are independent of patient heath, so you can just look at the numbers directly across many healthcare systems and several decades. Vastly more data generally trumps better analysis.

6gvONxR4sf7o 6y ago
> Vastly more data generally trumps better analysis. If you mean more data in the sense of more covariates, sure. If you mean more data in terms of just more observations, no way. Not when it comes to causal inference.
- lowdose 6y ago
  Could you elaborate on this point "no way", for more observations?
  - 6gvONxR4sf7o 6y ago
    Think about getting a helicopter to the hospital. It's probably pretty serious, right? You're more likely to die if you get a helicopter evac than if you don't. If all you have recorded is whether they got to the hospital via helicopter or not, and whether they died or not, then you're going to see that helicopters are associated with death. If you could control for what's wrong with the person, you might see that helicopters save lives. If you don't have that data, then just having more observations of (helicopter y/n, died y/n) will just look like helivac is murder. No matter what you do with that data, you just can't control for what you'd need to, because what you need was an unobserved confounder. When you're trying to establish causation, you have to rule out unobserved confounders, which is tricky. There's always something where someone might say "well what if it was X?" and the data doesn't contain that answer. In this particular case, that commenter said that conference dates are independent enough from patient health to say that aside from the important things, everything else is equal. As in, the only difference between the treatments is who is doing the treatment. I disagree with that assertion, but that's the kind of argument you need for causation: Once we've dealt with/controlled for XYZ, the only difference left is the one we're interested in. It's very difficult to demonstrate that the only difference left is what you care about when you can't even see certain variables. Someone says "what if people in group A are more likely to be left handed and it's patient handedness instead of doctor quality that's causing death here" but you didn't measure left handedness. More and more observations of group A without measuring handedness can't rule out that maybe they were lefties. So you either measure handedness, or argue that your setup will even it out (via randomization for example), or argue that handedness just doesn't matter here. And not just for handedness, but for everything imaginable.
    - Retric 6y ago
      While intuitively that reasoning seems valid, having more data makes signals more clear. If your sample sizes is say 99.8% of all doctors that’s going to be representative in ways a carefully crafted representative sample of 1,000 doctors simply isn’t. Sampling always loses out to having the entire population. But you say missing data makes the interpretation incorrect. That’s always possible but only in ways that also fit the original data. Thus, you can then preform a study that improves upon the understanding of the correlation, but it’s not going to disprove the original correlation. PS: And even here it’s easy to test with more data, just compare total deaths for the month around conferences.
smallnamespace 6y ago
Is that true? A pandemic affects both patient health and conference dates.
- Retric 6y ago
  Attendance may be linked, but the proposed dates should only link to minor seasonal variations not specific days.