4 ms·

> I am reasonably confident that there is a group out there that was more careful and has something that might have been useful What is that confidence based o

> I am reasonably confident that there is a group out there that was more careful and has something that might have been useful What is that confidence based on? It seems like the researchers here made a reasonably thorough effort to find "something that might have been useful" and only turned up two that were even worth further investigation: "She and her colleagues have looked at 232 algorithms for diagnosing patients or predicting how sick those with the disease might get. They found that none of them were fit for clinical use. Just two have been singled out as being promising enough for future testing." So are you reasonably confident that these researchers just didn't look hard enough to find the good stuff?

Grimm1 5y ago
Why not? Plenty of researchers aren’t thorough. I’m not saying that’s the case here but it’s entirely plausible from what I know about academia and wouldn’t be a crazy claim to make.
- Jweb_Guru 5y ago
  If a model did particularly well while almost every other model did badly, it would have gained attention in the community (not to mention, the researchers who worked on it would be actively marketing this!). Even if the researchers who worked on the paper weren't aware of it, it would almost certainly be brought up during peer review. The only way that such a model would not end up on such an extensive list is deliberate omission, not lack of rigor.
  - dontreact 5y ago
    I don’t think this is true. Medical practices change slowly and it takes time to build trust. It’s possible a covid model will become clinically useful, but it probably requires having enough buy in to run a clinical trial and then for the economics to line up so that using it leads to more money for hospitals somehow
dontreact 5y ago
I think the issue is not with the training step but with the validation step. The only way to build enough trust is to carefully run a prospective clinical trial, and before that you probably need enough retrospective evidence to get some sort of FDA approval. These processes take a lot of time and it’s not surprising that a year and a half after COVID-19 became widespread in the US, we still don’t have a fully validated model.
- dontreact 5y ago
  In fact the claim the paper makes is that no model has been sufficiently clinically validated. Problem is: how do you get enough buy in from clinicians to run a clinical trial with YOUR model, when there are hundreds of crappy models out there? It takes a lot of time to run these studies for this reason to start with and then depending on the size of the clinical trial, it takes time for the trial to run it’s course.

Grimm1 5y ago

Why not? Plenty of researchers aren’t thorough. I’m not saying that’s the case here but it’s entirely plausible from what I know about academia and wouldn’t be a crazy claim to make.

Jweb_Guru 5y ago
If a model did particularly well while almost every other model did badly, it would have gained attention in the community (not to mention, the researchers who worked on it would be actively marketing this!). Even if the researchers who worked on the paper weren't aware of it, it would almost certainly be brought up during peer review. The only way that such a model would not end up on such an extensive list is deliberate omission, not lack of rigor.
- dontreact 5y ago
  I don’t think this is true. Medical practices change slowly and it takes time to build trust. It’s possible a covid model will become clinically useful, but it probably requires having enough buy in to run a clinical trial and then for the economics to line up so that using it leads to more money for hospitals somehow

dontreact 5y ago

I think the issue is not with the training step but with the validation step. The only way to build enough trust is to carefully run a prospective clinical trial, and before that you probably need enough retrospective evidence to get some sort of FDA approval. These processes take a lot of time and it’s not surprising that a year and a half after COVID-19 became widespread in the US, we still don’t have a fully validated model.

dontreact 5y ago
In fact the claim the paper makes is that no model has been sufficiently clinically validated. Problem is: how do you get enough buy in from clinicians to run a clinical trial with YOUR model, when there are hundreds of crappy models out there? It takes a lot of time to run these studies for this reason to start with and then depending on the size of the clinical trial, it takes time for the trial to run it’s course.