10 ms·

I don’t like relative risk and relative risk reduction because it tends to overestimate the effectiveness of the intervention. In this case, the absolute risk

I don’t like relative risk and relative risk reduction because it tends to overestimate the effectiveness of the intervention. In this case, the absolute risk when measuring for death in the GIM pre-intervention and GIM post-intervention are 0.0215 (2.15%) and 0.0146 (1.46%) with an absolute risk reduction of 0.0069 (.69%). While the relative risk is 26% across the pre- and post-intervention, the absolute risk reduction is only 0.69% with a NNT (number needed to treat) of 1/156. Which means that 1 patient in 156 was helped by this intervention. In addition, they had 2 false alarms for each true alarm and could suggest that interventions were performed in patients who did not require it — more tests, medications and possibly increased risk from said interventions. This shows that the CHARTwatch ML/AI is not helping at all that much clinically.

swyx 2y ago
this was excellent and necessary context on all fluff pieces like the OP. how can we automate this kind of analysis?
- esoleyman 2y ago
  You can't automate it. You have to look at the data and charts to figure out the specifics you want and then you plug and chug. I haven't looked deeply at this though but whenever researchers use relative risk and it shows a profound effect, I always calculate the absolute risk to make sure that the intervention is effective. Many researchers go to relative risk because it shows better results!
- netruk44 2y ago
  I know everyone hates "I asked ChatGPT" comments but...I feel it's relevant here. It came to roughly the same conclusion as the gp comment when provided with the study PDF. https://chatgpt.com/share/66eb09e3-7a74-8008-afa8-3b60161d2497 https://chatgpt.com/share/66eb09e3-7a74-8008-afa8-3b60161d24... (Though obviously this approach still requires you to go and look at the PDF yourself to make sure it isn't making anything up)
  - staticman2 2y ago
    I think that ChatGPT result is a Rorschach test, it wrote things like "The percentage reduction could be exaggerated based..." Could is doing a lot of work in letting you interpret what it's saying however you like.
- moralestapia 2y ago
  >How can we automate this kind of analysis? Happy to talk about it. Are you in the Healthcare industry?
  - moralestapia 2y ago
    Update (3 days later): It was bs, as usual.
vessenes 2y ago
I like this analysis, although I come to a different conclusion: if AI can give early warning to nursing staff, telling them 'look closer', and over 1/3 of the time, it was right, that seems great. Right now in a 30 bed unit, nurses have to keep track of 30 sets of data. With this, they could focus in on 3 sets when an alarm goes off. I believe these systems will get better over time as well. But, as a patient, I'd 100% take a ward that early AI warning with 66% chance of false positives over one with no such tech. Wouldn't you?
- rscho 2y ago
  No, many people working in clinical units wouldn't. Because of what might happen on false alarms. What GP said: more meds, more interventions. It's not clear at all whether such systems would help with current workflows and current technology. One of the most famous books about medicine says that good medicine is doing nothing as much as possible. It's still very true in 2024, and probably for a long time still.
- _aavaa_ 2y ago
  I would not. High false alarm rates are a problem in all sorts of industry when it comes to warnings and alerts. Too many alerts, or too many false positive alerts cause operators (or nurses in this example) to start ignoring such warnings.
  - tcmart14 2y ago
    This is the real problem. In a perfect world, everyone pays attention to alarms with the same attentiveness all the time. But it just isn't reality. Before going into building software, I was in the Navy and after that did work as a chemical system tech. In the Navy, I worked in JP-5 pumprooms. In both environments we had alarms and in both environments we learned what were nuisance alarms and what weren't, or just took alarms with a grain of salt and there for never paid proper attention to them. That is always the issue with alarms. You have a fine line to walk. Too many alarms and people become complacent and learn to ignore alarms. Too few alarms and you don't draw the attention that is needed.
  - 2y ago
hammock 2y ago
That’s a good point, a similar conversation was had around the Covid jabs, with some research re vaccine mandates concluding “heads of governments, schools, healthcare facilities, and private businesses (were) misled by the vaccines’ reported 95% relative risk reduction” https://www.sciencedirect.com/science/article/pii/S2772653322000740 https://www.sciencedirect.com/science/article/pii/S277265332...
fsckboy 2y ago
>1 patient in 156 was helped by this intervention the headline says we're talking about death: does that mean 1 life was saved for every 156 patients? >In addition, they had 2 false alarms for each true alarm and ... and possibly increased risk from said interventions but wouldn't this study have captured any deaths from those interventions, so the 1 out of 156 life-savings was net?
- rscho 2y ago
  Would you suffer serious nonlethal complications from false alarm to (maybe) save your room neighbour that you've never met before? This wouldn't be captured.
  - fsckboy 2y ago
    an individual would probably not make that choice, but the population could easily, the insurance company might, religious leaders might, etc. this study was measuring deaths and what you are suggesting would be outside this study, but it could be measured also.

swyx 2y ago

this was excellent and necessary context on all fluff pieces like the OP. how can we automate this kind of analysis?

esoleyman 2y ago
You can't automate it. You have to look at the data and charts to figure out the specifics you want and then you plug and chug. I haven't looked deeply at this though but whenever researchers use relative risk and it shows a profound effect, I always calculate the absolute risk to make sure that the intervention is effective. Many researchers go to relative risk because it shows better results!
netruk44 2y ago
I know everyone hates "I asked ChatGPT" comments but...I feel it's relevant here. It came to roughly the same conclusion as the gp comment when provided with the study PDF. https://chatgpt.com/share/66eb09e3-7a74-8008-afa8-3b60161d2497 https://chatgpt.com/share/66eb09e3-7a74-8008-afa8-3b60161d24... (Though obviously this approach still requires you to go and look at the PDF yourself to make sure it isn't making anything up)
- staticman2 2y ago
  I think that ChatGPT result is a Rorschach test, it wrote things like "The percentage reduction could be exaggerated based..." Could is doing a lot of work in letting you interpret what it's saying however you like.
moralestapia 2y ago
>How can we automate this kind of analysis? Happy to talk about it. Are you in the Healthcare industry?
- moralestapia 2y ago
  Update (3 days later): It was bs, as usual.

vessenes 2y ago

I like this analysis, although I come to a different conclusion: if AI can give early warning to nursing staff, telling them 'look closer', and over 1/3 of the time, it was right, that seems great. Right now in a 30 bed unit, nurses have to keep track of 30 sets of data. With this, they could focus in on 3 sets when an alarm goes off. I believe these systems will get better over time as well. But, as a patient, I'd 100% take a ward that early AI warning with 66% chance of false positives over one with no such tech. Wouldn't you?

rscho 2y ago
No, many people working in clinical units wouldn't. Because of what might happen on false alarms. What GP said: more meds, more interventions. It's not clear at all whether such systems would help with current workflows and current technology. One of the most famous books about medicine says that good medicine is doing nothing as much as possible. It's still very true in 2024, and probably for a long time still.
_aavaa_ 2y ago
I would not. High false alarm rates are a problem in all sorts of industry when it comes to warnings and alerts. Too many alerts, or too many false positive alerts cause operators (or nurses in this example) to start ignoring such warnings.
- tcmart14 2y ago
  This is the real problem. In a perfect world, everyone pays attention to alarms with the same attentiveness all the time. But it just isn't reality. Before going into building software, I was in the Navy and after that did work as a chemical system tech. In the Navy, I worked in JP-5 pumprooms. In both environments we had alarms and in both environments we learned what were nuisance alarms and what weren't, or just took alarms with a grain of salt and there for never paid proper attention to them. That is always the issue with alarms. You have a fine line to walk. Too many alarms and people become complacent and learn to ignore alarms. Too few alarms and you don't draw the attention that is needed.
- 2y ago

hammock 2y ago

That’s a good point, a similar conversation was had around the Covid jabs, with some research re vaccine mandates concluding “heads of governments, schools, healthcare facilities, and private businesses (were) misled by the vaccines’ reported 95% relative risk reduction” https://www.sciencedirect.com/science/article/pii/S2772653322000740 https://www.sciencedirect.com/science/article/pii/S277265332...

fsckboy 2y ago

>1 patient in 156 was helped by this intervention the headline says we're talking about death: does that mean 1 life was saved for every 156 patients? >In addition, they had 2 false alarms for each true alarm and ... and possibly increased risk from said interventions but wouldn't this study have captured any deaths from those interventions, so the 1 out of 156 life-savings was net?

rscho 2y ago
Would you suffer serious nonlethal complications from false alarm to (maybe) save your room neighbour that you've never met before? This wouldn't be captured.
- fsckboy 2y ago
  an individual would probably not make that choice, but the population could easily, the insurance company might, religious leaders might, etc. this study was measuring deaths and what you are suggesting would be outside this study, but it could be measured also.