6 ms·

A popular algorithm to predict sepsis misses most cases and sends false alarms

120 points by hhs 5y ago

bimboler38 5y ago
Paywalled
- ameetgaitonde 5y ago
  Here's an alternative article on the same topic: https://labblog.uofmhealth.org/lab-report/popular-sepsis-prediction-tool-less-accurate-than-claimed https://labblog.uofmhealth.org/lab-report/popular-sepsis-pre...
  - lstamour 5y ago
    Or https://www.wired.com/story/algorithm-predicts-deadly-infections-often-flawed https://www.wired.com/story/algorithm-predicts-deadly-infect...
mooneater 5y ago
Podcast on this was good: https://edhub.ama-assn.org/jn-learning/audio-player/18620183 https://edhub.ama-assn.org/jn-learning/audio-player/18620183
MattGaiser 5y ago
The problem of looking at supposedly representative (but not actually) proxies for actual information is widespread in society, whether it be in hiring or medicine. Here they just replicated all the same issues the current model of diagnosis faced.
- tgv 5y ago
  Looks like they picked data that didn't require any work on their part.
  - MattGaiser 5y ago
    I work for an AI company, so I can easily see that.
- raducu 5y ago
  My wife recently had a burst apendice with perithonitis and a lot of puss in her abdomen and she had no fever. After the surgery my parents(both doctors) told me to watch for fever as a sign of reinfection. I had to remind them there was no fever in the first place and that was likely a bad proxy for infection in her case.
  - tuukkah 5y ago
    As CRP testing is easy and cheap, why do we still cling to heuristics such as measuring fever instead of measuring the infection?
    - rscho 5y ago
      Would it maybe mean that CRP means nothing in isolation? Like a majority of what we test for?
      - tuukkah 5y ago
        As it's easy and cheap, you have no reason to use a measurement in isolation. For instance, you can track it every day to see if the infection is growing or shrinking. Here's a recent example regarding COVID-19: "We realized that whereas a single CRP lab value from hospital admission wasn't very practical as a predictor of who might get sicker, tracking the rate of change from Day 1 to Day 2 or 3 was a very powerful and very clinically predictive test" https://www.sciencedaily.com/releases/2020/11/201105112947.htm https://www.sciencedaily.com/releases/2020/11/201105112947.h...
        rscho 5y ago
        Wow, incredible!
    - raducu 5y ago
      My wife did receive a CRP test that confirmed her infection in the hospital(it was over 200) -- that was among the first things the emergency department checked, then an ultrasound revealed the burst apendice, but I don't think there's any easy way to check it at home. She already had a lot of burst veins from the hospital, so we decided to just do what her surgeon said, basically take amoxicillin for another 7 days and just about it.
- deleted 5y ago
  [deleted]
skmurphy 5y ago
Original article https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2781307 https://jamanetwork.com/journals/jamainternalmedicine/fullar... Not that "Epic" is the name of an Electronic Health Record system that embeds this proprietary algorithm for detecting sepsis. Key Points Question How accurately does the Epic Sepsis Model, a proprietary sepsis prediction model implemented at hundreds of US hospitals, predict the onset of sepsis? Findings In this cohort study of 27 697 patients undergoing 38 455 hospitalizations, sepsis occurred in 7% of the hosptalizations. The Epic Sepsis Model predicted the onset of sepsis with an area under the curve of 0.63, which is substantially worse than the performance reported by its developer. Meaning This study suggests that the Epic Sepsis Model poorly predicts sepsis; its widespread adoption despite poor performance raises fundamental concerns about sepsis management on a national level. Results We identified 27 697 patients who had 38 455 hospitalizations (21 904 women [57%]; median age, 56 years [interquartile range, 35-69 years]) meeting inclusion criteria, of whom sepsis occurred in 2552 (7%). The ESM had a hospitalization-level area under the receiver operating characteristic curve of 0.63 (95% CI, 0.62-0.64). The ESM identified 183 of 2552 patients with sepsis (7%) who did not receive timely administration of antibiotics, highlighting the low sensitivity of the ESM in comparison with contemporary clinical practice. The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue. Conclusions and Relevance This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis. The widespread adoption of the ESM despite its poor performance raises fundamental concerns about sepsis management on a national level.
farseer 5y ago
So that's euphemism for the algorithm doesn't work.
- tuukkah 5y ago
  This case is worse than just not working: the algorithm generates alert fatigue and misleads doctors who would do a better job and come to better decisions without the algorithm. In the hospital context, a fair characterisation might be that the algorithm causes deaths.
  - modmans2nd 5y ago
    Prior to this algorithm most Epic customers had an alert for providers to check for Sepsis in a patient. In most cases the Epic tool is an improvement in over alerting.
    - tuukkah 5y ago
      That provides important context. However, based on the news, Epic has not been honest in their marketing of this feature. If you provide something that is less harmful but still harmful, you have to disclose that the real solution would be something else - fixing the insurance system, liabilities?
  - cratermoon 5y ago
    Just like the days when every program would pop up warning after warning asking, "are you sure?". After a while users started to treat them as annoyances interfering with the task and automatically close them. They became worse than useless because among the noise there would sometimes be serious events that would lead to corruption or loss of data.
stuartbman 5y ago
The issue is that sepsis is very poorly defined, difficult to distinguish from other conditions, and very serious if missed. This means that I get automated alerts for patients all day querying sepsis, but probably only 5% (maybe fewer) have any signs of infection clinically. This reflects clinical practice where we overtreat initially and then row back treatment if things improve. I can go for a run on a hot day- I'll be pyrexic, tachypnoeic, and tachycardic- enough to trigger a sepsis alert. Whereas an elderly patient who is a bit more confused than normal may have a very severe raging infection with few changes in their markers.
- mjburgess 5y ago
  It seems like this should be a sepsis-risk calculation then, based on extrinsic factors (age, admittance reason, etc.); rather than intrinsic factors (hr, bp..). The problem here then, i'd guess, is how many of those extrinsic factors are unmeasurable in practice; and how little data we'd have on them.
  - joshgel 5y ago
    Basically everyone admitted to the hospital is at some level or risk from sepsis which is why the proxy markers become important.
  - stuartbman 5y ago
    Sure, and there's a lot of work in this area to predict sepsis in specific subgroups, but as with a lot of AI I think we're a long, long way off a 'general purpose' sepsis detector. For various reasons sepsis just presents differently in different patient groups so one person's risk factor is another's protective factor
- joshgel 5y ago
  Sepsis is pretty well defined between SIRS and qSOFA. But the definition has a subjective component which has very poor inter-rater reliability. And you’re 100% right that the definition overlaps with other very common conditions, such as pain or having just had an operation.
- 5y ago
dr_house 5y ago
check out these guys: https://www.luminaremed.com https://www.luminaremed.com ; started because of the false alarms

120 points by hhs 5y ago

Paywalled

ameetgaitonde 5y ago
Here's an alternative article on the same topic: https://labblog.uofmhealth.org/lab-report/popular-sepsis-prediction-tool-less-accurate-than-claimed https://labblog.uofmhealth.org/lab-report/popular-sepsis-pre...
- lstamour 5y ago
  Or https://www.wired.com/story/algorithm-predicts-deadly-infections-often-flawed https://www.wired.com/story/algorithm-predicts-deadly-infect...

Podcast on this was good: https://edhub.ama-assn.org/jn-learning/audio-player/18620183 https://edhub.ama-assn.org/jn-learning/audio-player/18620183

MattGaiser 5y ago

The problem of looking at supposedly representative (but not actually) proxies for actual information is widespread in society, whether it be in hiring or medicine. Here they just replicated all the same issues the current model of diagnosis faced.

tgv 5y ago
Looks like they picked data that didn't require any work on their part.
- MattGaiser 5y ago
  I work for an AI company, so I can easily see that.
raducu 5y ago
My wife recently had a burst apendice with perithonitis and a lot of puss in her abdomen and she had no fever. After the surgery my parents(both doctors) told me to watch for fever as a sign of reinfection. I had to remind them there was no fever in the first place and that was likely a bad proxy for infection in her case.
- tuukkah 5y ago
  As CRP testing is easy and cheap, why do we still cling to heuristics such as measuring fever instead of measuring the infection?
  - rscho 5y ago
    Would it maybe mean that CRP means nothing in isolation? Like a majority of what we test for?
    - tuukkah 5y ago
      As it's easy and cheap, you have no reason to use a measurement in isolation. For instance, you can track it every day to see if the infection is growing or shrinking. Here's a recent example regarding COVID-19: "We realized that whereas a single CRP lab value from hospital admission wasn't very practical as a predictor of who might get sicker, tracking the rate of change from Day 1 to Day 2 or 3 was a very powerful and very clinically predictive test" https://www.sciencedaily.com/releases/2020/11/201105112947.htm https://www.sciencedaily.com/releases/2020/11/201105112947.h...
      - rscho 5y ago
        Wow, incredible!
  - raducu 5y ago
    My wife did receive a CRP test that confirmed her infection in the hospital(it was over 200) -- that was among the first things the emergency department checked, then an ultrasound revealed the burst apendice, but I don't think there's any easy way to check it at home. She already had a lot of burst veins from the hospital, so we decided to just do what her surgeon said, basically take amoxicillin for another 7 days and just about it.
deleted 5y ago
[deleted]

skmurphy 5y ago

Original article https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2781307 https://jamanetwork.com/journals/jamainternalmedicine/fullar... Not that "Epic" is the name of an Electronic Health Record system that embeds this proprietary algorithm for detecting sepsis. Key Points Question How accurately does the Epic Sepsis Model, a proprietary sepsis prediction model implemented at hundreds of US hospitals, predict the onset of sepsis? Findings In this cohort study of 27 697 patients undergoing 38 455 hospitalizations, sepsis occurred in 7% of the hosptalizations. The Epic Sepsis Model predicted the onset of sepsis with an area under the curve of 0.63, which is substantially worse than the performance reported by its developer. Meaning This study suggests that the Epic Sepsis Model poorly predicts sepsis; its widespread adoption despite poor performance raises fundamental concerns about sepsis management on a national level. Results We identified 27 697 patients who had 38 455 hospitalizations (21 904 women [57%]; median age, 56 years [interquartile range, 35-69 years]) meeting inclusion criteria, of whom sepsis occurred in 2552 (7%). The ESM had a hospitalization-level area under the receiver operating characteristic curve of 0.63 (95% CI, 0.62-0.64). The ESM identified 183 of 2552 patients with sepsis (7%) who did not receive timely administration of antibiotics, highlighting the low sensitivity of the ESM in comparison with contemporary clinical practice. The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue. Conclusions and Relevance This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis. The widespread adoption of the ESM despite its poor performance raises fundamental concerns about sepsis management on a national level.

farseer 5y ago

So that's euphemism for the algorithm doesn't work.

tuukkah 5y ago
This case is worse than just not working: the algorithm generates alert fatigue and misleads doctors who would do a better job and come to better decisions without the algorithm. In the hospital context, a fair characterisation might be that the algorithm causes deaths.
- modmans2nd 5y ago
  Prior to this algorithm most Epic customers had an alert for providers to check for Sepsis in a patient. In most cases the Epic tool is an improvement in over alerting.
  - tuukkah 5y ago
    That provides important context. However, based on the news, Epic has not been honest in their marketing of this feature. If you provide something that is less harmful but still harmful, you have to disclose that the real solution would be something else - fixing the insurance system, liabilities?
- cratermoon 5y ago
  Just like the days when every program would pop up warning after warning asking, "are you sure?". After a while users started to treat them as annoyances interfering with the task and automatically close them. They became worse than useless because among the noise there would sometimes be serious events that would lead to corruption or loss of data.

stuartbman 5y ago

The issue is that sepsis is very poorly defined, difficult to distinguish from other conditions, and very serious if missed. This means that I get automated alerts for patients all day querying sepsis, but probably only 5% (maybe fewer) have any signs of infection clinically. This reflects clinical practice where we overtreat initially and then row back treatment if things improve. I can go for a run on a hot day- I'll be pyrexic, tachypnoeic, and tachycardic- enough to trigger a sepsis alert. Whereas an elderly patient who is a bit more confused than normal may have a very severe raging infection with few changes in their markers.

mjburgess 5y ago
It seems like this should be a sepsis-risk calculation then, based on extrinsic factors (age, admittance reason, etc.); rather than intrinsic factors (hr, bp..). The problem here then, i'd guess, is how many of those extrinsic factors are unmeasurable in practice; and how little data we'd have on them.
- joshgel 5y ago
  Basically everyone admitted to the hospital is at some level or risk from sepsis which is why the proxy markers become important.
- stuartbman 5y ago
  Sure, and there's a lot of work in this area to predict sepsis in specific subgroups, but as with a lot of AI I think we're a long, long way off a 'general purpose' sepsis detector. For various reasons sepsis just presents differently in different patient groups so one person's risk factor is another's protective factor
joshgel 5y ago
Sepsis is pretty well defined between SIRS and qSOFA. But the definition has a subjective component which has very poor inter-rater reliability. And you’re 100% right that the definition overlaps with other very common conditions, such as pain or having just had an operation.
5y ago

dr_house 5y ago

check out these guys: https://www.luminaremed.com https://www.luminaremed.com ; started because of the false alarms