7 ms·

Apple Watch can detect arrhythmia with 97% accuracy, study says

342 points by brandonb 9y ago

openasocket 9y ago
They aren't clear what they mean by "97% accuracy". Does that mean 97% of people with arrhythmia are correctly diagnosed, or 97% of people are correctly diagnosed or not-diagnosed? If it's the latter, it's not very helpful at all. The number of people in the general population with arrhythmia is significantly less than 3%, so if this Apple Watch test says you have arrhythmia it is far more likely to be a false positive than a true positive.
- brandonb 9y ago
  Here, 97% accuracy refers to a c-statistic (area under the ROC curve) of 0.9740. An example operating point would be 98% sensitivity with 90% specificity. For reducing false positives, rather than starting with the general population, it'd be natural to start with a higher risk sub-group, e.g., people with a high CHA2DS2-Vasc score. Earlier this week, Circulation published a review screening for atrial fibrillation: circ.ahajournals.org/content/135/19/1851.full?ijkey=StzSPk8eljGaP2G&keytype=ref
- bluGill 9y ago
  No, accuracy doesn't tell us anything about usefulness. If the test is cheap and it brings in people for a better test and thus saves lives it is useful. The only real danger is if people decide that since the watch says there is nothing wrong so they can ignore other symptoms. It is something like chest pain: most of the time chest pain is not a symptom of a heart attack, but it is best indicator we have so you go to the emergency room when you have chest pain. Doctors there can evaluate your situation.
  - nraynaud 9y ago
    On a related note, I had a very rare stroke last week, only symptom: headache for 3 days. The doctor sent me for a CT scan without any idea of what to look for. The only clue: it was not a migraine because I missed any neurological anomaly. On the other hand, stroke prevalence is rare, 1.3% for men, and generally don't look like mine, and my kind is 2% of all strokes, it would have been completely weird to suspect it with such a low probability and such a non-specific evidence. Long story short, I almost got sent home with an aspirin while I had an extensive cerebral veinous thrombosis.
    - TheOtherHobbes 9y ago
      Having spent a couple of days in a stroke ward over Christmas, the low prevalence of strokes should never be used as a reason not to test where possible. I suspect a lot of people don't understand how serious and crippling - mentally and physically - a bad stroke can be. If you're lucky a bad stroke kills you. If you're not lucky you lose a good part of your brain and motor function. In practice this means you can be left unable to move some or all of your limbs, unable to talk, unable to hear, unable to understand what's happening around you, and perhaps unable to see. It's no exaggeration to say that it can turn life into a nightmare. Anything that makes this less common and less likely is a good thing.
  - mattybrennan 9y ago
    Disagree. Note that both mammogram and colonoscopy frequency recommendations have both been reduced recently, both because the cost of screening and because of the consequences of false positives resulting in unnecessary medical intervention
    - bluGill 9y ago
      That is a useful point, but not reason to disagree. In this case a cheap test that indicates you need a better test is useful. The risk of false positives needs to be considered, but that is for medical professionals. Even if the ultimate answer is for most people nothing more is done it still gets people into the doctor to explain what else they need to look for before a test is done.
brandonb 9y ago
(Cardiogram Co-Founder here) Let me know if any of you have questions on the study, app, or deep learning algorithm. My colleague Avesh wrote a post with a little more technical detail here: https://blog.cardiogr.am/applying-artificial-intelligence-in-medicine-our-early-results-78bfe7605d32 https://blog.cardiogr.am/applying-artificial-intelligence-in...
- deleted 9y ago
  [deleted]
- cheath 9y ago
  This is really exciting! Quick question for you. Have you experimented at all with using the Apple Watch to measure blood pressure? I have done some reading that suggests the optical sensor could measure blood pressure with some accuracy, but that Apple is hesitant to release it as a feature due to regulatory and accuracy concerns. It's my #1 wished for feature.
  - brandonb 9y ago
    Is it possible you're talking about blood oxygen saturation? http://www.cultofmac.com/320322/apple-watch-sensors-are-capable-of-measuring-blood-oxygen/ http://www.cultofmac.com/320322/apple-watch-sensors-are-capa...
  - JshWright 9y ago
    Measuring blood pressure (non-invasively) requires applying pressure to the artery and determining at what point that pressure obstructs blood flow. That would be a) challenging to do with a watch, and b) really annoying to the user.
    - captainmuon 9y ago
      Not entirely true, there is a clever method that works without (pulse transit time measurement). You measure the electrical signal of the heartbeat, and the acoustic signal, and take advantage of the fact that the acoustic signal propagates slower in blood than the electrical signal propagates through nerves. This difference is directly related to blood pressure! I don't know if you can get away with one measurement on the wrist + some clever calibration, or if you need two points (e.g. one close to the heart). If you take two measurements, you can apparently also calculate the aortal pressure, which is different from the pressure in your arm, and hard to obtain non-invasively. (I'm not an expert so I'm sorry if I got something slightly wrong :-) .) There was a startup that worked on using this, but they failed (due to financial but also regulatory reasons), and then remotely bricked all their sold devices... I really wish someone would develop this further. Even if it is not as accurate as a normal measurement, the implications would be huge. There are so many people running around with hypertension who have no idea. I also don't see a risk in false positives in this case, since in principle everybody is recommended to have their blood pressure checked - false positives who go to the doctor are then just like people who read an article and go to the doctor, and are weeded out there. False negatives might be a problem - but if you don't advertise it as a blood pressure measurement tool, but just implement it as an additional warning in a smart watch, you'd reduce the false sense of security people would get if it didn't work properly.
jasonmp85 9y ago
Probably want to update the title, which still refers to the more general 'arrhythmia'. It sounds like Cardiogram's work is mostly focused on afib?
- brandonb 9y ago
  First validation study is on atrial fibrillation, although we've had users who have discovered other arrhythmias through the app. Here's on example of a person who discovered supraventricular tachycardia: https://blog.cardiogr.am/my-apple-watch-saved-my-life-a6125663b2bb https://blog.cardiogr.am/my-apple-watch-saved-my-life-a61256...
threeseed 9y ago
This is really great news but I wish it was actually available. The Apple Watch right now is largely useless. It captures all of these health metrics but then does absolutely nothing with it. It really is desperate for some actual killer health use cases.
- brandonb 9y ago
  Thanks! You can use the Cardiogram app today to understand your heart rate data: https://itunes.apple.com/us/app/cardiogram/id1000017994?ls=1&mt=8 https://itunes.apple.com/us/app/cardiogram/id1000017994?ls=1... We'll be incorporating these results into the app itself over time. But as with anything in medicine... it's ready, aim, aim, aim, aim, aim... fire! :)
peterjlee 9y ago
For those who's going to mention Bayes' Theorem regarding medical tests, here's a link to save you a Google search https://en.wikipedia.org/wiki/Bayes%27_theorem#Drug_testing https://en.wikipedia.org/wiki/Bayes%27_theorem#Drug_testing
pak 9y ago
We need to see the full, published study and its methods (particularly around recruitment and exclusion criteria) before we can judge it properly. Until then, the presented statistics about accuracy, sensitivity, and specificity potentially bear no relation to real world usage, if the cohort and data quality were tightly controlled, as you'd expect for an initial study involving the makers of the algorithm. A few other thoughts: 1. Even at 98% sensitivity and 90% specificity [0], which I don't think would hold up with real world usage in casual, healthy users, if AFib has a prevalence of roughly 2-3% [1] then by a quick back of the envelope calculation a positive test result is still 5× more likely to be a false positive than a true positive. With those odds, I don't think many cardiologists are going to answer the phone. You'd still need an EKG to diagnose AFib. 2. There is huge variance among people's real world use of wearable sensors, and also among the quality of the sensors. (Imagine people that wear the watch looser, sweat more, have different skin, move it around a lot, etc.) You'd likely need to do an open, third-party validation study of the accuracy of the sensors in the Apple Watch before you can expect doctors to use the data. My understanding is that the Apple Watch sensors are actually pretty good compared to other wearable sensors, but I don't know of any rigorous study of that compares them to an EKG. 3. Obviously, this is only for AFib. AFib is a sweet corner case in terms of extrapolating from heart rate to arrhythmia, because it's a rapid & irregular rhythm that probably contains some subpatterns in beats that are hard for humans to appreciate. As others—including Cardiogram themselves [2]—have pointed out previously, many serious arrhythmias are not possible to detect with only an optical heart rate sensor. [0]: https://blog.cardiogr.am/applying-artificial-intelligence-in-medicine-our-early-results-78bfe7605d32 https://blog.cardiogr.am/applying-artificial-intelligence-in... [1]: https://www.ncbi.nlm.nih.gov/pubmed/24966695 https://www.ncbi.nlm.nih.gov/pubmed/24966695 [2]: https://blog.cardiogr.am/what-do-normal-and-abnormal-heart-rhythms-look-like-on-apple-watch-7b33b4a8ecfa#fabf https://blog.cardiogr.am/what-do-normal-and-abnormal-heart-r...
- seizethecheese 9y ago
  > by a quick back of the envelope calculation a positive test result is still 5× more likely to be a false positive than a true positive. With those odds, I don't think many cardiologists are going to answer the phone. You'd still need an EKG to diagnose AFib. This is a good point, and certainly nobody should go directly to a cardiologist based on these results. It seems that this would be a good system to recommend that people get an EKG done, though.
robbiep 9y ago
With due respect to the Cardiogram developers (hi guys) as a doctor I really can't see a huge amount of value in this - my patients who present to emergency departments with paroxysmal AF are all anticoagulanted or on rate control and there has only been one instance in the last 3 years and many thousand patients when someone has presented to emergency and we have had to run through the full spectrum of echo -> anticoagulante -> cardiovert. To the founders: what do you see as being he end game here? Are you just looking for validation (of the concept in itself, not the app- I use it on my watch)? Is the US market so different that this is particularly useful and cost effective for detection? Do you see this as eventually displaying early warning in the instance of an early warning? Thanks and I don't mean to denegrate your efforts, but I do see lots of Consumer Med tech as solving a problem that really isn't creating value (i.e. Proliferation of devices,wearables and algorithms that proclaim the ability to help with X but are really marginally helpful at best) and I'm wondering if I'm missing something about the actual medical benefit, or whether what I feel like is true- that they aren't after being a medical device at all but instead are chasing the consumer dollars by making medical claims
- pm90 9y ago
  Not the developer or founder, but a user of Apple Watch. I would like to see your questions answered as well. But as a hopeful consumer, I hope wearable devices / medtech startups direct more attention and funding to improving sensors and associated detection algorithms to such an extent that at some point in the future, it would actually be useful to real doctors and offer real medical benefits.
- ithinkinstereo 9y ago
  Wouldn't this help with pre-stroke detection? The article mentioned that 1 in 4 strokes are caused by AFib. I would imagine this would be helpful for in-patient Neuro-ICU and stroke hospitalists. On the patient side, shouldn't something like this - assuming the monitoring is in real-time - provide a preventative/detection benefits for patients with a high-risk for stroke?
  - brandonb 9y ago
revelation 9y ago
Always great to have percentages reported for a sample size <100.
- matt4077 9y ago
  ? I don't get that logic... And neither do half of my friends. The other 50% actually don't get it, either. But I did only ask <100 people.
deepsun 9y ago
Am I the only one who thinks that 97% accuracy (1/30 chance of an error) is very bad in medical diagnosis? At least I'd expect something like 99.9% accuracy (1/1000 chance of an error) when someone gives me my own heart diagnosis.
- brandonb 9y ago
  You'll find most most medical algorithms pretty disappointing. :) For example, the algorithm in implantable cardioverter defibrillators generates unnecessary shocks in 1 in 6 patients. Its accuracy is getting worse over time: http://www.reuters.com/article/us-untimely-jolts-idUSTRE70O7IP20110125 http://www.reuters.com/article/us-untimely-jolts-idUSTRE70O7...
nonbel 9y ago
Accuracy is a useless metric for something like this. If you have binary data filled with 97% zeros (ie most of the time it is not arrhythmia) you can use the sophisticated machine learning technique of: if(TRUE) return(0) This will give you 97% accuracy. EDIT: I just read the headline earlier. Now after checking: >"The study involved 6,158 participants recruited through the Cardiogram app on Apple Watch. Most of the participants in the UCSF Health eHeart study had normal EKG readings. However, 200 of them had been diagnosed with paroxysmal atrial fibrillation (an abnormal heartbeat). Engineers then trained a deep neural network to identify these abnormal heart rhythms from Apple Watch heart rate data." So 1 - 200/6158 = 0.9675219. My method performs just as well as theirs if we round to the nearest percent. This is ridiculous.
- linkregister 9y ago
  Then it's missed 100% of the arrythmatic cases. But I appreciate the sentiment of what you're trying to say.
- ericjang 9y ago
  I think the article is not precise with their wording, but the 97% is actually recall (i.e. detects 97% of the positives).
  - nonbel 9y ago
    Can you quote the part that leads you to think that? At first I was just commenting on the title, but see the edit. I would agree, my interpretation makes this the most ridiculous hype I have ever seen, so maybe I missed something.
    - ericjang 9y ago
      No quote in particular, but it seems unlikely that they would miss such a blatant dataset bias and fail to instrument their ML models with metrics besides accuracy. But then again, welcome to Silicon Valley :) 1 - 200/6158 = 97% is indeed a pretty suspicious coincidence though. I would assume/hope that they've shuffled a big dataset of recorded heart events (like the image in the TC article) and that the 200 people diagnosed with paroxysmal atrial fibrillation only rarely experience AF, so the number of true positives is probably far smaller than 1% of the dataset.
chris_va 9y ago
If I am reading this correctly, there were 6,158 patents with ~200 true positives, so approximately 3% of the population. 98.04% sensitivity (recall) and 90.2% specificity (%true negatives) leads to... ~4 false positives for each true positive. That isn't bad, all things considered, but still a long way to go.
dennyabraham 9y ago
Despite all the caveats around this work, early detection is one of the reasons HR trackers are a great investment. You can't manage what you can't monitor
zobzu 9y ago
I've arrhythmia and I can see it on my Garmin watch by looking at the data, as well as .. my phone. It doesn't help much though, because I don't know if its good or bad (well, actually I know but not because of the watch data). Doctors are still needed for this, and generally that includes a bunch of controlled tests and people listening to your heart while also gathering data (similarly to the watch albeit with a more precise apparatus) I guess it can help to tell people they might wanna see a doctor if they haven't though.
paul7986 9y ago
I'm seeing more and more people wearing Apple watches and more similar PR stories like this. I saw one story that talked about a guy whose car flipped and he was unable to reach his phone but thanks to his watch he was able to call for help.
caycep 9y ago
the one question I have is - how much better is the DNN vs. just simple rhythm analysis, i.e. periodicity, etc.

342 points by brandonb 9y ago

They aren't clear what they mean by "97% accuracy". Does that mean 97% of people with arrhythmia are correctly diagnosed, or 97% of people are correctly diagnosed or not-diagnosed? If it's the latter, it's not very helpful at all. The number of people in the general population with arrhythmia is significantly less than 3%, so if this Apple Watch test says you have arrhythmia it is far more likely to be a false positive than a true positive.

brandonb 9y ago
Here, 97% accuracy refers to a c-statistic (area under the ROC curve) of 0.9740. An example operating point would be 98% sensitivity with 90% specificity. For reducing false positives, rather than starting with the general population, it'd be natural to start with a higher risk sub-group, e.g., people with a high CHA2DS2-Vasc score. Earlier this week, Circulation published a review screening for atrial fibrillation: circ.ahajournals.org/content/135/19/1851.full?ijkey=StzSPk8eljGaP2G&keytype=ref
bluGill 9y ago
No, accuracy doesn't tell us anything about usefulness. If the test is cheap and it brings in people for a better test and thus saves lives it is useful. The only real danger is if people decide that since the watch says there is nothing wrong so they can ignore other symptoms. It is something like chest pain: most of the time chest pain is not a symptom of a heart attack, but it is best indicator we have so you go to the emergency room when you have chest pain. Doctors there can evaluate your situation.
- nraynaud 9y ago
  On a related note, I had a very rare stroke last week, only symptom: headache for 3 days. The doctor sent me for a CT scan without any idea of what to look for. The only clue: it was not a migraine because I missed any neurological anomaly. On the other hand, stroke prevalence is rare, 1.3% for men, and generally don't look like mine, and my kind is 2% of all strokes, it would have been completely weird to suspect it with such a low probability and such a non-specific evidence. Long story short, I almost got sent home with an aspirin while I had an extensive cerebral veinous thrombosis.
  - TheOtherHobbes 9y ago
    Having spent a couple of days in a stroke ward over Christmas, the low prevalence of strokes should never be used as a reason not to test where possible. I suspect a lot of people don't understand how serious and crippling - mentally and physically - a bad stroke can be. If you're lucky a bad stroke kills you. If you're not lucky you lose a good part of your brain and motor function. In practice this means you can be left unable to move some or all of your limbs, unable to talk, unable to hear, unable to understand what's happening around you, and perhaps unable to see. It's no exaggeration to say that it can turn life into a nightmare. Anything that makes this less common and less likely is a good thing.
- mattybrennan 9y ago
  Disagree. Note that both mammogram and colonoscopy frequency recommendations have both been reduced recently, both because the cost of screening and because of the consequences of false positives resulting in unnecessary medical intervention
  - bluGill 9y ago
    That is a useful point, but not reason to disagree. In this case a cheap test that indicates you need a better test is useful. The risk of false positives needs to be considered, but that is for medical professionals. Even if the ultimate answer is for most people nothing more is done it still gets people into the doctor to explain what else they need to look for before a test is done.

brandonb 9y ago

(Cardiogram Co-Founder here) Let me know if any of you have questions on the study, app, or deep learning algorithm. My colleague Avesh wrote a post with a little more technical detail here: https://blog.cardiogr.am/applying-artificial-intelligence-in-medicine-our-early-results-78bfe7605d32 https://blog.cardiogr.am/applying-artificial-intelligence-in...

deleted 9y ago
[deleted]
cheath 9y ago
This is really exciting! Quick question for you. Have you experimented at all with using the Apple Watch to measure blood pressure? I have done some reading that suggests the optical sensor could measure blood pressure with some accuracy, but that Apple is hesitant to release it as a feature due to regulatory and accuracy concerns. It's my #1 wished for feature.
- brandonb 9y ago
  Is it possible you're talking about blood oxygen saturation? http://www.cultofmac.com/320322/apple-watch-sensors-are-capable-of-measuring-blood-oxygen/ http://www.cultofmac.com/320322/apple-watch-sensors-are-capa...
- JshWright 9y ago
  Measuring blood pressure (non-invasively) requires applying pressure to the artery and determining at what point that pressure obstructs blood flow. That would be a) challenging to do with a watch, and b) really annoying to the user.
  - captainmuon 9y ago
    Not entirely true, there is a clever method that works without (pulse transit time measurement). You measure the electrical signal of the heartbeat, and the acoustic signal, and take advantage of the fact that the acoustic signal propagates slower in blood than the electrical signal propagates through nerves. This difference is directly related to blood pressure! I don't know if you can get away with one measurement on the wrist + some clever calibration, or if you need two points (e.g. one close to the heart). If you take two measurements, you can apparently also calculate the aortal pressure, which is different from the pressure in your arm, and hard to obtain non-invasively. (I'm not an expert so I'm sorry if I got something slightly wrong :-) .) There was a startup that worked on using this, but they failed (due to financial but also regulatory reasons), and then remotely bricked all their sold devices... I really wish someone would develop this further. Even if it is not as accurate as a normal measurement, the implications would be huge. There are so many people running around with hypertension who have no idea. I also don't see a risk in false positives in this case, since in principle everybody is recommended to have their blood pressure checked - false positives who go to the doctor are then just like people who read an article and go to the doctor, and are weeded out there. False negatives might be a problem - but if you don't advertise it as a blood pressure measurement tool, but just implement it as an additional warning in a smart watch, you'd reduce the false sense of security people would get if it didn't work properly.

jasonmp85 9y ago

Probably want to update the title, which still refers to the more general 'arrhythmia'. It sounds like Cardiogram's work is mostly focused on afib?

brandonb 9y ago
First validation study is on atrial fibrillation, although we've had users who have discovered other arrhythmias through the app. Here's on example of a person who discovered supraventricular tachycardia: https://blog.cardiogr.am/my-apple-watch-saved-my-life-a6125663b2bb https://blog.cardiogr.am/my-apple-watch-saved-my-life-a61256...

threeseed 9y ago

This is really great news but I wish it was actually available. The Apple Watch right now is largely useless. It captures all of these health metrics but then does absolutely nothing with it. It really is desperate for some actual killer health use cases.

brandonb 9y ago
Thanks! You can use the Cardiogram app today to understand your heart rate data: https://itunes.apple.com/us/app/cardiogram/id1000017994?ls=1&mt=8 https://itunes.apple.com/us/app/cardiogram/id1000017994?ls=1... We'll be incorporating these results into the app itself over time. But as with anything in medicine... it's ready, aim, aim, aim, aim, aim... fire! :)

peterjlee 9y ago

For those who's going to mention Bayes' Theorem regarding medical tests, here's a link to save you a Google search https://en.wikipedia.org/wiki/Bayes%27_theorem#Drug_testing https://en.wikipedia.org/wiki/Bayes%27_theorem#Drug_testing

pak 9y ago

We need to see the full, published study and its methods (particularly around recruitment and exclusion criteria) before we can judge it properly. Until then, the presented statistics about accuracy, sensitivity, and specificity potentially bear no relation to real world usage, if the cohort and data quality were tightly controlled, as you'd expect for an initial study involving the makers of the algorithm. A few other thoughts: 1. Even at 98% sensitivity and 90% specificity [0], which I don't think would hold up with real world usage in casual, healthy users, if AFib has a prevalence of roughly 2-3% [1] then by a quick back of the envelope calculation a positive test result is still 5× more likely to be a false positive than a true positive. With those odds, I don't think many cardiologists are going to answer the phone. You'd still need an EKG to diagnose AFib. 2. There is huge variance among people's real world use of wearable sensors, and also among the quality of the sensors. (Imagine people that wear the watch looser, sweat more, have different skin, move it around a lot, etc.) You'd likely need to do an open, third-party validation study of the accuracy of the sensors in the Apple Watch before you can expect doctors to use the data. My understanding is that the Apple Watch sensors are actually pretty good compared to other wearable sensors, but I don't know of any rigorous study of that compares them to an EKG. 3. Obviously, this is only for AFib. AFib is a sweet corner case in terms of extrapolating from heart rate to arrhythmia, because it's a rapid & irregular rhythm that probably contains some subpatterns in beats that are hard for humans to appreciate. As others—including Cardiogram themselves [2]—have pointed out previously, many serious arrhythmias are not possible to detect with only an optical heart rate sensor. [0]: https://blog.cardiogr.am/applying-artificial-intelligence-in-medicine-our-early-results-78bfe7605d32 https://blog.cardiogr.am/applying-artificial-intelligence-in... [1]: https://www.ncbi.nlm.nih.gov/pubmed/24966695 https://www.ncbi.nlm.nih.gov/pubmed/24966695 [2]: https://blog.cardiogr.am/what-do-normal-and-abnormal-heart-rhythms-look-like-on-apple-watch-7b33b4a8ecfa#fabf https://blog.cardiogr.am/what-do-normal-and-abnormal-heart-r...

seizethecheese 9y ago
> by a quick back of the envelope calculation a positive test result is still 5× more likely to be a false positive than a true positive. With those odds, I don't think many cardiologists are going to answer the phone. You'd still need an EKG to diagnose AFib. This is a good point, and certainly nobody should go directly to a cardiologist based on these results. It seems that this would be a good system to recommend that people get an EKG done, though.

robbiep 9y ago

With due respect to the Cardiogram developers (hi guys) as a doctor I really can't see a huge amount of value in this - my patients who present to emergency departments with paroxysmal AF are all anticoagulanted or on rate control and there has only been one instance in the last 3 years and many thousand patients when someone has presented to emergency and we have had to run through the full spectrum of echo -> anticoagulante -> cardiovert. To the founders: what do you see as being he end game here? Are you just looking for validation (of the concept in itself, not the app- I use it on my watch)? Is the US market so different that this is particularly useful and cost effective for detection? Do you see this as eventually displaying early warning in the instance of an early warning? Thanks and I don't mean to denegrate your efforts, but I do see lots of Consumer Med tech as solving a problem that really isn't creating value (i.e. Proliferation of devices,wearables and algorithms that proclaim the ability to help with X but are really marginally helpful at best) and I'm wondering if I'm missing something about the actual medical benefit, or whether what I feel like is true- that they aren't after being a medical device at all but instead are chasing the consumer dollars by making medical claims

pm90 9y ago
Not the developer or founder, but a user of Apple Watch. I would like to see your questions answered as well. But as a hopeful consumer, I hope wearable devices / medtech startups direct more attention and funding to improving sensors and associated detection algorithms to such an extent that at some point in the future, it would actually be useful to real doctors and offer real medical benefits.
ithinkinstereo 9y ago
Wouldn't this help with pre-stroke detection? The article mentioned that 1 in 4 strokes are caused by AFib. I would imagine this would be helpful for in-patient Neuro-ICU and stroke hospitalists. On the patient side, shouldn't something like this - assuming the monitoring is in real-time - provide a preventative/detection benefits for patients with a high-risk for stroke?
- brandonb 9y ago

revelation 9y ago

Always great to have percentages reported for a sample size <100.

matt4077 9y ago
? I don't get that logic... And neither do half of my friends. The other 50% actually don't get it, either. But I did only ask <100 people.

deepsun 9y ago

Am I the only one who thinks that 97% accuracy (1/30 chance of an error) is very bad in medical diagnosis? At least I'd expect something like 99.9% accuracy (1/1000 chance of an error) when someone gives me my own heart diagnosis.

brandonb 9y ago
You'll find most most medical algorithms pretty disappointing. :) For example, the algorithm in implantable cardioverter defibrillators generates unnecessary shocks in 1 in 6 patients. Its accuracy is getting worse over time: http://www.reuters.com/article/us-untimely-jolts-idUSTRE70O7IP20110125 http://www.reuters.com/article/us-untimely-jolts-idUSTRE70O7...

nonbel 9y ago

Accuracy is a useless metric for something like this. If you have binary data filled with 97% zeros (ie most of the time it is not arrhythmia) you can use the sophisticated machine learning technique of: if(TRUE) return(0) This will give you 97% accuracy. EDIT: I just read the headline earlier. Now after checking: >"The study involved 6,158 participants recruited through the Cardiogram app on Apple Watch. Most of the participants in the UCSF Health eHeart study had normal EKG readings. However, 200 of them had been diagnosed with paroxysmal atrial fibrillation (an abnormal heartbeat). Engineers then trained a deep neural network to identify these abnormal heart rhythms from Apple Watch heart rate data." So 1 - 200/6158 = 0.9675219. My method performs just as well as theirs if we round to the nearest percent. This is ridiculous.

linkregister 9y ago
Then it's missed 100% of the arrythmatic cases. But I appreciate the sentiment of what you're trying to say.
ericjang 9y ago
I think the article is not precise with their wording, but the 97% is actually recall (i.e. detects 97% of the positives).
- nonbel 9y ago
  Can you quote the part that leads you to think that? At first I was just commenting on the title, but see the edit. I would agree, my interpretation makes this the most ridiculous hype I have ever seen, so maybe I missed something.
  - ericjang 9y ago
    No quote in particular, but it seems unlikely that they would miss such a blatant dataset bias and fail to instrument their ML models with metrics besides accuracy. But then again, welcome to Silicon Valley :) 1 - 200/6158 = 97% is indeed a pretty suspicious coincidence though. I would assume/hope that they've shuffled a big dataset of recorded heart events (like the image in the TC article) and that the 200 people diagnosed with paroxysmal atrial fibrillation only rarely experience AF, so the number of true positives is probably far smaller than 1% of the dataset.

chris_va 9y ago

If I am reading this correctly, there were 6,158 patents with ~200 true positives, so approximately 3% of the population. 98.04% sensitivity (recall) and 90.2% specificity (%true negatives) leads to... ~4 false positives for each true positive. That isn't bad, all things considered, but still a long way to go.

dennyabraham 9y ago

Despite all the caveats around this work, early detection is one of the reasons HR trackers are a great investment. You can't manage what you can't monitor

zobzu 9y ago

I've arrhythmia and I can see it on my Garmin watch by looking at the data, as well as .. my phone. It doesn't help much though, because I don't know if its good or bad (well, actually I know but not because of the watch data). Doctors are still needed for this, and generally that includes a bunch of controlled tests and people listening to your heart while also gathering data (similarly to the watch albeit with a more precise apparatus) I guess it can help to tell people they might wanna see a doctor if they haven't though.

paul7986 9y ago

I'm seeing more and more people wearing Apple watches and more similar PR stories like this. I saw one story that talked about a guy whose car flipped and he was unable to reach his phone but thanks to his watch he was able to call for help.

caycep 9y ago

the one question I have is - how much better is the DNN vs. just simple rhythm analysis, i.e. periodicity, etc.