5 ms·

97% accuracy seems high, but with an approximate sales number of 10 million Apple watches, it will give 300 thousand users an incorrect diagnosis.

outworlder 9y ago
Which is not so bad. They can follow up and figure it out. Some people will go for decades before getting a proper diagnostic - or even die before they get one.
- ekianjo 9y ago
  > Which is not so bad. They can follow up and figure it out. Following up is not free. So that's cost incurred for the ones who have no problem in the first place.
  - aequitas 9y ago
    Think about the cost of not following up. Personal health, productivity, life quality. Also here in the Netherlands we have healthcare. I'd rather see these emerging devices offer good warnings to go see a doctor then people not going at all or going to often for every little thing that they feel and googled on the internet because they are to unsure.
    - vidarh 9y ago
      See my long response elsewhere. Depending on incidence rates and whether or not the effect of following up earlier is sufficiently large, there can - and sometimes is - a net detrimental effect to wrongfully diagnosing someone with a condition. Even when following up is relatively simple and risk free, and lives are saved by finding conditions early. E.g. cause an extra million doctors visits due to wrongly diagnosing people, and the incident rate of e.g. extra infections transmitted from another patient, or compounded misdiagnosis leading to extra interventions which have some risk of failure, or even just accidents on the way to the doctors office, very quickly leads to extra harm just from the sheer volume. That extra harm might be worth it, if the early diagnosis of others save enough lives - the same way seat belts kills some people but save far more. But it also might not, e.g. if the condition is rare enough, or early detection have little enough effect in improving outcomes.
- edejong 9y ago
  Unfortunately, most arrhythmias are benign and do not lead to further treatment from a physician. So you are taking 300,000 worried people wrongly to a general practitioner, who in fact can do nothing but write down the event and perhaps correlate it with other data.
  - vidarh 9y ago
    It's worth always keeping this in mind, and not just taking extra worried people to the gp, but also giving unwarranted peace of mind to those who have a problem. Even just having them get in a car and go to their GP, or increase their stress levels, have the potential to cause harm. It may be an exceedingly low risk, but applied to a large enough number of people and it starts adding up. When applied to conditions that not everyone has, this can easily tilt the balance. Consider a hypothetical condition and a population of 1000000. If 1% has the condition at any one time, and the accuracy of the test is 97%, then 10,000 should have the condition. Of those we'd expect 9,700 to get the correct diagnosis; 300 would remain undiagnosed; But 3% of those who don't have the condition - an additional 29,700 - will be wrongly diagnosed. Let's say this is a condition where 10% would die untreated without this new intervention, it'd be tempting to suggest we've saved 970 people. But early detection might not make that much of a difference - for many things the most severe cases will be noticed anyway and/or late detection is still sufficient to save the patient. Let's assume for this hypothetical condition, earlier detection will save 10% more of those. In the above that'd save 97 people. Awesome. But lets say it causes 10% more of the undiagnosed ones to die because they wait longer. That's 3 more dead. So net 94? But if only 0.33% of those wrongfully diagnosed end up dying as the result of the misdiagnosis, be it car crashes on the way to the GP, or the visit leading to a follow up error leading to an intervention that is generally but not always safe, then the wrongful diagnosis would kill 98 people. I'm not saying that would be the case here. But it's an illustration of why it is important to consider not how many problems are detected, but what benefits early detection gives vs. the cost of wrongly diagnosing people, even when it seems benign enough (just another test, and they'll confirm it was a mistake?) E.g. there's been pushback against large scale breast cancer screening for this very reason: They're very successful at finding cancer early, but most gets found eventually anyway, and the improvement in long term survival rates from the earlier detection is reasonably small. Couple this with low overall rates of occurrence across the total population at any given point in time (cancer rates look high when considering "will X get cancer at some point in their life?", but most people will be free of cancer at any given single point in their life; even most people who have had cancer), and the impact of false positives and potential health implications of a (very small but still existent) risk involved in even something "simple" like a biopsy, and it's unclear if large scale screening is ethical (over general populations; screening of groups with sufficiently high risk factors is another matter entirely).
- Someone 9y ago
  Only 300k would have to, if this device would tag its diagnosis with “but it is incorrect”. Otherwise, up to the full 100% would have to “follow up and figure it out”. Also, of those 10M, where would those told “your blood pressure is OK” or “you don’t have sleep apnea” get the hint to “follow up and figure it out”?
- edejong 9y ago
  Very rough back-of-the-envelope calculation: we have 3000 GP's in the Netherlands. Lets say the proposal will cause 30.000 patients in the Netherlands to go to the GP incorrectly per year. That will cause 8 man-year of time (20 minutes per consult, including paperwork), 4 workable consulting-hours per GP, 300 days per year. That's 0.3% of all available GP time, wasted. That time needs to come from somewhere: patients having actionable issues. Healthcare is not an unlimited resource.
Blazespinnaker 9y ago
Give 1000 people apple watches with the app versus 1000 people other smart watches without the capability. At the end of the year, find out how well diagnosed the two groups are.
- brandonb 9y ago
  Yep—this is a good sketch of a randomized control trial to measure the primary outcome, change in diagnosis rate. Secondary outcomes you might look at are overall change in medical spend, e.g., is this intervention cost-saving, or at least cost-effective? And change in treatment, which may be independent of diagnosis rate. The tweak I'd make is that you don't need to give out Apple Watches, Garmins, Fitbits, or Android Wear watches. People are buying these devices on their own already. Given ubiquity of hardware, continuous screening may be achievable at close to zero marginal cost.
- edejong 9y ago
  Before doing an expensive and inaccurate field experiment, we can calculate the expected cost per QALJ [1] by taking the following variables: - Estimated accuracy and recall of the test - Estimated probability of user doing a follow-up based on feedback from watch - Estimated price of a doctors visit - Estimated probability of a follow-up test using medical grade equipment (based on additional data) - Estimated cost per QALJ of the extra test and follow-ups - Estimated prevalence of arrhythmia among Apple Watch users based on age-group I expect, but do not know, the Apple Watch tests to prove to be a net-negative compared to doing nothing. [1] Quality Adjusted Life Year
- Someone 9y ago
  “Versus Apple watches without the app” is the better control, I would think. It would make it easier to keep their GP’s in the dark as to who is in the control group and who isn’t.
- davidwihl 9y ago
  Aetna, the US health insurance company, is planning to give away or offer a reduced price for Apple Watches to its 23 M customers. Apparently, following prescriptions is one of several planned apps to help increase health and reduce costs. https://www.engadget.com/2017/08/15/aetna-in-talks-for-discounted-apple-watch/ https://www.engadget.com/2017/08/15/aetna-in-talks-for-disco...
extrapickles 9y ago
It would help as its more of a “you should see a doctor” test rather than a official diagnosis. I’m fine with it providing the marketing does not mislead people into thinking it’s foolproof. Basically, they need to say it can only detect obvious cases, and minor/rare cases are missed.
bertil 9y ago
For this project like for self-driving cars, I believe that releasing machine-learning early in the wild “too early” has a dynamic benefit: we learn earlier and the bump in misdiagnostics pays for better diagnostic down the line. To argue that, you need an estimate of the rate of progress with and without a deployment which I would assume is not a stretch.

outworlder 9y ago

Which is not so bad. They can follow up and figure it out. Some people will go for decades before getting a proper diagnostic - or even die before they get one.

ekianjo 9y ago
> Which is not so bad. They can follow up and figure it out. Following up is not free. So that's cost incurred for the ones who have no problem in the first place.
- aequitas 9y ago
  Think about the cost of not following up. Personal health, productivity, life quality. Also here in the Netherlands we have healthcare. I'd rather see these emerging devices offer good warnings to go see a doctor then people not going at all or going to often for every little thing that they feel and googled on the internet because they are to unsure.
  - vidarh 9y ago
    See my long response elsewhere. Depending on incidence rates and whether or not the effect of following up earlier is sufficiently large, there can - and sometimes is - a net detrimental effect to wrongfully diagnosing someone with a condition. Even when following up is relatively simple and risk free, and lives are saved by finding conditions early. E.g. cause an extra million doctors visits due to wrongly diagnosing people, and the incident rate of e.g. extra infections transmitted from another patient, or compounded misdiagnosis leading to extra interventions which have some risk of failure, or even just accidents on the way to the doctors office, very quickly leads to extra harm just from the sheer volume. That extra harm might be worth it, if the early diagnosis of others save enough lives - the same way seat belts kills some people but save far more. But it also might not, e.g. if the condition is rare enough, or early detection have little enough effect in improving outcomes.
edejong 9y ago
Unfortunately, most arrhythmias are benign and do not lead to further treatment from a physician. So you are taking 300,000 worried people wrongly to a general practitioner, who in fact can do nothing but write down the event and perhaps correlate it with other data.
- vidarh 9y ago
  It's worth always keeping this in mind, and not just taking extra worried people to the gp, but also giving unwarranted peace of mind to those who have a problem. Even just having them get in a car and go to their GP, or increase their stress levels, have the potential to cause harm. It may be an exceedingly low risk, but applied to a large enough number of people and it starts adding up. When applied to conditions that not everyone has, this can easily tilt the balance. Consider a hypothetical condition and a population of 1000000. If 1% has the condition at any one time, and the accuracy of the test is 97%, then 10,000 should have the condition. Of those we'd expect 9,700 to get the correct diagnosis; 300 would remain undiagnosed; But 3% of those who don't have the condition - an additional 29,700 - will be wrongly diagnosed. Let's say this is a condition where 10% would die untreated without this new intervention, it'd be tempting to suggest we've saved 970 people. But early detection might not make that much of a difference - for many things the most severe cases will be noticed anyway and/or late detection is still sufficient to save the patient. Let's assume for this hypothetical condition, earlier detection will save 10% more of those. In the above that'd save 97 people. Awesome. But lets say it causes 10% more of the undiagnosed ones to die because they wait longer. That's 3 more dead. So net 94? But if only 0.33% of those wrongfully diagnosed end up dying as the result of the misdiagnosis, be it car crashes on the way to the GP, or the visit leading to a follow up error leading to an intervention that is generally but not always safe, then the wrongful diagnosis would kill 98 people. I'm not saying that would be the case here. But it's an illustration of why it is important to consider not how many problems are detected, but what benefits early detection gives vs. the cost of wrongly diagnosing people, even when it seems benign enough (just another test, and they'll confirm it was a mistake?) E.g. there's been pushback against large scale breast cancer screening for this very reason: They're very successful at finding cancer early, but most gets found eventually anyway, and the improvement in long term survival rates from the earlier detection is reasonably small. Couple this with low overall rates of occurrence across the total population at any given point in time (cancer rates look high when considering "will X get cancer at some point in their life?", but most people will be free of cancer at any given single point in their life; even most people who have had cancer), and the impact of false positives and potential health implications of a (very small but still existent) risk involved in even something "simple" like a biopsy, and it's unclear if large scale screening is ethical (over general populations; screening of groups with sufficiently high risk factors is another matter entirely).
Someone 9y ago
Only 300k would have to, if this device would tag its diagnosis with “but it is incorrect”. Otherwise, up to the full 100% would have to “follow up and figure it out”. Also, of those 10M, where would those told “your blood pressure is OK” or “you don’t have sleep apnea” get the hint to “follow up and figure it out”?
edejong 9y ago
Very rough back-of-the-envelope calculation: we have 3000 GP's in the Netherlands. Lets say the proposal will cause 30.000 patients in the Netherlands to go to the GP incorrectly per year. That will cause 8 man-year of time (20 minutes per consult, including paperwork), 4 workable consulting-hours per GP, 300 days per year. That's 0.3% of all available GP time, wasted. That time needs to come from somewhere: patients having actionable issues. Healthcare is not an unlimited resource.

Blazespinnaker 9y ago

Give 1000 people apple watches with the app versus 1000 people other smart watches without the capability. At the end of the year, find out how well diagnosed the two groups are.

brandonb 9y ago
Yep—this is a good sketch of a randomized control trial to measure the primary outcome, change in diagnosis rate. Secondary outcomes you might look at are overall change in medical spend, e.g., is this intervention cost-saving, or at least cost-effective? And change in treatment, which may be independent of diagnosis rate. The tweak I'd make is that you don't need to give out Apple Watches, Garmins, Fitbits, or Android Wear watches. People are buying these devices on their own already. Given ubiquity of hardware, continuous screening may be achievable at close to zero marginal cost.
edejong 9y ago
Before doing an expensive and inaccurate field experiment, we can calculate the expected cost per QALJ [1] by taking the following variables: - Estimated accuracy and recall of the test - Estimated probability of user doing a follow-up based on feedback from watch - Estimated price of a doctors visit - Estimated probability of a follow-up test using medical grade equipment (based on additional data) - Estimated cost per QALJ of the extra test and follow-ups - Estimated prevalence of arrhythmia among Apple Watch users based on age-group I expect, but do not know, the Apple Watch tests to prove to be a net-negative compared to doing nothing. [1] Quality Adjusted Life Year
Someone 9y ago
“Versus Apple watches without the app” is the better control, I would think. It would make it easier to keep their GP’s in the dark as to who is in the control group and who isn’t.
davidwihl 9y ago
Aetna, the US health insurance company, is planning to give away or offer a reduced price for Apple Watches to its 23 M customers. Apparently, following prescriptions is one of several planned apps to help increase health and reduce costs. https://www.engadget.com/2017/08/15/aetna-in-talks-for-discounted-apple-watch/ https://www.engadget.com/2017/08/15/aetna-in-talks-for-disco...

extrapickles 9y ago

It would help as its more of a “you should see a doctor” test rather than a official diagnosis. I’m fine with it providing the marketing does not mislead people into thinking it’s foolproof. Basically, they need to say it can only detect obvious cases, and minor/rare cases are missed.

bertil 9y ago

For this project like for self-driving cars, I believe that releasing machine-learning early in the wild “too early” has a dynamic benefit: we learn earlier and the bump in misdiagnostics pays for better diagnostic down the line. To argue that, you need an estimate of the rate of progress with and without a deployment which I would assume is not a stretch.