5 ms·

The company is saying you don't need a specialist, but after bayes theorem (using 90%TN 87%TP and D(A)= 200,000 complication / 29,100,000 diabetes), the chance

The company is saying you don't need a specialist, but after bayes theorem (using 90%TN 87%TP and D(A)= 200,000 complication / 29,100,000 diabetes), the chance you have this condition after the machine says you do is 0.83%.

wycs 8y ago
Why are specialists immune to those same statistics?
- marcosdumay 8y ago
  They aren't. With some luck their errors do not correlate with the ones of the machine, that gives us the best of both worlds: cheap diagnosis for healthy people and precise diagnostics for unhealty ones.
jadedhacker 8y ago
I think you might have inverted the parameters. I think the numbers given are TN and TP.
Eridrus 8y ago
I think the number is closer to 6%. 90% of people are accurately detected as not having the disease, i.e. 10% FP rate. So ~3m people would be falsely diagnosed as having the complication, to 200k/3m have the disease. 10% isn't a great number, but it isn't clear from this coverage whether this complication is generally asymptomatic or not. If there are symptoms to go with it, the numbers may be far better.
- epmaybe 8y ago
  Also keep in mind that if there are positives general practitioners would refer to a specialist anyways for treatment. These specialists would be more than equipped to detect false positives. Teleretina imaging is becoming more and more prevalent as well with eyePACS and Welch-Allyn having dedicated interpretation services, so patients wouldn't have to necessarily go somewhere for verification. I'm more worried about the 87% accurately detected as having the disease, i.e. 13% of false negatives (FN). I don't know how many general practitioners would actually send a patient to a specialist if the device did not detect changes. The retina seems well suited to AI approaches, though, so I'd be interested in what comes next from companies like this, DeepMind, and other researchers/organizations (look out for Lee et al over at the University of Washington)
  - lallysingh 8y ago
    I guess the baseline for comparing the 13% would be the nonspecialist rate of the same.
    - epmaybe 8y ago
      Perhaps, that's one way to compare non-inferiority. Few if any primary care physicians take the time to look in ones eye (don't know how to use direct ophthalmoscope, don't have other specialized fundus cameras in clinic). Given that, if this tool forces them to take more retinal photographs of all patients, maybe we could detect diabetic disease before it is usually seen. The standard practice today is that if a patient is determined to be diabetic then they get referred to an ophthalmologist visit once a year. In that case would comparing those rates of diagnosis be useful?
maxerickson 8y ago
They are marketing the device strictly to make referrals to specialists. The FDA press release says as much anyway: https://www.fda.gov/NewsEvents/Newsroom/PressAnnouncements/ucm604357.htm https://www.fda.gov/NewsEvents/Newsroom/PressAnnouncements/u... If the images are of sufficient quality, the software provides the doctor with one of two results: (1) “more than mild diabetic retinopathy detected: refer to an eye care professional” or (2) “negative for more than mild diabetic retinopathy; rescreen in 12 months.” If a positive result is detected, patients should see an eye care provider for further diagnostic evaluation and possible treatment as soon as possible. Are you interpreting the Verge or talking about some other statement?
- asperous 8y ago
  Yeah from the article: > IDx-DR founder Michael Abràmoff told Science News. “It makes the clinical decision on its own.” But I would guess a specialist would still need to be involved since it's not a fool-proof system. A specialist might take other symptoms or variables into account when making the diagnosis or order further tests. While this tool might be useful for blanket screening considering that it is harmless, it seems like it's hardly going to be "making the decision on its own" and prescribing treatment.
  - maxerickson 8y ago
    It's specifically making the clinical decision I quote from the FDA press release, whether or not to refer to a specialist for further diagnosis. It's not something primary care doctors currently do.
amelius 8y ago
I'm more worried about the false-negatives. What if you have the condition, the GP runs the test, and it turns out negative, and then the GP decides to not send you to a specialist ... Can anybody infer what the confusion matrix looks like from reading the text?
asperous 8y ago
Yeah I was wrong on the math: P(H) = .0069 P(D|H) = .87 P(D|H')= .1 (10% chance the test says you have it but you don't) P(H|D) = 0.057 or about 6% probability you have complication after machine says you do
raverbashing 8y ago
Any machine or specialist that wants to do a diagnostic based on a test result will have to consider other factors apart from the result. The false positive bayesian math is a good illustrative example, but reality is more complicated. And no doctor will base their diagnostic solely on one number.
pontus 8y ago
Keep in mind that the test wouldn't be administered on any randomly chosen diabetic patient but presumably rather on those who already exhibit some type of vision imparement consistent with the disease. As a result, the prior of 200k / 29M is not quite right and I'm guessing that the true prior is likely much higher.
- drta 8y ago
  >Keep in mind that the test wouldn't be administered on any randomly chosen diabetic patient but presumably rather on those who already exhibit some type of vision imparement consistent with the disease. Not true. The aim is to treat patients before they become symptomatic. Outcomes are much worse otherwise. 2min promo on Diabetic Eye Screening: https://www.youtube.com/watch?v=PK1Y-1BKFn0 https://www.youtube.com/watch?v=PK1Y-1BKFn0
  - pontus 8y ago
    I didn't know that. Thanks for pointing this out!
YeGoblynQueenne 8y ago
Results are reported as .87 accuracy, not true positive rate, at least in the Verge article. I guess they may be misreporting sensitivity as accuracy.

wycs 8y ago

Why are specialists immune to those same statistics?

marcosdumay 8y ago
They aren't. With some luck their errors do not correlate with the ones of the machine, that gives us the best of both worlds: cheap diagnosis for healthy people and precise diagnostics for unhealty ones.

jadedhacker 8y ago

I think you might have inverted the parameters. I think the numbers given are TN and TP.

Eridrus 8y ago

I think the number is closer to 6%. 90% of people are accurately detected as not having the disease, i.e. 10% FP rate. So ~3m people would be falsely diagnosed as having the complication, to 200k/3m have the disease. 10% isn't a great number, but it isn't clear from this coverage whether this complication is generally asymptomatic or not. If there are symptoms to go with it, the numbers may be far better.

epmaybe 8y ago
Also keep in mind that if there are positives general practitioners would refer to a specialist anyways for treatment. These specialists would be more than equipped to detect false positives. Teleretina imaging is becoming more and more prevalent as well with eyePACS and Welch-Allyn having dedicated interpretation services, so patients wouldn't have to necessarily go somewhere for verification. I'm more worried about the 87% accurately detected as having the disease, i.e. 13% of false negatives (FN). I don't know how many general practitioners would actually send a patient to a specialist if the device did not detect changes. The retina seems well suited to AI approaches, though, so I'd be interested in what comes next from companies like this, DeepMind, and other researchers/organizations (look out for Lee et al over at the University of Washington)
- lallysingh 8y ago
  I guess the baseline for comparing the 13% would be the nonspecialist rate of the same.
  - epmaybe 8y ago
    Perhaps, that's one way to compare non-inferiority. Few if any primary care physicians take the time to look in ones eye (don't know how to use direct ophthalmoscope, don't have other specialized fundus cameras in clinic). Given that, if this tool forces them to take more retinal photographs of all patients, maybe we could detect diabetic disease before it is usually seen. The standard practice today is that if a patient is determined to be diabetic then they get referred to an ophthalmologist visit once a year. In that case would comparing those rates of diagnosis be useful?

maxerickson 8y ago

They are marketing the device strictly to make referrals to specialists. The FDA press release says as much anyway: https://www.fda.gov/NewsEvents/Newsroom/PressAnnouncements/ucm604357.htm https://www.fda.gov/NewsEvents/Newsroom/PressAnnouncements/u... If the images are of sufficient quality, the software provides the doctor with one of two results: (1) “more than mild diabetic retinopathy detected: refer to an eye care professional” or (2) “negative for more than mild diabetic retinopathy; rescreen in 12 months.” If a positive result is detected, patients should see an eye care provider for further diagnostic evaluation and possible treatment as soon as possible. Are you interpreting the Verge or talking about some other statement?

asperous 8y ago
Yeah from the article: > IDx-DR founder Michael Abràmoff told Science News. “It makes the clinical decision on its own.” But I would guess a specialist would still need to be involved since it's not a fool-proof system. A specialist might take other symptoms or variables into account when making the diagnosis or order further tests. While this tool might be useful for blanket screening considering that it is harmless, it seems like it's hardly going to be "making the decision on its own" and prescribing treatment.
- maxerickson 8y ago
  It's specifically making the clinical decision I quote from the FDA press release, whether or not to refer to a specialist for further diagnosis. It's not something primary care doctors currently do.

amelius 8y ago

I'm more worried about the false-negatives. What if you have the condition, the GP runs the test, and it turns out negative, and then the GP decides to not send you to a specialist ... Can anybody infer what the confusion matrix looks like from reading the text?

asperous 8y ago

Yeah I was wrong on the math: P(H) = .0069 P(D|H) = .87 P(D|H')= .1 (10% chance the test says you have it but you don't) P(H|D) = 0.057 or about 6% probability you have complication after machine says you do

raverbashing 8y ago

Any machine or specialist that wants to do a diagnostic based on a test result will have to consider other factors apart from the result. The false positive bayesian math is a good illustrative example, but reality is more complicated. And no doctor will base their diagnostic solely on one number.

pontus 8y ago

Keep in mind that the test wouldn't be administered on any randomly chosen diabetic patient but presumably rather on those who already exhibit some type of vision imparement consistent with the disease. As a result, the prior of 200k / 29M is not quite right and I'm guessing that the true prior is likely much higher.

drta 8y ago
>Keep in mind that the test wouldn't be administered on any randomly chosen diabetic patient but presumably rather on those who already exhibit some type of vision imparement consistent with the disease. Not true. The aim is to treat patients before they become symptomatic. Outcomes are much worse otherwise. 2min promo on Diabetic Eye Screening: https://www.youtube.com/watch?v=PK1Y-1BKFn0 https://www.youtube.com/watch?v=PK1Y-1BKFn0
- pontus 8y ago
  I didn't know that. Thanks for pointing this out!

YeGoblynQueenne 8y ago

Results are reported as .87 accuracy, not true positive rate, at least in the Verge article. I guess they may be misreporting sensitivity as accuracy.