4 ms·
>> In one clinical trial that used more than 900 images, IDx-DR correctly detected retinopathy about 87 percent of the time, and could correctly identify those
by YeGoblynQueenne 9y ago
>> In one clinical trial that used more than 900 images, IDx-DR correctly detected retinopathy about 87 percent of the time, and could correctly identify those who didn’t have the disease about 90 percent of the time.
I read that as .87 accuracy, .9 specificity (True Negative Rate). However, I can't find in the link provided in the article above the sensitivity (recall or True Positive Rate).
I'm guessing it goes a bit like this (assuming perfectly balanced classes which in reality they aren't):
Predicted + Predicted - Total
Actual + 378 72 450
Actual - 45 405 450
-------------------------------------
Total 423 477 900
Accuracy: 0.8700
Error: 0.1300
True Positive Rate: 0.8400
True Negative Rate: 0.9000
Precision: 0.8936
Recall (TPR): 0.8400
F-Score: 0.8660
I'm not sure how good or bad is 10% false positives and 16% false negatives are for that kind of diagnosis. The linked trial page says that 40% of diabetes patients have some degree of diabetic retinopathy (DR), that early treatment reduces vision loss "by as much as 52%" and that only some 50%-60% of people with diabetes have a yearly eye exam.
Off the top of my head, it looks like automated screening will do some good and probably more good than harm, but without knowing how doctors judge good vs harm there's no way to know for sure how useful this device will really be.
- kendallpark 9y ago"For moderate or worse DR, the majority decision of ophthalmologists had a sensitivity of 0.838 and specificity of 0.981" Note that this is "majority decision" of multiple ophthalmologists, which is going to be better than the average ophthalmologist's single diagnosis. http://www.aaojournal.org/article/S0161-6420(17)32698-2/abstract http://www.aaojournal.org/article/S0161-6420(17)32698-2/abst...
- ksolanki 9y agoThis is not completely correct. Sensitivity (87% in this case) is not same as accuracy, nor is specificity (89.5% in this case) same as true negative rate. Sensitivity generally gives an indication of safety of a medical device, and specificity gives a general indication of effectiveness.
- YeGoblynQueenne 9y agoThe link changed since I posted my comment :) The original article on Verge reported "87% accuracy" and 90% what sounded like TNR. The new link points to an FDA page that makes it more likely that ".87" is actually sensitivity: Dx-DR was able to correctly identify the presence of more than mild diabetic retinopathy 87.4 percent of the time and was able to correctly identify those patients who did not have more than mild diabetic retinopathy 89.5 percent of the time. So, I guess, something like this: Predicted + Predicted - Total Actual + 393 57 450 Actual - 47 403 450 ------------------------------------- Total 440 460 900 Accuracy: 0.8844 Error: 0.1156 True Positive Rate: 0.8733 True Negative Rate: 0.8956 Precision: 0.8932 Recall (TPR): 0.8733 F-Score: 0.8831 Closest I can get with exactly 900 cases :0
- ksolanki 9y agoThanks for this analysis. The balance in real world is more like 20-80, ie 20% of typically screened patients would have referable retinopathy (screen positive).