2 ms·

"Yeah, but the point here was that radiologists on average fared even worse." Except they don't. See the table in the original post. Also, comparing the "avera

by timr 8y ago

"Yeah, but the point here was that radiologists on average fared even worse." Except they don't. See the table in the original post. Also, comparing the "average" radiologist by F1 scores from a single experiment (as you've done in other comments here) is meaningless. Unless my doctor is exactly average (and isn't incorporating additional information, or smart enough to be optimizing for false positive/negative rates relative to cost), comparison to average statistics is academic. But I don't really need to tell you this -- your comment has so many caveats that you're clearly already aware of the limits of the method.

bitL 8y ago
What would be the good metrics then? Of course metrics are just indicators that can be interpreted incorrectly. Still, we have to measure something tangible. What would you propose? I am aware of limitations and would gladly use something better... Some people mention Matthews correlation coefficients, Youden's J statistic, Cohen's kappa etc. but I haven't seen them in any Deep Learning paper so far and I bet they have large blindspots as well.
bitL 8y ago
> Unless my doctor is exactly average Just to get back to this point: what if the vision system of your doctor is below average and you augment her by giving her a statistically better vision system, while allowing her to use the additional sources as she sees fit. Wouldn't be that an improvement? We are talking about vision subsystem here, not the whole "reasoning package" human doctors posses.
- timr 8y ago
  Again, check that table. It says a lot: https://stanfordmlgroup.github.io/competitions/mura/ https://stanfordmlgroup.github.io/competitions/mura/ On just about every test set, the model is beaten by radiologists. Even the mean performance is underwhelming.
  - bitL 8y ago
    I was referring mainly to this one (from the same group and it actually surpassed humans on average): https://stanfordmlgroup.github.io/projects/chexnet/ https://stanfordmlgroup.github.io/projects/chexnet/ In their paper they even used "weaker" DenseNet-121 instead of DenseNet-169 for Mura/bones. DenseNet-BC I tried is another refinement of the same approach.
    - timr 8y ago
      Those are some sketchy statistics. The evaluation procedure is questionable (F1 against the other 4 as ground truth? Mean of means?), and the 95% CI overlap pretty substantially. Even if their bootstrap sampling said the difference is significant, I don't believe them. Basically, I see this as "everyone sucks, but the AI maybe sucks a little less than the worst of our radiologists, on average"
sh33mp 8y ago
This thread is a microcosm of this whole issue of overhyping. On one hand, we have one commenter saying he can train a model to do a specific thing with a specific quantitative metric, to demonstrate how deep learning can incredibly powerful/useful. On the other hand, we have another commenter saying "But this won't replace my doctor!" and therefore deep learning is overhyped. The two sides aren't even talking about the same thing.
- timr 8y ago
  Agree that the thread is a microcosm of the debate, but ironically, I'm not trying to say anything like "this won't replace my doctor". That kind of hyperventilating stuff is easy to brush off. The problem with deep-learning hype is that comments like "my classifier gets a ROC/AUC score of 0.8 with barely any work!" are presented as meaningful. The difference between a 0.8 AUC and a usable medical technology means that most of the work is ahead of you.
  - sh33mp 8y ago
    Agreed. I think it comes down to the presentation/interpretation of results. The response to "My classifier gets score of X" can be either "wow, that's a good score for a classifier, this method has merit" or "but X is not a good measure of [actual objective]". So I think it's come down to conflict between 1. Which the author is trying to present 2. What an astute reader might interpret it as 3. What an astute reader might worry an uninformed reader might interpret it as And my feeling is that, given all the talk about hype in pop-sci, we're actually on point 3 now, even when the author and reader are actually talking about something reasonable. Whereas personally I'm more interested in the research and interpretations from experts, which I find tend to be not so problematic.

by timr 8y ago

bitL 8y ago

What would be the good metrics then? Of course metrics are just indicators that can be interpreted incorrectly. Still, we have to measure something tangible. What would you propose? I am aware of limitations and would gladly use something better... Some people mention Matthews correlation coefficients, Youden's J statistic, Cohen's kappa etc. but I haven't seen them in any Deep Learning paper so far and I bet they have large blindspots as well.

bitL 8y ago

> Unless my doctor is exactly average Just to get back to this point: what if the vision system of your doctor is below average and you augment her by giving her a statistically better vision system, while allowing her to use the additional sources as she sees fit. Wouldn't be that an improvement? We are talking about vision subsystem here, not the whole "reasoning package" human doctors posses.

timr 8y ago
Again, check that table. It says a lot: https://stanfordmlgroup.github.io/competitions/mura/ https://stanfordmlgroup.github.io/competitions/mura/ On just about every test set, the model is beaten by radiologists. Even the mean performance is underwhelming.
- bitL 8y ago
  I was referring mainly to this one (from the same group and it actually surpassed humans on average): https://stanfordmlgroup.github.io/projects/chexnet/ https://stanfordmlgroup.github.io/projects/chexnet/ In their paper they even used "weaker" DenseNet-121 instead of DenseNet-169 for Mura/bones. DenseNet-BC I tried is another refinement of the same approach.
  - timr 8y ago
    Those are some sketchy statistics. The evaluation procedure is questionable (F1 against the other 4 as ground truth? Mean of means?), and the 95% CI overlap pretty substantially. Even if their bootstrap sampling said the difference is significant, I don't believe them. Basically, I see this as "everyone sucks, but the AI maybe sucks a little less than the worst of our radiologists, on average"

sh33mp 8y ago

This thread is a microcosm of this whole issue of overhyping. On one hand, we have one commenter saying he can train a model to do a specific thing with a specific quantitative metric, to demonstrate how deep learning can incredibly powerful/useful. On the other hand, we have another commenter saying "But this won't replace my doctor!" and therefore deep learning is overhyped. The two sides aren't even talking about the same thing.

timr 8y ago
Agree that the thread is a microcosm of the debate, but ironically, I'm not trying to say anything like "this won't replace my doctor". That kind of hyperventilating stuff is easy to brush off. The problem with deep-learning hype is that comments like "my classifier gets a ROC/AUC score of 0.8 with barely any work!" are presented as meaningful. The difference between a 0.8 AUC and a usable medical technology means that most of the work is ahead of you.
- sh33mp 8y ago
  Agreed. I think it comes down to the presentation/interpretation of results. The response to "My classifier gets score of X" can be either "wow, that's a good score for a classifier, this method has merit" or "but X is not a good measure of [actual objective]". So I think it's come down to conflict between 1. Which the author is trying to present 2. What an astute reader might interpret it as 3. What an astute reader might worry an uninformed reader might interpret it as And my feeling is that, given all the talk about hype in pop-sci, we're actually on point 3 now, even when the author and reader are actually talking about something reasonable. Whereas personally I'm more interested in the research and interpretations from experts, which I find tend to be not so problematic.