3 ms·

It would be nice to see more genuine, enthusiastic scientific curiosity to understand how the ML algorithms are doing this, rather than just abject terror and a

It would be nice to see more genuine, enthusiastic scientific curiosity to understand how the ML algorithms are doing this, rather than just abject terror and alarm.

SpicyLemonZest 4y ago
It seems like the reason the researchers in this paper are concerned is precisely that they tried and failed to understand how the ML algorithms are doing this. If they’d discovered that white people have a subtly distinctive vertebra shape the model was detecting, it would have been much more of “oh, we discovered a neat fact”.
- civilized 4y ago
  I don't think they tried very hard at all. I see no meaningful use of modern explanation tools. There are lots of known ways in which people of different races are different physiologically. Probably even more unknown ways. There could also be differences in imaging technology used in different communities, as others have suggested. I'd be a bit surprised if something like that could create such a strong signal but it's on the table.
  - SpicyLemonZest 4y ago
    For those of us less familiar with this space, what are these modern explanation tools? (I certainly agree that it's plausible the model is seeing a physiological difference, and the researchers seem to have considered a few concrete hypotheses on that dimension.)
    - civilized 4y ago
      Here's an introduction to one technique: https://cloud.google.com/blog/products/ai-machine-learning/explaining-model-predictions-on-image-data https://cloud.google.com/blog/products/ai-machine-learning/e... This is a cutting edge subfield of ML, so it's understandable that one paper in a medical journal isn't going to be on that cutting edge, but I think they should at least acknowledge that their investigations barely scratched the surface.
      - SpicyLemonZest 4y ago
        Thanks! This looks pretty neat, I'll have to dig more into their explainable AI product later.

It would be nice to see more genuine, enthusiastic scientific curiosity to understand how the ML algorithms are doing this, rather than just abject terror and alarm.

SpicyLemonZest 4y ago

It seems like the reason the researchers in this paper are concerned is precisely that they tried and failed to understand how the ML algorithms are doing this. If they’d discovered that white people have a subtly distinctive vertebra shape the model was detecting, it would have been much more of “oh, we discovered a neat fact”.

civilized 4y ago
I don't think they tried very hard at all. I see no meaningful use of modern explanation tools. There are lots of known ways in which people of different races are different physiologically. Probably even more unknown ways. There could also be differences in imaging technology used in different communities, as others have suggested. I'd be a bit surprised if something like that could create such a strong signal but it's on the table.
- SpicyLemonZest 4y ago
  For those of us less familiar with this space, what are these modern explanation tools? (I certainly agree that it's plausible the model is seeing a physiological difference, and the researchers seem to have considered a few concrete hypotheses on that dimension.)
  - civilized 4y ago
    Here's an introduction to one technique: https://cloud.google.com/blog/products/ai-machine-learning/explaining-model-predictions-on-image-data https://cloud.google.com/blog/products/ai-machine-learning/e... This is a cutting edge subfield of ML, so it's understandable that one paper in a medical journal isn't going to be on that cutting edge, but I think they should at least acknowledge that their investigations barely scratched the surface.
    - SpicyLemonZest 4y ago
      Thanks! This looks pretty neat, I'll have to dig more into their explainable AI product later.