6 ms·

I don't see why this is necessarily bad. An ML model is picking up on subtle anatomical or physiological differences between races. So what, that doesn't automa

by umvi 5y ago

I don't see why this is necessarily bad. An ML model is picking up on subtle anatomical or physiological differences between races. So what, that doesn't automatically mean the AI is racist or biased...

ceejayoz 5y ago
It's not necessarily bad, if it's actually working. The fact that it works on an 8x8 massively pixelated version of the x-ray points to the possibility that it's not actually working, which would be bad if you based patient treatment decisions on an training set that was actually teaching the AI something else entirely.
- cubano 5y ago
  Huh? What do you mean, not working? That the AI was randomly choosing the correct race 82% of the time by luck? I'm confused by what your implying because it would seem to me that the authors went through many steps to try to pinpoint how the AI was doing this identification and how baffling it was to everyone that even with a lot of x-ray information removed (8x8 pixels compared to say 4k), it somehow was still correctly picking the race. What would this "something else entirely" that you are implying actually be?
  - ceejayoz 5y ago
    > That the AI was randomly choosing the correct race 82% of the time by luck? No; as with the article I linked elsewhere in the thread (https://techcrunch.com/2018/12/31/this-clever-ai-hid-data-from-its-creators-to-cheat-at-its-appointed-task/ https://techcrunch.com/2018/12/31/this-clever-ai-hid-data-fr...), that the AI might have found some other indicator, like filenames in the data set, or metadata in the images that included patient name, or differences in the length of patient name (often redacted by black rectangles in x-rays in training data), or any number of other factors. This happens all the time in science. As another recent example of "whoops, turned out we were measuring the wrong thing", https://en.wikipedia.org/wiki/Faster-than-light_neutrino_anomaly https://en.wikipedia.org/wiki/Faster-than-light_neutrino_ano... Another example around AI: https://www.vox.com/recode/2019/12/12/20993665/artificial-intelligence-ai-job-screen https://www.vox.com/recode/2019/12/12/20993665/artificial-in... > One such résumé-screening tool identified being named Jared and having played lacrosse in high school as the best predictors of job performance, as Quartz reported. Are lacrosse players naturally better workers? Probably not. Are they probably whiter, wealthier, better networks, etc. than the average population? Probably. These sorts of things - as with the 8x8 pixel example - start to point to confounding variables that need to be worked out and accounted for.
    - dexen 5y ago
      >the AI might have found some other indicator, like filenames in the data set The paper quite explicitly goes into testing and disseminating what exactly the AI detects. Two observations: - the classification clearly was primarily based on the visual content rather than spurious metadata, because various transformations of the visual content had the expected impact on classification correctness - the classification clearly wasn't based on one specific feature of the visual content but rather on multiple factors in the visuals, because various transformations to features (including masking out specific features like bone density) produced results matching expectations (usually gradual decrease in accuracy, with some thresholds). Conversely, if the classification was primarily based on factors other than the visual content, the visual transformations would have had negligible effect - possibly up to a threshold, and then would throw the AI completely off.
      - ceejayoz 5y ago
        The faster-than-light neutrino experiment similarly went "we've tried to account for everything we can think of and still can't figure it out" when they published. It turned out to be a measurement error. The same may be true here, and I think it's the most likely explanation. I'd be interested in whether the same model can be trained to predict patient wealth, hair color, style of clothing, religion, etc. from the same x-ray data sets.
        dexen 5y ago
        I am dismayed by your example that runs counter to the modern science. While "faster than light neutrino" was highly unexpected and rather suspect from the start, the "bone geometry differs slightly between ethnic groups" is well established among the anthropologists of humans. There are also parallels in wider biology of animals - mentioning that to underscore it's as scientifically expected, and not merely construed for humans alone. The question here was how exactly is AI detecting it this well from chest X-rays; the question centered around AI and possibly if it would unexpectedly influence the medical processes - rather than around the bone geometry itself. For sake of example, a random link from google search: https://www.researchgate.net/publication/24427702_Ethnic_differences_in_bone_geometry_and_strength_are_apparent_in_childhood https://www.researchgate.net/publication/24427702_Ethnic_dif...
  - FeepingCreature 5y ago
    I think the idea is that it's picking up on a coincidental correlational bias in the source data.
  - CWuestefeld 5y ago
    I'm just making this up, but... Perhaps hospitals that treat a disproportionate share of poor people (which themselves are disproportionately not white), tend to use a different brand of X-ray film, and that brand has different contrast ratios than that of the brand preferred by rich hospitals. Thus, they'd be detecting the different brand of X-ray film rather than anything about the patients themselves. Of course, at this level it's still hard to imagine generating that 82% hit rate. But maybe there are multiple factors along these lines.
    - lostlogin 5y ago
      > tend to use a different brand of X-ray film Most of us radiology folk abandoned film 20 years ago and went to digital systems (CR or DR). This doesn’t negate your query though, as vendors do have different technologies and their images do not look the same.
    - kovek 5y ago
      That sounds like a great idea and they can test for it! Classify the scans on the “type of film” and then alter the scan and see if the model recognizes it
- Cycl0ps 5y ago
  Since it's effectively the same anatomical structure, presented in a grayscale image, I think bone density would change the average color of the image regardless of what level of pixelation was applied. The chance of detection through some other method isn't ruled out though. It could be hitting off of something mundane like the margins on the edge of an image, how centered the torso is in frame, foreign objects in the body from procedures, any random thing that could bias the data can also lead to false positives. The only way to verify would be to use a new data set, ideally from new hospitals, and see if it has similar results.
macksd 5y ago
But it does mean that AI models trained on data might end up just perpetuating racial correlations even when you don't think it's aware of race. It's another example of why interpretability is important: the model might end up depending on correlations you don't mean it to.

by umvi 5y ago

ceejayoz 5y ago

It's not necessarily bad, if it's actually working. The fact that it works on an 8x8 massively pixelated version of the x-ray points to the possibility that it's not actually working, which would be bad if you based patient treatment decisions on an training set that was actually teaching the AI something else entirely.

cubano 5y ago
Huh? What do you mean, not working? That the AI was randomly choosing the correct race 82% of the time by luck? I'm confused by what your implying because it would seem to me that the authors went through many steps to try to pinpoint how the AI was doing this identification and how baffling it was to everyone that even with a lot of x-ray information removed (8x8 pixels compared to say 4k), it somehow was still correctly picking the race. What would this "something else entirely" that you are implying actually be?
- ceejayoz 5y ago
  > That the AI was randomly choosing the correct race 82% of the time by luck? No; as with the article I linked elsewhere in the thread (https://techcrunch.com/2018/12/31/this-clever-ai-hid-data-from-its-creators-to-cheat-at-its-appointed-task/ https://techcrunch.com/2018/12/31/this-clever-ai-hid-data-fr...), that the AI might have found some other indicator, like filenames in the data set, or metadata in the images that included patient name, or differences in the length of patient name (often redacted by black rectangles in x-rays in training data), or any number of other factors. This happens all the time in science. As another recent example of "whoops, turned out we were measuring the wrong thing", https://en.wikipedia.org/wiki/Faster-than-light_neutrino_anomaly https://en.wikipedia.org/wiki/Faster-than-light_neutrino_ano... Another example around AI: https://www.vox.com/recode/2019/12/12/20993665/artificial-intelligence-ai-job-screen https://www.vox.com/recode/2019/12/12/20993665/artificial-in... > One such résumé-screening tool identified being named Jared and having played lacrosse in high school as the best predictors of job performance, as Quartz reported. Are lacrosse players naturally better workers? Probably not. Are they probably whiter, wealthier, better networks, etc. than the average population? Probably. These sorts of things - as with the 8x8 pixel example - start to point to confounding variables that need to be worked out and accounted for.
  - dexen 5y ago
    >the AI might have found some other indicator, like filenames in the data set The paper quite explicitly goes into testing and disseminating what exactly the AI detects. Two observations: - the classification clearly was primarily based on the visual content rather than spurious metadata, because various transformations of the visual content had the expected impact on classification correctness - the classification clearly wasn't based on one specific feature of the visual content but rather on multiple factors in the visuals, because various transformations to features (including masking out specific features like bone density) produced results matching expectations (usually gradual decrease in accuracy, with some thresholds). Conversely, if the classification was primarily based on factors other than the visual content, the visual transformations would have had negligible effect - possibly up to a threshold, and then would throw the AI completely off.
    - ceejayoz 5y ago
      The faster-than-light neutrino experiment similarly went "we've tried to account for everything we can think of and still can't figure it out" when they published. It turned out to be a measurement error. The same may be true here, and I think it's the most likely explanation. I'd be interested in whether the same model can be trained to predict patient wealth, hair color, style of clothing, religion, etc. from the same x-ray data sets.
      - dexen 5y ago
        I am dismayed by your example that runs counter to the modern science. While "faster than light neutrino" was highly unexpected and rather suspect from the start, the "bone geometry differs slightly between ethnic groups" is well established among the anthropologists of humans. There are also parallels in wider biology of animals - mentioning that to underscore it's as scientifically expected, and not merely construed for humans alone. The question here was how exactly is AI detecting it this well from chest X-rays; the question centered around AI and possibly if it would unexpectedly influence the medical processes - rather than around the bone geometry itself. For sake of example, a random link from google search: https://www.researchgate.net/publication/24427702_Ethnic_differences_in_bone_geometry_and_strength_are_apparent_in_childhood https://www.researchgate.net/publication/24427702_Ethnic_dif...
- FeepingCreature 5y ago
  I think the idea is that it's picking up on a coincidental correlational bias in the source data.
- CWuestefeld 5y ago
  I'm just making this up, but... Perhaps hospitals that treat a disproportionate share of poor people (which themselves are disproportionately not white), tend to use a different brand of X-ray film, and that brand has different contrast ratios than that of the brand preferred by rich hospitals. Thus, they'd be detecting the different brand of X-ray film rather than anything about the patients themselves. Of course, at this level it's still hard to imagine generating that 82% hit rate. But maybe there are multiple factors along these lines.
  - lostlogin 5y ago
    > tend to use a different brand of X-ray film Most of us radiology folk abandoned film 20 years ago and went to digital systems (CR or DR). This doesn’t negate your query though, as vendors do have different technologies and their images do not look the same.
  - kovek 5y ago
    That sounds like a great idea and they can test for it! Classify the scans on the “type of film” and then alter the scan and see if the model recognizes it
Cycl0ps 5y ago
Since it's effectively the same anatomical structure, presented in a grayscale image, I think bone density would change the average color of the image regardless of what level of pixelation was applied. The chance of detection through some other method isn't ruled out though. It could be hitting off of something mundane like the margins on the edge of an image, how centered the torso is in frame, foreign objects in the body from procedures, any random thing that could bias the data can also lead to false positives. The only way to verify would be to use a new data set, ideally from new hospitals, and see if it has similar results.

macksd 5y ago

But it does mean that AI models trained on data might end up just perpetuating racial correlations even when you don't think it's aware of race. It's another example of why interpretability is important: the model might end up depending on correlations you don't mean it to.