3 ms·

There is no reason to believe that an X-ray model's correct estimation of age is due to "spurious correlation". Rather, it seems to be "undesirable correlation"

There is no reason to believe that an X-ray model's correct estimation of age is due to "spurious correlation". Rather, it seems to be "undesirable correlation".

TeMPOraL 2y ago
> Rather, it seems to be "undesirable correlation". More specifically, politically undesirable correlation - as in, "it's there, but its existence upsets some people". It's pretty obvious and self-evident that there are meaningful biological differences related to age, sex, and other demographics. Whether or not they're clinically relevant for a specific diagnosis under question is one thing, but they are clinically relevant for great many diagnoses; trying to "de-bias" reality here will only lead to unnecessary suffering and loss of life.
- knallfrosch 2y ago
  It seems that the models not only use bone structure (or similar) itself, but improve their prediction with "forbidden" knowledge, such as "green people have an overall lower or higher rate of bone cancer than others" or "people who come to this specialized hospital have bone cancer anyway, so I don't even look at the image" Now you can say that this is perfectly fine and represents the most likely real-world use case. Or you might prefer a model that looks at the image only, with the implicit assumption that this "forbidden knowledge" will be added by human doctors later on in the pipeline. This is beneficial because the "forbidden knowledge", such as whether patients from Hospital A always have bone cancer, might change overnight! Imagine the hospital gets assigned a new name in the system and the prediction is shit now. This second, "unbiased" AI system will always have a worse performance, because you lobotomize it when you kill the forbidden knowledge with a sledgehammer. This study just showed that "group fairness" is at odds with optimal predictions" and how much it is at odds. PS: You might even prefer a society where everyone is worse off, but every protected group is equally bad off. You'd also ban the humans from applying the forbidden knowledge. Whether that is desirable, is, of course, out of the scope of the paper.
  - carbocation 2y ago
    > Or you might prefer a model that looks at the image only These models are only looking at the images. They are inferring demographics.
- RandomLensman 2y ago
  Yes, these things can be a factor for a specific diagnosis but why use AI when (just) going back to conditional probabilities based on groups instead of making a true individual diagnosis? You want each diagnosis to be correct and not just a good average.
  - TeMPOraL 2y ago
    You always diagnose on conditional probabilities. The diagnosis is conditioned on your belief in occurrence of symptoms, which is conditioned on the observations and results of tests you make. In an ideal case, you observe well enough to make a definite diagnosis; in real case, there's always some uncertainty, plus you can't do all the tests simultaneously - which is where all those proxy factors like demographics are useful: they help prioritize tests and narrow down the diagnosis quicker.
    - RandomLensman 2y ago
      You don't always diagnose on conditional probabilities (a simple example is looking at an x-ray of a broken bone - no priors needed to spot the broken bone). Your knowledge guides, but it also doesn't (or shouldn't) blind you.
pbhjpbhj 2y ago
Do the sorts of diseases ML is being used to detect have a flat incidence profile over age? Even if they do, negating other diagnoses that are age dependent would still mean ML models would acquire a measure of patient age, say.
- carbocation 2y ago
  Almost every noncommunicable disease of adulthood becomes more common with age. (The diagnoses in this paper were things like "cardiomegaly" which are mostly just X-ray findings and, while they have ICD codes, are not a meaningful diagnosis that a practicing physician would care about.)

There is no reason to believe that an X-ray model's correct estimation of age is due to "spurious correlation". Rather, it seems to be "undesirable correlation".

TeMPOraL 2y ago

> Rather, it seems to be "undesirable correlation". More specifically, politically undesirable correlation - as in, "it's there, but its existence upsets some people". It's pretty obvious and self-evident that there are meaningful biological differences related to age, sex, and other demographics. Whether or not they're clinically relevant for a specific diagnosis under question is one thing, but they are clinically relevant for great many diagnoses; trying to "de-bias" reality here will only lead to unnecessary suffering and loss of life.

knallfrosch 2y ago
It seems that the models not only use bone structure (or similar) itself, but improve their prediction with "forbidden" knowledge, such as "green people have an overall lower or higher rate of bone cancer than others" or "people who come to this specialized hospital have bone cancer anyway, so I don't even look at the image" Now you can say that this is perfectly fine and represents the most likely real-world use case. Or you might prefer a model that looks at the image only, with the implicit assumption that this "forbidden knowledge" will be added by human doctors later on in the pipeline. This is beneficial because the "forbidden knowledge", such as whether patients from Hospital A always have bone cancer, might change overnight! Imagine the hospital gets assigned a new name in the system and the prediction is shit now. This second, "unbiased" AI system will always have a worse performance, because you lobotomize it when you kill the forbidden knowledge with a sledgehammer. This study just showed that "group fairness" is at odds with optimal predictions" and how much it is at odds. PS: You might even prefer a society where everyone is worse off, but every protected group is equally bad off. You'd also ban the humans from applying the forbidden knowledge. Whether that is desirable, is, of course, out of the scope of the paper.
- carbocation 2y ago
  > Or you might prefer a model that looks at the image only These models are only looking at the images. They are inferring demographics.
RandomLensman 2y ago
Yes, these things can be a factor for a specific diagnosis but why use AI when (just) going back to conditional probabilities based on groups instead of making a true individual diagnosis? You want each diagnosis to be correct and not just a good average.
- TeMPOraL 2y ago
  You always diagnose on conditional probabilities. The diagnosis is conditioned on your belief in occurrence of symptoms, which is conditioned on the observations and results of tests you make. In an ideal case, you observe well enough to make a definite diagnosis; in real case, there's always some uncertainty, plus you can't do all the tests simultaneously - which is where all those proxy factors like demographics are useful: they help prioritize tests and narrow down the diagnosis quicker.
  - RandomLensman 2y ago
    You don't always diagnose on conditional probabilities (a simple example is looking at an x-ray of a broken bone - no priors needed to spot the broken bone). Your knowledge guides, but it also doesn't (or shouldn't) blind you.

pbhjpbhj 2y ago

Do the sorts of diseases ML is being used to detect have a flat incidence profile over age? Even if they do, negating other diagnoses that are age dependent would still mean ML models would acquire a measure of patient age, say.

carbocation 2y ago
Almost every noncommunicable disease of adulthood becomes more common with age. (The diagnoses in this paper were things like "cardiomegaly" which are mostly just X-ray findings and, while they have ICD codes, are not a meaningful diagnosis that a practicing physician would care about.)