6 ms·

Keep in mind ML models are really great at cheating to get answers: Maybe it's detecting that women have to tilt their head up more to reach the retinal photogr

by drcode 5y ago

Keep in mind ML models are really great at cheating to get answers: Maybe it's detecting that women have to tilt their head up more to reach the retinal photography machine, because they're shorter on average. Maybe, some of the images come from an optometrist that specializes in women's glass frames, and his retinal photography machine has a slightly dimmer bulb. Maybe, men are more likely to get a retinograph only when they have more severe disease already, so the retinas look different for that reason.

Baeocystin 5y ago
To wit- the apocryphal Tank story: https://www.gwern.net/Tanks https://www.gwern.net/Tanks
- sgt101 5y ago
  apocryphal ehh?
- B1FF_PSUVM 5y ago
  """ Once upon a time—I’ve seen this story in several versions and several places, sometimes cited as fact, but I’ve never tracked down an original source—once upon a time, I say, the US Army wanted to use neural networks to automatically detect camouflaged enemy tanks. The researchers trained a neural net on 50 photos of camouflaged tanks amid trees, and 50 photos of trees without tanks. Using standard techniques for supervised learning, the researchers trained the neural network to a weighting that correctly loaded the training set—output “yes” for the 50 photos of camouflaged tanks, and output “no” for the 50 photos of forest. Now this did not prove, or even imply, that new examples would be classified correctly. The neural network might have “learned” 100 special cases that wouldn’t generalize to new problems. Not, “camouflaged tanks versus forest”, but just, “photo-1 positive, photo-2 negative, photo-3 negative, photo-4 positive…” But wisely, the researchers had originally taken 200 photos, 100 photos of tanks and 100 photos of trees, and had used only half in the training set. The researchers ran the neural network on the remaining 100 photos, and without further training the neural network classified all remaining photos correctly. Success confirmed! The researchers handed the finished work to the Pentagon, which soon handed it back, complaining that in their own tests the neural network did no better than chance at discriminating photos. It turned out that in the researchers’ data set, photos of camouflaged tanks had been taken on cloudy days, while photos of plain forest had been taken on sunny days. The neural network had learned to distinguish cloudy days from sunny days, instead of distinguishing camouflaged tanks from empty forest. """
  - SubiculumCode 5y ago
    That is a pretty novice mistake for a paid scientist, imo.
    - m4x 5y ago
      The story apparently dates from 1963. I wouldn't judge any scientist training neural networks in the 60s too harshly.
      - SubiculumCode 5y ago
        I certainly did not expect the down votes. I've designed a number psychology experiments and I and my colleagues were always careful to balance experimental stimuli and conditions on a number of factors, ranging from luminosity to semantic association, and especially order. When I was tasked to create stimuli or to take measurements on a series of items it was always important to try to eliminate systematic differences on non-interest, most typically thru randomization of the order of creation or measurements.
        gwern 5y ago
        I would point out that the study which appears to be the closest thing to a tiny seed of truth to the tank story (Kanal & Randall 1964) actually did balance its photographs by construction, by subcropping large aerial photographs to have tank/no-tank sections and training on that. That necessarily controls for time-of-day, weather, luminosity etc.
ed25519FUUU 5y ago
Reproducibility is one thing I like about AI research. If they provide the model, I can take on my own computer and test it against whatever I want and judge it. Most things in science is almost impossible to reproduce because of cost or specialized equipment.
- sombremesa 5y ago
  > test it against whatever I want and judge it Is that what reproducibility means?
  - deleted 5y ago
    [deleted]
acituan 5y ago
Their use of an external validation dataset eliminates many, if not all, of those concerns. Regarding external validation set: > This dataset differed from the UK Biobank development set with respect to both fundus camera used, and in sourcing from a pathology-rich population at a tertiary ophthalmic referral center. Regarding UK Biobank set (training set) > UK Biobank dataset, which is an observational study in the United Kingdom that began in 2006 and has recruited over 500,000 participants—85,262 of which received eye imaging38. Eye imaging was obtained at 6 centers in the UK and comprises over 10 terabytes of data39. Participants volunteered to provide data including other medical imaging, laboratory results, and detailed subjective questionnaires.
- nerdponx 5y ago
  Still, until we have a better sense of what features the model is extracting, it's a surprising result and ought to be treated with caution.
  - slver 5y ago
    I wish we treated our own conclusions this way.
    - ithkuil 5y ago
      Science's secret weapon is the innate pleasure that many academics have in proving their colleagues wrong.
      - raihansaputra 5y ago
        That same spirit is also what's behind shorts in the market to keep overly optimistic players in check. As with crypto-ransom operations against corporations to keep their system security on alert. Adversarial actions (within reasonable boundaries) is the most important balance mechanism.
        slver 5y ago
max_ 5y ago
Thank you very much for this comment. My problem with modern scientific publications is that they focus more on the "discovery" instead of describing the logical rigor as to why their "discovery" could be true or false.
- _nothing 5y ago
  What kind of publications are you referring to? Yeah, many pop-sci articles often overstate the implications of a given discovery, but actual research papers (such as the one linked) typically have a discussion section in which they describe the limitations of the discovery and what kind of future research could help expound on the pitfalls of the current research. The paper in this post itself has its own Limitations section.
  - max_ 5y ago
    > The paper in this post itself has its own Limitations section. I haven't actually read the paper. I was reacting to the title. > what kind of future research could help expound on the pitfalls of the current research. I think a better way of documenting research to people is by describing what scientific boxes where checked. In the given case for example one of the boxes maybe, "Is there an anatomical distinction between retinas of sexes?" — If that is true then we can see if machine learning can detect such differences. Take computer science for example. A publication with the title "Professors create a machine that can think" would maybe published as "Professors create machine that passes Turing test [0]" Another example, can be that is medicine. The research in question maybe if a microbe A causes a flue. Instead of a publication being "Microbe A causes disease B". A better Publication IMO should revolve around, "Microbe A had passed Koch's postulates [1] for disease B" [0]:https://en.m.wikipedia.org/wiki/Turing_test https://en.m.wikipedia.org/wiki/Turing_test [1]:https://en.wikipedia.org/wiki/Koch's_postulates https://en.wikipedia.org/wiki/Koch's_postulates
robocat 5y ago
Slight reflections of eyelashes? Especially if blurred?
X6S1x6Okd1st 5y ago
Yup. I learned this the hard way on the last model I trained at scale. It was evaluating fit between two heterogeneous classes I sampled training & test split off of a large time window and got to work. It performed extremely well on test & training. Too good. I pulled a third sample from a completely different time window and it performed terribly. It turned out that both datasets were dominated by class A being sorted into always selecting great fit or poor fit, so the ML model learned to memorize the class A instances. This problem when away when I subselected down to only instances of class A that had examples of both good fit and poor fit.
8note 5y ago
I'm uncertain how they're defining sex in this. I think a good control would be to see how this model treats trans people at varying stages of transition
- read_if_gay_ 5y ago
  I don’t think its confidence will necessarily match the degree to which a trans person has transitioned. But indeed it would be interesting to see how trans people are treated at all.
- Spooky23 5y ago
  It depends on what you mean, as that community has a pretty vast and complex taxonomy. Estrogen levels affect the eyes, so given enough samples and context it may be surprising accurate for people getting that type of therapy.
- AussieWog93 5y ago
  I think the problem with that line of thinking is that even after fully transitioning, trans people still tend to have strong biological markers of the sex they've transitioned from. For a ML model to correctly guess gender identity, it'd need other cues that indicate a person would prefer to be referred to by a certain pronoun, such as clothing or facial hair.
  - elil17 5y ago
    It really depends on the bio marker. Most bio markers are effected: fat distribution, hair texture (curly vs. straight), and even immune system function. I’d think that learning whether this model works on people who’ve taken hormones for gender transition could give some insight into what it’s detecting. Perhaps there are retinal features whose function or form are impacted by the presence of testosterone, for instance, or maybe it’s a different that forms before birth.
0x000000E2 5y ago
My theory: retinal photos often contain parts of eyelashes. Women's are statistically longer. Their eyes are also more almond shaped. Retinal photographs are hard to take. Often they contain significant amount of eyelashes and surrounding structure
- ncmncm 5y ago
  Serious question: Do you know what a retina is?
  - antonvs 5y ago
    I'm not sure he knows what eyes are.
m463 5y ago
I read somewhere that an ML model could detect cancerous tumors, but it was keying off the ruler used to measure tumor size.
- stevelosh 5y ago
  Paper: https://www.sciencedirect.com/science/article/pii/S0022202X18322930 https://www.sciencedirect.com/science/article/pii/S0022202X1... > For instance, in our work, we noted that the algorithm appeared more likely to interpret images with rulers as malignant. Why? In our dataset, images with rulers were more likely to be malignant; thus the algorithm inadvertently “learned” that rulers are malignant.
raverbashing 5y ago
Yeah, when I read the headline I thought Cornea/Iris, not Retina, and there are a couple of ways an ML could "cheat" there. But still, there are ways of getting the resulting NN and applying some techniques to figure out which variable/patterns are responsible for most of the output.
ASalazarMX 5y ago
ML needs an EXPLAIN command so it can point the features that determined its answer.
- mousepilot 5y ago
  with neural networks, I'd imagine this feature would be difficult to implement.
  - ASalazarMX 5y ago
    NN: This is a woman's retina. Scientist: How do you know that? NN: I just do, trust me.

by drcode 5y ago

Baeocystin 5y ago

To wit- the apocryphal Tank story: https://www.gwern.net/Tanks https://www.gwern.net/Tanks

sgt101 5y ago
apocryphal ehh?
B1FF_PSUVM 5y ago
""" Once upon a time—I’ve seen this story in several versions and several places, sometimes cited as fact, but I’ve never tracked down an original source—once upon a time, I say, the US Army wanted to use neural networks to automatically detect camouflaged enemy tanks. The researchers trained a neural net on 50 photos of camouflaged tanks amid trees, and 50 photos of trees without tanks. Using standard techniques for supervised learning, the researchers trained the neural network to a weighting that correctly loaded the training set—output “yes” for the 50 photos of camouflaged tanks, and output “no” for the 50 photos of forest. Now this did not prove, or even imply, that new examples would be classified correctly. The neural network might have “learned” 100 special cases that wouldn’t generalize to new problems. Not, “camouflaged tanks versus forest”, but just, “photo-1 positive, photo-2 negative, photo-3 negative, photo-4 positive…” But wisely, the researchers had originally taken 200 photos, 100 photos of tanks and 100 photos of trees, and had used only half in the training set. The researchers ran the neural network on the remaining 100 photos, and without further training the neural network classified all remaining photos correctly. Success confirmed! The researchers handed the finished work to the Pentagon, which soon handed it back, complaining that in their own tests the neural network did no better than chance at discriminating photos. It turned out that in the researchers’ data set, photos of camouflaged tanks had been taken on cloudy days, while photos of plain forest had been taken on sunny days. The neural network had learned to distinguish cloudy days from sunny days, instead of distinguishing camouflaged tanks from empty forest. """
- SubiculumCode 5y ago
  That is a pretty novice mistake for a paid scientist, imo.
  - m4x 5y ago
    The story apparently dates from 1963. I wouldn't judge any scientist training neural networks in the 60s too harshly.
    - SubiculumCode 5y ago
      I certainly did not expect the down votes. I've designed a number psychology experiments and I and my colleagues were always careful to balance experimental stimuli and conditions on a number of factors, ranging from luminosity to semantic association, and especially order. When I was tasked to create stimuli or to take measurements on a series of items it was always important to try to eliminate systematic differences on non-interest, most typically thru randomization of the order of creation or measurements.
      - gwern 5y ago
        I would point out that the study which appears to be the closest thing to a tiny seed of truth to the tank story (Kanal & Randall 1964) actually did balance its photographs by construction, by subcropping large aerial photographs to have tank/no-tank sections and training on that. That necessarily controls for time-of-day, weather, luminosity etc.

ed25519FUUU 5y ago

Reproducibility is one thing I like about AI research. If they provide the model, I can take on my own computer and test it against whatever I want and judge it. Most things in science is almost impossible to reproduce because of cost or specialized equipment.

sombremesa 5y ago
> test it against whatever I want and judge it Is that what reproducibility means?
- deleted 5y ago
  [deleted]

acituan 5y ago

Their use of an external validation dataset eliminates many, if not all, of those concerns. Regarding external validation set: > This dataset differed from the UK Biobank development set with respect to both fundus camera used, and in sourcing from a pathology-rich population at a tertiary ophthalmic referral center. Regarding UK Biobank set (training set) > UK Biobank dataset, which is an observational study in the United Kingdom that began in 2006 and has recruited over 500,000 participants—85,262 of which received eye imaging38. Eye imaging was obtained at 6 centers in the UK and comprises over 10 terabytes of data39. Participants volunteered to provide data including other medical imaging, laboratory results, and detailed subjective questionnaires.

nerdponx 5y ago
Still, until we have a better sense of what features the model is extracting, it's a surprising result and ought to be treated with caution.
- slver 5y ago
  I wish we treated our own conclusions this way.
  - ithkuil 5y ago
    Science's secret weapon is the innate pleasure that many academics have in proving their colleagues wrong.
    - raihansaputra 5y ago
      That same spirit is also what's behind shorts in the market to keep overly optimistic players in check. As with crypto-ransom operations against corporations to keep their system security on alert. Adversarial actions (within reasonable boundaries) is the most important balance mechanism.
      - slver 5y ago

max_ 5y ago

Thank you very much for this comment. My problem with modern scientific publications is that they focus more on the "discovery" instead of describing the logical rigor as to why their "discovery" could be true or false.

_nothing 5y ago
What kind of publications are you referring to? Yeah, many pop-sci articles often overstate the implications of a given discovery, but actual research papers (such as the one linked) typically have a discussion section in which they describe the limitations of the discovery and what kind of future research could help expound on the pitfalls of the current research. The paper in this post itself has its own Limitations section.
- max_ 5y ago
  > The paper in this post itself has its own Limitations section. I haven't actually read the paper. I was reacting to the title. > what kind of future research could help expound on the pitfalls of the current research. I think a better way of documenting research to people is by describing what scientific boxes where checked. In the given case for example one of the boxes maybe, "Is there an anatomical distinction between retinas of sexes?" — If that is true then we can see if machine learning can detect such differences. Take computer science for example. A publication with the title "Professors create a machine that can think" would maybe published as "Professors create machine that passes Turing test [0]" Another example, can be that is medicine. The research in question maybe if a microbe A causes a flue. Instead of a publication being "Microbe A causes disease B". A better Publication IMO should revolve around, "Microbe A had passed Koch's postulates [1] for disease B" [0]:https://en.m.wikipedia.org/wiki/Turing_test https://en.m.wikipedia.org/wiki/Turing_test [1]:https://en.wikipedia.org/wiki/Koch's_postulates https://en.wikipedia.org/wiki/Koch's_postulates

robocat 5y ago

Slight reflections of eyelashes? Especially if blurred?

X6S1x6Okd1st 5y ago

Yup. I learned this the hard way on the last model I trained at scale. It was evaluating fit between two heterogeneous classes I sampled training & test split off of a large time window and got to work. It performed extremely well on test & training. Too good. I pulled a third sample from a completely different time window and it performed terribly. It turned out that both datasets were dominated by class A being sorted into always selecting great fit or poor fit, so the ML model learned to memorize the class A instances. This problem when away when I subselected down to only instances of class A that had examples of both good fit and poor fit.

8note 5y ago

I'm uncertain how they're defining sex in this. I think a good control would be to see how this model treats trans people at varying stages of transition

read_if_gay_ 5y ago
I don’t think its confidence will necessarily match the degree to which a trans person has transitioned. But indeed it would be interesting to see how trans people are treated at all.
Spooky23 5y ago
It depends on what you mean, as that community has a pretty vast and complex taxonomy. Estrogen levels affect the eyes, so given enough samples and context it may be surprising accurate for people getting that type of therapy.
AussieWog93 5y ago
I think the problem with that line of thinking is that even after fully transitioning, trans people still tend to have strong biological markers of the sex they've transitioned from. For a ML model to correctly guess gender identity, it'd need other cues that indicate a person would prefer to be referred to by a certain pronoun, such as clothing or facial hair.
- elil17 5y ago
  It really depends on the bio marker. Most bio markers are effected: fat distribution, hair texture (curly vs. straight), and even immune system function. I’d think that learning whether this model works on people who’ve taken hormones for gender transition could give some insight into what it’s detecting. Perhaps there are retinal features whose function or form are impacted by the presence of testosterone, for instance, or maybe it’s a different that forms before birth.

0x000000E2 5y ago

My theory: retinal photos often contain parts of eyelashes. Women's are statistically longer. Their eyes are also more almond shaped. Retinal photographs are hard to take. Often they contain significant amount of eyelashes and surrounding structure

ncmncm 5y ago
Serious question: Do you know what a retina is?
- antonvs 5y ago
  I'm not sure he knows what eyes are.

m463 5y ago

I read somewhere that an ML model could detect cancerous tumors, but it was keying off the ruler used to measure tumor size.

stevelosh 5y ago
Paper: https://www.sciencedirect.com/science/article/pii/S0022202X18322930 https://www.sciencedirect.com/science/article/pii/S0022202X1... > For instance, in our work, we noted that the algorithm appeared more likely to interpret images with rulers as malignant. Why? In our dataset, images with rulers were more likely to be malignant; thus the algorithm inadvertently “learned” that rulers are malignant.

raverbashing 5y ago

Yeah, when I read the headline I thought Cornea/Iris, not Retina, and there are a couple of ways an ML could "cheat" there. But still, there are ways of getting the resulting NN and applying some techniques to figure out which variable/patterns are responsible for most of the output.

ASalazarMX 5y ago

ML needs an EXPLAIN command so it can point the features that determined its answer.

mousepilot 5y ago
with neural networks, I'd imagine this feature would be difficult to implement.
- ASalazarMX 5y ago
  NN: This is a woman's retina. Scientist: How do you know that? NN: I just do, trust me.