3 ms·

The null hypothesis is that the participants are just as able to find those cat-like features in either perturbation of the image, and would pick the “right” on

by riwsky 3y ago

The null hypothesis is that the participants are just as able to find those cat-like features in either perturbation of the image, and would pick the “right” one only 50% of the time.

glitchc 3y ago
No, the null hypothesis is that neither image elicits a cat-like response. It requires asking an open-ended question such as "what does this image look like to you?" Once you prime the subject, you have artificially restricted the responses to "Not really, sure, kinda?" Remember, the ML model is objectively selecting cat with (very) high probability out of the entire corpus of possible responses. The human should be given the same range of possible responses to objectively establish bias. Since no human would say 'cat-like' for any of those images, it suggests a fairly large gap between human and machine perception. We've got a long road ahead of us.
- krisoft 3y ago
  > It requires asking an open-ended question such as "what does this image look like to you?" Because the answer to that is simply “it looks like flovers in a vase”. There is no question about human’s ability to tell what the image is. So much so that if you ask the humans to describe the images they would probably say something along the lines of “two identical images of the same flowers”. So you would think if you ask them which one is more cat like they will shrug and pick one at random. Since it is a nonsense question. Yet people were able to pick up the manipulated image as more cat-like. Which means there is some signal they are able to pick up on. > it suggests a fairly large gap between human and machine perception Naturally. That is not at dispute, neither is it the subject of this study.
  - glitchc 3y ago
    > Because the answer to that is simply “it looks like flovers in a vase”. There is no question about human’s ability to tell what the image is. Thank you. Because that right there means that any bias being measured is one that's introduced by the researchers. Ergo the study is useless.
    - krisoft 3y ago
      > bias being measured is one that's introduced by the researchers Yes. Intentionally. > Ergo the study is useless. No, it is not useless. It just not studying what you seem to think it does. There was a previous very well estabilished finding that you can manipulate images such that they look basically identical to humans but they completely change classification for the neural network. This is an estabilished fact. Have you heard about this? Are you aware of this? Because if you are not that would explain your confusion. This research is not proving that phenomenon. It has been proven before. It follows up on it and investigates what that “basically identical” means. There are two assumptions one can make: 1; The difference between the manipulated and the not manipulated image is so small humans won’t be able to tell which one is wich. 2; Since the manipulations appear to be neural network specific there is no reason to expect that a human will preceive them as the same class as the neural network. For all we know the humans might see the original image as more cat like, or the one we manipulated to look like a truck will appear more cat like to a human. These are the two statements they wanted to investigate.
- riwsky 3y ago
  “Neither image elicits a cat-like response” is a special case of both pictures being equally cat-like, and still gives you the 50/50 prior. And “not really, sure, kinda” is not a possible response to “which of these two copies of a vase picture looks more cat-like?”. I agree we have a long road ahead of us, but you clearly do not understand the design of this experiment.
  - glitchc 3y ago
    The design of the experiment is clear, it's the value that's circumspect. I suspect neither you nor the researchers have a firm grasp of survey methodology or statistics. All this experiment measures is the impact of a priming effect.

by riwsky 3y ago

The null hypothesis is that the participants are just as able to find those cat-like features in either perturbation of the image, and would pick the “right” one only 50% of the time.

glitchc 3y ago

No, the null hypothesis is that neither image elicits a cat-like response. It requires asking an open-ended question such as "what does this image look like to you?" Once you prime the subject, you have artificially restricted the responses to "Not really, sure, kinda?" Remember, the ML model is objectively selecting cat with (very) high probability out of the entire corpus of possible responses. The human should be given the same range of possible responses to objectively establish bias. Since no human would say 'cat-like' for any of those images, it suggests a fairly large gap between human and machine perception. We've got a long road ahead of us.

krisoft 3y ago
> It requires asking an open-ended question such as "what does this image look like to you?" Because the answer to that is simply “it looks like flovers in a vase”. There is no question about human’s ability to tell what the image is. So much so that if you ask the humans to describe the images they would probably say something along the lines of “two identical images of the same flowers”. So you would think if you ask them which one is more cat like they will shrug and pick one at random. Since it is a nonsense question. Yet people were able to pick up the manipulated image as more cat-like. Which means there is some signal they are able to pick up on. > it suggests a fairly large gap between human and machine perception Naturally. That is not at dispute, neither is it the subject of this study.
- glitchc 3y ago
  > Because the answer to that is simply “it looks like flovers in a vase”. There is no question about human’s ability to tell what the image is. Thank you. Because that right there means that any bias being measured is one that's introduced by the researchers. Ergo the study is useless.
  - krisoft 3y ago
    > bias being measured is one that's introduced by the researchers Yes. Intentionally. > Ergo the study is useless. No, it is not useless. It just not studying what you seem to think it does. There was a previous very well estabilished finding that you can manipulate images such that they look basically identical to humans but they completely change classification for the neural network. This is an estabilished fact. Have you heard about this? Are you aware of this? Because if you are not that would explain your confusion. This research is not proving that phenomenon. It has been proven before. It follows up on it and investigates what that “basically identical” means. There are two assumptions one can make: 1; The difference between the manipulated and the not manipulated image is so small humans won’t be able to tell which one is wich. 2; Since the manipulations appear to be neural network specific there is no reason to expect that a human will preceive them as the same class as the neural network. For all we know the humans might see the original image as more cat like, or the one we manipulated to look like a truck will appear more cat like to a human. These are the two statements they wanted to investigate.
riwsky 3y ago
“Neither image elicits a cat-like response” is a special case of both pictures being equally cat-like, and still gives you the 50/50 prior. And “not really, sure, kinda” is not a possible response to “which of these two copies of a vase picture looks more cat-like?”. I agree we have a long road ahead of us, but you clearly do not understand the design of this experiment.
- glitchc 3y ago
  The design of the experiment is clear, it's the value that's circumspect. I suspect neither you nor the researchers have a firm grasp of survey methodology or statistics. All this experiment measures is the impact of a priming effect.