2 ms·

A bit of an odd paper. I kept expecting it to reveal specific methodology problems, but they hardly do. Instead they accept that the ML papers are done correctl

A bit of an odd paper. I kept expecting it to reveal specific methodology problems, but they hardly do. Instead they accept that the ML papers are done correctly and find real correlations in the data. Their criticisms are things like "correlation is not causation", although the original papers didn't tend to argue that, or "what even is crime anyway?" (post-modernism). The closest to a solid criticism is one where they argue that facial feature differences were actually proxies for head tilt direction, which is fine, but hardly justifies a paper as grandiloquent as this one is.

grahamrow 2y ago
Agreed the prose was a bit over the top, but I expect such is the expectation in the authors’ circles. Much was general, but I did appreciate the simple reasoning on labeling inaccuracy from just scraping the web for pictures of individuals with autism spectrum disorders and mostly getting pictures of those who also had chromosomal disorders.
UncleMeat 2y ago
"What even is crime anyway" is not just froo-froo postmodernism. It is an extremely critical question when constructing or evaluating these sorts of systems and the absence of at least some discussion of it in papers building these systems is a clear epistemic problem, especially in a world where the authors of these systems are claiming that ML-based approaches achieve a goal of reducing bias.
- naasking 2y ago
  Crime is defined as a violation of specific laws of a specific country, province or municipality. What is the epistemic problem you think needs to be solved here?
jltsiren 2y ago
Methodological issues are usually good issues, because they are easy to spot. Here the issues are more about the data itself. If you use perfect methodology and put garbage in, you still get garbage out. Politicians often make the mistake of assuming that if they do something in order to get X, the outcome will be X. Researchers who are more familiar with the methodology than the subject matter often make similar mistakes. The data that is supposed to measure X never actually measures X. It measures something related but subtly different. If you want to make conclusions from the data, you need to understand those differences. You need to understand what the data is exactly and the process how it was collected. Including the details you think are irrelevant but aren't. The last part is particularly problematic for people who are not subject matter experts.

grahamrow 2y ago

Agreed the prose was a bit over the top, but I expect such is the expectation in the authors’ circles. Much was general, but I did appreciate the simple reasoning on labeling inaccuracy from just scraping the web for pictures of individuals with autism spectrum disorders and mostly getting pictures of those who also had chromosomal disorders.

UncleMeat 2y ago

"What even is crime anyway" is not just froo-froo postmodernism. It is an extremely critical question when constructing or evaluating these sorts of systems and the absence of at least some discussion of it in papers building these systems is a clear epistemic problem, especially in a world where the authors of these systems are claiming that ML-based approaches achieve a goal of reducing bias.

naasking 2y ago
Crime is defined as a violation of specific laws of a specific country, province or municipality. What is the epistemic problem you think needs to be solved here?

jltsiren 2y ago

Methodological issues are usually good issues, because they are easy to spot. Here the issues are more about the data itself. If you use perfect methodology and put garbage in, you still get garbage out. Politicians often make the mistake of assuming that if they do something in order to get X, the outcome will be X. Researchers who are more familiar with the methodology than the subject matter often make similar mistakes. The data that is supposed to measure X never actually measures X. It measures something related but subtly different. If you want to make conclusions from the data, you need to understand those differences. You need to understand what the data is exactly and the process how it was collected. Including the details you think are irrelevant but aren't. The last part is particularly problematic for people who are not subject matter experts.