7 ms·

Troubling trends in machine learning scholarship (2018)

197 points by scottlocklin 7y ago

joe_the_user 7y ago
"In this paper, we focus on the following four patterns that appear to us to be trending in ML scholarship: (i) failure to distinguish between explanation and speculation; (ii) failure to identify the sources of empirical gains, e.g., emphasizing unnecessary modifications to neural architectures when gains actually stem from hyper-parameter tuning; (iii) mathiness: the use of mathematics that obfuscates or impresses rather than clarifies, e.g., by confusing technical and non-technical concepts; and (iv) misuse of language, e.g., by choosing terms of art with colloquial connotations or by overloading established technical terms. " To their credit, the authors actually own-up to doing this themselves in various papers. It seems like a way to describe the situation is that neural nets have become such computational monsters that talking about them exactly becomes very difficult with the language opaque and ambiguous.
- gmueckl 7y ago
  I'd say a lack of a proper fundamental understanding of trained neural networks is the main cause. People throw NNs at any problem they can think of, get good results and when they want to publish, they come up with an explanation that is more esoteric than founded in solid theory because the monster they generated is so inscrutable.
  - recursive 7y ago
    The stuff is what the stuff is, brother. https://youtu.be/ajGX7odA87k?t=931 https://youtu.be/ajGX7odA87k?t=931
    - gmueckl 7y ago
      Thanks! This is a great talk.
  - joe_the_user 7y ago
    What you're saying is a bit tautological in a way you may not intend. What the paper describes is those research papers which aim at, that, giving a fundamental understanding of a trained neural network. That the papers are satisfied with "it works" stands in the way of anyone having this fundamental understanding.
  - netcan 7y ago
    Re:inscrutable We'll have to see where this all leads, to ver the next few years/decades. Maybe someone will manage to combine "a proper fundamental understanding of trained neural networks," and good results. That'll lead to (perhaps) good theories, to explain the good results. If "good results" continue to outpace our understanding of wtf the useful NN is up to... It'll have to be studied expirementaly, like the way we study biology. Ie, we might see CS theory adapt from "mathematical," to "scientific" to in its methods and theories. The current trajectory seems to be heading here. There is tremendous interest and resources in NNs. As they become more commercially important, interest and resources dedicated to developing them increases. They only need the NNs to work, not to be scriptable. Scientists are not just going to give up though. They'll study NNs expirementaly as black boxes if that's all they have.
- im3w1l 7y ago
  Regarding (i), isn't that just because it's an immature field? People don't really know where the border between knowledge and speculation lies.
  - wish5031 7y ago
    It’s questionable how “immature” ML really is. Most methods that get used were initially designed 50+ years ago, with various improvements over time. E.g., neural networks were invented in the 1950s, backprop was introduced in the 80s, architectures like LSTM and CNN in the 90s, etc. The only thing that’s really new is the amount of computational power at our hands. That has allowed us to shift from relatively simpler methods to more powerful but opaque methods like NNs. They just don’t lend themselves to easy analysis because it’s a lot harder to explain why inputs to these ML systems map to their respective outputs. Hence, attempts at drawing the connection between inputs and outputs become more speculative.
    - im3w1l 7y ago
      Sure people had these ideas, but there was no consensus that they were the right ones. As such they weren't studied as much as they are now.
  - jononor 7y ago
    The people who make a paper have to know where the border is for their particular paper. That is, which things in the paper are claims with evidence to back it up - and which things are speculations about what might be an explanation. That some things are speculative is not such a big deal, as long as it is clearly marked as such. Then someone else can investigate it properly in another paper. Or people can use it in another work, by treating it as an assumption that they can verify whether holds, and then make use of.
ramraj07 7y ago
This is not just isolated to machine learning, but most technical fields (at least one other to my knowledge). I used to work in microscopy image analysis and the papers often would obfuscate the fact that they were not exactly doing anything new by using what looks like fancy math and some trendy names. One of the most outrageous examples is this "high profile" paper that says it does compressive sensing with superresolution microscopy - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3477591/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3477591/ except I don't think they do; the math when you remove the bullshit sounds more like deconvolution than anything else (and the results are only as good). Yet, it got reviewed and accepted by Nature Methods, and is cited by 360 papers already. Why? Apparently no one in this field knows what compressive sensing really means. At least one professor in the field when I confronted him, just said he doesn't have time to go through compressive sensing literature first before evaluating this paper. What's the root cause? Frankly I'd argue the majority of professors nowadays aren't smart in innovation but smart in hustling. Because hustlers are who become professors in today's academic climate. They are able to publish good papers still if the field isn't mature, but if the field is saturated, you have to be really smart to make meaningful progress, and these hustlers are not. So they just try to find some way to wrap meaningless progress in fancy math and shove it in papers. The papers also go to reviewers who are similar hustlers (not every paper can be reviewed by Hinton ) so they either don't notice the problem or they do but let it slide because it's just their colleague (yay for journals asking for "suggested reviewers" to the authors itself).
- deepnotderp 7y ago
  I'm somewhat familiar with compressed sensing, but I only glanced through the linked paper for like a minute, but it looks like you're right. They seem to be taking the overlapped returns and doing some sort of weird sparse basis deconvolution. Specifically "Using compressed sensing, we analyzed a simulated image with 100 molecules randomly distributed in a 4 µm × 4 µm region (Fig. 1a). Although the images of individual molecules completely overlapped, we could identify almost all of the molecules. " Unless that is a close enough approximation to a random projection? I'm not that familiar with STORM.
  - 7y ago
inertially 7y ago
"It is difficult to get a man to understand something when his salary depends upon his not understanding it." - Upton Sinclai
thanks1banks2 7y ago
> "...overloading established technical terms. " Subtle. ;)
cosmic_ape 7y ago
previous discussion here: https://news.ycombinator.com/item?id=17497273 https://news.ycombinator.com/item?id=17497273

197 points by scottlocklin 7y ago

"In this paper, we focus on the following four patterns that appear to us to be trending in ML scholarship: (i) failure to distinguish between explanation and speculation; (ii) failure to identify the sources of empirical gains, e.g., emphasizing unnecessary modifications to neural architectures when gains actually stem from hyper-parameter tuning; (iii) mathiness: the use of mathematics that obfuscates or impresses rather than clarifies, e.g., by confusing technical and non-technical concepts; and (iv) misuse of language, e.g., by choosing terms of art with colloquial connotations or by overloading established technical terms. " To their credit, the authors actually own-up to doing this themselves in various papers. It seems like a way to describe the situation is that neural nets have become such computational monsters that talking about them exactly becomes very difficult with the language opaque and ambiguous.

gmueckl 7y ago
I'd say a lack of a proper fundamental understanding of trained neural networks is the main cause. People throw NNs at any problem they can think of, get good results and when they want to publish, they come up with an explanation that is more esoteric than founded in solid theory because the monster they generated is so inscrutable.
- recursive 7y ago
  The stuff is what the stuff is, brother. https://youtu.be/ajGX7odA87k?t=931 https://youtu.be/ajGX7odA87k?t=931
  - gmueckl 7y ago
    Thanks! This is a great talk.
- joe_the_user 7y ago
  What you're saying is a bit tautological in a way you may not intend. What the paper describes is those research papers which aim at, that, giving a fundamental understanding of a trained neural network. That the papers are satisfied with "it works" stands in the way of anyone having this fundamental understanding.
- netcan 7y ago
  Re:inscrutable We'll have to see where this all leads, to ver the next few years/decades. Maybe someone will manage to combine "a proper fundamental understanding of trained neural networks," and good results. That'll lead to (perhaps) good theories, to explain the good results. If "good results" continue to outpace our understanding of wtf the useful NN is up to... It'll have to be studied expirementaly, like the way we study biology. Ie, we might see CS theory adapt from "mathematical," to "scientific" to in its methods and theories. The current trajectory seems to be heading here. There is tremendous interest and resources in NNs. As they become more commercially important, interest and resources dedicated to developing them increases. They only need the NNs to work, not to be scriptable. Scientists are not just going to give up though. They'll study NNs expirementaly as black boxes if that's all they have.
im3w1l 7y ago
Regarding (i), isn't that just because it's an immature field? People don't really know where the border between knowledge and speculation lies.
- wish5031 7y ago
  It’s questionable how “immature” ML really is. Most methods that get used were initially designed 50+ years ago, with various improvements over time. E.g., neural networks were invented in the 1950s, backprop was introduced in the 80s, architectures like LSTM and CNN in the 90s, etc. The only thing that’s really new is the amount of computational power at our hands. That has allowed us to shift from relatively simpler methods to more powerful but opaque methods like NNs. They just don’t lend themselves to easy analysis because it’s a lot harder to explain why inputs to these ML systems map to their respective outputs. Hence, attempts at drawing the connection between inputs and outputs become more speculative.
  - im3w1l 7y ago
    Sure people had these ideas, but there was no consensus that they were the right ones. As such they weren't studied as much as they are now.
- jononor 7y ago
  The people who make a paper have to know where the border is for their particular paper. That is, which things in the paper are claims with evidence to back it up - and which things are speculations about what might be an explanation. That some things are speculative is not such a big deal, as long as it is clearly marked as such. Then someone else can investigate it properly in another paper. Or people can use it in another work, by treating it as an assumption that they can verify whether holds, and then make use of.

ramraj07 7y ago

This is not just isolated to machine learning, but most technical fields (at least one other to my knowledge). I used to work in microscopy image analysis and the papers often would obfuscate the fact that they were not exactly doing anything new by using what looks like fancy math and some trendy names. One of the most outrageous examples is this "high profile" paper that says it does compressive sensing with superresolution microscopy - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3477591/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3477591/ except I don't think they do; the math when you remove the bullshit sounds more like deconvolution than anything else (and the results are only as good). Yet, it got reviewed and accepted by Nature Methods, and is cited by 360 papers already. Why? Apparently no one in this field knows what compressive sensing really means. At least one professor in the field when I confronted him, just said he doesn't have time to go through compressive sensing literature first before evaluating this paper. What's the root cause? Frankly I'd argue the majority of professors nowadays aren't smart in innovation but smart in hustling. Because hustlers are who become professors in today's academic climate. They are able to publish good papers still if the field isn't mature, but if the field is saturated, you have to be really smart to make meaningful progress, and these hustlers are not. So they just try to find some way to wrap meaningless progress in fancy math and shove it in papers. The papers also go to reviewers who are similar hustlers (not every paper can be reviewed by Hinton ) so they either don't notice the problem or they do but let it slide because it's just their colleague (yay for journals asking for "suggested reviewers" to the authors itself).

deepnotderp 7y ago
I'm somewhat familiar with compressed sensing, but I only glanced through the linked paper for like a minute, but it looks like you're right. They seem to be taking the overlapped returns and doing some sort of weird sparse basis deconvolution. Specifically "Using compressed sensing, we analyzed a simulated image with 100 molecules randomly distributed in a 4 µm × 4 µm region (Fig. 1a). Although the images of individual molecules completely overlapped, we could identify almost all of the molecules. " Unless that is a close enough approximation to a random projection? I'm not that familiar with STORM.
- 7y ago

inertially 7y ago

"It is difficult to get a man to understand something when his salary depends upon his not understanding it." - Upton Sinclai

thanks1banks2 7y ago

> "...overloading established technical terms. " Subtle. ;)

cosmic_ape 7y ago

previous discussion here: https://news.ycombinator.com/item?id=17497273 https://news.ycombinator.com/item?id=17497273