6 ms·

Some good points in the article, but I disagree with the tone and the conclusion. > we’ve rewarded and lauded incremental researchers as innovators, increased

by b3kart 6y ago

Some good points in the article, but I disagree with the tone and the conclusion. > we’ve rewarded and lauded incremental researchers as innovators, increased their budgets so they can do even more incremental research There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning). In no way is machine learning unique in this. Sounds like the author is simply disappointed that, like in any other profession, day-to-day of a researcher is a slog and not a perennial intellectual festival. We've been in an exciting deep learning craze for a while, but it's silly to expect it to last forever. Back to the grind now. > Machine Learning Researchers can now engage in risk-free, high-income, high-prestige work Not sure what author means by "risk-free". Yes, if you're not publishing enough you're most likely not going to starve. Is that a bad thing? Is the survival instinct the only good motivator for doing good research? I would argue that there's plenty of risk, in that people who don't publish good research don't get very far in their academic careers, which in my view is good enough motivation. "They must do good research or starve" is a rather cynical take, especially from someone who seems to not be doing too badly for themselves. I'd rather more fields provided similar benefits. Maybe then going into science wouldn't be associated with so much sacrifice, so more smart people would choose science over investment banking or such, and we'd make more scientific progress faster. > CNNs use convolutions which are a generalization of matrix multiplication. A nitpick: CNNs are most definitely not a generalization of matrix multiplication. In fact, the opposite: you can view CNNs as a matrix multiplication with a particular matrix structure.

AndrewKemendo 6y ago
>We've been in an exciting deep learning craze for a while, but it's silly to expect it to last forever. Back to the grind now. This was effectively my response to hardmaru when this topic came up on reddit [1] Basically 2010-2018 was an open field for ML/DL research with old(ish) methods being rapidly applied to low hanging fruit and large datasets with newly cheap compute. Deepmind and others are actually making new methods but by and large are remixes of those same old approaches. The majority of different research out there trying other approaches (Numenta, OpenCog, Causal Calculus, anything Schmidhuber etc...) don't really get any love because it doesn't fit within the mass tensorflow/torch framework. [1]https://twitter.com/AndrewKemendo/status/1349387455552745473 https://twitter.com/AndrewKemendo/status/1349387455552745473
- counters 6y ago
  > Basically 2010-2018 was an open field for ML/DL research with old(ish) methods being rapidly applied to low hanging fruit and large datasets with newly cheap compute. Absolutely! And what many folks need to continue to remember is that many scientific disciplines and domains are really just starting to wrestle with the utility and implications of this first generation of deep learning tools and applications. I graduated with my PhD in atmospheric science from an R1 just over 4 years ago; at that time, very few people were looking at how DL provided useful tools for their work. These days, the field is inundated with folks playing with these tools and knocking tons of low-hanging fruit off the tree - it might not be "deep", revolutionary research, but it's fomenting a mini-revolution with respect to R2O and real applications of what had previously been somewhat niche science. There's no reason to think this trend won't continue. New tools let new generations of scientists take new stabs at their discipline, and of course the low-hanging fruit drops first as folks get their bearing, build skills/experience, and - most importantly - prove efficacy so that they can get funding for more ambitious work.
  - jabl 6y ago
    This seems to be the trend in every computational science field. Take something compute intensive, apply DL magic and get results in a fraction of the CPU time as the old approach. Similarly on the commercial side, plenty of opportunity to solve existing problems with DL. So while it may be true that DL research progress has slowed, still plenty to do in applying existing DL.
- p1esk 6y ago
  Those other approaches you mentioned don’t get much love because they don’t work. Their advocates worked on them for many years and have nothing to show for it.
  - b3kart 6y ago
    >they don't work. In the same way that ANNs didn't work for quite some time, until we've had the compute and the data to train them successfully? I get that it's important to prove that an idea is worthwhile, and the easiest way to do that is to use it to solve a practical problem. At the same time, I am conscious that we shouldn't put all our eggs in the deep learning basket: who knows where the ceiling is going to be. Don't get me wrong, I like deep learning, and you have to be silly not to admit how successful it has been. But the field would be so much more boring if not for the people with alternative views and ideas.
dadoge 6y ago
Couldn’t agree more. The author cites no compelling trend that ML is stagnating, esp relative to other disciplines. I’d also add: is it BAD if universities are churning out highly skilled workers when there is high demand for them? This seems to be more a rant of ML becoming more mainstream and accessible to a wider variety of students than anything. They admit that some of the most “innovative and challenging” problems are still around, but many do not focus on that. Ok, so what, there are maybe more tiers and splitting of work into subdomains, some more “vocational” and some more “research-y” in nature. Is this a BAD thing?
adamc 6y ago
Having a sinecure from which you can make forays into new ideas is far from the reality of most jobs. It is more akin to the "gentlemen scientists" of yore. Whether this is actually a good way to stimulate knowledge accumulation seems like something that should be studied rather than assumed.
ppod 6y ago
Agree, and the article sets up a weird dichotomy between empiricism and innovation. The empirical work of adapting a Transformer architecture to be smaller or faster or work better on a particular task is scientific work. Like a lot of Comp Sci, a lot of it is engineering, but there is a clear scientific component in that you come up with a model first, then implement and test it (using a lot of pre-built components). It's certainly as scientific as e.g. designing a medicine.
MaxBarraclough 6y ago
See also the recent discussion on Peer-reviewed papers are getting increasingly boring. https://news.ycombinator.com/item?id=25605156 https://news.ycombinator.com/item?id=25605156
jjtheblunt 6y ago
agreed with the nitpick wholeheartedly
tracyhenry 6y ago
> It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning). Is the deep learning really the result of incremental research? The SOTA chasing frenzy comes after the discovery of deep learning. The motivation of incremental research can hardly be justified as to discover the next deep learning, although they might do.
- b3kart 6y ago
  Of course it is, if we agree that incremental research isn't necessarily "SOTA chasing". Deep learning, and in fact all scientific breakthroughs, did not materialize out of nowhere. They're a culmination of a long sequence of scientific papers (some more incremental than others), conference presentations, poster sessions, hallway conversations, even coffee chats. It's not some dude disappearing into the forest for a few years and coming back with a revolutionary idea (maybe in movies). Everything that helped shape the scientist's thinking has contributed to the idea, even if sometimes the idea is so revolutionary that it's hard to see the direct link. Besides, show me a scientific paper with no references. :-) It's a shame that this is rarely acknowledged and more often than not we talk about "that guy who invented X and got a Nobel prize for it". But us mere mortals can take solace in knowing that even if we didn't change the world with our own ideas, it's possible that we've influenced someone who has.
- BlueTemplar 6y ago
  What does SOTA means ? "State of the Art" ?
  - b3kart 6y ago
    Yes. Alternatively, “look, ma, my number is the biggest number”. Apologise for the sass, just not very happy that SOTA chasing is often discouraging original contributions that would actually move the field forward.
dkislyuk 6y ago
>> CNNs use convolutions which are a generalization of matrix multiplication. > A nitpick: CNNs are most definitely not a generalization of matrix multiplication. In fact, the opposite: you can view CNNs as a matrix multiplication with a particular matrix structure. Neither is really a generalization. Any matrix multiplication can be implemented with a convolution function, and yet any convolution can be represented as a matrix multiplication via im2col.
- bnegreve 6y ago
  Discrete convolutions are a special case of matrix multiplication in the sense that a discrete convolution can be formulated as the product between a Toeplitz matrix and the input signal. [1] https://en.wikipedia.org/wiki/Toeplitz_matrix#Discrete_convolution https://en.wikipedia.org/wiki/Toeplitz_matrix#Discrete_convo...
JPKab 6y ago
Just to be clear, there's a lot of researchers making money that aren't producing squat. There are many AI ethicists I admire, and there are many that can't code, can't produce, and just spend their time getting into arguments with people like Yann LeCunn, who actually have produced groundbreaking research.
- SilkStocking 6y ago
  LeCon's work wasn't groundbreaking; he was saved from oblivion only by Moore's Law. LeCun stood still doing the same thing for decades. But when wave after wave of faster hardware arrived he was finally declared successful. So if you want to credit success to dumb luck, or to stubbornness, or insanity (as Einstein said: "doing the same thing over and over again and expecting different results." then LeCon is your man. I don't know of any quote where LeCun says something to the effect of: "Well, I knew the hardware would speed up and then my neural net software would work exactly as I designed it to and manifest AI. So I waited twenty years." IOW It wasn't the LeCunn's surfboard(his software) that handed him success, it was the wave(advancing hardware speeds): https://www.wired.com/2017/04/race-make-ai-chips-everything-heating-fast/ https://www.wired.com/2017/04/race-make-ai-chips-everything-...
  - jpfr 6y ago
    I hope calling him a “con” wasn’t intentional. He speaks French after all...
  - njanirudh 6y ago
    Are you telling me CNN are not groundbreaking?
  - BlueTemplar 6y ago
    > "Well, I knew the hardware would speed up and then my neural net software would work exactly as I designed it to and manifest AI. So I waited twenty years." You're talking like we only discovered Moore's law now, instead of it being a good bet for the last half-century...
kkylin 6y ago
I think you are exactly right, and want to add a couple things (): - It's easy to recognize scientific / technological revolutions in hindsight, but at the time they're anything but. I think most recognize the importance of persistence in the progenitor of an idea. What's often missed in these discussions is how important the subsequent incremental progress is: working out consequences of a theoretical insight, figuring out what you can build on top of a new tech, etc. - I don't have a strong opinion about the optimal amount of "risk," but I do think making risk existential (as in one would starve without a research breakthrough) would have the opposite effect, because while we all say we like people to take risks, what we really mean is like people to take risks and suceeed. And yes, too little risk can breed complacency. And yes, convolutions are a specific kind of linear transformation, whereas matrices represent linear transformations on vector spaces. The specific structure is that convolutions represent linear transformations that are translation-invariant, a property that many types of data (e.g., images) have, at least approximately. () My perspective as an academic but not a computer scientist.
- dcow 6y ago
  Where in the world did you come across the () ... () style? I like it.
  - kkylin 6y ago
    Thanks. I actually type "(asterisk)", forgetting every single time that HN uses asterisks for formatting, and too lazy to go back and fix it...
  - valzam 6y ago
    Found the LISP programmer
  - deleted 6y ago
    [deleted]
joe_the_user 6y ago
There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning). In no way is machine learning unique in this. Except ... while machine learning is great, has made important and significant strides, it's not a yet science. It involves essentially a series of sophisticated, mathematically informed recipes for feeding data to giant algorithms and having them create something useful (maybe very useful but still). An analogy from a couple years ago is bridge building before physics. You accumulate rules of thumb, you get a vague understand what works. You get better. But you aren't producing a systematic field. And that implies merely advancement isn't necessarily progress (which isn't to say there's no progress but building larger SOTA isn't that as the article notes).
- refactor_master 6y ago
  > An analogy from a couple years ago is bridge building before physics. You accumulate rules of thumb, you get a vague understand what works. You get better. But you aren't producing a systematic field. Sounds like science to me; systematic recording of (perceived) cause and effect. Arguably, deep learning and cell biology both appear like equal parts pure wizardry and flailing in the dark, but maybe that’s just because we haven’t gathered enough pieces yet, and not necessarily because people are doing the wrong things, thus failing to advance?
- b3kart 6y ago
  I would agree that as a science machine learning is in its infancy. There's statistical learning theory, but arguably that's a fairly small subfield, if not a separate field completely. ML has largely been an engineering enterprise so far, but in my experience the interest in developing underlying theory has ramped up in recent years. What we should be careful about is to not be too strict when defining "science". In my view the goal of any scientific field is to build understanding that is useful for predicting outcomes of experiments. Now, this understanding could be defined mathematically, but it doesn't have to be. Don't see why building heuristics can't be a part of this, assuming such heuristics reliably predict outcomes of experiments.
- 6y ago
spullara 6y ago
I'd argue that deep learning isn't even new. I was doing multi-layer neural networks back in 1992. The compute power though was woefully underpowered to actually do what they are using it for today.
- BlueTemplar 6y ago
  Yeah, "multilayer perceptrons" date back to the 80's...

by b3kart 6y ago

AndrewKemendo 6y ago

>We've been in an exciting deep learning craze for a while, but it's silly to expect it to last forever. Back to the grind now. This was effectively my response to hardmaru when this topic came up on reddit [1] Basically 2010-2018 was an open field for ML/DL research with old(ish) methods being rapidly applied to low hanging fruit and large datasets with newly cheap compute. Deepmind and others are actually making new methods but by and large are remixes of those same old approaches. The majority of different research out there trying other approaches (Numenta, OpenCog, Causal Calculus, anything Schmidhuber etc...) don't really get any love because it doesn't fit within the mass tensorflow/torch framework. [1]https://twitter.com/AndrewKemendo/status/1349387455552745473 https://twitter.com/AndrewKemendo/status/1349387455552745473

counters 6y ago
> Basically 2010-2018 was an open field for ML/DL research with old(ish) methods being rapidly applied to low hanging fruit and large datasets with newly cheap compute. Absolutely! And what many folks need to continue to remember is that many scientific disciplines and domains are really just starting to wrestle with the utility and implications of this first generation of deep learning tools and applications. I graduated with my PhD in atmospheric science from an R1 just over 4 years ago; at that time, very few people were looking at how DL provided useful tools for their work. These days, the field is inundated with folks playing with these tools and knocking tons of low-hanging fruit off the tree - it might not be "deep", revolutionary research, but it's fomenting a mini-revolution with respect to R2O and real applications of what had previously been somewhat niche science. There's no reason to think this trend won't continue. New tools let new generations of scientists take new stabs at their discipline, and of course the low-hanging fruit drops first as folks get their bearing, build skills/experience, and - most importantly - prove efficacy so that they can get funding for more ambitious work.
- jabl 6y ago
  This seems to be the trend in every computational science field. Take something compute intensive, apply DL magic and get results in a fraction of the CPU time as the old approach. Similarly on the commercial side, plenty of opportunity to solve existing problems with DL. So while it may be true that DL research progress has slowed, still plenty to do in applying existing DL.
p1esk 6y ago
Those other approaches you mentioned don’t get much love because they don’t work. Their advocates worked on them for many years and have nothing to show for it.
- b3kart 6y ago
  >they don't work. In the same way that ANNs didn't work for quite some time, until we've had the compute and the data to train them successfully? I get that it's important to prove that an idea is worthwhile, and the easiest way to do that is to use it to solve a practical problem. At the same time, I am conscious that we shouldn't put all our eggs in the deep learning basket: who knows where the ceiling is going to be. Don't get me wrong, I like deep learning, and you have to be silly not to admit how successful it has been. But the field would be so much more boring if not for the people with alternative views and ideas.

dadoge 6y ago

Couldn’t agree more. The author cites no compelling trend that ML is stagnating, esp relative to other disciplines. I’d also add: is it BAD if universities are churning out highly skilled workers when there is high demand for them? This seems to be more a rant of ML becoming more mainstream and accessible to a wider variety of students than anything. They admit that some of the most “innovative and challenging” problems are still around, but many do not focus on that. Ok, so what, there are maybe more tiers and splitting of work into subdomains, some more “vocational” and some more “research-y” in nature. Is this a BAD thing?

adamc 6y ago

Having a sinecure from which you can make forays into new ideas is far from the reality of most jobs. It is more akin to the "gentlemen scientists" of yore. Whether this is actually a good way to stimulate knowledge accumulation seems like something that should be studied rather than assumed.

ppod 6y ago

Agree, and the article sets up a weird dichotomy between empiricism and innovation. The empirical work of adapting a Transformer architecture to be smaller or faster or work better on a particular task is scientific work. Like a lot of Comp Sci, a lot of it is engineering, but there is a clear scientific component in that you come up with a model first, then implement and test it (using a lot of pre-built components). It's certainly as scientific as e.g. designing a medicine.

MaxBarraclough 6y ago

See also the recent discussion on Peer-reviewed papers are getting increasingly boring. https://news.ycombinator.com/item?id=25605156 https://news.ycombinator.com/item?id=25605156

jjtheblunt 6y ago

agreed with the nitpick wholeheartedly

tracyhenry 6y ago

> It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning). Is the deep learning really the result of incremental research? The SOTA chasing frenzy comes after the discovery of deep learning. The motivation of incremental research can hardly be justified as to discover the next deep learning, although they might do.

b3kart 6y ago
Of course it is, if we agree that incremental research isn't necessarily "SOTA chasing". Deep learning, and in fact all scientific breakthroughs, did not materialize out of nowhere. They're a culmination of a long sequence of scientific papers (some more incremental than others), conference presentations, poster sessions, hallway conversations, even coffee chats. It's not some dude disappearing into the forest for a few years and coming back with a revolutionary idea (maybe in movies). Everything that helped shape the scientist's thinking has contributed to the idea, even if sometimes the idea is so revolutionary that it's hard to see the direct link. Besides, show me a scientific paper with no references. :-) It's a shame that this is rarely acknowledged and more often than not we talk about "that guy who invented X and got a Nobel prize for it". But us mere mortals can take solace in knowing that even if we didn't change the world with our own ideas, it's possible that we've influenced someone who has.
BlueTemplar 6y ago
What does SOTA means ? "State of the Art" ?
- b3kart 6y ago
  Yes. Alternatively, “look, ma, my number is the biggest number”. Apologise for the sass, just not very happy that SOTA chasing is often discouraging original contributions that would actually move the field forward.

dkislyuk 6y ago

>> CNNs use convolutions which are a generalization of matrix multiplication. > A nitpick: CNNs are most definitely not a generalization of matrix multiplication. In fact, the opposite: you can view CNNs as a matrix multiplication with a particular matrix structure. Neither is really a generalization. Any matrix multiplication can be implemented with a convolution function, and yet any convolution can be represented as a matrix multiplication via im2col.

bnegreve 6y ago
Discrete convolutions are a special case of matrix multiplication in the sense that a discrete convolution can be formulated as the product between a Toeplitz matrix and the input signal. [1] https://en.wikipedia.org/wiki/Toeplitz_matrix#Discrete_convolution https://en.wikipedia.org/wiki/Toeplitz_matrix#Discrete_convo...

JPKab 6y ago

Just to be clear, there's a lot of researchers making money that aren't producing squat. There are many AI ethicists I admire, and there are many that can't code, can't produce, and just spend their time getting into arguments with people like Yann LeCunn, who actually have produced groundbreaking research.

SilkStocking 6y ago
LeCon's work wasn't groundbreaking; he was saved from oblivion only by Moore's Law. LeCun stood still doing the same thing for decades. But when wave after wave of faster hardware arrived he was finally declared successful. So if you want to credit success to dumb luck, or to stubbornness, or insanity (as Einstein said: "doing the same thing over and over again and expecting different results." then LeCon is your man. I don't know of any quote where LeCun says something to the effect of: "Well, I knew the hardware would speed up and then my neural net software would work exactly as I designed it to and manifest AI. So I waited twenty years." IOW It wasn't the LeCunn's surfboard(his software) that handed him success, it was the wave(advancing hardware speeds): https://www.wired.com/2017/04/race-make-ai-chips-everything-heating-fast/ https://www.wired.com/2017/04/race-make-ai-chips-everything-...
- jpfr 6y ago
  I hope calling him a “con” wasn’t intentional. He speaks French after all...
- njanirudh 6y ago
  Are you telling me CNN are not groundbreaking?
- BlueTemplar 6y ago
  > "Well, I knew the hardware would speed up and then my neural net software would work exactly as I designed it to and manifest AI. So I waited twenty years." You're talking like we only discovered Moore's law now, instead of it being a good bet for the last half-century...

kkylin 6y ago

I think you are exactly right, and want to add a couple things (): - It's easy to recognize scientific / technological revolutions in hindsight, but at the time they're anything but. I think most recognize the importance of persistence in the progenitor of an idea. What's often missed in these discussions is how important the subsequent incremental progress is: working out consequences of a theoretical insight, figuring out what you can build on top of a new tech, etc. - I don't have a strong opinion about the optimal amount of "risk," but I do think making risk existential (as in one would starve without a research breakthrough) would have the opposite effect, because while we all say we like people to take risks, what we really mean is like people to take risks and suceeed. And yes, too little risk can breed complacency. And yes, convolutions are a specific kind of linear transformation, whereas matrices represent linear transformations on vector spaces. The specific structure is that convolutions represent linear transformations that are translation-invariant, a property that many types of data (e.g., images) have, at least approximately. () My perspective as an academic but not a computer scientist.

dcow 6y ago
Where in the world did you come across the () ... () style? I like it.
- kkylin 6y ago
  Thanks. I actually type "(asterisk)", forgetting every single time that HN uses asterisks for formatting, and too lazy to go back and fix it...
- valzam 6y ago
  Found the LISP programmer
- deleted 6y ago
  [deleted]

joe_the_user 6y ago

There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning). In no way is machine learning unique in this. Except ... while machine learning is great, has made important and significant strides, it's not a yet science. It involves essentially a series of sophisticated, mathematically informed recipes for feeding data to giant algorithms and having them create something useful (maybe very useful but still). An analogy from a couple years ago is bridge building before physics. You accumulate rules of thumb, you get a vague understand what works. You get better. But you aren't producing a systematic field. And that implies merely advancement isn't necessarily progress (which isn't to say there's no progress but building larger SOTA isn't that as the article notes).

refactor_master 6y ago
> An analogy from a couple years ago is bridge building before physics. You accumulate rules of thumb, you get a vague understand what works. You get better. But you aren't producing a systematic field. Sounds like science to me; systematic recording of (perceived) cause and effect. Arguably, deep learning and cell biology both appear like equal parts pure wizardry and flailing in the dark, but maybe that’s just because we haven’t gathered enough pieces yet, and not necessarily because people are doing the wrong things, thus failing to advance?
b3kart 6y ago
I would agree that as a science machine learning is in its infancy. There's statistical learning theory, but arguably that's a fairly small subfield, if not a separate field completely. ML has largely been an engineering enterprise so far, but in my experience the interest in developing underlying theory has ramped up in recent years. What we should be careful about is to not be too strict when defining "science". In my view the goal of any scientific field is to build understanding that is useful for predicting outcomes of experiments. Now, this understanding could be defined mathematically, but it doesn't have to be. Don't see why building heuristics can't be a part of this, assuming such heuristics reliably predict outcomes of experiments.
6y ago

spullara 6y ago

I'd argue that deep learning isn't even new. I was doing multi-layer neural networks back in 1992. The compute power though was woefully underpowered to actually do what they are using it for today.

BlueTemplar 6y ago
Yeah, "multilayer perceptrons" date back to the 80's...