4 ms·

So what we're seeing is these fields pre-deep learning were mathematical disciplines making steady progress on well-understood foundations. Then deep learning

by habitue 10y ago

So what we're seeing is these fields pre-deep learning were mathematical disciplines making steady progress on well-understood foundations. Then deep learning came in, was exceptionally effective at problems that had been difficult to crack, and people shifted focus because it seems weird to be diddling around with incremental gains on techniques that are significantly less effective. What this created though, was a clean break from a field with a theoretical foundation like what we expect in CS usually, to an empirical foundation, like what we see in psychology, sociology, medicine etc. The theory will eventually catch up, but the fact that "it works" really trumps having a full understanding of why it works (at least for now).

reader5000 10y ago
Also, deep net problems are different than traditional CS because we can mathematically define when say a list of numbers is sorted but not when "this image looks photorealistic". Effectively, deep nets are good at tricking human visual cortex, so I'm not sure theres a deeper mathematical reason they work well, in the same way that the reason mp3 compression works so well is that they tested it against human hearing. But yes, all deep learning is based on gradient descent which is a greedy heuristic algorithm and it's not a priori apparent it should do anything interesting. But it does.
- mistercow 10y ago
  > in the same way that the reason mp3 compression works so well But there are deeper reasons that mp3 works well, besides "tested against humans". It's just that those reasons are about psychoacoustics. The most obvious example is that humans are bad at discerning amplitude and phase differences at high frequencies. I expect that as we get deeper into the theory around why particular algorithms work, we'll find similar answers. We'll be able to say things like "when an image has the following statistical properties, we're likely to interpret it as a cat". Of course, right now we can trivially say that if by "the following statistical properties" we mean "activates this specifically trained network in a particular way", but the point is that we can probably boil that down into deeper (and simpler) statements about how human image recognition works. I can't find a link about it now, but I remember reading that for their Translate app, Google created a way to lossily compress their character recognition neural net. That kind of research seems likely to me to lead to a better understanding of how to boil down the insights that these learning algorithms are gleaning. This Two Minute Papers video has some thinking along the same lines: https://www.youtube.com/watch?v=ZBWTD2aNb_o https://www.youtube.com/watch?v=ZBWTD2aNb_o But I'd also point out that the human visual cortex is itself a bunch of models about the real world. It's gotten that way through a combination of evolution and learning. When engineers train neural networks to do tricks like TFA shows, those networks are effectively modeling the real world in two different ways: directly, and by proxy in modeling human visual systems. So the insights we can find by studying them are likely to tell us not only about our own vision, but about the world itself. Finally, it's worth noting that graphics algorithms have always been about human visual systems and the real physical world, as opposed to deep mathematical truths. Look at this paper for example: http://www.jiansun.org/papers/Dehaze_CVPR2009.pdf http://www.jiansun.org/papers/Dehaze_CVPR2009.pdf The oversimplified plain English insight those researchers had is "For a photo that doesn't contain haze, you usually can find a fully saturated pixel within some reasonably small neighborhood of any point in the image." There's no deep mathematical insight there, but there is an important insight into the statistical nature of natural images – a fact about the physical world. And they use that insight to great effect: they can remove haze from photographs, and even use the haze as a way of estimating a depth map. So sure, while you're right that there probably aren't many deep mathematical insights to find from the results of deep learning algorithms (which is generally true of graphics algorithms anyway), I wouldn't gloss over the deep practical insights that they encode.
scott_s 10y ago
Agreed with the first paragraph, but I think this sort of behavior is normal for most disciplines. That is, it's not CS looking like psychology or sociology, it's normal for CS to do this. I think we're seeing a similar thing right now in biology with CRISPR.
- polyfractal 10y ago
  FWIW, CRISPR isn't a great analogy. It's a well understood mechanism, and not really all that different from standard genome editing techniques like restriction enzymes, zinc-finger nucleases, etc. It's new and flashy and super powerful -- and the applications of CRISPR are still being discovered -- but the mechanism is understood and simple.
  - scott_s 10y ago
    That wasn't the aspect I was going for. More for a period of incremental improvements in understanding, from which emerges a particular thing with enormous potential, and then the field is dominated by applications of that new thing.
kcorbitt 10y ago
> The theory will eventually catch up Maybe... but the emergent behavior in a complicated system (and these networks are only getting more complicated, not less) is likely to quickly become more complex than the human mind can reasonably be expected to understand, given any amount of time. We actually know a lot less about biology, for example, than your typical "biology 101" course would lead you to believe. It's pretty cool that we figured out how DNA translates into proteins, but actually figuring out how a cell decides which DNA to express at any given time, much less what any arbitrary protein actually does, takes you right back into guess-and-check territory. There are heuristics you can use to eliminate some guesses more quickly than others, but we're so far away from a unified explanatory theory of biology that I'm doubtful one will ever be developed.
- jszymborski 10y ago
  I mean, that's the nature of empirical sciences. You establish a model that fits observations you've made, but inevitably their is some chaotic, sometimes imperceptible noise that gets averaged out that represents some further level of complexity. I wouldn't argue that it means we don't know much, in the end you can argue we don't know anything because all empirical sciences are beholden to the belief of abstractions that can be attributed to some imperceptible abstraction until you (don't) reach the bottom of the stack of turtles.
- deleted 10y ago
  [deleted]
VikingCoder 10y ago
It's data-driven... rather than theory-driven. In some respects this is good, in other respects it's awful.

by habitue 10y ago

reader5000 10y ago

Also, deep net problems are different than traditional CS because we can mathematically define when say a list of numbers is sorted but not when "this image looks photorealistic". Effectively, deep nets are good at tricking human visual cortex, so I'm not sure theres a deeper mathematical reason they work well, in the same way that the reason mp3 compression works so well is that they tested it against human hearing. But yes, all deep learning is based on gradient descent which is a greedy heuristic algorithm and it's not a priori apparent it should do anything interesting. But it does.

mistercow 10y ago
> in the same way that the reason mp3 compression works so well But there are deeper reasons that mp3 works well, besides "tested against humans". It's just that those reasons are about psychoacoustics. The most obvious example is that humans are bad at discerning amplitude and phase differences at high frequencies. I expect that as we get deeper into the theory around why particular algorithms work, we'll find similar answers. We'll be able to say things like "when an image has the following statistical properties, we're likely to interpret it as a cat". Of course, right now we can trivially say that if by "the following statistical properties" we mean "activates this specifically trained network in a particular way", but the point is that we can probably boil that down into deeper (and simpler) statements about how human image recognition works. I can't find a link about it now, but I remember reading that for their Translate app, Google created a way to lossily compress their character recognition neural net. That kind of research seems likely to me to lead to a better understanding of how to boil down the insights that these learning algorithms are gleaning. This Two Minute Papers video has some thinking along the same lines: https://www.youtube.com/watch?v=ZBWTD2aNb_o https://www.youtube.com/watch?v=ZBWTD2aNb_o But I'd also point out that the human visual cortex is itself a bunch of models about the real world. It's gotten that way through a combination of evolution and learning. When engineers train neural networks to do tricks like TFA shows, those networks are effectively modeling the real world in two different ways: directly, and by proxy in modeling human visual systems. So the insights we can find by studying them are likely to tell us not only about our own vision, but about the world itself. Finally, it's worth noting that graphics algorithms have always been about human visual systems and the real physical world, as opposed to deep mathematical truths. Look at this paper for example: http://www.jiansun.org/papers/Dehaze_CVPR2009.pdf http://www.jiansun.org/papers/Dehaze_CVPR2009.pdf The oversimplified plain English insight those researchers had is "For a photo that doesn't contain haze, you usually can find a fully saturated pixel within some reasonably small neighborhood of any point in the image." There's no deep mathematical insight there, but there is an important insight into the statistical nature of natural images – a fact about the physical world. And they use that insight to great effect: they can remove haze from photographs, and even use the haze as a way of estimating a depth map. So sure, while you're right that there probably aren't many deep mathematical insights to find from the results of deep learning algorithms (which is generally true of graphics algorithms anyway), I wouldn't gloss over the deep practical insights that they encode.

scott_s 10y ago

Agreed with the first paragraph, but I think this sort of behavior is normal for most disciplines. That is, it's not CS looking like psychology or sociology, it's normal for CS to do this. I think we're seeing a similar thing right now in biology with CRISPR.

polyfractal 10y ago
FWIW, CRISPR isn't a great analogy. It's a well understood mechanism, and not really all that different from standard genome editing techniques like restriction enzymes, zinc-finger nucleases, etc. It's new and flashy and super powerful -- and the applications of CRISPR are still being discovered -- but the mechanism is understood and simple.
- scott_s 10y ago
  That wasn't the aspect I was going for. More for a period of incremental improvements in understanding, from which emerges a particular thing with enormous potential, and then the field is dominated by applications of that new thing.

kcorbitt 10y ago

> The theory will eventually catch up Maybe... but the emergent behavior in a complicated system (and these networks are only getting more complicated, not less) is likely to quickly become more complex than the human mind can reasonably be expected to understand, given any amount of time. We actually know a lot less about biology, for example, than your typical "biology 101" course would lead you to believe. It's pretty cool that we figured out how DNA translates into proteins, but actually figuring out how a cell decides which DNA to express at any given time, much less what any arbitrary protein actually does, takes you right back into guess-and-check territory. There are heuristics you can use to eliminate some guesses more quickly than others, but we're so far away from a unified explanatory theory of biology that I'm doubtful one will ever be developed.

jszymborski 10y ago
I mean, that's the nature of empirical sciences. You establish a model that fits observations you've made, but inevitably their is some chaotic, sometimes imperceptible noise that gets averaged out that represents some further level of complexity. I wouldn't argue that it means we don't know much, in the end you can argue we don't know anything because all empirical sciences are beholden to the belief of abstractions that can be attributed to some imperceptible abstraction until you (don't) reach the bottom of the stack of turtles.
deleted 10y ago
[deleted]

VikingCoder 10y ago

It's data-driven... rather than theory-driven. In some respects this is good, in other respects it's awful.