3 ms·

Part of the problem in writing a deep learning book, is that very little that warrants being in a book, is actually known. The mainstream deep learning academic

Part of the problem in writing a deep learning book, is that very little that warrants being in a book, is actually known. The mainstream deep learning academic community welcomes theoretical work but articles on new techniques which beat SOTA are given much more attention than articles on setting up a theoretical structure of the models. Consequently, that line of research is also under-cited and under-developed. In some ways, 'deep learning' is in a different Kuhnian paradigm altogether. So, people who are used to learning about all the intricacies of classical ML models fail to appreciate deep learning, because the metrics with which you would judge a deep learning model are different from the ones you would use to judge a classical ML model. The same applies for the books in different Kuhnian paradigm.

colllectorof 11y ago
>The mainstream deep learning academic community welcomes theoretical work but articles on new techniques which beat SOTA are given much more attention than articles on setting up a theoretical structure of the models. The fact that beating an accuracy test on some arbitrary data set is seen as more important than understanding why and when your methods works (or do not work) is deeply disturbing.
- ogrisel 11y ago
  It's not an arbitrary dataset. To be taken seriously you need to beat SOTA on several standard benchmark datasets where only reaching SOTA on one is already hard. Or alternatively, reach comparable level of validation accuracy with significantly less compute or memory usage (computational complexity). Or alternatively, reach comparable level of validation accuracy with only a subset of labeled samples + unlabeled samples (sample complexity).
  - eli_gottlieb 11y ago
    That still amounts to publishing when you can say, "We know it does work" as opposed to "We know why it works".
    - aminorex 11y ago
      ...which is very valuable. After all, we don't know when, if ever, a satisfactory explanation (by what criteria?) will derive. To defer all progress in the field until it should meet the standards of some other field would be a great loss, and might preclude any number of future further discoveries.
      - eli_gottlieb 11y ago
        As far as I understand the concept of "science", an empirical observation (including "it works from an engineering perspective, kinda") is only ever an observation. Scientific progress, by definition, involves advancing our understanding of hows and whys.

colllectorof 11y ago

>The mainstream deep learning academic community welcomes theoretical work but articles on new techniques which beat SOTA are given much more attention than articles on setting up a theoretical structure of the models. The fact that beating an accuracy test on some arbitrary data set is seen as more important than understanding why and when your methods works (or do not work) is deeply disturbing.

ogrisel 11y ago
It's not an arbitrary dataset. To be taken seriously you need to beat SOTA on several standard benchmark datasets where only reaching SOTA on one is already hard. Or alternatively, reach comparable level of validation accuracy with significantly less compute or memory usage (computational complexity). Or alternatively, reach comparable level of validation accuracy with only a subset of labeled samples + unlabeled samples (sample complexity).
- eli_gottlieb 11y ago
  That still amounts to publishing when you can say, "We know it does work" as opposed to "We know why it works".
  - aminorex 11y ago
    ...which is very valuable. After all, we don't know when, if ever, a satisfactory explanation (by what criteria?) will derive. To defer all progress in the field until it should meet the standards of some other field would be a great loss, and might preclude any number of future further discoveries.
    - eli_gottlieb 11y ago
      As far as I understand the concept of "science", an empirical observation (including "it works from an engineering perspective, kinda") is only ever an observation. Scientific progress, by definition, involves advancing our understanding of hows and whys.