6 ms·

I spent a few weeks closely reading this book and I have to disagree with the majority here. I didn't like the book at all. And I am an advanced math geek. My

by j2kun 11y ago

I spent a few weeks closely reading this book and I have to disagree with the majority here. I didn't like the book at all. And I am an advanced math geek. My main issue is that the book tells you all about the different parameter tweaks, but passes little concrete wisdom to the reader. It doesn't distinguish between modeling assumptions, and it replaces very simple explanations of concepts with complicated paragraphs that I can't make sense of. I think it boils down to something that I have been feeling and hearing a lot in the past few years: the statistical jargon is so overwhelming that the authors can't explain things clearly. I can point to many examples in this book that I feel are unnecessary stumbling blocks, but the fact is that I'll spend an hour or two discussing parts of this book with a room full of smart machine learning researchers, and at the end we'll all agree we don't understand the material better than we did at the start. On the other hand, I'll read research papers that don't force the statistical perspective down the reader's throat (e.g. http://arxiv.org/abs/1602.04485v1 http://arxiv.org/abs/1602.04485v1) and find them very easy to understand by comparison. It might be a cultural difference, but I've heard this complaint enough from experts who straddle both sides of the computational/statistical machine learning divide that I don't think it's just me.

gherkin0 11y ago
Do you know of any books that don't have the problems you speak of?
p1esk 11y ago
That's disappointing. I was really looking forward to reading the book.
- kleiba 11y ago
  You still should, I think. I mean, at least give it a try. Different people can have completely different experiences when reading the same material. Perhaps you will love it, perhaps you will agree with the OP. In any event, it's worth getting it straight from the horse's mouth.
  - j2kun 11y ago
    I agree. Everyone should give it a try if they're genuinely interested in learning it. But if you find yourself struggling with it, know you're not the only one :)
neurologic 11y ago
Statistics [edit: and especially ML] is different from math. It is important to let go of one's preconceived notions and to try to absorb the new field as is. Its jargon exists for a reason --- the normal math jargon doesn't quite have the right concepts --- so it is worthwhile spending the effort to understand it.
- j2kun 11y ago
  Yeah, but I did my PhD in theoretical computer science and have published research papers in ML. So I shouldn't have such a hard time. (It could just be me, but maybe it's a weak signal)
KVFinn 11y ago
What are you favorite books on the subject?
otoburb 11y ago
>And I am an advanced math geek. For those that may not know, j2kun also writes an on-going expository machine learning series on his mathematics blog[1]. [1] http://jeremykun.com/2012/08/04/machine-learning-introduction/ http://jeremykun.com/2012/08/04/machine-learning-introductio...
khuss 11y ago
>> it replaces very simple explanations of concepts with complicated paragraphs that I can't make sense of It is good to see critical views but it will be even better if you could give concrete examples for statements like the above. Also, what other books do you recommend?
- j2kun 11y ago
  > Also, what other books do you recommend? This has been my frustration. I've been wanting to grok deep learning, but haven't found any source that can explain it in a way that doesn't overcomplicate simple things (which I can only tell it's doing when I already understand the topic). I also don't have the time or incentive to really dig in and do the math myself from scratch, since so much of it is wide open research directions. I've also had this experience with much much simpler areas of statistics and statistical ML (cf. Markov chain monte carlo), so this is a sort of recurring theme for me. Considering how simple MCMC is now that I do understand it, it's difficult to dispel the nagging feeling that all the statistical ML literature is (likely unintentionally) obfuscated. I could give concrete examples of excerpts and entire sections of the book that don't make sense to me, but I don't think it's all that productive because a lot of it boils down to organizational disagreements, cultural assumptions behind the math, and differences in priorities. Individually it seems like nitpicking, but they add up quickly to general muddled confusion. This is especially true when, for example, almost always the right answer to the question "Why does this particular technique work?" is, "We have no clue, but here is some anecdotal evidence and half-substantiated oversimplified theories." Instead these answers are passed as well-known fact.
  - bronxbomber92 11y ago
    It is valuable to cite examples because it will help others determine if they have the same organizational disagreements and cultural assumptions that would make the book burdensome to read. On the other hand, if we find the cited examples clear due to e.g. different backgrounds, expectations, perspectives, etc, then that might be a good signal that we should look into the book.
  - kevinskii 11y ago
deleted 11y ago
[deleted]
sherjilozair 11y ago
Part of the problem in writing a deep learning book, is that very little that warrants being in a book, is actually known. The mainstream deep learning academic community welcomes theoretical work but articles on new techniques which beat SOTA are given much more attention than articles on setting up a theoretical structure of the models. Consequently, that line of research is also under-cited and under-developed. In some ways, 'deep learning' is in a different Kuhnian paradigm altogether. So, people who are used to learning about all the intricacies of classical ML models fail to appreciate deep learning, because the metrics with which you would judge a deep learning model are different from the ones you would use to judge a classical ML model. The same applies for the books in different Kuhnian paradigm.
- colllectorof 11y ago
  >The mainstream deep learning academic community welcomes theoretical work but articles on new techniques which beat SOTA are given much more attention than articles on setting up a theoretical structure of the models. The fact that beating an accuracy test on some arbitrary data set is seen as more important than understanding why and when your methods works (or do not work) is deeply disturbing.
  - ogrisel 11y ago
    It's not an arbitrary dataset. To be taken seriously you need to beat SOTA on several standard benchmark datasets where only reaching SOTA on one is already hard. Or alternatively, reach comparable level of validation accuracy with significantly less compute or memory usage (computational complexity). Or alternatively, reach comparable level of validation accuracy with only a subset of labeled samples + unlabeled samples (sample complexity).
    - eli_gottlieb 11y ago
      That still amounts to publishing when you can say, "We know it does work" as opposed to "We know why it works".
xamuel 11y ago
I, too, spent some time reading this book, and arrived at the conclusion that it's terribly written. From a mathematical perspective, the book does not define hardly anything. Take, for example, the important chapter on Deep Feedforward Networks. They hem and haw saying things like "quintessential deep learning models" but never actually explain _what a deep feedforward network is_. There are no definitions: they give what amounts to preamble of a definition, and then move right on to using the notion, but don't actually give the definition itself. In the deep feedforward network example, the closest thing to a definition is, "A feedforward network defines a mapping y=f(x;theta)". That doesn't really say anything though. If you're asked to define "dog", you can say, "A dog has four legs", but how does the reader know that a cat isn't a dog? I find that for basic machine learning it's much better to just go through machine learning framework tutorials. For advanced learning, I'm not there yet so I can't speak from experience but I'll be shocked if this book fits well in any role except maybe last-resort reference material. If that.
SixSigma 11y ago
Phew, I thought it was just me. I found http://neuralnetworksanddeeplearning.com/ http://neuralnetworksanddeeplearning.com/ far more accessible.
mjw 11y ago
If anything, to me a lot of deep learning literature seems to lack the statistical insight and theory that's available to other subfields in machine learning (whether the Bayesian/graphical models camp, the statistical learning theory camp...) If this book is trying to do more to bring statistical or probabilistic insights to bear on deep learning than I think that's a very good thing. It might make it less accessible to those coming from a pure computer science background, but potentially more so to those who like to think about machine learning from a probabilistic modelling perspective. If they're using stats jargon in a gratuitous way that doesn't actually cast any light on the material then that's another thing, but from a quick skim I didn't see anything particularly bad on this front. Do you have any examples of the kind of jargon you're talking about? To others reading, I just wanted to emphasise that statistics is really important in machine learning! Deep learning lets you get away with less of it than you might need elsewhere, but that doesn't mean one can treat it as an unnecessary inconvenience. It's a language you need to learn, especially if you want to try and get to the bottom of how and why aspects of deep learning work the way they do. As opposed to just an empirical "using GPU clusters to throw lots of clever shit at the wall and see what sticks" engineering field. Bengio seems very interested in these kinds of questions and I'm glad he's leading research in that direction, even if clear answers and intuition aren't always easy to come by at this point.
wodenokoto 11y ago
This is the first time I have ever heard anyone say that a text book is harder to understand and is less clear than research papers. Journal papers are the pinnacle of unclear, convoluted context depend writing I know of. This is an extremely damning critique.
- woodson 11y ago
  Except for conference papers, which are even worse. There's so much "secret sauce", only few are independently reproducible. Sadly, in some research areas they are the dominant way of publishing research output.

by j2kun 11y ago

gherkin0 11y ago

Do you know of any books that don't have the problems you speak of?

p1esk 11y ago

That's disappointing. I was really looking forward to reading the book.

kleiba 11y ago
You still should, I think. I mean, at least give it a try. Different people can have completely different experiences when reading the same material. Perhaps you will love it, perhaps you will agree with the OP. In any event, it's worth getting it straight from the horse's mouth.
- j2kun 11y ago
  I agree. Everyone should give it a try if they're genuinely interested in learning it. But if you find yourself struggling with it, know you're not the only one :)

neurologic 11y ago

Statistics [edit: and especially ML] is different from math. It is important to let go of one's preconceived notions and to try to absorb the new field as is. Its jargon exists for a reason --- the normal math jargon doesn't quite have the right concepts --- so it is worthwhile spending the effort to understand it.

j2kun 11y ago
Yeah, but I did my PhD in theoretical computer science and have published research papers in ML. So I shouldn't have such a hard time. (It could just be me, but maybe it's a weak signal)

KVFinn 11y ago

What are you favorite books on the subject?

otoburb 11y ago

>And I am an advanced math geek. For those that may not know, j2kun also writes an on-going expository machine learning series on his mathematics blog[1]. [1] http://jeremykun.com/2012/08/04/machine-learning-introduction/ http://jeremykun.com/2012/08/04/machine-learning-introductio...

khuss 11y ago

>> it replaces very simple explanations of concepts with complicated paragraphs that I can't make sense of It is good to see critical views but it will be even better if you could give concrete examples for statements like the above. Also, what other books do you recommend?

j2kun 11y ago
> Also, what other books do you recommend? This has been my frustration. I've been wanting to grok deep learning, but haven't found any source that can explain it in a way that doesn't overcomplicate simple things (which I can only tell it's doing when I already understand the topic). I also don't have the time or incentive to really dig in and do the math myself from scratch, since so much of it is wide open research directions. I've also had this experience with much much simpler areas of statistics and statistical ML (cf. Markov chain monte carlo), so this is a sort of recurring theme for me. Considering how simple MCMC is now that I do understand it, it's difficult to dispel the nagging feeling that all the statistical ML literature is (likely unintentionally) obfuscated. I could give concrete examples of excerpts and entire sections of the book that don't make sense to me, but I don't think it's all that productive because a lot of it boils down to organizational disagreements, cultural assumptions behind the math, and differences in priorities. Individually it seems like nitpicking, but they add up quickly to general muddled confusion. This is especially true when, for example, almost always the right answer to the question "Why does this particular technique work?" is, "We have no clue, but here is some anecdotal evidence and half-substantiated oversimplified theories." Instead these answers are passed as well-known fact.
- bronxbomber92 11y ago
  It is valuable to cite examples because it will help others determine if they have the same organizational disagreements and cultural assumptions that would make the book burdensome to read. On the other hand, if we find the cited examples clear due to e.g. different backgrounds, expectations, perspectives, etc, then that might be a good signal that we should look into the book.
- kevinskii 11y ago

deleted 11y ago

[deleted]

sherjilozair 11y ago

Part of the problem in writing a deep learning book, is that very little that warrants being in a book, is actually known. The mainstream deep learning academic community welcomes theoretical work but articles on new techniques which beat SOTA are given much more attention than articles on setting up a theoretical structure of the models. Consequently, that line of research is also under-cited and under-developed. In some ways, 'deep learning' is in a different Kuhnian paradigm altogether. So, people who are used to learning about all the intricacies of classical ML models fail to appreciate deep learning, because the metrics with which you would judge a deep learning model are different from the ones you would use to judge a classical ML model. The same applies for the books in different Kuhnian paradigm.

colllectorof 11y ago
>The mainstream deep learning academic community welcomes theoretical work but articles on new techniques which beat SOTA are given much more attention than articles on setting up a theoretical structure of the models. The fact that beating an accuracy test on some arbitrary data set is seen as more important than understanding why and when your methods works (or do not work) is deeply disturbing.
- ogrisel 11y ago
  It's not an arbitrary dataset. To be taken seriously you need to beat SOTA on several standard benchmark datasets where only reaching SOTA on one is already hard. Or alternatively, reach comparable level of validation accuracy with significantly less compute or memory usage (computational complexity). Or alternatively, reach comparable level of validation accuracy with only a subset of labeled samples + unlabeled samples (sample complexity).
  - eli_gottlieb 11y ago
    That still amounts to publishing when you can say, "We know it does work" as opposed to "We know why it works".

xamuel 11y ago

I, too, spent some time reading this book, and arrived at the conclusion that it's terribly written. From a mathematical perspective, the book does not define hardly anything. Take, for example, the important chapter on Deep Feedforward Networks. They hem and haw saying things like "quintessential deep learning models" but never actually explain _what a deep feedforward network is_. There are no definitions: they give what amounts to preamble of a definition, and then move right on to using the notion, but don't actually give the definition itself. In the deep feedforward network example, the closest thing to a definition is, "A feedforward network defines a mapping y=f(x;theta)". That doesn't really say anything though. If you're asked to define "dog", you can say, "A dog has four legs", but how does the reader know that a cat isn't a dog? I find that for basic machine learning it's much better to just go through machine learning framework tutorials. For advanced learning, I'm not there yet so I can't speak from experience but I'll be shocked if this book fits well in any role except maybe last-resort reference material. If that.

SixSigma 11y ago

Phew, I thought it was just me. I found http://neuralnetworksanddeeplearning.com/ http://neuralnetworksanddeeplearning.com/ far more accessible.

mjw 11y ago

If anything, to me a lot of deep learning literature seems to lack the statistical insight and theory that's available to other subfields in machine learning (whether the Bayesian/graphical models camp, the statistical learning theory camp...) If this book is trying to do more to bring statistical or probabilistic insights to bear on deep learning than I think that's a very good thing. It might make it less accessible to those coming from a pure computer science background, but potentially more so to those who like to think about machine learning from a probabilistic modelling perspective. If they're using stats jargon in a gratuitous way that doesn't actually cast any light on the material then that's another thing, but from a quick skim I didn't see anything particularly bad on this front. Do you have any examples of the kind of jargon you're talking about? To others reading, I just wanted to emphasise that statistics is really important in machine learning! Deep learning lets you get away with less of it than you might need elsewhere, but that doesn't mean one can treat it as an unnecessary inconvenience. It's a language you need to learn, especially if you want to try and get to the bottom of how and why aspects of deep learning work the way they do. As opposed to just an empirical "using GPU clusters to throw lots of clever shit at the wall and see what sticks" engineering field. Bengio seems very interested in these kinds of questions and I'm glad he's leading research in that direction, even if clear answers and intuition aren't always easy to come by at this point.

wodenokoto 11y ago

This is the first time I have ever heard anyone say that a text book is harder to understand and is less clear than research papers. Journal papers are the pinnacle of unclear, convoluted context depend writing I know of. This is an extremely damning critique.

woodson 11y ago
Except for conference papers, which are even worse. There's so much "secret sauce", only few are independently reproducible. Sadly, in some research areas they are the dominant way of publishing research output.