3 ms·

Hmm, so I'm reading The Hundred Page Machine Learning Book [0] which gives an overview of the field, and one of the sections is on convolutional neural network

Hmm, so I'm reading The Hundred Page Machine Learning Book [0] which gives an overview of the field, and one of the sections is on convolutional neural networks, and it was very crisp and clear (like the whole book, really; highly recommended). I understand since it's a brief summary it's probably eliding stuff, but it's so much shorter and more understandable than this blog post that it makes me wonder if it's really eliding a lot of crucial stuff, or if this post is just... not clear. In the book, it walks you through the convolution of an input matrix with a smaller filter matrix. You start with lining up the smaller matrix in the top left corner of the input matrix, and you multiply each overlapping square together and add them all up. That's the value of the top left element of the result (the result is a matrix). Then you slide the filter to the right, repeat the calculation, and that's the value one to the right in the result matrix. You repeat, panning and scanning across the input matrix and in this fashion fill in the the whole result matrix. I thought I had it, but this blog post just confused things. I do remember learning about convolutions back in undergrad and being confused then, too, so that sounds about right. When I read the ML handbook, I had a "wait, is that it?" thought and was confused why I was confused back in the day. But I guess that's just one particular way of doing it, or one use case or something. [0] https://www.amazon.com/Hundred-Page-Machine-Learning-Book/dp/199957950X https://www.amazon.com/Hundred-Page-Machine-Learning-Book/dp...

currymj 8y ago
your understanding is not wrong, for the basic version of convolution typically used in neural networks. however, in practice the most optimized neural net libraries don't actually implement the operation in that way. as mentioned at the end of the post, if you understand things in terms of functions rather than just multiplying with a sliding window, you can then make use of Fourier transforms, and this enables much faster algorithms to compute the same result.
p1esk 8y ago
It’s because you looked at the wrong post. You want this one: http://colah.github.io/posts/2014-07-Conv-Nets-Modular http://colah.github.io/posts/2014-07-Conv-Nets-Modular
adamnemecek 8y ago
Convolution is essentially polynomial multiplication. That's how I think about it. You remember (a + b)(c + d) = ac + ad + bc + bd? Or (2x^2 + 1)(4x^3 + x) can be calculated by doing conv([0,2,0,1], [4,0,1,0]) (the numbers represent the coefficients). The results are the coefficients of the resulting polynomial. As the polynomials grow larger this gets harder and harder computationally. Convolution theorem lets you use Fourier to you convert this multiplication into a pairwise multiplication so that fft(f * g) = fft(f) .* fft(g) where * is convolution and .* is pairwise multiplication.
- deleted 8y ago
  [deleted]

currymj 8y ago

your understanding is not wrong, for the basic version of convolution typically used in neural networks. however, in practice the most optimized neural net libraries don't actually implement the operation in that way. as mentioned at the end of the post, if you understand things in terms of functions rather than just multiplying with a sliding window, you can then make use of Fourier transforms, and this enables much faster algorithms to compute the same result.

p1esk 8y ago

It’s because you looked at the wrong post. You want this one: http://colah.github.io/posts/2014-07-Conv-Nets-Modular http://colah.github.io/posts/2014-07-Conv-Nets-Modular

adamnemecek 8y ago

Convolution is essentially polynomial multiplication. That's how I think about it. You remember (a + b)(c + d) = ac + ad + bc + bd? Or (2x^2 + 1)(4x^3 + x) can be calculated by doing conv([0,2,0,1], [4,0,1,0]) (the numbers represent the coefficients). The results are the coefficients of the resulting polynomial. As the polynomials grow larger this gets harder and harder computationally. Convolution theorem lets you use Fourier to you convert this multiplication into a pairwise multiplication so that fft(f * g) = fft(f) .* fft(g) where * is convolution and .* is pairwise multiplication.

deleted 8y ago
[deleted]