4 ms·

The terminology comes from signal processing, where a convolution in the frequency domain is equivalent to a multiplication in the time domain. I don't think an

The terminology comes from signal processing, where a convolution in the frequency domain is equivalent to a multiplication in the time domain. I don't think anyone is thinking about the frequency domain in deep-learning, but they still call the operators convolution kernels.

YetAnotherNick 6y ago
Multiplying in frequency domain is convolution, in DL terminology convolution is that convolution with the weights rotated by 180 degree.
- r_c_a_d 6y ago
  Convolution is an integration. It isn't itself a property of a domain https://en.wikipedia.org/wiki/Convolution https://en.wikipedia.org/wiki/Convolution
  - YetAnotherNick 6y ago
    Try to first understand before commenting Convolution of f and g at t is: integration of f(x) * g(t - x) Cross correlation(which is termed convolution in DL) at t is: integration of f(x) * g(t + x) See the figure in the wikipedia page you shared.
    - srean 6y ago
      I dont see why this comment was downvoted so much.
  - suyash 6y ago
    This is still very low level, the whole article (although very comprehensive) missed the simple definition which it should have mentioned first. Going into jargons only add to complications. Look it up in a dictionary first for English definition and then try to understand how it has been applied in different domains. 1 : a form or shape that is folded in curved or tortuous windings e.g the convolutions of the intestines 2 : one of the irregular ridges on the surface of the brain and especially of the cerebrum of higher mammals 3 : a complication or intricacy of form, design, or structure … societies in which the convolutions of power and the caprices of the powerful are ever-present dangers to survival. After this is clear read the mathematical idea on wikipedia. After reading that, do google scholar search on the AI papers that first mentioned it. That is the way to go.
XMPPwocky 6y ago
Fuck the frequency domain, here- "Convolution with a kernel K" describes a system whose impulse response is K. In discrete time, suppose you have K=[1,2] and convolve [0,1,2,0] with it- you wind up with [0,1,3,2,0], if I'm awake enough for arithmetic. Correlation with a kernel K is convolution with K time-reversed (i.e. [2,1])- you'd get [0,2,5,2,0] (again if I'm awake). Note that 5- right there, the input signal "lines up just right" with the kernel- 2x2 + 1x1. That's why it's called correlation- its output is big when the input looks like the kernel.
qppo 6y ago
I mean ultimately it comes from functional analysis and differential equations (not signal processing). It's a binary operator on functions that yields a third function. It has a lot of useful properties and equivalences, like that it can be described as the product of two Fourier transforms (although that's very roundabout). You're actually introduced to convolution in middle school when you're taught how to multiply monomials to build a polynomial (at my middle school they called it "FOIL").
- elcritch 6y ago
  It appears to be a discrete Fourier, no? Does it apply to all convolutions or just a specific instance or subset? As in id there a proof showing that as sample size N goes to a limit it approaches a continuous limit? I still natively think in continuous convolutions from Physics. The whole discretization of these operators is oddly harder for me despite it technically being simpler to compute.
  - qppo 6y ago
    No, it's true of both continuous and discrete time/domain Fourier. Convolution in time is multiplication in frequency, and vice versa. You don't need to prove this with limits directly, just use the definition of the convolution integral and Fourier transform integral. > technically being simpler to compute. They're equivalent, since the only meaningful way to "compute" a continuous convolution is symbolically, and discrete convolutions obey most of the same identities. If one can place a lower bound on the time step resolution of a simulation then continuous convolutions are evaluated using discrete convolutions, which can represent the continuous case exactly via the Nyquist-Shannon sampling theorem. Interestingly enough, to prove the Sampling Theorem you need to rely on the identity that multiplication in frequency is convolution in time, and to prove that it can't be realized in a physical system (breaks causality, since you multiply by a superposition of Heavisides which of course are infinitely long sinc functions in both directions of time). And more interesting is that signals and systems is mostly applied dynamics and statistics, so it shouldn't be surprising if there's overlap.

YetAnotherNick 6y ago

Multiplying in frequency domain is convolution, in DL terminology convolution is that convolution with the weights rotated by 180 degree.

r_c_a_d 6y ago
Convolution is an integration. It isn't itself a property of a domain https://en.wikipedia.org/wiki/Convolution https://en.wikipedia.org/wiki/Convolution
- YetAnotherNick 6y ago
  Try to first understand before commenting Convolution of f and g at t is: integration of f(x) * g(t - x) Cross correlation(which is termed convolution in DL) at t is: integration of f(x) * g(t + x) See the figure in the wikipedia page you shared.
  - srean 6y ago
    I dont see why this comment was downvoted so much.
- suyash 6y ago
  This is still very low level, the whole article (although very comprehensive) missed the simple definition which it should have mentioned first. Going into jargons only add to complications. Look it up in a dictionary first for English definition and then try to understand how it has been applied in different domains. 1 : a form or shape that is folded in curved or tortuous windings e.g the convolutions of the intestines 2 : one of the irregular ridges on the surface of the brain and especially of the cerebrum of higher mammals 3 : a complication or intricacy of form, design, or structure … societies in which the convolutions of power and the caprices of the powerful are ever-present dangers to survival. After this is clear read the mathematical idea on wikipedia. After reading that, do google scholar search on the AI papers that first mentioned it. That is the way to go.

XMPPwocky 6y ago

Fuck the frequency domain, here- "Convolution with a kernel K" describes a system whose impulse response is K. In discrete time, suppose you have K=[1,2] and convolve [0,1,2,0] with it- you wind up with [0,1,3,2,0], if I'm awake enough for arithmetic. Correlation with a kernel K is convolution with K time-reversed (i.e. [2,1])- you'd get [0,2,5,2,0] (again if I'm awake). Note that 5- right there, the input signal "lines up just right" with the kernel- 2x2 + 1x1. That's why it's called correlation- its output is big when the input looks like the kernel.

qppo 6y ago

I mean ultimately it comes from functional analysis and differential equations (not signal processing). It's a binary operator on functions that yields a third function. It has a lot of useful properties and equivalences, like that it can be described as the product of two Fourier transforms (although that's very roundabout). You're actually introduced to convolution in middle school when you're taught how to multiply monomials to build a polynomial (at my middle school they called it "FOIL").

elcritch 6y ago
It appears to be a discrete Fourier, no? Does it apply to all convolutions or just a specific instance or subset? As in id there a proof showing that as sample size N goes to a limit it approaches a continuous limit? I still natively think in continuous convolutions from Physics. The whole discretization of these operators is oddly harder for me despite it technically being simpler to compute.
- qppo 6y ago
  No, it's true of both continuous and discrete time/domain Fourier. Convolution in time is multiplication in frequency, and vice versa. You don't need to prove this with limits directly, just use the definition of the convolution integral and Fourier transform integral. > technically being simpler to compute. They're equivalent, since the only meaningful way to "compute" a continuous convolution is symbolically, and discrete convolutions obey most of the same identities. If one can place a lower bound on the time step resolution of a simulation then continuous convolutions are evaluated using discrete convolutions, which can represent the continuous case exactly via the Nyquist-Shannon sampling theorem. Interestingly enough, to prove the Sampling Theorem you need to rely on the identity that multiplication in frequency is convolution in time, and to prove that it can't be realized in a physical system (breaks causality, since you multiply by a superposition of Heavisides which of course are infinitely long sinc functions in both directions of time). And more interesting is that signals and systems is mostly applied dynamics and statistics, so it shouldn't be surprising if there's overlap.