It's a pretty minor difference, really. Over functions (which is the same as over vectors/matrices, except the latter are taken at discrete points), the autocor

It's a pretty minor difference, really. Over functions (which is the same as over vectors/matrices, except the latter are taken at discrete points), the autocorrelation and the convolution differ by a sign. The definition of convolution is conv(x(t), y(t))(q) = integral (x(t)·y(q-t) dt) where x(t), y(t) are functions. Or, with sums and x, y are vectors conv(x, y)(q) = sum(over i + j = q+1) of (x(i)·y(j)) whereas the definition of the autocorrelation is "simpler" ac(x(t), y(t))(q) = int(x(t)·y(t-q)) In other words, it replaces y(t) with y(-t) (i.e., the autocorrelation of x(t) and y(t) is the same as the convolution of x(t) and y(-t)). A similar transformation happens for the discrete case. Turns out, in the context of learning, these are equivalent since it just corresponds to a relabelling of the indices (which is an invertible linear transformation). In fact, even more strongly, independent of the order of the method (so long as it is >0th order!), this will cause no change in the descent direction or objective value, unlike arbitrary invertible linear transformations (for which this is only true of ≥2nd order methods).