5 ms·

A good paper mentioned in the sources is "Neural Tangent Kernel: Convergence and Generalization in Neural Networks". > At initialization, artificial neural net

by bhl 7y ago

A good paper mentioned in the sources is "Neural Tangent Kernel: Convergence and Generalization in Neural Networks". > At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit, thus connecting them to kernel methods. I'm not familiar with ignoring higher order terms, besides approximations and bounds like Chebyshev's inequality and Chernoffs.

antognini 7y ago
If you take the sum of a large, but finite number of random variates from a probability distribution you can show that the resulting distribution is approximately a Gaussian perturbed by Hermite polynomials. The size of these perturbations decays inversely with the order of the polynomial divided by two minus one. This is the so-called Edgeworth expansion. I found this chapter to have a good explanation of the Edgeworth expansion if you're interested in more detail: http://web.math.ku.dk/~erhansen/bootstrap_05/doku/noter/Edgeworth_24_01.pdf http://web.math.ku.dk/~erhansen/bootstrap_05/doku/noter/Edge... I had a little workshop paper earlier this year showing that you can apply the Edgeworth expansion to wide, but finite neural networks: https://arxiv.org/abs/1908.10030 https://arxiv.org/abs/1908.10030