3 ms·
Is there a coherent theory of ML I could read somewhere? Something higher level than "here is how we use tanh", yet not hand wavy. All these "we've attached th
by akomtu 4y ago
Is there a coherent theory of ML I could read somewhere? Something higher level than "here is how we use tanh", yet not hand wavy.
All these "we've attached this random piece to our model and it makes things better for reasons we don't understand" leaves a poor impression. To an outsider, the GPT model looks a lot like a IIR / FIR filter made of linear operators (matrix multiplications) and convolutions (matrix multiplications in the Fourier space) that processes a sequence of vectors, instead of numbers (like a normal FIR filter would). Perhaps the extensive theory of filters apply here, and perhaps we can even apply the Z transform on the GPT filter to analyze its convergence?
- fooblaster 4y agoIf I understand correctly z transform analysis can't work as these systems are fundamentally non linear. In fact, the "non linear" part, i.e. the activation functions, are the part of these networks that allow for them to be universal function approximators and learn arbitrary things, versus just becoming a simple linear system.
- PartiallyTyped 4y agoI am a big fan of “Geometric Deep Learning” and the view the authors propose. There is also a series of lectures. https://arxiv.org/abs/2104.13478 https://arxiv.org/abs/2104.13478