Andrej covers this in https://github.com/karpathy/nn-zero-to-hero https://github.com/karpathy/nn-zero-to-hero. He explains things in multiple ways, both the mat

Andrej covers this in https://github.com/karpathy/nn-zero-to-hero https://github.com/karpathy/nn-zero-to-hero. He explains things in multiple ways, both the matrix multiplications as well as the "programmer's" way of thinking of it - i.e. the lookups. The downside is it takes a while to get through those lectures. I would say for each 1 hour you need another 10 to looks stuff up and practice, unless you are fresh out of calculus and linear algebra classes. Other idioms I can think of, in my words: Softmax = take the maximum (but in a differentiable way) tanh/sigmoid/relu = a switch. "activation" cross entropy loss = average(-log(probability you gave to the right answer)). Averaged over the current batch you are training on for this step. (Sorry that is still quite mathy).