As the author mentions, the CS231n course notes may be what you want. They build up many of the neural network primitives using just numpy and basic operations,

As the author mentions, the CS231n course notes may be what you want. They build up many of the neural network primitives using just numpy and basic operations, so it's very easy to see exactly how it works at the code level.