I actually always describe neural networks as a robust taylor series. What we are trying to do by calculating the weights are matching the "plane" of your data

I actually always describe neural networks as a robust taylor series. What we are trying to do by calculating the weights are matching the "plane" of your data space. In the case of classification, you may have multiple taylor series generated, where each one represents a class, such that the highest value output of all the functions concatenated is your classification. Once you understand that a neural network "learns" the underlying data space, it is easy to understand why/how you can just change the output layer of a trained network in many cases to solve a different problem (on the same dataset).