The Michael Nielsen online text is very well-regarded and mostly fits what you describe for 1, although it may have too much code for your taste in addition to

The Michael Nielsen online text is very well-regarded and mostly fits what you describe for 1, although it may have too much code for your taste in addition to the mathematical basics. However the code is not "how to use TensorFlow" but rather "implement backpropagation from scratch". It covers some interesting topics including the universal approximation proof. It has some reasonable exercises but no solutions, but I think people have posted their own solutions online around the web. http://neuralnetworksanddeeplearning.com http://neuralnetworksanddeeplearning.com I think this is your best bet, to be honest. Meat & potatoes neural networks doesn't really require any super-deep mathematical knowledge (just linear algebra and a very hand-wavy ability to do basic matrix calculus), and the more advanced topics are moving way too fast for a textbook to cover them (Goodfellow et al. is already getting out of date). The recommendation of Strang's new book is probably also pretty good.