Principles of Statistics by M.G. Bulmer is a nice introduction to the mathematical side of things. It's part of Dover's classic textbook series, so it's inexpen

Principles of Statistics by M.G. Bulmer is a nice introduction to the mathematical side of things. It's part of Dover's classic textbook series, so it's inexpensive compared to newer textbooks, and also concise and well-written. It does assume you already have a solid understanding of calculus and combinatorics, though. Which I think is fair. Discrete statistics is arguably just applied combinatorics, and continuous statistics applied calculus, so if you have a strong foundation in those two subjects then you're already 90% of the way there. (And, if you don't, stop the cart and let the horse catch up.)