I wouldn't recommend Szeliski for that - the book is basically a literature review (and is now a little out of date given how fast CV moves). It's great if you

I wouldn't recommend Szeliski for that - the book is basically a literature review (and is now a little out of date given how fast CV moves). It's great if you need a background survey on how things are done, but not for learning from. The level of maths required varies a lot. Linear algebra is used heavily everywhere, you see a lot of optimisation (e.g. Levenberg Marquadt) and in some places graph theory. However if you're just using tools like OpenCV then you can get by with a fairly poor understanding of the maths (say 2nd year undergrad of an engineering degree). Part of the challenge is reading past the maths. If you read a paper from Pattern Analysis there's a lot of, frankly, obtuse notation. So you see images described as discrete mapping blah blah. A lot of it is fairly straightforward set theory. It's necessary for proving things, but when it comes to actually implementing this stuff it's nowhere near as complicated as it looks.