"Challenges" is kind of subjective. GEB is really a bit fluffy (as I said in another comment here), and SICP is only challenging if you've never seen a function

"Challenges" is kind of subjective. GEB is really a bit fluffy (as I said in another comment here), and SICP is only challenging if you've never seen a functional language before. PLP is also not particularly challenging as such, but it is a satisfyingly informative treatment of the practicalities involved in implementing a conventional programming language from the ground up. On the other hand, while I've not read all of Types and Programming Languages, it definitely starts to get into some challenging (and very interesting!) material.