Surprised nobody has mentioned ASP websites yet - definitely among the hardest. Those sites carry so much state in cookies rather than URLs, so you have to foll

Surprised nobody has mentioned ASP websites yet - definitely among the hardest. Those sites carry so much state in cookies rather than URLs, so you have to follow all the UI interactions in order to get to the result you're trying to parse. The markup is also typically really bloated and filled with randomly-generated IDs.