Usually the problem is that the APIs are like typical XML apis which are a royal pain to use requiring various hoop jumping in order to say which tags are of in

Usually the problem is that the APIs are like typical XML apis which are a royal pain to use requiring various hoop jumping in order to say which tags are of interest in which container and relationships with siblings. If you are lucky you have to learn some sort of XPATH like syntax. Meanwhile you do a view source, and can trivially see exactly which lines you want, and say sod all that nonsense - I'll use a regex. My recommendation is to find a library that provides jQuery/CSS selector style syntax and semantics, and then suddenly it is a lot easier to deal with the document. For example for Python there is soupselect or cssselect. Amusingly the latter shows that the selector "div.content" translates into the XPATH "descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' content ')]".