Whilst it's true in principal that you should always parse xml/html with a proper parser, in practise it can be a little different. On the web you'll find just

Whilst it's true in principal that you should always parse xml/html with a proper parser, in practise it can be a little different. On the web you'll find just as much malformed html that will throw an error in any parser, as you will bad html that will throw a regex. As long as you have proper verification of the link at the end, you'll be safe enough. Then when you factor in that constructing and extracting links from the DOM might take 500ms vs. the 5ms of a regex - a 100x increase in crawl speed in exchange for a little potential dirtiness is more than acceptable; particularly if you have a million pages to crawl. That's the difference between a crawl taking 1 week, vs. 2 years... And anyway, all web crawls are inevitably going to end up dirty and scrappy, that's just the nature of the web. I once crawled a website that returned "_ontent-Type" instead of "Content-Type", making it impossible to tell what kind of file was about to be downloaded by my polite, "text only" crawler.