I attacked it from both directions: an arguably supervised dataset of urls (hundreds of GB from various internet survey projects), and unit testing the insane m

I attacked it from both directions: an arguably supervised dataset of urls (hundreds of GB from various internet survey projects), and unit testing the insane markov gibberish the parser makes when you cram random data where it doesn't belong and run it in reverse. The dataset was way less helpful for a strictly conforming design, but I imagine it would be useful for a mostly correct parser.