Thanks for the feedback! I halfway implemented traversing whole sitemaps, but never finished because the waits were getting so long that I would need to move a

Thanks for the feedback! I halfway implemented traversing whole sitemaps, but never finished because the waits were getting so long that I would need to move away from django and towards something like twisted where I could incrementally update the results and work asynchronously. Would you rather give a sitemap or a representative list of URLs?