I've developed a Twisted Python crawler that does something very similar to that. The possibility that it would work well seemed dubious at first, but I've been

I've developed a Twisted Python crawler that does something very similar to that. The possibility that it would work well seemed dubious at first, but I've been pleasantly surprised with the results. Email me at johnwehr@gmail.com - I'd be happy to discuss the technology and progress I've made so far.