I'm rewriting a web crawler that I created a decade ago. Part of it is to see how much my development has changed. Another part is to create a general purpose c

I'm rewriting a web crawler that I created a decade ago. Part of it is to see how much my development has changed. Another part is to create a general purpose crawler infrastructure that I can easily reuse for new projects.