You can do it in python but one has to dig into how python do reference counting and how that interacts with threads. Personally I prefer using processes rathe

You can do it in python but one has to dig into how python do reference counting and how that interacts with threads. Personally I prefer using processes rather than threads, with a worker pool and a message bus rather than shared memory. That solution has its own drawbacks (and a bit more overhead), but you don't need to worry so much about memory issues. Processes also seems a better match for crawlers since the number of processes will be fairly constant and the work the processes do is fairly independent.