3 ms·

> Turns out that a Python program with ~500 threads will start to chew up memory over time. Anyone have more info about this issue? I've got a scraper myself i

• xyzeva 3y ago
  Python just isn't the language for this, really. Rewriting it is the only real solution, I don't know your exact problem.
• belorn 3y ago
  You can do it in python but one has to dig into how python do reference counting and how that interacts with threads. Personally I prefer using processes rather than threads, with a worker pool and a message bus rather than shared memory. That solution has its own drawbacks (and a bit more overhead), but you don't need to worry so much about memory issues. Processes also seems a better match for crawlers since the number of processes will be fairly constant and the work the processes do is fairly independent.
• dist-epoch 3y ago
  You really should be using ayncio. It's a perfect fit for your usecase.
• teddyh 3y ago
  import multiprocessing as threading