19 ms·

I have a web scraping script written in Python that I want to make multi-threaded. It scrapes web pages from a list, and enters results into a DB. Can someone (

• twopoint718 18y ago
  That's really a job for google.
• russell 18y ago
  Go to python.org and read the documentation on threading, concurrency. The mailing list under community is very good, but they expect you to have read the docs and goodled first.
• lacker 18y ago
  Check out the docs on the "multiprocessing" or "threading" module. In particular multiprocessing.Pool is handy for controlling the number of parallel things you have going on at once. # Assume we have functions GetUrls() that retrieves a list # of the urls we want to get, and Download(url) which # downloads the content of a url and sticks it in the # database. import multiprocessing pool = multiprocessing.Pool(processes=100) urls = GetUrls() pool.map(Download, urls) See: http://docs.python.org/library/multiprocessing.html http://docs.python.org/library/multiprocessing.html http://docs.python.org/library/threading.html http://docs.python.org/library/threading.html Also, if you're entering results into a database, the easiest way may be just to spawn multiple python processes from the command line.