4 ms·
My suggestion for learning: re-write this using futures. Here's a quick example of using similar parallelism to search structured json in gzip'd files: https://
by storyinmemo 9y ago
My suggestion for learning: re-write this using futures. Here's a quick example of using similar parallelism to search structured json in gzip'd files: https://github.com/autocracy/snippets/blob/master/search_cloudtrail.py https://github.com/autocracy/snippets/blob/master/search_clo...
- emj 9y agoBasically xargs in python, and I love xargs. There are some cool examples of places were xargs fails in the docs, this got me very excited I hope it works in 2.7. https://docs.python.org/3/library/concurrent.futures.html https://docs.python.org/3/library/concurrent.futures.html
- d0mine 9y agoThere is futures module for Python 2.7 on PyPI. In simple cases, you could use multiprocessing.Pool(): results = pool.map(processfile, getpaths()) There is also ThreadPool. There are known issues with multiprocessing especially on Python 2.7 e.g., Ctrl+C handling. Though you don't need threading to manage concurrent subprocesses or IO (Gevent, Twisted or curio if you use Python 3).
- mkesper 9y agoAnd alwyays use if __name__ == '__main__' if you want be able to test your code.
- marlynm 9y agoI'm learning. Thanks for letting me know. Will use that.
- marlynm 9y agoThanks. Will learn and use.