3 ms·
or choose both and compose with pipes! good for prototyping, maybe escalate to a single executable when the design stabilizes.
by nathants 6y ago
or choose both and compose with pipes! good for prototyping, maybe escalate to a single executable when the design stabilizes.
- mattbillenstein 6y agoLeads into serialization/deserialization and allocator overhead when using pipes. I usually went the route of parallelizing using the shell (xargs -P) or using multiprocessing. I typically had tens to hundreds of thousands of objects in S3 to process. But one neat trick that is pipe-like us using subprocess to do the gunzip step -- subprocess.Popen('gunzip -c ' + filename, ...) -- this would offload work from the python process and give me a decent wall clock speedup.