Leads into serialization/deserialization and allocator overhead when using pipes. I usually went the route of parallelizing using the shell (xargs -P) or using

Leads into serialization/deserialization and allocator overhead when using pipes. I usually went the route of parallelizing using the shell (xargs -P) or using multiprocessing. I typically had tens to hundreds of thousands of objects in S3 to process. But one neat trick that is pipe-like us using subprocess to do the gunzip step -- subprocess.Popen('gunzip -c ' + filename, ...) -- this would offload work from the python process and give me a decent wall clock speedup.