3 ms·
Yeah, it's a lot of work to try and avoid copies and allocations. cpython I think comes from a very different era where cpu speeds were much closer to memory s
by mattbillenstein 6y ago
Yeah, it's a lot of work to try and avoid copies and allocations.
cpython I think comes from a very different era where cpu speeds were much closer to memory speeds and you had a lot less latency between the two in terms of cpu cycles. Like reference counting would probably not be a design you'd start at today if you were designing a new runtime.
That being said, it's kinda amazing to me that cpython is only 4x slower in some of these benchmarks - and you can actually get within 2x with a different runtime that has the same guarantees and a JIT. It makes me think you have to have really good reasons not to choose Python for a lot of tasks given developer productivity and maturity of the runtime and libraries.
- nathants 6y agoor choose both and compose with pipes! good for prototyping, maybe escalate to a single executable when the design stabilizes.
- mattbillenstein 6y agoLeads into serialization/deserialization and allocator overhead when using pipes. I usually went the route of parallelizing using the shell (xargs -P) or using multiprocessing. I typically had tens to hundreds of thousands of objects in S3 to process. But one neat trick that is pipe-like us using subprocess to do the gunzip step -- subprocess.Popen('gunzip -c ' + filename, ...) -- this would offload work from the python process and give me a decent wall clock speedup.