3 ms·

i agree, pypy is really good. most interesting to me is how it behaves very differently than cpython when trying to avoid allocations and optimize. these kind

i agree, pypy is really good. most interesting to me is how it behaves very differently than cpython when trying to avoid allocations and optimize. these kind of changes where x2 faster with pypy, but way slower in cpython. similar wins with go, rust, and c. things like: https://github.com/nathants/bsv/blob/master/experiments/cut/cut.py https://github.com/nathants/bsv/blob/master/experiments/cut/...

mattbillenstein 6y ago
Yeah, it's a lot of work to try and avoid copies and allocations. cpython I think comes from a very different era where cpu speeds were much closer to memory speeds and you had a lot less latency between the two in terms of cpu cycles. Like reference counting would probably not be a design you'd start at today if you were designing a new runtime. That being said, it's kinda amazing to me that cpython is only 4x slower in some of these benchmarks - and you can actually get within 2x with a different runtime that has the same guarantees and a JIT. It makes me think you have to have really good reasons not to choose Python for a lot of tasks given developer productivity and maturity of the runtime and libraries.
- nathants 6y ago
  or choose both and compose with pipes! good for prototyping, maybe escalate to a single executable when the design stabilizes.
  - mattbillenstein 6y ago
    Leads into serialization/deserialization and allocator overhead when using pipes. I usually went the route of parallelizing using the shell (xargs -P) or using multiprocessing. I typically had tens to hundreds of thousands of objects in S3 to process. But one neat trick that is pipe-like us using subprocess to do the gunzip step -- subprocess.Popen('gunzip -c ' + filename, ...) -- this would offload work from the python process and give me a decent wall clock speedup.

3 ms·

i agree, pypy is really good. most interesting to me is how it behaves very differently than cpython when trying to avoid allocations and optimize. these kind

by nathants 6y ago

mattbillenstein 6y ago
Yeah, it's a lot of work to try and avoid copies and allocations. cpython I think comes from a very different era where cpu speeds were much closer to memory speeds and you had a lot less latency between the two in terms of cpu cycles. Like reference counting would probably not be a design you'd start at today if you were designing a new runtime. That being said, it's kinda amazing to me that cpython is only 4x slower in some of these benchmarks - and you can actually get within 2x with a different runtime that has the same guarantees and a JIT. It makes me think you have to have really good reasons not to choose Python for a lot of tasks given developer productivity and maturity of the runtime and libraries.
- nathants 6y ago
  or choose both and compose with pipes! good for prototyping, maybe escalate to a single executable when the design stabilizes.
  - mattbillenstein 6y ago
    Leads into serialization/deserialization and allocator overhead when using pipes. I usually went the route of parallelizing using the shell (xargs -P) or using multiprocessing. I typically had tens to hundreds of thousands of objects in S3 to process. But one neat trick that is pipe-like us using subprocess to do the gunzip step -- subprocess.Popen('gunzip -c ' + filename, ...) -- this would offload work from the python process and give me a decent wall clock speedup.