5 ms·

I am a C/C++ programmer mostly. Is there good documentation for python development when I care about performance and memory footprint out there as a book or som

I am a C/C++ programmer mostly. Is there good documentation for python development when I care about performance and memory footprint out there as a book or something that anyone can recommend? It was fun reading this but I like to know more.

BerislavLopac 7y ago
This article is a solid start: https://rushter.com/blog/numba-cython-python-optimization/ https://rushter.com/blog/numba-cython-python-optimization/
- myth_drannon 7y ago
  High Performance Python is a good book. https://www.oreilly.com/library/view/high-performance-python/9781449361747/ https://www.oreilly.com/library/view/high-performance-python...
drej 7y ago
High Performance Python by Ian Ozsvald and Micha Gorelick - fairly outdated at this point, but still quite relevant (I believe the authors are working on a new edition).
- BerislavLopac 7y ago
  They have started work on the new edition if I'm not mistaken.
  - agumonkey 7y ago
    No draft online ?
    - BerislavLopac 7y ago
      I don't think so. This is quoted from Ian's latest mailing list [0] posting: Second Edition of High Performance Python in the works I'm very pleased to say that Micha and I have started work on a Second Edition of High Performance Python with O'Reilly, planned for early 2020. This book will use Python 3.7 and will add tools that barely existed 4 years ago including Dask and probably some Tensorflow, amongst lots of other goodies. I'll let you know how the book progresses via this list. So far I've updated the Profiling chapter, added some advice on 'being a highly performant developer' and have rebuilt the Cython/Numba/PyPy code for the Compiling chapter. [0] https://ianozsvald.com/data-science-jobs/ https://ianozsvald.com/data-science-jobs/
- ausjke 7y ago
  it's written in 2014 so not that old I think
sixplusone 7y ago
I thought the usual mantra for performance-critical functions is to implement them in C and call from py.
- hermitdev 7y ago
  Not always necessary. I do a lot of ETL type work in python, and generally, the idea is to never pull everything into memory at once if you don't need to. This means leveraging generators quite a bit. Read a row, process a row, generate a row, pass it along the pipeline. I've written scripts that can process a 1k line csv file with the same memory footprint as a 10M line csv. If you have to read everything into memory, because you're doing some sort of transforms like a list to a dict or such, explicitly deleting variables can help, as well, rather than waiting for them to go out of scope, but this should be pretty rare.
- pjmlp 7y ago
  Or just port to Julia, Scheme or Common Lisp, while enjoying the power of dynamic languages with compilation to native code.
njharman 7y ago
I'm a Python "snob", going on 24 years and If I ever "I care about performance and memory footprint" I'd not use Python. Python is good enough 90%. You get faster 2x 10x, etc code by picking better algorythms or solutions. In Python, you should not be caring about 10% or 30% speed improvement. It's not worth it, It's not Python's strength. When you need faster you go to C based libs (the python libs that need to go fast are already in C), https://en.wikipedia.org/wiki/Cython https://en.wikipedia.org/wiki/Cython https://en.wikipedia.org/wiki/Numba https://en.wikipedia.org/wiki/Numba etc.
- jerven 7y ago
  Or go for [1]pypy or [2]graalpython. I know that equivalent code in graal java ee produces equivalent assembly/performance to C+current gcc, which really shows that yes changing to a JIT can pay off. The python in graalvm is early stages but it shows that the old lesson of just write the hot spot in C is no longer true. Chris Seaton shows that for Ruby/C the language Ruby interpreter/C switch is expensive. I think the same is true for Python interpreter/C switches. Something that GraalVM+Python|Ruby can optimize out. [1] https://www.pypy.org https://www.pypy.org [2] https://www.graalvm.org/docs/reference-manual/languages/python/ https://www.graalvm.org/docs/reference-manual/languages/pyth...
  - jashmatthews 7y ago
    LuaJIT has great performance through the C FFI. It’s more that traditional JITs can’t optimize across FFI boundaries. Hopefully Chris will correct me if I’m full of shit. There are other approaches which can work too, like compiling C to LLVM IR using Clang so a function can be inlined by an LLVM based JIT at runtime.
- edflsafoiewq 7y ago
  Like with JS, you might not have a choice. I mostly use Python for extending programs that use it as a embedded scripting language. Performance is still important.
zerkten 7y ago
See Mike Mueller's tutorials from PyCon at https://www.youtube.com/watch?v=DGrS0uwMuHY https://www.youtube.com/watch?v=DGrS0uwMuHY. That's the 2018 one, but I'm guessing there is a similar one. There always seems to be a high perf tutorial of some kind.

BerislavLopac 7y ago

This article is a solid start: https://rushter.com/blog/numba-cython-python-optimization/ https://rushter.com/blog/numba-cython-python-optimization/

myth_drannon 7y ago
High Performance Python is a good book. https://www.oreilly.com/library/view/high-performance-python/9781449361747/ https://www.oreilly.com/library/view/high-performance-python...

drej 7y ago

High Performance Python by Ian Ozsvald and Micha Gorelick - fairly outdated at this point, but still quite relevant (I believe the authors are working on a new edition).

BerislavLopac 7y ago
They have started work on the new edition if I'm not mistaken.
- agumonkey 7y ago
  No draft online ?
  - BerislavLopac 7y ago
    I don't think so. This is quoted from Ian's latest mailing list [0] posting: Second Edition of High Performance Python in the works I'm very pleased to say that Micha and I have started work on a Second Edition of High Performance Python with O'Reilly, planned for early 2020. This book will use Python 3.7 and will add tools that barely existed 4 years ago including Dask and probably some Tensorflow, amongst lots of other goodies. I'll let you know how the book progresses via this list. So far I've updated the Profiling chapter, added some advice on 'being a highly performant developer' and have rebuilt the Cython/Numba/PyPy code for the Compiling chapter. [0] https://ianozsvald.com/data-science-jobs/ https://ianozsvald.com/data-science-jobs/
ausjke 7y ago
it's written in 2014 so not that old I think

sixplusone 7y ago

I thought the usual mantra for performance-critical functions is to implement them in C and call from py.

hermitdev 7y ago
Not always necessary. I do a lot of ETL type work in python, and generally, the idea is to never pull everything into memory at once if you don't need to. This means leveraging generators quite a bit. Read a row, process a row, generate a row, pass it along the pipeline. I've written scripts that can process a 1k line csv file with the same memory footprint as a 10M line csv. If you have to read everything into memory, because you're doing some sort of transforms like a list to a dict or such, explicitly deleting variables can help, as well, rather than waiting for them to go out of scope, but this should be pretty rare.
pjmlp 7y ago
Or just port to Julia, Scheme or Common Lisp, while enjoying the power of dynamic languages with compilation to native code.

njharman 7y ago

I'm a Python "snob", going on 24 years and If I ever "I care about performance and memory footprint" I'd not use Python. Python is good enough 90%. You get faster 2x 10x, etc code by picking better algorythms or solutions. In Python, you should not be caring about 10% or 30% speed improvement. It's not worth it, It's not Python's strength. When you need faster you go to C based libs (the python libs that need to go fast are already in C), https://en.wikipedia.org/wiki/Cython https://en.wikipedia.org/wiki/Cython https://en.wikipedia.org/wiki/Numba https://en.wikipedia.org/wiki/Numba etc.

jerven 7y ago
Or go for [1]pypy or [2]graalpython. I know that equivalent code in graal java ee produces equivalent assembly/performance to C+current gcc, which really shows that yes changing to a JIT can pay off. The python in graalvm is early stages but it shows that the old lesson of just write the hot spot in C is no longer true. Chris Seaton shows that for Ruby/C the language Ruby interpreter/C switch is expensive. I think the same is true for Python interpreter/C switches. Something that GraalVM+Python|Ruby can optimize out. [1] https://www.pypy.org https://www.pypy.org [2] https://www.graalvm.org/docs/reference-manual/languages/python/ https://www.graalvm.org/docs/reference-manual/languages/pyth...
- jashmatthews 7y ago
  LuaJIT has great performance through the C FFI. It’s more that traditional JITs can’t optimize across FFI boundaries. Hopefully Chris will correct me if I’m full of shit. There are other approaches which can work too, like compiling C to LLVM IR using Clang so a function can be inlined by an LLVM based JIT at runtime.
edflsafoiewq 7y ago
Like with JS, you might not have a choice. I mostly use Python for extending programs that use it as a embedded scripting language. Performance is still important.

zerkten 7y ago

See Mike Mueller's tutorials from PyCon at https://www.youtube.com/watch?v=DGrS0uwMuHY https://www.youtube.com/watch?v=DGrS0uwMuHY. That's the 2018 one, but I'm guessing there is a similar one. There always seems to be a high perf tutorial of some kind.