3 ms·
i have seldom seen data engineers write raw python loops the way you did with your examples. they usually use numpy, scipy, etc.
by blondin 5y ago
i have seldom seen data engineers write raw python loops the way you did with your examples. they usually use numpy, scipy, etc.
- shoo 5y agothat can be a good approach where the computation maps cleanly onto some C or fortran code prepared earlier and wrapped as a numpy or scipy primitive. but sometimes expressing what you want to do as a bunch of numpy operations becomes a lot harder to read and perhaps also slower than if you just directly wrote the raw loops over some arrays in C. in cases where what you want to do doesn't exactly fit standard operations, cython can be pretty nice. e.g. 200x -- 1000x speedups for translating C-oriented number crunching code from python to cython. but if you do want performance, you have to think about it while writing the code (avoid needlessly allocating memory in tight loops, data-oriented programming with simple arrays, statically type all of your variables, ...).