9 ms·

Learning C is easy, but getting a piece of C integrated with your Python code isn't. It can be done well, but bad software is littered with things that could ha

Learning C is easy, but getting a piece of C integrated with your Python code isn't. It can be done well, but bad software is littered with things that could have been done well. I recently had a project at work of ripping out a bad C implementation of a high-performance section of code and replacing it with faster and shorter piece of numpy, so this is definitely a practical issue.

mlthoughts2018 8y ago
I find the opposite to be true. With numba & Cython, it’s ridiculously simple to target a segment of Python code for compiled specialization, often required for custom pieces of work not directly expressible in NumPy, especially when processing non-ndarray data structures. I find I am frequently removing sections of inscrutable “vectorized” NumPy code that is inefficient, not because NumPy is inefficient but because conforming the data to a structure amenable to NumPy is too inefficient in the first place, and rewriting in straightforward, simple Cython functions.
- carlmr 8y ago
  >I find I am frequently removing sections of inscrutable “vectorized” NumPy code that is inefficient, not because NumPy is inefficient but because conforming the data to a structure amenable to NumPy is too inefficient in the first place Also the next maintainer will thank you for it.
yvdriess 8y ago
To generalize, learning C is easy, but doing anything with it is not. To be clear, this is not meant as a critique on C, on the contrary. Put simply, it is just that making computers do stuff is hard. This is especially true if you want it to do it in a very particular way such as in high-performance code. If you cannot already leverage the work of other experts, you need to know about modern optimizing compilers, language runtimes, modern processor architectures, OS/system, ...) To continue your Numpy example, that framework entombs the know-how and man-years of work of specialist in a certain area. If you need to do simple dense algebra, you will a hard time beating Numpy/Julia/Eigen/MKL etc. If your problem goes slightly off-track (e.g. sparse problems) where you cannot easily leverage the hard work and expertise of others, you better start rolling up your sleeves and open that main.c
- comicjk 8y ago
  I used scipy's sparse matrices - that was actually a key part of the project.

mlthoughts2018 8y ago

I find the opposite to be true. With numba & Cython, it’s ridiculously simple to target a segment of Python code for compiled specialization, often required for custom pieces of work not directly expressible in NumPy, especially when processing non-ndarray data structures. I find I am frequently removing sections of inscrutable “vectorized” NumPy code that is inefficient, not because NumPy is inefficient but because conforming the data to a structure amenable to NumPy is too inefficient in the first place, and rewriting in straightforward, simple Cython functions.

carlmr 8y ago
>I find I am frequently removing sections of inscrutable “vectorized” NumPy code that is inefficient, not because NumPy is inefficient but because conforming the data to a structure amenable to NumPy is too inefficient in the first place Also the next maintainer will thank you for it.

yvdriess 8y ago

To generalize, learning C is easy, but doing anything with it is not. To be clear, this is not meant as a critique on C, on the contrary. Put simply, it is just that making computers do stuff is hard. This is especially true if you want it to do it in a very particular way such as in high-performance code. If you cannot already leverage the work of other experts, you need to know about modern optimizing compilers, language runtimes, modern processor architectures, OS/system, ...) To continue your Numpy example, that framework entombs the know-how and man-years of work of specialist in a certain area. If you need to do simple dense algebra, you will a hard time beating Numpy/Julia/Eigen/MKL etc. If your problem goes slightly off-track (e.g. sparse problems) where you cannot easily leverage the hard work and expertise of others, you better start rolling up your sleeves and open that main.c

comicjk 8y ago
I used scipy's sparse matrices - that was actually a key part of the project.