4 ms·
The thing is that the current ecosystem (numpy + scipy + pandas + PyCharm) fits our company structure perfectly (very few software engineers, mostly test engine
by ZuLuuuuuu 6y ago
The thing is that the current ecosystem (numpy + scipy + pandas + PyCharm) fits our company structure perfectly (very few software engineers, mostly test engineers who are not very proficient coders). And we already have tens of thousands lines of code. So changing the whole ecosystem just because 5% of the problems are slow is too big of a jump for us. For now, it is easier for us to write a bit of C code for places where we absolutely need speed. But it would have been so convenient if we could just have native threads, so that we don't even need that 5% C code.
- mhh__ 6y agoDo you know what you are actually stalled on? i.e. Memory or Instructions, if it's the latter you can probably eek out some more performance just by fiddling with compiler flags (you'd be surprised simultaneously how clever and how utterly braindead some compilers are if you play with them on Compiler Explorer for a bit) Also, if you are running on Intel you might have some luck with Intel's profiling tools as, although I've never used them for python they are by far and away the best things in this area, they do support Python. I don't know what details they give you in from Python code directly, but if you've used perf to profile code at a low level before, vTune is like a Tom & Jerry shotgun in comparison.
- pantsforbirds 6y agoYou don't have to replace the core python parts. We moved a decent amount of our data pre-processing into Scala, Golang, and Data Base queries and then just store the processed data for the python models to use to train/run (which ends up using the super efficient C/Fortran/Cuda code).