4 ms·

I've used numpy a little and I enjoyed it and it performed ok, but is python really a good choice for writing this kind of code? I'm really genuinely interested

I've used numpy a little and I enjoyed it and it performed ok, but is python really a good choice for writing this kind of code? I'm really genuinely interested.

iskander 14y ago
I've been doing data analysis in Python for the past 2ish years and I think it's a great choice. Before Python I worked in Matlab, which simplifies matrix operations at the expense of making everything else terrible. The main other contenders here are R and Mathematica, both of which will fail you when you need do something that isn't strictly statistical/mathematical. Python gives you predictable decent performance and the NumPy ecosystem is awesome for numerical libraries. I've never come across a machine learning library nearly as well designed as scikit-learn and pandas dataframes are a lot snappier than R's equivalents. My only gripe is the paucity of good plotting libraries (matplotlib is impoverished and ugly compared with R's sexy plotting routines). Now, I haven't said a word about the faster statically compiled languages: C, C++, Java, C#, F#, OCaml, Haskell, etc... The trouble with static languages is that they either lack essential libraries or don't allow for rapid prototyping (or in some cases, both). Now, if you're implementing the heart of a numerically intensive algorithm and your code can't be decomposed into a few already implemented primitives, it makes sense to write it in C. The first thing to do, though, is to wrap that native code with a Python interface and test it from python.
- carterschonwald 14y ago
  There will be very nice tools rolling out for Haskell early this fall. :-). (I can't elaborate much because I'm busy writing theM presently, but stay tuned and I think y'all will like what you'll see when the public release lands) (one sexy hint though: the value add of these works in progress is enough that I'll be able to hire folks full time to work on it with me starting mid September or October . )
  - carterschonwald 14y ago
    Folks who are intrigued (whether as hypothetical users/customers, or as future colleagues / collaborators, shoot me an email!)
    - carterschonwald 14y ago
      Loving the emails :-)
ogrisel 14y ago
Yes as long as you profile and make sure that the bottleneck is a blas / atlas call (when doing numpy vector operation) or a compiled extension of your own (e.g. a piece of Cython or c code) if you need a for loop that cannot be vectorized easily with numpy. cython is very nice for numerical computation because it offers a nice syntax for dealing with array operations very efficiently (e.g. see typed memory views: http://docs.cython.org/src/userguide/memoryviews.html http://docs.cython.org/src/userguide/memoryviews.html) and has good integration with numpy datastructures and C-API.
- gajomi 14y ago
  >Yes as long as you profile and make sure that the bottleneck is a blas / atlas call I agree. Where Python really shines, I think, is where you have these sorts of bottlenecks + something extra that MATLAB performs terribly at, either as a consequence of language/VM design or lack of libraries.

I've used numpy a little and I enjoyed it and it performed ok, but is python really a good choice for writing this kind of code? I'm really genuinely interested.

iskander 14y ago

I've been doing data analysis in Python for the past 2ish years and I think it's a great choice. Before Python I worked in Matlab, which simplifies matrix operations at the expense of making everything else terrible. The main other contenders here are R and Mathematica, both of which will fail you when you need do something that isn't strictly statistical/mathematical. Python gives you predictable decent performance and the NumPy ecosystem is awesome for numerical libraries. I've never come across a machine learning library nearly as well designed as scikit-learn and pandas dataframes are a lot snappier than R's equivalents. My only gripe is the paucity of good plotting libraries (matplotlib is impoverished and ugly compared with R's sexy plotting routines). Now, I haven't said a word about the faster statically compiled languages: C, C++, Java, C#, F#, OCaml, Haskell, etc... The trouble with static languages is that they either lack essential libraries or don't allow for rapid prototyping (or in some cases, both). Now, if you're implementing the heart of a numerically intensive algorithm and your code can't be decomposed into a few already implemented primitives, it makes sense to write it in C. The first thing to do, though, is to wrap that native code with a Python interface and test it from python.

carterschonwald 14y ago
There will be very nice tools rolling out for Haskell early this fall. :-). (I can't elaborate much because I'm busy writing theM presently, but stay tuned and I think y'all will like what you'll see when the public release lands) (one sexy hint though: the value add of these works in progress is enough that I'll be able to hire folks full time to work on it with me starting mid September or October . )
- carterschonwald 14y ago
  Folks who are intrigued (whether as hypothetical users/customers, or as future colleagues / collaborators, shoot me an email!)
  - carterschonwald 14y ago
    Loving the emails :-)

ogrisel 14y ago

Yes as long as you profile and make sure that the bottleneck is a blas / atlas call (when doing numpy vector operation) or a compiled extension of your own (e.g. a piece of Cython or c code) if you need a for loop that cannot be vectorized easily with numpy. cython is very nice for numerical computation because it offers a nice syntax for dealing with array operations very efficiently (e.g. see typed memory views: http://docs.cython.org/src/userguide/memoryviews.html http://docs.cython.org/src/userguide/memoryviews.html) and has good integration with numpy datastructures and C-API.

gajomi 14y ago
>Yes as long as you profile and make sure that the bottleneck is a blas / atlas call I agree. Where Python really shines, I think, is where you have these sorts of bottlenecks + something extra that MATLAB performs terribly at, either as a consequence of language/VM design or lack of libraries.