5 ms·

Pandas 0.15 has been released

116 points by jvm 12y ago

japaget 12y ago
Change log: http://pandas.pydata.org/pandas-docs/version/0.15.0/whatsnew.html http://pandas.pydata.org/pandas-docs/version/0.15.0/whatsnew... In particular, note that NumPy 1.7.0 or newer is required.
Bootvis 12y ago
I have some experience using Pandas but I'd love to read about peoples experience using it in production or in large teams. Please share!
- PudgePacket 12y ago
  It seems quite quick to do work on large datasets, I've found the documentation lacking however. There are no links in docs to the types that are being referenced, some types do not have documentation, some documentation is the function header with no other info ie no documentation, functions that take string formatting info eg '5min' do not have their argument possibilities documented anywhere I can find.
  - raymondh 12y ago
    Really? You found the documentation to be lacking. FWIW, there are over 1500 pages in the docs including a short-over view, tutorials, extensive feature coverage, and interaction with other tools: http://pandas.pydata.org/pandas-docs/version/0.15.0/pandas.pdf http://pandas.pydata.org/pandas-docs/version/0.15.0/pandas.p... The docs may have some issues, but they certainly can't be characterized as lacking.
    - Bootvis 12y ago
      In my eyes, you both make a valid point: yes there's plenty of documentation but sometimes I just can't find what I'm looking for. I like how the book[1] is structured, it really helped me but it isn't complete. Don't get me wrong, I'm grateful for all the work and I know I haven't contributed much but I think the online could be improved with more examples and recipes. Edit: There really is no excuse, getting started is easy[2]. [1]: http://shop.oreilly.com/product/0636920023784.do http://shop.oreilly.com/product/0636920023784.do [2]: http://pandas.pydata.org/developers.html http://pandas.pydata.org/developers.html
      - EmlynC 12y ago
        Correct me if I'm wrong but perhaps you had the same issue as me. The documentation is plentiful, lots of good examples and the book, similarly, increases with a nice linear complexity from basic "how do I select a (cell | row | column) ..." to full blown how do I do timeseries analysis on a dataseries pulled in from a remote source. The issue I had was not the documentation but the language of pandas mirrors the language used in R (I think this is something Wes McKinney intentional did) and it's the burden of all that new verbage that makes the documentation harder to sift through. Some choice exampels; "melt", "stack/unstack" and "reindex" — necessary, I grant you, so that functions can be aptly named and in turn encapsulate vectorised procedures that are composable. I found that the documentation was harder to search because I lacked the domain language and the documentation, for better for worse, doesn't dawdle with educating the reader about the verbage — worked examples often provide a easier route. It reads like a mathematical proof rather than prose and I used to think that the documentation was too terse but now I appreciate that probably just succinct.
  - RayVR 12y ago
    Documentation is mostly organized around trying to explain how to use some specific feature, which is usually not the best format for me, but it may be for others. The argument possibilities has always been an issue for me. In general, I have found, if you have non-homogenous data, Pandas is your best bet due to how general it is, even if it is sometimes frustrating when it forces a generalization on your data (e.g., try doing type conversions on numpy.datetime64, it lacks any sort of intuition). The distinction between a Series and a dataframe is something I always found pretty silly/frustrating and I wonder if it was the result of an early implementation issue rather than a logical simplification. maybe I'm particularly ignorant of some issues since my use case is perhaps more straightforward than some others but I've built an entire labelled data library for my team and it is easier for us to operate under similar primitive beliefs to the numpy ndarray, i.e., it's always an ndarray, adding a column (or dimension) does not change the type of the object and the associated methods and indexing in one particular way vs another (df['a'] vs df[['a']]) does not change the type of your object. If I'm missing the point of Series I would love to see them justified or a use case referenced.
- ehurrell 12y ago
  I find it excellent in production, and it's one of the backbones of Python as a 'data science' language. Being able to leverage dataframes in the same environment you build the webserver that serves the results is a really powerful thing. There have been times when the documentation has been difficult (mostly when it comes to already difficult to search for operations though).
- plafl 12y ago
  It's a good library, but I have the feeling that it tries to do too many things, or that you can do the same thing in too many different ways.
  - easytiger 12y ago
    Build & installation from scratch is far too complex. Relies on fortran libraries of numpy which I can't get built inside my corporate domain
    - elliott34 12y ago
      Why is that panda's problem? If you have scipy and numpy installed, it's just sudo pip install pandas....
      - easytiger 12y ago
        You can't do that in a closed envrionment
        ionforce 12y ago
        > You can't do that in a closed envrionment This is the real killer. Any leads on making it less closed? Also, I'm unfamiliar with how pip works, but you can't even install into your local user profile (i.e. w/o root)?
        damon_c 12y ago
        Sometimes, even when installing to a local virtual environment, if there are required libraries that require, for example a Fortran compiler, you'll need to install that on the system.
        mynegation 12y ago
        Use Anaconda Python distribution (https://store.continuum.io/cshop/anaconda/ https://store.continuum.io/cshop/anaconda/). Comes bundled with pandas, numpy, scipy (and much more). Does not require admin privileges. Updates and installations do not require working compiler (binary packages for your platform are downloaded).
elliott34 12y ago
I work in pandas 95% of my day doing data automation tasks, manipulating sql queries, and data-munging for machine learning. It is literally life changing for someone like me. I used to program solely in R, but after discovering pandas I really have no need to go back to R. My project workflow consists of several IPython notebooks+pandas+sklearn. Works extremely well in production, as in, on a flask web server, as well.
- minimaxir 12y ago
  For the particular tasks of "data automation tasks, manipulating sql queries, and data-munging for machine learning," Python is indeed better than R, especially with sklearn For other applications (especially charting and data manipulation with ggplot2 and dplyr respectively), R has an edge.
  - tlmr 12y ago
    I don't think so. Have you checked out seaborn, blaze and bokeh?
    - tdaltonc 12y ago
      Not OP, but I've also been look for a way to leave ggplot2 behind and make the jump to 100% python for data analysis. These look neat. Seaborn - http://web.stanford.edu/~mwaskom/software/seaborn/ http://web.stanford.edu/~mwaskom/software/seaborn/ Blaze - http://blaze.pydata.org/docs/v_0_6_5/index.html http://blaze.pydata.org/docs/v_0_6_5/index.html Bokeh - http://bokeh.pydata.org/ http://bokeh.pydata.org/
      - hadley 12y ago
        There's also the confusingly named ggplot (for python): https://github.com/yhat/ggplot https://github.com/yhat/ggplot
        tdaltonc 12y ago
makmanalp 12y ago
No discussion of the new features? Categorical is awesome, and equivalent to R's c() iirc. This should make plots easier in terms of automatically deciding whether to facet something, or showing legends nicely etc. The memory usage feature is super neat. Also for those of us stuck with STATA, the to_stata() and read_stata() just got much better. I'm eagerly awaiting a numpy native NA value instead of np.NaN.

116 points by jvm 12y ago

Change log: http://pandas.pydata.org/pandas-docs/version/0.15.0/whatsnew.html http://pandas.pydata.org/pandas-docs/version/0.15.0/whatsnew... In particular, note that NumPy 1.7.0 or newer is required.

Bootvis 12y ago

I have some experience using Pandas but I'd love to read about peoples experience using it in production or in large teams. Please share!

PudgePacket 12y ago
It seems quite quick to do work on large datasets, I've found the documentation lacking however. There are no links in docs to the types that are being referenced, some types do not have documentation, some documentation is the function header with no other info ie no documentation, functions that take string formatting info eg '5min' do not have their argument possibilities documented anywhere I can find.
- raymondh 12y ago
  Really? You found the documentation to be lacking. FWIW, there are over 1500 pages in the docs including a short-over view, tutorials, extensive feature coverage, and interaction with other tools: http://pandas.pydata.org/pandas-docs/version/0.15.0/pandas.pdf http://pandas.pydata.org/pandas-docs/version/0.15.0/pandas.p... The docs may have some issues, but they certainly can't be characterized as lacking.
  - Bootvis 12y ago
    In my eyes, you both make a valid point: yes there's plenty of documentation but sometimes I just can't find what I'm looking for. I like how the book[1] is structured, it really helped me but it isn't complete. Don't get me wrong, I'm grateful for all the work and I know I haven't contributed much but I think the online could be improved with more examples and recipes. Edit: There really is no excuse, getting started is easy[2]. [1]: http://shop.oreilly.com/product/0636920023784.do http://shop.oreilly.com/product/0636920023784.do [2]: http://pandas.pydata.org/developers.html http://pandas.pydata.org/developers.html
    - EmlynC 12y ago
      Correct me if I'm wrong but perhaps you had the same issue as me. The documentation is plentiful, lots of good examples and the book, similarly, increases with a nice linear complexity from basic "how do I select a (cell | row | column) ..." to full blown how do I do timeseries analysis on a dataseries pulled in from a remote source. The issue I had was not the documentation but the language of pandas mirrors the language used in R (I think this is something Wes McKinney intentional did) and it's the burden of all that new verbage that makes the documentation harder to sift through. Some choice exampels; "melt", "stack/unstack" and "reindex" — necessary, I grant you, so that functions can be aptly named and in turn encapsulate vectorised procedures that are composable. I found that the documentation was harder to search because I lacked the domain language and the documentation, for better for worse, doesn't dawdle with educating the reader about the verbage — worked examples often provide a easier route. It reads like a mathematical proof rather than prose and I used to think that the documentation was too terse but now I appreciate that probably just succinct.
- RayVR 12y ago
  Documentation is mostly organized around trying to explain how to use some specific feature, which is usually not the best format for me, but it may be for others. The argument possibilities has always been an issue for me. In general, I have found, if you have non-homogenous data, Pandas is your best bet due to how general it is, even if it is sometimes frustrating when it forces a generalization on your data (e.g., try doing type conversions on numpy.datetime64, it lacks any sort of intuition). The distinction between a Series and a dataframe is something I always found pretty silly/frustrating and I wonder if it was the result of an early implementation issue rather than a logical simplification. maybe I'm particularly ignorant of some issues since my use case is perhaps more straightforward than some others but I've built an entire labelled data library for my team and it is easier for us to operate under similar primitive beliefs to the numpy ndarray, i.e., it's always an ndarray, adding a column (or dimension) does not change the type of the object and the associated methods and indexing in one particular way vs another (df['a'] vs df[['a']]) does not change the type of your object. If I'm missing the point of Series I would love to see them justified or a use case referenced.
ehurrell 12y ago
I find it excellent in production, and it's one of the backbones of Python as a 'data science' language. Being able to leverage dataframes in the same environment you build the webserver that serves the results is a really powerful thing. There have been times when the documentation has been difficult (mostly when it comes to already difficult to search for operations though).
plafl 12y ago
It's a good library, but I have the feeling that it tries to do too many things, or that you can do the same thing in too many different ways.
- easytiger 12y ago
  Build & installation from scratch is far too complex. Relies on fortran libraries of numpy which I can't get built inside my corporate domain
  - elliott34 12y ago
    Why is that panda's problem? If you have scipy and numpy installed, it's just sudo pip install pandas....
    - easytiger 12y ago
      You can't do that in a closed envrionment
      - ionforce 12y ago
        > You can't do that in a closed envrionment This is the real killer. Any leads on making it less closed? Also, I'm unfamiliar with how pip works, but you can't even install into your local user profile (i.e. w/o root)?
        damon_c 12y ago
        Sometimes, even when installing to a local virtual environment, if there are required libraries that require, for example a Fortran compiler, you'll need to install that on the system.
        mynegation 12y ago
        Use Anaconda Python distribution (https://store.continuum.io/cshop/anaconda/ https://store.continuum.io/cshop/anaconda/). Comes bundled with pandas, numpy, scipy (and much more). Does not require admin privileges. Updates and installations do not require working compiler (binary packages for your platform are downloaded).

elliott34 12y ago

I work in pandas 95% of my day doing data automation tasks, manipulating sql queries, and data-munging for machine learning. It is literally life changing for someone like me. I used to program solely in R, but after discovering pandas I really have no need to go back to R. My project workflow consists of several IPython notebooks+pandas+sklearn. Works extremely well in production, as in, on a flask web server, as well.

minimaxir 12y ago
For the particular tasks of "data automation tasks, manipulating sql queries, and data-munging for machine learning," Python is indeed better than R, especially with sklearn For other applications (especially charting and data manipulation with ggplot2 and dplyr respectively), R has an edge.
- tlmr 12y ago
  I don't think so. Have you checked out seaborn, blaze and bokeh?
  - tdaltonc 12y ago
    Not OP, but I've also been look for a way to leave ggplot2 behind and make the jump to 100% python for data analysis. These look neat. Seaborn - http://web.stanford.edu/~mwaskom/software/seaborn/ http://web.stanford.edu/~mwaskom/software/seaborn/ Blaze - http://blaze.pydata.org/docs/v_0_6_5/index.html http://blaze.pydata.org/docs/v_0_6_5/index.html Bokeh - http://bokeh.pydata.org/ http://bokeh.pydata.org/
    - hadley 12y ago
      There's also the confusingly named ggplot (for python): https://github.com/yhat/ggplot https://github.com/yhat/ggplot
      - tdaltonc 12y ago

makmanalp 12y ago

No discussion of the new features? Categorical is awesome, and equivalent to R's c() iirc. This should make plots easier in terms of automatically deciding whether to facet something, or showing legends nicely etc. The memory usage feature is super neat. Also for those of us stuck with STATA, the to_stata() and read_stata() just got much better. I'm eagerly awaiting a numpy native NA value instead of np.NaN.