8 ms·

Pandas 1.0

510 points by kylebarron 7y ago

natalyarostova 7y ago
This is a huge accomplishment. The maintainers work very hard at keeping data science running for a substantial chunk of the field. Congratulations!
jzwinck 7y ago
I am looking forward to a decade of fewer API breaking changes. However, 1.0 introduces a new column type for strings, recommends its use over the old "object" column type, yet says it is "Experimental and may change at any time." How are we supposed to interpret this in light of the promise that there will be no more API breakages until 2.0? It reads as if this promise does not apply to string data, which impacts rather a lot of use cases.
- jellyksong 7y ago
  It states in the deprecation policy that "API-breaking changes will be made only in major releases (except for experimental features)".
  - s_Hogg 7y ago
    That seems like a rather large carve-out. Fortunately a lot of use cases won't really need to worry about something like that (e.g. exploratory analysis), in the context of it breaking your company.
  - jzwinck 7y ago
    So Pandas 1.0's recommended API for strings is unstable. Seems strange.
deleted 7y ago
[deleted]
mmahemoff 7y ago
"pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language." If anyone else is wondering what this is. (Source: project homepage
- smabie 7y ago
  The only true part is that it’s open source and BSD licensed. It’s one of those unfortunate libraries (like matplotlib) that are both very useful and not very fun to use.
dzonga 7y ago
one of the best python | data libraries out there. For analysis alone, no tool comes close.
_coveredInBees 7y ago
Great accomplishment and kudos to the dedicated maintainers. That being said, I've always had a love-hate relationship with pandas. It is a very powerful library and does a ton, but yet the API is all over the place and unless you use it regularly for a long period of time, it is almost impossible to get fluent with it. Every time I am away from it for a couple of months, I find even doing the most basic things to be complicated/confusing and find myself on stackoverflow way too often. By comparison, the API of something like Pytorch is an absolute pleasure to use and even though I'm not using it all the time, I almost have no trouble every time I begin training models/trying out new things in Pytorch. All that being said, this is definitely a step in the right direction and hopefully the API gets a bit more coherent over time.
- oarabbus_ 7y ago
  Couldn't agree more. Fantastically powerful library, but my goodness the syntax is often unfriendly, inconsistent, etc.
- Aperocky 7y ago
  > yet the API is all over the place This so much. I've answered quite a bit of pandas on SO, and I have to say the APIs are a mess. There are always multiple ways to do things, there are hidden traps that can lead to huge run times, and stuff that are just wildly un-pythonic. It's still the best general data processing has to offer. But a smaller, cleaner package might just take the cake.
  - spectramax 7y ago
    I can see a future for pytonic interface to pandas. It would run pandas in the backend but would be just as easy as manipulating list, dicts, sets and tuples...we'd have a 5th type - tables.
- 322849a8d87 7y ago
  > yet the API is all over the place Agreed. In particular one might have hoped that 1.0 would fix indexing. .ix (deprecated), .loc, .iloc and "[" is an example of what people mean by saying the API is (a) a mess and (b) "deeply unpythonic". Shouldn't "[" be removed entirely if .loc and .iloc are recommended, given the odd and unpredictable edge cases with "["? > unless you use it regularly for a long period of time, it is almost impossible to get fluent with it. Agreed. I know a huge amount of valiant, voluntary, open source work has gone into it, but it is a shame that the primary data-frame library in the Python ecosystem lacks a clean, pythonic API. Having been negative I don't want to obscure the fact that it does have some great and powerful code behind its API.
kmax12 7y ago
I know I'm not the only one, but it's hard to imagine doing my job the last several year without Pandas. Even though Pandas has been used in production by many people as basically a 1.0.0 release for a long time, this an amazing milestone and I think everyone in my office smiled when they saw the release news. I think it's worth it to acknowledge the great stewardship of the community by all the Pandas developers (and the rest of people in the PyData ecosystem). It has been an inspiration for me as I create and contribute to open source libraries for data science [0][1]. [0] https://github.com/FeatureLabs/featuretools/ https://github.com/FeatureLabs/featuretools/ [1] https://github.com/FeatureLabs/compose/ https://github.com/FeatureLabs/compose/
- alexpetralia 7y ago
  I basically owe my career to pandas - it made me the go-to resource for any data analysis that couldn't be done in Excel. Once I became useful in that respect, I was strongly encouraged to further develop my programming knowledge.
closed 7y ago
I've had to dive into the pandas code over the last year for a project [0], and my attitude has shifted dramatically from... * old attitude: why does pandas have to make things so hard * new attitude: pandas has a crazy difficult job I think this is most apparent in the functions that decide what "[d]type" a Block--the most basic thing that stores data in pandas--should be. https://github.com/pandas-dev/pandas/blob/4edcc5541ff3f6470f5e3c083cb83136119e6f0c/pandas/core/internals/blocks.py#L2973 https://github.com/pandas-dev/pandas/blob/4edcc5541ff3f6470f... And then, for the ubiquitous Object dtype, often figure out which of the many possible more specific types to cast it to. If you think that is easy, ask yourself what this outputs: import numpy as np np.array([np.nan, 'a']) Lo and behold--it produces an array where the np.nan has been converted to the string "nan". And yet import pandas as pd pd.Series([np.nan, "a"]) Knows this, has your back, and does not stringify it. It also has a pathological fixation on when it tries to convert dtypes, since avoiding all the bad conversion outcomes is a relatively time intensive process (compared to e.g. creating a numpy array). I realize things could be much easier in pandas user facing interface, but really appreciate the sheer amount of effort that has gone into its dtype wrangling. [0]: http://github.com/machow/siuba http://github.com/machow/siuba
- deleted 7y ago
  [deleted]
- _coveredInBees 7y ago
  I think another area pandas has done a lot of work on is with datetimes. Numpy's datetime objects are pretty deficient when you need to perform computations / data wrangling with them and utilizing python's native datetime objects would slow things down a decent bit. So they have done a lot of work to create their own datetime implementation that helps a lot when dealing with tabular data and performing date/time based arithmetic/manipulations.
  - spectramax 7y ago
    Datetimes are still a mess. Now we have 3 datetime objects and converting between them is not obvious or trivial: https://i.stack.imgur.com/uiXQd.png https://i.stack.imgur.com/uiXQd.png
squaresmile 7y ago
convert_dtypes is pretty nice. I wonder how soon the new dtypes will be the defaults.
iandinwoodie 7y ago
"We’ve added to_markdown() for creating a markdown table" That's awesome!
lapnitnelav 7y ago
Congratulations to the Pandas team. You lot have saved my bacon so many times over the last few years, I owe you many breakfasts. Long live the King.
mrfusion 7y ago
Assuming you want to use python. What’s the alternative to pandas? Kind of a brew up your own code kind of thing? Csv module I guess?
- jaifraic 7y ago
  Well, depends on what you want to achieve.
- pottertheotter 7y ago
  Dask is one option.
  - Pinus 7y ago
    Wildly off topic, for which I apologize, but whenever I see “Dask”, I think of this: https://en.wikipedia.org/wiki/DASK https://en.wikipedia.org/wiki/DASK . You’re not going to manage any large datasets on that! :-)
louis8799 7y ago
I have 30K line of codes in production using pandas 0.23.4. Should I consider updating.
- eesmith 7y ago
  Do you plan to stay at 0.23.4 forever? Then no. Else, yes.
  - poooood 7y ago
    This could be more pythonic
snicker7 7y ago
No mention of vaex? https://vaex.readthedocs.io/en/latest/ https://vaex.readthedocs.io/en/latest/ It has a cleaner, leaner API + the ability to use memory-mapped files.
drej 7y ago
I've been waiting for this release for years and I hoped for one thing and one thing only - for pandas to have a proper way of dealing with NULLs. And it does have it... OPTIONALLY. It's great that the whole thing with extension arrays, custom types etc. has lead to this, but when the devs have, after 10+ years, the biggest chance for a backward incompatible change, this is the one to make. By making it optional, they are fixing it for the very few that know of its existence. I love pandas and a sizeable part of my career depended on it - and while I don't use it anymore (partly because of the NULLs), I wish it the best and I hope there will be a future release with this breaking change.
alpineidyll3 7y ago
Pandas is my least favorite necessary evil. It's always changing, far too expansive API costs me about an hour a week. Whenever one can use a utc epoch column for time indexed data in a raw numpy array instead, one should.
ppod 7y ago
Could we collect some recommendations for really good books, online guides, tutorials, and recipes for current Pandas? There are quite a few complaints here about the interface being confusing and difficult to use, and I feel like some of this is due to there being significant differences between versions. I would love to read a medium-length online free tutorial on Pandas 1.0, but it seems like most of what turns on up google are short idiosyncratic tutorials on specific tasks in various versions.
- squaresmile 7y ago
  I find pandas tutorials great https://pandas.pydata.org/pandas-docs/stable/getting_started/index.html https://pandas.pydata.org/pandas-docs/stable/getting_started... https://pandas.pydata.org/pandas-docs/stable/getting_started/10min.html https://pandas.pydata.org/pandas-docs/stable/getting_started...
this_is_not_you 7y ago
Any word for when the RC will be "properly" released?

510 points by kylebarron 7y ago

This is a huge accomplishment. The maintainers work very hard at keeping data science running for a substantial chunk of the field. Congratulations!

jzwinck 7y ago

I am looking forward to a decade of fewer API breaking changes. However, 1.0 introduces a new column type for strings, recommends its use over the old "object" column type, yet says it is "Experimental and may change at any time." How are we supposed to interpret this in light of the promise that there will be no more API breakages until 2.0? It reads as if this promise does not apply to string data, which impacts rather a lot of use cases.

jellyksong 7y ago
It states in the deprecation policy that "API-breaking changes will be made only in major releases (except for experimental features)".
- s_Hogg 7y ago
  That seems like a rather large carve-out. Fortunately a lot of use cases won't really need to worry about something like that (e.g. exploratory analysis), in the context of it breaking your company.
- jzwinck 7y ago
  So Pandas 1.0's recommended API for strings is unstable. Seems strange.

deleted 7y ago

[deleted]

mmahemoff 7y ago

"pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language." If anyone else is wondering what this is. (Source: project homepage

smabie 7y ago
The only true part is that it’s open source and BSD licensed. It’s one of those unfortunate libraries (like matplotlib) that are both very useful and not very fun to use.

dzonga 7y ago

one of the best python | data libraries out there. For analysis alone, no tool comes close.

_coveredInBees 7y ago

Great accomplishment and kudos to the dedicated maintainers. That being said, I've always had a love-hate relationship with pandas. It is a very powerful library and does a ton, but yet the API is all over the place and unless you use it regularly for a long period of time, it is almost impossible to get fluent with it. Every time I am away from it for a couple of months, I find even doing the most basic things to be complicated/confusing and find myself on stackoverflow way too often. By comparison, the API of something like Pytorch is an absolute pleasure to use and even though I'm not using it all the time, I almost have no trouble every time I begin training models/trying out new things in Pytorch. All that being said, this is definitely a step in the right direction and hopefully the API gets a bit more coherent over time.

oarabbus_ 7y ago
Couldn't agree more. Fantastically powerful library, but my goodness the syntax is often unfriendly, inconsistent, etc.
Aperocky 7y ago
> yet the API is all over the place This so much. I've answered quite a bit of pandas on SO, and I have to say the APIs are a mess. There are always multiple ways to do things, there are hidden traps that can lead to huge run times, and stuff that are just wildly un-pythonic. It's still the best general data processing has to offer. But a smaller, cleaner package might just take the cake.
- spectramax 7y ago
  I can see a future for pytonic interface to pandas. It would run pandas in the backend but would be just as easy as manipulating list, dicts, sets and tuples...we'd have a 5th type - tables.
322849a8d87 7y ago
> yet the API is all over the place Agreed. In particular one might have hoped that 1.0 would fix indexing. .ix (deprecated), .loc, .iloc and "[" is an example of what people mean by saying the API is (a) a mess and (b) "deeply unpythonic". Shouldn't "[" be removed entirely if .loc and .iloc are recommended, given the odd and unpredictable edge cases with "["? > unless you use it regularly for a long period of time, it is almost impossible to get fluent with it. Agreed. I know a huge amount of valiant, voluntary, open source work has gone into it, but it is a shame that the primary data-frame library in the Python ecosystem lacks a clean, pythonic API. Having been negative I don't want to obscure the fact that it does have some great and powerful code behind its API.

kmax12 7y ago

I know I'm not the only one, but it's hard to imagine doing my job the last several year without Pandas. Even though Pandas has been used in production by many people as basically a 1.0.0 release for a long time, this an amazing milestone and I think everyone in my office smiled when they saw the release news. I think it's worth it to acknowledge the great stewardship of the community by all the Pandas developers (and the rest of people in the PyData ecosystem). It has been an inspiration for me as I create and contribute to open source libraries for data science [0][1]. [0] https://github.com/FeatureLabs/featuretools/ https://github.com/FeatureLabs/featuretools/ [1] https://github.com/FeatureLabs/compose/ https://github.com/FeatureLabs/compose/

alexpetralia 7y ago
I basically owe my career to pandas - it made me the go-to resource for any data analysis that couldn't be done in Excel. Once I became useful in that respect, I was strongly encouraged to further develop my programming knowledge.

closed 7y ago

I've had to dive into the pandas code over the last year for a project [0], and my attitude has shifted dramatically from... * old attitude: why does pandas have to make things so hard * new attitude: pandas has a crazy difficult job I think this is most apparent in the functions that decide what "[d]type" a Block--the most basic thing that stores data in pandas--should be. https://github.com/pandas-dev/pandas/blob/4edcc5541ff3f6470f5e3c083cb83136119e6f0c/pandas/core/internals/blocks.py#L2973 https://github.com/pandas-dev/pandas/blob/4edcc5541ff3f6470f... And then, for the ubiquitous Object dtype, often figure out which of the many possible more specific types to cast it to. If you think that is easy, ask yourself what this outputs: import numpy as np np.array([np.nan, 'a']) Lo and behold--it produces an array where the np.nan has been converted to the string "nan". And yet import pandas as pd pd.Series([np.nan, "a"]) Knows this, has your back, and does not stringify it. It also has a pathological fixation on when it tries to convert dtypes, since avoiding all the bad conversion outcomes is a relatively time intensive process (compared to e.g. creating a numpy array). I realize things could be much easier in pandas user facing interface, but really appreciate the sheer amount of effort that has gone into its dtype wrangling. [0]: http://github.com/machow/siuba http://github.com/machow/siuba

deleted 7y ago
[deleted]
_coveredInBees 7y ago
I think another area pandas has done a lot of work on is with datetimes. Numpy's datetime objects are pretty deficient when you need to perform computations / data wrangling with them and utilizing python's native datetime objects would slow things down a decent bit. So they have done a lot of work to create their own datetime implementation that helps a lot when dealing with tabular data and performing date/time based arithmetic/manipulations.
- spectramax 7y ago
  Datetimes are still a mess. Now we have 3 datetime objects and converting between them is not obvious or trivial: https://i.stack.imgur.com/uiXQd.png https://i.stack.imgur.com/uiXQd.png

squaresmile 7y ago

convert_dtypes is pretty nice. I wonder how soon the new dtypes will be the defaults.

iandinwoodie 7y ago

"We’ve added to_markdown() for creating a markdown table" That's awesome!

lapnitnelav 7y ago

Congratulations to the Pandas team. You lot have saved my bacon so many times over the last few years, I owe you many breakfasts. Long live the King.

mrfusion 7y ago

Assuming you want to use python. What’s the alternative to pandas? Kind of a brew up your own code kind of thing? Csv module I guess?

jaifraic 7y ago
Well, depends on what you want to achieve.
pottertheotter 7y ago
Dask is one option.
- Pinus 7y ago
  Wildly off topic, for which I apologize, but whenever I see “Dask”, I think of this: https://en.wikipedia.org/wiki/DASK https://en.wikipedia.org/wiki/DASK . You’re not going to manage any large datasets on that! :-)

louis8799 7y ago

I have 30K line of codes in production using pandas 0.23.4. Should I consider updating.

eesmith 7y ago
Do you plan to stay at 0.23.4 forever? Then no. Else, yes.
- poooood 7y ago
  This could be more pythonic

snicker7 7y ago

No mention of vaex? https://vaex.readthedocs.io/en/latest/ https://vaex.readthedocs.io/en/latest/ It has a cleaner, leaner API + the ability to use memory-mapped files.

drej 7y ago

I've been waiting for this release for years and I hoped for one thing and one thing only - for pandas to have a proper way of dealing with NULLs. And it does have it... OPTIONALLY. It's great that the whole thing with extension arrays, custom types etc. has lead to this, but when the devs have, after 10+ years, the biggest chance for a backward incompatible change, this is the one to make. By making it optional, they are fixing it for the very few that know of its existence. I love pandas and a sizeable part of my career depended on it - and while I don't use it anymore (partly because of the NULLs), I wish it the best and I hope there will be a future release with this breaking change.

alpineidyll3 7y ago

Pandas is my least favorite necessary evil. It's always changing, far too expansive API costs me about an hour a week. Whenever one can use a utc epoch column for time indexed data in a raw numpy array instead, one should.

ppod 7y ago

Could we collect some recommendations for really good books, online guides, tutorials, and recipes for current Pandas? There are quite a few complaints here about the interface being confusing and difficult to use, and I feel like some of this is due to there being significant differences between versions. I would love to read a medium-length online free tutorial on Pandas 1.0, but it seems like most of what turns on up google are short idiosyncratic tutorials on specific tasks in various versions.

squaresmile 7y ago
I find pandas tutorials great https://pandas.pydata.org/pandas-docs/stable/getting_started/index.html https://pandas.pydata.org/pandas-docs/stable/getting_started... https://pandas.pydata.org/pandas-docs/stable/getting_started/10min.html https://pandas.pydata.org/pandas-docs/stable/getting_started...

this_is_not_you 7y ago

Any word for when the RC will be "properly" released?