4 ms·

2017 Outlook: Pandas, Arrow, Feather, Parquet, Spark, Ibis

78 points by TheAlchemist 10y ago

pixelmonkey 10y ago
Thank you for all you do for the Python and data community, Wes! Arrow and "native" parquet support for Python are very exciting developments.
- lintiness 10y ago
  i've migrated back to r for various reasons, but wes really is a beast in the python finance world.
  - peatmoss 10y ago
    I'd expect that improvements to Arrow, Feather, Parquet etc. will help lift lots of data communities including those of us who also do R. I'm really looking forward to reusable in-memory dataframe structures like Arrow to be the foundations for R/Pandas-like data munging in lots of languages that aren't R or Python
ttub 10y ago
Very cool, but does anyone know more about the plan to offer a "lib pandas" mentioned here?https://mail.python.org/pipermail/pandas-dev/2015-December/000395.html https://mail.python.org/pipermail/pandas-dev/2015-December/0...
- peatmoss 10y ago
  Hoping Wes will chime in here if I'm wrong, but I'm guessing that the technologies he discusses here will more or less take the place of a lib pandas.
  - wesm 10y ago
    No, it's all closely inter-related. pandas as a project is responsible for data manipulation and in-memory analytics. These other projects are all complementary technologies.
- shoyer 10y ago
  "lib pandas" has become pandas 2.0. See https://github.com/pandas-dev/pandas2 https://github.com/pandas-dev/pandas2 for ongoing design discussions and development.
Nexxxeh 10y ago
I was expecting a post about Microsoft's email client. An annoying side effect of using commonly used words as product names. Signal being the most egregious offender recently. The capital O of Outlook didn't help. I know nothing of Python, R and big data, but things are looking up by the sound of it?
filereaper 10y ago
Found out about Apache Kudu, written in C++... Are these systems moving away from the JVM now?
aorth 10y ago
My goal is to deliver the same quality pandas user experience on 10x as much data. pandas works well on 1GB of data, but less well on 10GB. This has to change for pandas to remain a relevant tool in the future. Awesome! Great to see such honest commitment. Pandas is already a great library, and I can't wait to see the progress!
SolarNet 10y ago
At first I thought this was a list of animals going extinct or something.
deleted 10y ago
[deleted]
xapata 10y ago
I hope the API becomes more Pythonic and less NumPy-ic. A little more purity would be practical in this case.
mojoe 10y ago
Feather is awesome, and I really hope that Dask (http://dask.pydata.org/en/latest/ http://dask.pydata.org/en/latest/) can eventually integrate it so that we have a lightweight way to write to/from disk in parallel.
ipstone2014 10y ago
Nice road-map and Kudos to Wes for the great work/tools for the data community!

78 points by TheAlchemist 10y ago

Thank you for all you do for the Python and data community, Wes! Arrow and "native" parquet support for Python are very exciting developments.

lintiness 10y ago
i've migrated back to r for various reasons, but wes really is a beast in the python finance world.
- peatmoss 10y ago
  I'd expect that improvements to Arrow, Feather, Parquet etc. will help lift lots of data communities including those of us who also do R. I'm really looking forward to reusable in-memory dataframe structures like Arrow to be the foundations for R/Pandas-like data munging in lots of languages that aren't R or Python

ttub 10y ago

Very cool, but does anyone know more about the plan to offer a "lib pandas" mentioned here?https://mail.python.org/pipermail/pandas-dev/2015-December/000395.html https://mail.python.org/pipermail/pandas-dev/2015-December/0...

peatmoss 10y ago
Hoping Wes will chime in here if I'm wrong, but I'm guessing that the technologies he discusses here will more or less take the place of a lib pandas.
- wesm 10y ago
  No, it's all closely inter-related. pandas as a project is responsible for data manipulation and in-memory analytics. These other projects are all complementary technologies.
shoyer 10y ago
"lib pandas" has become pandas 2.0. See https://github.com/pandas-dev/pandas2 https://github.com/pandas-dev/pandas2 for ongoing design discussions and development.

Nexxxeh 10y ago

I was expecting a post about Microsoft's email client. An annoying side effect of using commonly used words as product names. Signal being the most egregious offender recently. The capital O of Outlook didn't help. I know nothing of Python, R and big data, but things are looking up by the sound of it?

filereaper 10y ago

Found out about Apache Kudu, written in C++... Are these systems moving away from the JVM now?

aorth 10y ago

My goal is to deliver the same quality pandas user experience on 10x as much data. pandas works well on 1GB of data, but less well on 10GB. This has to change for pandas to remain a relevant tool in the future. Awesome! Great to see such honest commitment. Pandas is already a great library, and I can't wait to see the progress!

SolarNet 10y ago

At first I thought this was a list of animals going extinct or something.

deleted 10y ago

[deleted]

xapata 10y ago

I hope the API becomes more Pythonic and less NumPy-ic. A little more purity would be practical in this case.

mojoe 10y ago

Feather is awesome, and I really hope that Dask (http://dask.pydata.org/en/latest/ http://dask.pydata.org/en/latest/) can eventually integrate it so that we have a lightweight way to write to/from disk in parallel.

ipstone2014 10y ago

Nice road-map and Kudos to Wes for the great work/tools for the data community!