4 ms·

Wes seems to be very focused on performance and big data applications these days, and of course it'd be great if Pandas could be used for bigger datasets, but w

Wes seems to be very focused on performance and big data applications these days, and of course it'd be great if Pandas could be used for bigger datasets, but when I hear people complain about Pandas they complain about: 1. the weird handling of types and null values (#4) 2. the verbosity of filtering like `dataframe[dataframe.column == x]` and transformations like `dataframe.col_a - dataframe.col_b`, compared to `dplyr` in R 3. warts on the indexing system (including MultiIndex, which is very powerful but confusing) For those of us who use Pandas as an alternative to R, these usability shortcomings matter way more than memory efficiency.

vonnik 9y ago
People interested in perf and big data ETL for the JVM may want to check out DataVec: https://github.com/deeplearning4j/datavec https://github.com/deeplearning4j/datavec https://deeplearning4j.org/datavec https://deeplearning4j.org/datavec It vectorizes/tensorizes most major data types to put them in shape for machine learning. It also lets you save the data pipeline as a reusable object.
- deleted 9y ago
  [deleted]
timClicks 9y ago
I too would welcome a friendlier pandas library, but every time I've tried to think of an API that would work I fail. Well actually, I keep on wanting pandas to understand SQL.
- lars512 9y ago
  There’s a pandasql library which lets you execute SQL on dataframes. It’s a little slower because it needs to serialize via SQLite, but it’s a quick way to get going. https://pypi.python.org/pypi/pandasql https://pypi.python.org/pypi/pandasql
chasedehan 9y ago
I will definitely echo (2). dplyr is amazing and works in far, far fewer lines of code than pandas. That was my largest issue when migrating over from R. There is dplython, but it doesn't quite work the same so I don't use it much. https://github.com/dodger487/dplython https://github.com/dodger487/dplython
- has2k1 9y ago
  I created plydata, you may find it sufficient for your needs. https://github.com/has2k1/plydata https://github.com/has2k1/plydata
  - chasedehan 9y ago
    Is the only difference the placeholder X? I was running into issues with dplython in executing arbitrary functions outside the tidyverse. Can your package handle situations such as this? df %>% select(var1, var2) %>% rbind(df2) %>% na.omit() etc? That was the big benefit I saw from using dplyr.`
    - has2k1 9y ago
      Yes, no X placeholder. And at the moment you cannot pipe to arbitrary functions, Python limitations. I'll get around this by providing a helper function e.g df >> call(pd.dropna, axis=1)
paultopia 9y ago
#2 is a big issue for me. Filtering, subsetting, these are all really arcane-feeling transformations, there's a lot of weirdness with view vs copy, etc.
filmor 9y ago
You can write 2) as dataframe.query("column == x").

vonnik 9y ago

People interested in perf and big data ETL for the JVM may want to check out DataVec: https://github.com/deeplearning4j/datavec https://github.com/deeplearning4j/datavec https://deeplearning4j.org/datavec https://deeplearning4j.org/datavec It vectorizes/tensorizes most major data types to put them in shape for machine learning. It also lets you save the data pipeline as a reusable object.

deleted 9y ago
[deleted]

timClicks 9y ago

I too would welcome a friendlier pandas library, but every time I've tried to think of an API that would work I fail. Well actually, I keep on wanting pandas to understand SQL.

lars512 9y ago
There’s a pandasql library which lets you execute SQL on dataframes. It’s a little slower because it needs to serialize via SQLite, but it’s a quick way to get going. https://pypi.python.org/pypi/pandasql https://pypi.python.org/pypi/pandasql

chasedehan 9y ago

I will definitely echo (2). dplyr is amazing and works in far, far fewer lines of code than pandas. That was my largest issue when migrating over from R. There is dplython, but it doesn't quite work the same so I don't use it much. https://github.com/dodger487/dplython https://github.com/dodger487/dplython

has2k1 9y ago
I created plydata, you may find it sufficient for your needs. https://github.com/has2k1/plydata https://github.com/has2k1/plydata
- chasedehan 9y ago
  Is the only difference the placeholder X? I was running into issues with dplython in executing arbitrary functions outside the tidyverse. Can your package handle situations such as this? df %>% select(var1, var2) %>% rbind(df2) %>% na.omit() etc? That was the big benefit I saw from using dplyr.`
  - has2k1 9y ago
    Yes, no X placeholder. And at the moment you cannot pipe to arbitrary functions, Python limitations. I'll get around this by providing a helper function e.g df >> call(pd.dropna, axis=1)

paultopia 9y ago

#2 is a big issue for me. Filtering, subsetting, these are all really arcane-feeling transformations, there's a lot of weirdness with view vs copy, etc.

filmor 9y ago

You can write 2) as dataframe.query("column == x").