4 ms·

“Fancy array-of-associative-arrays with their own set of non-Python-native field data types and a lot of helper methods for sorting and import and export and su

“Fancy array-of-associative-arrays with their own set of non-Python-native field data types and a lot of helper methods for sorting and import and export and such, but weirdly always missing any helpers that would be both hard to write and very useful” “SQLite but only for Python and worse. Kind of.” “That annoying transitional step you have no direct need for but that you have to do anyway, because all data-wrangling tools in Python assume a dataframe” (I use this stuff daily) [edit] “Imagine if you could read in from a database or CSV and then work with the returned rows directly and as if they were still a table/spreadsheet. Except by ‘directly’ I mean ‘after turning it into a dataframe and discarding/fucking-up all your data type information’. And then with another fucking-everything-up step if you want to write it out so you can do real stuff with it elsewhere.”

dajt 3y ago
LOL. I have been programming for over 30 years on all sorts of systems and the Pandas DataFrame API is completely beyond me. Just trying to get a cell value seems way more difficult than it should be. Same with xarray datasets. I just loaded the same CSV into Pandas and Polars and Polars did a much better job of it.
- tomrod 3y ago
  What part of df.loc[index_val, column_val] is hard here? Oh you meant multiindex... yeah, slicing multiindexes sucks :)
  - ok_computer 3y ago
    Multi indexes is baking my data into the horrible pandas object model into weird tuples. Every gripe I’ve had with pandas starts with trying to do something “simple” then following the pandas object model to achieve it and over complicating things. Polars is awesome, it fits my numpy understanding of dataframes as dict labeled arrays. I even like the immutability aspect.
- vundercind 3y ago
  Polars is at least more consistent & sensible. The whole thing though is just shockingly unhelpful and half-baked for something that taken over so completely in its niche. I guess my perspective is that of someone trying to build reliable automation, though—it’s probably really nice if you’re just noodlin’ in notebooks or the repl or whatever.
- mmmmpancakes 3y ago
  To me, Polars feels like almost exactly how I would want to redesign Pandas interfaces for small - medium sized data processing, given my previous experience with Pandas and PySpark. Throw out all the custom multi index nonsense, throw out numpy and handle types properly, memory map arrow, focus on method chaining interface, do standard stuff like groupby and window functions in the standard way, and implement all the query optimizations under the hood that we know make stuff way faster. To be fair, Polars has the benefit of hindsight and designing their interfaces and syntax from scratch. The poor choices in Pandas were made long ago, and its adoption and evolution into the most popular dataframe library for python feels like mostly about timing the market than having the best software product.
- fbdab103 3y ago
  Accessing an individual cell value is slightly clunky, but that's not really where you use a DataFrame. A DataFrame is an object for where the entirety of the dataset is under study. Where you are typically interested in the broad distributions contained within the data. After you have highlighted trends (the majority of the work), then you might go spelunking at individual examples to see why something is funny.
ericjmorey 3y ago
Is DuckDB a better tool for your purposes? Dataframes are popular among people who are trying to use the data in a way that they don't intend to do anything more with the processed data after they've reported their findings.
- cmollis 3y ago
  I've been testing polars and duckdb recently.. Polars is an excellent dataframe package.. extremely memory efficient and fast. I've experienced some issues with a hive-partitioning of a large S3 dataset which DuckDB doesn't have. I've scanned multi-TB S3 parquet datasets with DuckDB on my m1 laptop executing some really hairy SQL (stuff I didn't think it could handle).. window functions, joins to other parquet datasets just as large, etc. Very impressive software. I haven't done the same types of things in Polars yet (simple selects).
  - cmollis 3y ago
    Also, duckdb allows you convert scan results directly to both pandas and pl dataframes.. so you can mix and match based on need.
  - theLiminator 3y ago
    Hmm, how are you getting acceptable performance doing that? Is your data in a delta table or something? I've actually personally found that DuckDB is tremendously slow against the cloud, though perhaps I'm going through the wrong API? I'm using https://duckdb.org/docs/guides/import/s3_import https://duckdb.org/docs/guides/import/s3_import. My data is hive partitioned, when I monitor my network throughput, I only get a few MB/s with DuckDB but can achieve 1-2GB/s through polars. Very possible it's a case of PEBKAC though.
mmmmpancakes 3y ago
i mean, if you are reading and writing csv then yeah, you've already fucked up.
- hermitcrab 3y ago
  CSV is a terrible format. But it is extensively used. See also: Why isn’t there a decent file format for tabular data? https://news.ycombinator.com/item?id=31220841 https://news.ycombinator.com/item?id=31220841
  - mmmmpancakes 3y ago
    parquet is perfectly fine > There is Parquet. It is very efficient with it’s columnar storage and compression. But it is binary, so can’t be viewed or edited with standard tools, which is a pain. I can open parquet in excel
    - hermitcrab 3y ago
      Using this connector? https://learn.microsoft.com/en-us/power-query/connectors/parquet https://learn.microsoft.com/en-us/power-query/connectors/par...

dajt 3y ago

LOL. I have been programming for over 30 years on all sorts of systems and the Pandas DataFrame API is completely beyond me. Just trying to get a cell value seems way more difficult than it should be. Same with xarray datasets. I just loaded the same CSV into Pandas and Polars and Polars did a much better job of it.

tomrod 3y ago
What part of df.loc[index_val, column_val] is hard here? Oh you meant multiindex... yeah, slicing multiindexes sucks :)
- ok_computer 3y ago
  Multi indexes is baking my data into the horrible pandas object model into weird tuples. Every gripe I’ve had with pandas starts with trying to do something “simple” then following the pandas object model to achieve it and over complicating things. Polars is awesome, it fits my numpy understanding of dataframes as dict labeled arrays. I even like the immutability aspect.
vundercind 3y ago
Polars is at least more consistent & sensible. The whole thing though is just shockingly unhelpful and half-baked for something that taken over so completely in its niche. I guess my perspective is that of someone trying to build reliable automation, though—it’s probably really nice if you’re just noodlin’ in notebooks or the repl or whatever.
mmmmpancakes 3y ago
To me, Polars feels like almost exactly how I would want to redesign Pandas interfaces for small - medium sized data processing, given my previous experience with Pandas and PySpark. Throw out all the custom multi index nonsense, throw out numpy and handle types properly, memory map arrow, focus on method chaining interface, do standard stuff like groupby and window functions in the standard way, and implement all the query optimizations under the hood that we know make stuff way faster. To be fair, Polars has the benefit of hindsight and designing their interfaces and syntax from scratch. The poor choices in Pandas were made long ago, and its adoption and evolution into the most popular dataframe library for python feels like mostly about timing the market than having the best software product.
fbdab103 3y ago
Accessing an individual cell value is slightly clunky, but that's not really where you use a DataFrame. A DataFrame is an object for where the entirety of the dataset is under study. Where you are typically interested in the broad distributions contained within the data. After you have highlighted trends (the majority of the work), then you might go spelunking at individual examples to see why something is funny.

ericjmorey 3y ago

Is DuckDB a better tool for your purposes? Dataframes are popular among people who are trying to use the data in a way that they don't intend to do anything more with the processed data after they've reported their findings.

cmollis 3y ago
I've been testing polars and duckdb recently.. Polars is an excellent dataframe package.. extremely memory efficient and fast. I've experienced some issues with a hive-partitioning of a large S3 dataset which DuckDB doesn't have. I've scanned multi-TB S3 parquet datasets with DuckDB on my m1 laptop executing some really hairy SQL (stuff I didn't think it could handle).. window functions, joins to other parquet datasets just as large, etc. Very impressive software. I haven't done the same types of things in Polars yet (simple selects).
- cmollis 3y ago
  Also, duckdb allows you convert scan results directly to both pandas and pl dataframes.. so you can mix and match based on need.
- theLiminator 3y ago
  Hmm, how are you getting acceptable performance doing that? Is your data in a delta table or something? I've actually personally found that DuckDB is tremendously slow against the cloud, though perhaps I'm going through the wrong API? I'm using https://duckdb.org/docs/guides/import/s3_import https://duckdb.org/docs/guides/import/s3_import. My data is hive partitioned, when I monitor my network throughput, I only get a few MB/s with DuckDB but can achieve 1-2GB/s through polars. Very possible it's a case of PEBKAC though.

mmmmpancakes 3y ago

i mean, if you are reading and writing csv then yeah, you've already fucked up.

hermitcrab 3y ago
CSV is a terrible format. But it is extensively used. See also: Why isn’t there a decent file format for tabular data? https://news.ycombinator.com/item?id=31220841 https://news.ycombinator.com/item?id=31220841
- mmmmpancakes 3y ago
  parquet is perfectly fine > There is Parquet. It is very efficient with it’s columnar storage and compression. But it is binary, so can’t be viewed or edited with standard tools, which is a pain. I can open parquet in excel
  - hermitcrab 3y ago
    Using this connector? https://learn.microsoft.com/en-us/power-query/connectors/parquet https://learn.microsoft.com/en-us/power-query/connectors/par...