3 ms·

I was going to say - it always feels so humbling seeing pages like this. "DataFrames for the new era" okay… maybe I know what data frames are? "Multi-threaded q

I was going to say - it always feels so humbling seeing pages like this. "DataFrames for the new era" okay… maybe I know what data frames are? "Multi-threaded query engine" ahh, so it’s like a database. A graph comparing it to things called pandas, modin, and vaex - I have no clue what any of these are either! I guess this really isn’t for me. It’s a shame because I like to read about new tech or project and try and learn more, even if I don’t understand it completely. But there’s just nothing here for me. This must be what normal people go through when I talk about my lowly web development work…

theLiminator 3y ago
It's pretty much just an alternative to SQL that's a lot easier/natural to use for more hardcore data analysis. You can much more easily compose the operations you want to run. Just think of it as an API for manipulating tabular data stored somewhere (often parquet files, though they can query many different data sources).
- riku_iki 3y ago
  > It's pretty much just an alternative to SQL Polars actually has SQL supported, so you can mix and match: e.g. construct DF from SQL.
- youainti 3y ago
  Data tables tend to also be a standard ingestion format for statistical tools in many cases.
- _dain_ 3y ago
  Dataframes and SQL have overlapping functionality, but I wouldn't say that dataframes are an "alternative" to SQL. The tradeoffs are very different. You don't have to worry about minimizing disk reads or think about concurrency issues like transactions or locks, because a dataframe is just an in-memory data structure like a list or a dict, rather than a database. Dataframes also aren't really about relational algebra like SQL is.
  - theLiminator 3y ago
    Have you tried polars? I agree that if pandas is all you've tried, that it's pretty far from an alternative frontend for query engines, but if you've tried polars it maps pretty cleanly to SQL, can be query optimized to a query plan similarly to SQL, I should've made it clear that I'm talking about an alternative to SQL used in an OLAP context, not for OLTP.

theLiminator 3y ago

It's pretty much just an alternative to SQL that's a lot easier/natural to use for more hardcore data analysis. You can much more easily compose the operations you want to run. Just think of it as an API for manipulating tabular data stored somewhere (often parquet files, though they can query many different data sources).

riku_iki 3y ago
> It's pretty much just an alternative to SQL Polars actually has SQL supported, so you can mix and match: e.g. construct DF from SQL.
youainti 3y ago
Data tables tend to also be a standard ingestion format for statistical tools in many cases.
_dain_ 3y ago
Dataframes and SQL have overlapping functionality, but I wouldn't say that dataframes are an "alternative" to SQL. The tradeoffs are very different. You don't have to worry about minimizing disk reads or think about concurrency issues like transactions or locks, because a dataframe is just an in-memory data structure like a list or a dict, rather than a database. Dataframes also aren't really about relational algebra like SQL is.
- theLiminator 3y ago
  Have you tried polars? I agree that if pandas is all you've tried, that it's pretty far from an alternative frontend for query engines, but if you've tried polars it maps pretty cleanly to SQL, can be query optimized to a query plan similarly to SQL, I should've made it clear that I'm talking about an alternative to SQL used in an OLAP context, not for OLTP.