6 ms·

> So, what is Polars? A short description would be “a query engine with a DataFrame frontend”. Guess that would help if I already knew what a dataframe is!

dist-epoch 3y ago
If you don't know what a dataframe is, it doesn't matter what Polars is then, not for you.
- appplication 3y ago
  Eh, plenty of SQL-first analysts could find it helpful. Not just for people coming from pandas/r.
- 369548684892826 3y ago
  With that attitude I'd never learn anything new
  - lopatin 3y ago
    Go learn about Dataframes then. There's no reason an article like this should start with a Dataframes 101.
  - amanzi 3y ago
    Would be quicker for you to just search for the word "dataframe" than to post on Hacker News that you don't know what a dataframe is.
    - stavros 3y ago
      It's useful feedback for whoever wrote the article, that there are terms they need to not take for granted.
      - vacuity 3y ago
        But articles shouldn't have to teach readers all the background needed. Otherwise even a well written article would get a lot of bloat. Even linking other guides would require some context, and the total breadth of knowledge could be quite high. I could agree if there was to be a brief "areas you should know about" with a few keywords. There are some areas where Hacker News might uniquely shine in answering questions, but "what are dataframes" isn't one of them. If someone sees "dataframe" and is confused, they have all the opportunity to search up "dataframes 101" and get up to speed.
- vundercind 3y ago
  “Fancy array-of-associative-arrays with their own set of non-Python-native field data types and a lot of helper methods for sorting and import and export and such, but weirdly always missing any helpers that would be both hard to write and very useful” “SQLite but only for Python and worse. Kind of.” “That annoying transitional step you have no direct need for but that you have to do anyway, because all data-wrangling tools in Python assume a dataframe” (I use this stuff daily) [edit] “Imagine if you could read in from a database or CSV and then work with the returned rows directly and as if they were still a table/spreadsheet. Except by ‘directly’ I mean ‘after turning it into a dataframe and discarding/fucking-up all your data type information’. And then with another fucking-everything-up step if you want to write it out so you can do real stuff with it elsewhere.”
  - dajt 3y ago
    LOL. I have been programming for over 30 years on all sorts of systems and the Pandas DataFrame API is completely beyond me. Just trying to get a cell value seems way more difficult than it should be. Same with xarray datasets. I just loaded the same CSV into Pandas and Polars and Polars did a much better job of it.
    - tomrod 3y ago
      What part of df.loc[index_val, column_val] is hard here? Oh you meant multiindex... yeah, slicing multiindexes sucks :)
      - ok_computer 3y ago
        Multi indexes is baking my data into the horrible pandas object model into weird tuples. Every gripe I’ve had with pandas starts with trying to do something “simple” then following the pandas object model to achieve it and over complicating things. Polars is awesome, it fits my numpy understanding of dataframes as dict labeled arrays. I even like the immutability aspect.
    - vundercind 3y ago
      Polars is at least more consistent & sensible. The whole thing though is just shockingly unhelpful and half-baked for something that taken over so completely in its niche. I guess my perspective is that of someone trying to build reliable automation, though—it’s probably really nice if you’re just noodlin’ in notebooks or the repl or whatever.
- riku_iki 3y ago
  Polars allows to use SQL I think?
- goku12 3y ago
  Dataframe is a potentially useful concept to even those who haven't heard about it. And Polars can be a reasonable introduction to it too.
- dylan604 3y ago
  But it is such a shame that us plebes can't learn about it to maybe find the hammer to this nail that is sticking up in our projects because the product can't tell people what it is. And it's only made worse by arrogant responses like yours.
jbverschoor 3y ago
Python framework for tables (arrays with column names). You can query/filter/sort and pick /transform data on multiple directions (row/column) It’s used for data mining, ML etc. It’s basically like working with a spreadsheet/sql table
- goku12 3y ago
  Note that dataframes aren't Python specific. R has it too, and is an inspiration for Pandas.
softwaredoug 3y ago
Logically its a SQL table or spreadsheet. Where each column can be of different types. A column of usernames (strings), their age (integers) etc. Every row is a distinct entity. Every column is usually stored / treated as its own distinct array.
- euroderf 3y ago
  So it's a sort of an embedded relational database table, but live, and maybe with some bells and whistles ? Sounds like a triumph of marketing. What am I missing ?
esafak 3y ago
It is a data structure used by data scientists. Polars is API compatible with the previous solution, but faster. If you don't analyze data, you can ignore it.
green0wl 3y ago
An earlier thread on Polars had this as the #1 thread (so the same complaint) - https://news.ycombinator.com/item?id=38920043 https://news.ycombinator.com/item?id=38920043
tomrod 3y ago
Wide-form data structure, often with an index or partition framework in place able to apply column- and row-transformations quickly
__MatrixMan__ 3y ago
Another bit of context: it's "polars" (like polar bears) because the popular python dataframe library is "pandas". My first assumption was that it was somehow related to polar coordinates.

dist-epoch 3y ago

If you don't know what a dataframe is, it doesn't matter what Polars is then, not for you.

appplication 3y ago
Eh, plenty of SQL-first analysts could find it helpful. Not just for people coming from pandas/r.
369548684892826 3y ago
With that attitude I'd never learn anything new
- lopatin 3y ago
  Go learn about Dataframes then. There's no reason an article like this should start with a Dataframes 101.
- amanzi 3y ago
  Would be quicker for you to just search for the word "dataframe" than to post on Hacker News that you don't know what a dataframe is.
  - stavros 3y ago
    It's useful feedback for whoever wrote the article, that there are terms they need to not take for granted.
    - vacuity 3y ago
      But articles shouldn't have to teach readers all the background needed. Otherwise even a well written article would get a lot of bloat. Even linking other guides would require some context, and the total breadth of knowledge could be quite high. I could agree if there was to be a brief "areas you should know about" with a few keywords. There are some areas where Hacker News might uniquely shine in answering questions, but "what are dataframes" isn't one of them. If someone sees "dataframe" and is confused, they have all the opportunity to search up "dataframes 101" and get up to speed.
vundercind 3y ago
“Fancy array-of-associative-arrays with their own set of non-Python-native field data types and a lot of helper methods for sorting and import and export and such, but weirdly always missing any helpers that would be both hard to write and very useful” “SQLite but only for Python and worse. Kind of.” “That annoying transitional step you have no direct need for but that you have to do anyway, because all data-wrangling tools in Python assume a dataframe” (I use this stuff daily) [edit] “Imagine if you could read in from a database or CSV and then work with the returned rows directly and as if they were still a table/spreadsheet. Except by ‘directly’ I mean ‘after turning it into a dataframe and discarding/fucking-up all your data type information’. And then with another fucking-everything-up step if you want to write it out so you can do real stuff with it elsewhere.”
- dajt 3y ago
  LOL. I have been programming for over 30 years on all sorts of systems and the Pandas DataFrame API is completely beyond me. Just trying to get a cell value seems way more difficult than it should be. Same with xarray datasets. I just loaded the same CSV into Pandas and Polars and Polars did a much better job of it.
  - tomrod 3y ago
    What part of df.loc[index_val, column_val] is hard here? Oh you meant multiindex... yeah, slicing multiindexes sucks :)
    - ok_computer 3y ago
      Multi indexes is baking my data into the horrible pandas object model into weird tuples. Every gripe I’ve had with pandas starts with trying to do something “simple” then following the pandas object model to achieve it and over complicating things. Polars is awesome, it fits my numpy understanding of dataframes as dict labeled arrays. I even like the immutability aspect.
  - vundercind 3y ago
    Polars is at least more consistent & sensible. The whole thing though is just shockingly unhelpful and half-baked for something that taken over so completely in its niche. I guess my perspective is that of someone trying to build reliable automation, though—it’s probably really nice if you’re just noodlin’ in notebooks or the repl or whatever.
riku_iki 3y ago
Polars allows to use SQL I think?
goku12 3y ago
Dataframe is a potentially useful concept to even those who haven't heard about it. And Polars can be a reasonable introduction to it too.
dylan604 3y ago
But it is such a shame that us plebes can't learn about it to maybe find the hammer to this nail that is sticking up in our projects because the product can't tell people what it is. And it's only made worse by arrogant responses like yours.

jbverschoor 3y ago

Python framework for tables (arrays with column names). You can query/filter/sort and pick /transform data on multiple directions (row/column) It’s used for data mining, ML etc. It’s basically like working with a spreadsheet/sql table

goku12 3y ago
Note that dataframes aren't Python specific. R has it too, and is an inspiration for Pandas.

softwaredoug 3y ago

Logically its a SQL table or spreadsheet. Where each column can be of different types. A column of usernames (strings), their age (integers) etc. Every row is a distinct entity. Every column is usually stored / treated as its own distinct array.

euroderf 3y ago
So it's a sort of an embedded relational database table, but live, and maybe with some bells and whistles ? Sounds like a triumph of marketing. What am I missing ?

esafak 3y ago

It is a data structure used by data scientists. Polars is API compatible with the previous solution, but faster. If you don't analyze data, you can ignore it.

green0wl 3y ago

An earlier thread on Polars had this as the #1 thread (so the same complaint) - https://news.ycombinator.com/item?id=38920043 https://news.ycombinator.com/item?id=38920043

tomrod 3y ago

Wide-form data structure, often with an index or partition framework in place able to apply column- and row-transformations quickly

__MatrixMan__ 3y ago

Another bit of context: it's "polars" (like polar bears) because the popular python dataframe library is "pandas". My first assumption was that it was somehow related to polar coordinates.