4 ms·

Actually pandas is not the original open source data frame library, perhaps only in Python. There is a very rich tradition in R on data.frames, which includes t

by dkga 3y ago

Actually pandas is not the original open source data frame library, perhaps only in Python. There is a very rich tradition in R on data.frames, which includes the unjustly neglected data.table.

7thaccount 3y ago
Yeah. I think Wes McKinney liked the data frames in R, but preferred the programming language of Python. I've heard somewhere that he also got a lot of inspiration from APL.
- Cacti 3y ago
  R is literally designed to do statistics and has first class support and language feature support for many specialized tasks in statistics and closely related fields. Python is literally designed to be easy to program with in general. Well, it turns out when you’re dealing with terabytes of data and TFLOPS, the programming becomes more important than the math. Not all R devs are happy about this and they are very loud about it. But it shouldn’t really surprise anyone. That is literally how those languages are designed. Most of the R devs I know like this are just butthurt they are paid less and refuse to switch because they’re obstinate, or they’re a little scared they’re being left behind. first group is all over the place, but the second group tends to skew older of course
  - disgruntledphd2 3y ago
    > Most of the R devs I know like this are just butthurt they are paid less and refuse to switch because they’re obstinate, or they’re a little scared they’re being left behind. first group is all over the place, but the second group tends to skew older of course Look, I started with R and use mostly Python these days, but this is not really a fair take. R is (still) much, much, much better for analytics and graphing (the only decent plotting library in python is a ggplot clone). The big change (and why Python ended up winning) is that integrating R with other tools (like web stuff, for example) is harder than just using Python. pandas (for instance) is like an unholy clone of the worst features from both R and Python. Polars is pretty rocking, though (mostly because it clones from Spark/dplyr/linc). It's another example of Python being the second best language for everything winning out in the marketplace. That being said, if I was starting a data focused company and needed to pick a language, I'd almost certainly build all the DS focused stuff in R as it would be many many times quicker, as long as I didn't need to hire too many people.
  - mjhay 3y ago
    R is heavily influenced by Scheme. Not only is it heavily functional, but it has metaprogramming capabilities allowing a high level of flexibility and expressiveness. The tidyverse libraries use this heavily to produce very nice composable APIs that aren't really practically possible in Python. R is fine. The issue is more in the ecosystem (with the aforementioned exception of the tidyverse).
    - RamblingCTO 3y ago
      Imho forecasting is still way better in R than in Python
p4ul 3y ago
Yep! Unless I'm mistaken, R (and its predecessor S) seems to have been the first to introduce the concept of a dataframe. One could also argue that dataframes are basically in-memory database tables. And in that case, S and SQL probably tie in terms of the creation timeline.
- ayhanfuat 3y ago
  The difference is dataframes can also be seen like matrices. You can do row operations, row + column operations, multiply rows and columns, multiply different matrices, transpose them etc. These kind of things don't really make sense in DB tables (and they are generally not supported and you jump through hoops to do similar things in DBs).
  - p4ul 3y ago
    Yes, that's totally fair; dataframes are more flexible in that sense.
    - p4ul 3y ago
      Oh, and another important difference is memory layout. The dataframe implementations mostly (or all) use column-major format. Whereas most conventional SQL implementations use row-major format, I believe.
      - theLiminator 3y ago
        I think most OLTP databases are row oriented whilst most OLAP are column.
  - kkoncevicius 3y ago
    > The difference is dataframes can also be seen like matrices. You can do row operations, row + column operations, multiply rows and columns, multiply different matrices, transpose them etc. I think this is overblowing the similarities to matrices. Matrices have elements all of the same type, while data.frames mix numbers, characters, factors, etc. You certainly cannot transpose a data.frame and still have a data.frame that makes sense. Multiplying rows would not make sense either, since within one row you will have different types of data. Unless you have a data.frame that is all numeric, but in that case one should probably be using a matrix in the first place.
Cacti 3y ago
My friend. You cannot make people like R. We all know about and study data.table, so it’s not neglected, we just don’t use that implementation. Mainly because R sucks for anything that isn’t statistics.
bonadrag 3y ago
> which includes the unjustly neglected data.table So so true. I was working on an adhoc project that needed a quick result by the end of the day. I had to pull this series of parquet files and do some quick and dirty analysis. My first reflex was to use python with pandas, quick and easy. Python could not handle the datasets, too large. I decided to give R and data.table a go and it went smoothly. I am usually a python user but from time to time I feel compelled to jump back to R and data.table. Phenomenal tool.
- jgalt212 3y ago
  Indeed. Python is great for getting things into a data.frame. But once in a data.frame, R is so much easier to work with for all things stats and ml-related.

by dkga 3y ago

Actually pandas is not the original open source data frame library, perhaps only in Python. There is a very rich tradition in R on data.frames, which includes the unjustly neglected data.table.

7thaccount 3y ago

Yeah. I think Wes McKinney liked the data frames in R, but preferred the programming language of Python. I've heard somewhere that he also got a lot of inspiration from APL.

Cacti 3y ago
R is literally designed to do statistics and has first class support and language feature support for many specialized tasks in statistics and closely related fields. Python is literally designed to be easy to program with in general. Well, it turns out when you’re dealing with terabytes of data and TFLOPS, the programming becomes more important than the math. Not all R devs are happy about this and they are very loud about it. But it shouldn’t really surprise anyone. That is literally how those languages are designed. Most of the R devs I know like this are just butthurt they are paid less and refuse to switch because they’re obstinate, or they’re a little scared they’re being left behind. first group is all over the place, but the second group tends to skew older of course
- disgruntledphd2 3y ago
  > Most of the R devs I know like this are just butthurt they are paid less and refuse to switch because they’re obstinate, or they’re a little scared they’re being left behind. first group is all over the place, but the second group tends to skew older of course Look, I started with R and use mostly Python these days, but this is not really a fair take. R is (still) much, much, much better for analytics and graphing (the only decent plotting library in python is a ggplot clone). The big change (and why Python ended up winning) is that integrating R with other tools (like web stuff, for example) is harder than just using Python. pandas (for instance) is like an unholy clone of the worst features from both R and Python. Polars is pretty rocking, though (mostly because it clones from Spark/dplyr/linc). It's another example of Python being the second best language for everything winning out in the marketplace. That being said, if I was starting a data focused company and needed to pick a language, I'd almost certainly build all the DS focused stuff in R as it would be many many times quicker, as long as I didn't need to hire too many people.
- mjhay 3y ago
  R is heavily influenced by Scheme. Not only is it heavily functional, but it has metaprogramming capabilities allowing a high level of flexibility and expressiveness. The tidyverse libraries use this heavily to produce very nice composable APIs that aren't really practically possible in Python. R is fine. The issue is more in the ecosystem (with the aforementioned exception of the tidyverse).
  - RamblingCTO 3y ago
    Imho forecasting is still way better in R than in Python

p4ul 3y ago

Yep! Unless I'm mistaken, R (and its predecessor S) seems to have been the first to introduce the concept of a dataframe. One could also argue that dataframes are basically in-memory database tables. And in that case, S and SQL probably tie in terms of the creation timeline.

ayhanfuat 3y ago
The difference is dataframes can also be seen like matrices. You can do row operations, row + column operations, multiply rows and columns, multiply different matrices, transpose them etc. These kind of things don't really make sense in DB tables (and they are generally not supported and you jump through hoops to do similar things in DBs).
- p4ul 3y ago
  Yes, that's totally fair; dataframes are more flexible in that sense.
  - p4ul 3y ago
    Oh, and another important difference is memory layout. The dataframe implementations mostly (or all) use column-major format. Whereas most conventional SQL implementations use row-major format, I believe.
    - theLiminator 3y ago
      I think most OLTP databases are row oriented whilst most OLAP are column.
- kkoncevicius 3y ago
  > The difference is dataframes can also be seen like matrices. You can do row operations, row + column operations, multiply rows and columns, multiply different matrices, transpose them etc. I think this is overblowing the similarities to matrices. Matrices have elements all of the same type, while data.frames mix numbers, characters, factors, etc. You certainly cannot transpose a data.frame and still have a data.frame that makes sense. Multiplying rows would not make sense either, since within one row you will have different types of data. Unless you have a data.frame that is all numeric, but in that case one should probably be using a matrix in the first place.

Cacti 3y ago

My friend. You cannot make people like R. We all know about and study data.table, so it’s not neglected, we just don’t use that implementation. Mainly because R sucks for anything that isn’t statistics.

bonadrag 3y ago

> which includes the unjustly neglected data.table So so true. I was working on an adhoc project that needed a quick result by the end of the day. I had to pull this series of parquet files and do some quick and dirty analysis. My first reflex was to use python with pandas, quick and easy. Python could not handle the datasets, too large. I decided to give R and data.table a go and it went smoothly. I am usually a python user but from time to time I feel compelled to jump back to R and data.table. Phenomenal tool.

jgalt212 3y ago
Indeed. Python is great for getting things into a data.frame. But once in a data.frame, R is so much easier to work with for all things stats and ml-related.