4 ms·

If I use python I have to write half of the algorithms I use myself. Worse, I have to write a gazillion helper functions nyself. It just won't do

by blahi 10y ago

Myrmornis 10y ago
I think you got downvoted because people didn't know what you meant. You're right though. Although R is laughably inferior to python as a programming language, it is vastly more work to try to do statistical data analysis in python than in R. I recommend using both languages and using csv or whatever format to exchange data sets.
- lima 10y ago
  With pandas, this is no longer true.
  - blahi 10y ago
    How pandas solves statistical problems or the thousand utility functions that are missing? Please, explain.
    - jnbiche 10y ago
      Have you even looked at pandas? I get the impression you haven't. Pandas has most of the statistical utility functions that R does, and for those few that it lacks, Python/Pandas also has available an easy-to-use FFI interface to R via rpy2. Take a look: http://pandas.pydata.org/pandas-docs/stable/ http://pandas.pydata.org/pandas-docs/stable/ EDIT: Specified name of interface (rpy2).
      - cwyers 10y ago
        > Pandas has most of the statistical utility functions that R does Pandas is not shipping with most of the stuff you can get on CRAN.
      - blahi 10y ago
        Have you even looked at statistics? I get the impression you haven't. Pandas has aggregation and that's it. I don't really like to start those language wars, but what the heck. Your precious pandas is actually worse than the R alternatives in every way. It is slower than data.table while having comparable syntax. Dplyr is comparable in speed while having WAY better syntax.
- jnbiche 10y ago
  > using csv or whatever format to exchange data sets Have you used Pandas in the past year or two? I'm curious why you would exchange data sets using csv or another format between Python and R when you could easily call the R function of interest from within Pandas (using rpy2) and not even worry about data interchange. It's definitely not vastly more work to do statistical analysis using Python Pandas than in R anymore, perhaps it was several years ago. EDIT: And carlmcqueen mentioned feathers in response, which is a collaboration between the developers of Pandas and R to create interoperable on-disk data frames for both languages. Point being, between rpy2 and feathers (and probably other projects), you definitely don't need to use intermediary csv files anymore to move data back and forth between R and Python.
  - Myrmornis 10y ago
    > It's definitely not vastly more work to do statistical analysis using Python Pandas than in R anymore, perhaps it was several years ago. Haha, what? What statistics can you do in pandas? You can do some statistics in python by cobbling together stuff from scipy and statsmodels (maybe I'm out of date, is there more?). I see a few modules for regression and stuff in pandas but they are marked as deprecated. I think perhaps you and I mean different things by "statistics". R provides a vast ecosystem covering, for example - Gold standard implementations of simulation, PDFs, quantiles of any probability distribution you can mention (in python you can find some of this in scipy; not pandas. But scipy is a real mess compared to R and not as comprehensive.) - Gold standard implementations of any classical hypothesis test you can mention - Gold standard implementations of computational methods for fitting generalized linear models, mixed models, frameworks for MCMC samplers, graphical models, HMMs, and a vast amount of other stuff I'm not clever enough to name right now let alone understand. Really any statistical procedure -- whether "classical" or "modern"/"computational statistics" -- in R you will find it, and furthermore it will be basically the reference implementation / gold standard. That's not mentioning the plotting tools and the numerical computing and clean linear algebra syntax. But that's it, no more: the people who go further and suggest using R for building a web server or web scraping or something mostly haven't used real programming languages. You're missing the point. The python ecosystem can't compete with R on the statistics front -- it would be crazy to try. That's certainly not the aim of pandas. > you definitely don't need to use intermediary csv files anymore to move data back and forth between R and Python. Perhaps not, but doesn't it please you to have a well-defined interface (a serialization format) between the two languages? I haven't tried Rpy2 for years. I don't like to have two different languages get their tentacles into each other like that if I can avoid it, but I'm sure it's a good project which has its use cases. EDIT: thanks, I hadn't seen feather. That looks like the thing to use.
- carlmcqueen 10y ago
  feather is a great way to move data between the two as it is blazingly fast and native in python and R.

If I use python I have to write half of the algorithms I use myself. Worse, I have to write a gazillion helper functions nyself. It just won't do

by blahi 10y ago

If I use python I have to write half of the algorithms I use myself. Worse, I have to write a gazillion helper functions nyself. It just won't do

Myrmornis 10y ago

I think you got downvoted because people didn't know what you meant. You're right though. Although R is laughably inferior to python as a programming language, it is vastly more work to try to do statistical data analysis in python than in R. I recommend using both languages and using csv or whatever format to exchange data sets.

lima 10y ago
With pandas, this is no longer true.
- blahi 10y ago
  How pandas solves statistical problems or the thousand utility functions that are missing? Please, explain.
  - jnbiche 10y ago
    Have you even looked at pandas? I get the impression you haven't. Pandas has most of the statistical utility functions that R does, and for those few that it lacks, Python/Pandas also has available an easy-to-use FFI interface to R via rpy2. Take a look: http://pandas.pydata.org/pandas-docs/stable/ http://pandas.pydata.org/pandas-docs/stable/ EDIT: Specified name of interface (rpy2).
    - cwyers 10y ago
      > Pandas has most of the statistical utility functions that R does Pandas is not shipping with most of the stuff you can get on CRAN.
    - blahi 10y ago
      Have you even looked at statistics? I get the impression you haven't. Pandas has aggregation and that's it. I don't really like to start those language wars, but what the heck. Your precious pandas is actually worse than the R alternatives in every way. It is slower than data.table while having comparable syntax. Dplyr is comparable in speed while having WAY better syntax.
jnbiche 10y ago
> using csv or whatever format to exchange data sets Have you used Pandas in the past year or two? I'm curious why you would exchange data sets using csv or another format between Python and R when you could easily call the R function of interest from within Pandas (using rpy2) and not even worry about data interchange. It's definitely not vastly more work to do statistical analysis using Python Pandas than in R anymore, perhaps it was several years ago. EDIT: And carlmcqueen mentioned feathers in response, which is a collaboration between the developers of Pandas and R to create interoperable on-disk data frames for both languages. Point being, between rpy2 and feathers (and probably other projects), you definitely don't need to use intermediary csv files anymore to move data back and forth between R and Python.
- Myrmornis 10y ago
  > It's definitely not vastly more work to do statistical analysis using Python Pandas than in R anymore, perhaps it was several years ago. Haha, what? What statistics can you do in pandas? You can do some statistics in python by cobbling together stuff from scipy and statsmodels (maybe I'm out of date, is there more?). I see a few modules for regression and stuff in pandas but they are marked as deprecated. I think perhaps you and I mean different things by "statistics". R provides a vast ecosystem covering, for example - Gold standard implementations of simulation, PDFs, quantiles of any probability distribution you can mention (in python you can find some of this in scipy; not pandas. But scipy is a real mess compared to R and not as comprehensive.) - Gold standard implementations of any classical hypothesis test you can mention - Gold standard implementations of computational methods for fitting generalized linear models, mixed models, frameworks for MCMC samplers, graphical models, HMMs, and a vast amount of other stuff I'm not clever enough to name right now let alone understand. Really any statistical procedure -- whether "classical" or "modern"/"computational statistics" -- in R you will find it, and furthermore it will be basically the reference implementation / gold standard. That's not mentioning the plotting tools and the numerical computing and clean linear algebra syntax. But that's it, no more: the people who go further and suggest using R for building a web server or web scraping or something mostly haven't used real programming languages. You're missing the point. The python ecosystem can't compete with R on the statistics front -- it would be crazy to try. That's certainly not the aim of pandas. > you definitely don't need to use intermediary csv files anymore to move data back and forth between R and Python. Perhaps not, but doesn't it please you to have a well-defined interface (a serialization format) between the two languages? I haven't tried Rpy2 for years. I don't like to have two different languages get their tentacles into each other like that if I can avoid it, but I'm sure it's a good project which has its use cases. EDIT: thanks, I hadn't seen feather. That looks like the thing to use.
carlmcqueen 10y ago
feather is a great way to move data between the two as it is blazingly fast and native in python and R.