8 ms·

I think Python is the biggest hidden gem in statistics. It's had a tremendous impact on machine learning and algorithm development, yet traditional statistician

I think Python is the biggest hidden gem in statistics. It's had a tremendous impact on machine learning and algorithm development, yet traditional statisticians still rely on SAS/R/Stata/MATLAB. All of these languages have libraries that produce the same results, the difficulty is mangling the data into the correct input format. Python's list comprehensions are much, much easier to use than MATLAB matrices, R's data frames, Java's ArrayLists, etc. I'd advise any new graduate student to learn how to plug data into traditional programs, but save yourself a headache and perform your data manipulation in Python. Eventually you can take the leap and do the analysis in Python as well.

moyok 10y ago
Julia is great too for this kind of work
jacobolus 10y ago
The even bigger advantage of python is everything else: web scraping, interfacing with weird APIs, hitting the database, consuming and emitting obscure formats, parsing text, calling operating system services, and a million other things. That and much much better abstractions for building larger systems out of reusable components and much better tooling for serious software engineering. R or Matlab can be fine at exploratory data analysis, making charts, running miscellaneous bits of non-programmer-grad-student code found on the internet, etc., but as soon as you want to do anything other than data analysis, they quickly become annoying.
- minimaxir 10y ago
  If you haven't looked at the R ecosystem in awhile, there is a package for each of those use cases. (scraping/API: rvest; database: dplyr; parsing text: stringr, etc). Yes, Hadley Wickham is primarily responsible for the popularity of R.
  - jacobolus 10y ago
    We’re talking about an order of magnitude difference in number of packages (82096 on PyPI vs. 8551 on CRAN) and their maturity, and such a naïve metric probably undersells the difference in variety of use cases. If you picked 100 random production python projects out of a hat, no more than a small handful of them would be remotely appropriate to build using R. And that's entirely fine. R is great at being a quick and dirty statistical analysis language for people doing data exploration.
    - yoplait_ 10y ago
      how about quick and friendly?
    - Houshalter 10y ago
      I suspect that the utility of more packages increases only logarithmically. Having 10x more packages doesn't mean it's 10 times more useful. If your obscure need isn't in the first 8,000 packages, it probably won't be in the next 80,000. That's just how power laws work. And any common task you can think of will probably be in the top 8,000.
      - hadley 10y ago
        I totally buy that argument, but there's probably not a huge overlap in the set of "common tasks" for R and python.
        baldfat 10y ago
        Pandas for Python is almost re-imaging of R (Base 0 instead of base 1 UGH) very closely and the "common task" match closely, but I still use R over Python.
        Houshalter 10y ago
        Well then all the more reason to use R, if those are the set of tasks you want to do.
  - ekianjo 10y ago
    Dplyr is not for databases at all. Its a piping operator that simplifies complex syntax a lot.
    - Malarkey73 10y ago
      Yes and no - the dplyr example code heavily used pipes but that's really a feature of Magrittr package. You can used dplyr without pipes. Plus dplyr essentially allows you to treat tables in a database as if they are data.frames.
    - minimaxir 10y ago
      Yes, you can use dplyr to connect and query data from a database. https://cran.r-project.org/web/packages/dplyr/vignettes/databases.html https://cran.r-project.org/web/packages/dplyr/vignettes/data...
  - micro_cam 10y ago
    Wickham has done a ton but but R had great popularity even before those projects, especially with projects like bioconductor.
minimaxir 10y ago
> All of these languages have libraries that produce the same results, the difficulty is mangling the data into the correct input format. Python's list comprehensions are much, much easier to use than MATLAB matrices, R's data frames, Java's ArrayLists, etc. At the end of the day, for machine learning applications, your data is in a tabular format. (in Python, a pandas data frame) Yes, Python has a few tricks like list comprehensions for speeding up data processing into that analyzable form. R has a few tricks for processing tabular data as well. (e.g. dplyr). There are tradeoffs and the skill is finding which works best. Using a single programming language is a bad philosophy even for non-statistical developers.
- collyw 10y ago
  Only learning a single language is a bad idea, but using a single one, it it fits the problem is a good one.
mziel 10y ago
I think R's data frames is by far the most intuitive abstraction. There's a reason why it was duplicated in Python, Spark or Julia. And R has a lot of utilities for converting input strings, dates, JSON etc.
- michaelsbradley 10y ago
  data.table and dplyr take data frames up a notch! https://github.com/Rdatatable/data.table/wiki/Getting-started https://github.com/Rdatatable/data.table/wiki/Getting-starte... https://cran.rstudio.com/web/packages/dplyr/vignettes/introduction.html https://cran.rstudio.com/web/packages/dplyr/vignettes/introd... xts is quite useful as well, when you need to do modeling/analysis in an especially time-oriented manner: http://joshuaulrich.github.io/xts/ http://joshuaulrich.github.io/xts/
blahi 10y ago
If I use python I have to write half of the algorithms I use myself. Worse, I have to write a gazillion helper functions nyself. It just won't do
- Myrmornis 10y ago
  I think you got downvoted because people didn't know what you meant. You're right though. Although R is laughably inferior to python as a programming language, it is vastly more work to try to do statistical data analysis in python than in R. I recommend using both languages and using csv or whatever format to exchange data sets.
  - lima 10y ago
    With pandas, this is no longer true.
    - blahi 10y ago
      How pandas solves statistical problems or the thousand utility functions that are missing? Please, explain.
      - jnbiche 10y ago
        Have you even looked at pandas? I get the impression you haven't. Pandas has most of the statistical utility functions that R does, and for those few that it lacks, Python/Pandas also has available an easy-to-use FFI interface to R via rpy2. Take a look: http://pandas.pydata.org/pandas-docs/stable/ http://pandas.pydata.org/pandas-docs/stable/ EDIT: Specified name of interface (rpy2).
        cwyers 10y ago
        > Pandas has most of the statistical utility functions that R does Pandas is not shipping with most of the stuff you can get on CRAN.
        10y ago
whyever 10y ago
R has the best plotting library (ggplot) by far.
compactmani 10y ago
Can you give some concrete examples of the benefits of wrangling in python over R and a general sense of how much time this would save a user?
- baldfat 10y ago
  I really would be interested because I started in Python and R and now use R exclusively due to manipulation. R (dplyr) really is the best manipulation system have ever used. Perhaps people try to do loops in R and other non-R like ways and give up. R is fundamentally a Functional Language and people try to band a square into a round hole and give up?
return0 10y ago
Whats the status of matpootlib nowadays? Graphs are a major major attractir/detractor
- pedrosorio 10y ago
  https://web.stanford.edu/~mwaskom/software/seaborn/ https://web.stanford.edu/~mwaskom/software/seaborn/
ldp01 10y ago
I'm glad I read this comment. After checking some of the docs I think I will have a go at Python for data wrangling. List comprehensions look... friendly. R still rules for plotting and running canned statistical procedures but sometimes I feel like if I stop programming R for a week I forget how to use it effectively... E.g. Forgetting to add stringsAsFactor=FALSE to everything, forgetting rbind() can overwrite column names, forgetting I have to define my own string concatenation operator in every script. If Python can save me some of the frustration involved in manipulating data frames that will be nice.
- hadley 10y ago
  I'd recommend reading http://r4ds.had.co.nz http://r4ds.had.co.nz for an R workflow that eliminates a lot of those pain points. (Except for infix string concatenation - I've never really understood why people prefer that to paste(). Maybe if you're not thinking in vectors?)
  - ldp01 10y ago
    A reply from the man himself! Thanks for the link. I'll have a go. I do like the look of the dplyr library a lot. Combining functions like select and group_by with the pipe operators creates code that is reminiscent of SQL- very nice for readability.
    - pedrosorio 10y ago
      You're going to love Spark if you haven't tried it yet.
      - ldp01 10y ago
        Another powerful tool I will keep in mind! I think this thread illustrates a kind of tension between those coming from an IT/big-data/web oriented background and the more traditional statistics/science/engineering side. The IT side bring a lot of very powerful and scalable tools to the table. However there are aspects of traditional work which I suspect are lost on some big-data people. For example, in my line of work (physical asset mgmt) we deal with a lot of very small datasets, very poor quality datasets (e.g. some guy's favourite spreadsheet) and also cultural issues (some engineers are inherently averse to changing systems, and spending decisions are inherently political). In this situation, there is a limit to the benefit of more powerful/scalable tools, and it is advantageous to use tools which are considered high quality and vetted by the community. R is in a good position here as it has the pedigree of being accepted by the academic stats community, as well as actually being a great tool.
daix 10y ago
I know few about other languages. But as the data size growing, I wonder to know if Python has a disadvantage on efficiency over others because it's interpreted?
micro_cam 10y ago
I'm primarily a python user and agree with most of your post but I do find myself going back to R for many of the more esoteric statistical methods. Ie if i want a specific sort of penalized regression it may not yet be implemented in python.
jsprogrammer 10y ago
I have heard of people that could not get the same results in Python as they get in R. Have you looked at R's source? Some common functions contain dozens of unsourced magic numbers (and the comments indicate they have been modified over time).
noelsusman 10y ago
List comprehension is nice, but Hadley's packages for R blow Python out of the water when it comes to data wrangling. Personally, I think anyone who works with data extensively should be familiar with both Python and R.
dbs 10y ago
I use both Python and R for analytics projects. Python falls quite short on statistics and time series, especially if one is doing exotic/advanced stuff. By the end of the day I just want to get my job done, and I select the best tool for the job. I have gained a lot by working with both languages. On a sidenote, one cannot talk about the success of R without mentioning RStudio (amazing IDE for working with data)
baldfat 10y ago
Python the second best option for just about everything. R has a wonderful ecosystem that does amazing data manipulation (dplyr for example) Python is a good choice but it isn't "clearly better."

moyok 10y ago

Julia is great too for this kind of work

jacobolus 10y ago

The even bigger advantage of python is everything else: web scraping, interfacing with weird APIs, hitting the database, consuming and emitting obscure formats, parsing text, calling operating system services, and a million other things. That and much much better abstractions for building larger systems out of reusable components and much better tooling for serious software engineering. R or Matlab can be fine at exploratory data analysis, making charts, running miscellaneous bits of non-programmer-grad-student code found on the internet, etc., but as soon as you want to do anything other than data analysis, they quickly become annoying.

minimaxir 10y ago
If you haven't looked at the R ecosystem in awhile, there is a package for each of those use cases. (scraping/API: rvest; database: dplyr; parsing text: stringr, etc). Yes, Hadley Wickham is primarily responsible for the popularity of R.
- jacobolus 10y ago
  We’re talking about an order of magnitude difference in number of packages (82096 on PyPI vs. 8551 on CRAN) and their maturity, and such a naïve metric probably undersells the difference in variety of use cases. If you picked 100 random production python projects out of a hat, no more than a small handful of them would be remotely appropriate to build using R. And that's entirely fine. R is great at being a quick and dirty statistical analysis language for people doing data exploration.
  - yoplait_ 10y ago
    how about quick and friendly?
  - Houshalter 10y ago
    I suspect that the utility of more packages increases only logarithmically. Having 10x more packages doesn't mean it's 10 times more useful. If your obscure need isn't in the first 8,000 packages, it probably won't be in the next 80,000. That's just how power laws work. And any common task you can think of will probably be in the top 8,000.
    - hadley 10y ago
      I totally buy that argument, but there's probably not a huge overlap in the set of "common tasks" for R and python.
      - baldfat 10y ago
        Pandas for Python is almost re-imaging of R (Base 0 instead of base 1 UGH) very closely and the "common task" match closely, but I still use R over Python.
      - Houshalter 10y ago
        Well then all the more reason to use R, if those are the set of tasks you want to do.
- ekianjo 10y ago
  Dplyr is not for databases at all. Its a piping operator that simplifies complex syntax a lot.
  - Malarkey73 10y ago
    Yes and no - the dplyr example code heavily used pipes but that's really a feature of Magrittr package. You can used dplyr without pipes. Plus dplyr essentially allows you to treat tables in a database as if they are data.frames.
  - minimaxir 10y ago
    Yes, you can use dplyr to connect and query data from a database. https://cran.r-project.org/web/packages/dplyr/vignettes/databases.html https://cran.r-project.org/web/packages/dplyr/vignettes/data...
- micro_cam 10y ago
  Wickham has done a ton but but R had great popularity even before those projects, especially with projects like bioconductor.

minimaxir 10y ago

> All of these languages have libraries that produce the same results, the difficulty is mangling the data into the correct input format. Python's list comprehensions are much, much easier to use than MATLAB matrices, R's data frames, Java's ArrayLists, etc. At the end of the day, for machine learning applications, your data is in a tabular format. (in Python, a pandas data frame) Yes, Python has a few tricks like list comprehensions for speeding up data processing into that analyzable form. R has a few tricks for processing tabular data as well. (e.g. dplyr). There are tradeoffs and the skill is finding which works best. Using a single programming language is a bad philosophy even for non-statistical developers.

collyw 10y ago
Only learning a single language is a bad idea, but using a single one, it it fits the problem is a good one.

mziel 10y ago

I think R's data frames is by far the most intuitive abstraction. There's a reason why it was duplicated in Python, Spark or Julia. And R has a lot of utilities for converting input strings, dates, JSON etc.

michaelsbradley 10y ago
data.table and dplyr take data frames up a notch! https://github.com/Rdatatable/data.table/wiki/Getting-started https://github.com/Rdatatable/data.table/wiki/Getting-starte... https://cran.rstudio.com/web/packages/dplyr/vignettes/introduction.html https://cran.rstudio.com/web/packages/dplyr/vignettes/introd... xts is quite useful as well, when you need to do modeling/analysis in an especially time-oriented manner: http://joshuaulrich.github.io/xts/ http://joshuaulrich.github.io/xts/

blahi 10y ago

If I use python I have to write half of the algorithms I use myself. Worse, I have to write a gazillion helper functions nyself. It just won't do

Myrmornis 10y ago
I think you got downvoted because people didn't know what you meant. You're right though. Although R is laughably inferior to python as a programming language, it is vastly more work to try to do statistical data analysis in python than in R. I recommend using both languages and using csv or whatever format to exchange data sets.
- lima 10y ago
  With pandas, this is no longer true.
  - blahi 10y ago
    How pandas solves statistical problems or the thousand utility functions that are missing? Please, explain.
    - jnbiche 10y ago
      Have you even looked at pandas? I get the impression you haven't. Pandas has most of the statistical utility functions that R does, and for those few that it lacks, Python/Pandas also has available an easy-to-use FFI interface to R via rpy2. Take a look: http://pandas.pydata.org/pandas-docs/stable/ http://pandas.pydata.org/pandas-docs/stable/ EDIT: Specified name of interface (rpy2).
      - cwyers 10y ago
        > Pandas has most of the statistical utility functions that R does Pandas is not shipping with most of the stuff you can get on CRAN.
      - 10y ago

whyever 10y ago

R has the best plotting library (ggplot) by far.

compactmani 10y ago

Can you give some concrete examples of the benefits of wrangling in python over R and a general sense of how much time this would save a user?

baldfat 10y ago
I really would be interested because I started in Python and R and now use R exclusively due to manipulation. R (dplyr) really is the best manipulation system have ever used. Perhaps people try to do loops in R and other non-R like ways and give up. R is fundamentally a Functional Language and people try to band a square into a round hole and give up?

return0 10y ago

Whats the status of matpootlib nowadays? Graphs are a major major attractir/detractor

pedrosorio 10y ago
https://web.stanford.edu/~mwaskom/software/seaborn/ https://web.stanford.edu/~mwaskom/software/seaborn/

ldp01 10y ago

I'm glad I read this comment. After checking some of the docs I think I will have a go at Python for data wrangling. List comprehensions look... friendly. R still rules for plotting and running canned statistical procedures but sometimes I feel like if I stop programming R for a week I forget how to use it effectively... E.g. Forgetting to add stringsAsFactor=FALSE to everything, forgetting rbind() can overwrite column names, forgetting I have to define my own string concatenation operator in every script. If Python can save me some of the frustration involved in manipulating data frames that will be nice.

hadley 10y ago
I'd recommend reading http://r4ds.had.co.nz http://r4ds.had.co.nz for an R workflow that eliminates a lot of those pain points. (Except for infix string concatenation - I've never really understood why people prefer that to paste(). Maybe if you're not thinking in vectors?)
- ldp01 10y ago
  A reply from the man himself! Thanks for the link. I'll have a go. I do like the look of the dplyr library a lot. Combining functions like select and group_by with the pipe operators creates code that is reminiscent of SQL- very nice for readability.
  - pedrosorio 10y ago
    You're going to love Spark if you haven't tried it yet.
    - ldp01 10y ago
      Another powerful tool I will keep in mind! I think this thread illustrates a kind of tension between those coming from an IT/big-data/web oriented background and the more traditional statistics/science/engineering side. The IT side bring a lot of very powerful and scalable tools to the table. However there are aspects of traditional work which I suspect are lost on some big-data people. For example, in my line of work (physical asset mgmt) we deal with a lot of very small datasets, very poor quality datasets (e.g. some guy's favourite spreadsheet) and also cultural issues (some engineers are inherently averse to changing systems, and spending decisions are inherently political). In this situation, there is a limit to the benefit of more powerful/scalable tools, and it is advantageous to use tools which are considered high quality and vetted by the community. R is in a good position here as it has the pedigree of being accepted by the academic stats community, as well as actually being a great tool.

daix 10y ago

I know few about other languages. But as the data size growing, I wonder to know if Python has a disadvantage on efficiency over others because it's interpreted?

micro_cam 10y ago

I'm primarily a python user and agree with most of your post but I do find myself going back to R for many of the more esoteric statistical methods. Ie if i want a specific sort of penalized regression it may not yet be implemented in python.

jsprogrammer 10y ago

I have heard of people that could not get the same results in Python as they get in R. Have you looked at R's source? Some common functions contain dozens of unsourced magic numbers (and the comments indicate they have been modified over time).

noelsusman 10y ago

List comprehension is nice, but Hadley's packages for R blow Python out of the water when it comes to data wrangling. Personally, I think anyone who works with data extensively should be familiar with both Python and R.

dbs 10y ago

I use both Python and R for analytics projects. Python falls quite short on statistics and time series, especially if one is doing exotic/advanced stuff. By the end of the day I just want to get my job done, and I select the best tool for the job. I have gained a lot by working with both languages. On a sidenote, one cannot talk about the success of R without mentioning RStudio (amazing IDE for working with data)

baldfat 10y ago

Python the second best option for just about everything. R has a wonderful ecosystem that does amazing data manipulation (dplyr for example) Python is a good choice but it isn't "clearly better."