6 ms·

I've been using R professionally for 4 years. I've been a Python programmer for 6 years. I've shipped large scale applications built on both languages to produc

I've been using R professionally for 4 years. I've been a Python programmer for 6 years. I've shipped large scale applications built on both languages to production. Findings: I honestly don't get the hype around R's plotting capabilities. ggplot2 is nice, but far too magical to be easily understood by beginners. Python offers more intuitive memory management than R (which is really more a statement on how bad R is at memory efficiency). R blows Python out of the water when it comes to expressing concise and readable linear algebra/stats computation. Done right, base R looks like mathematical pseudo-code. R's non-standard evaluation is an under-appreciated killer feature. I've seen complex and powerful DSLs implemented implemented in base R. Python has some support for this kind of thing, but nowhere near the level of R. I tried the Tidyverse and found it to be too magical and unstable. In general, the R package ecosystem is weird. Look at Stack Overflow and you'll find people touting all sorts of different packages to do simple operations. Needing to download a million random packages from grad students of the internet is a recipe for disaster.

prionassembly 6y ago
Do you have any insights on Julia?
- throwaway29103 6y ago
  I am not the person you are asking, but having used octave, matlab, R, Python for many years and julia for about a month I think that for numerical/algebraic stuff, julia is the one to go; the only reason holding me back is that communities using octave-matlab-R need to be convinced and their codebases need to be ported.
throwaway29103 6y ago
Genuinely curious: what do you mean with respect to "R's non-standard evaluation" and its relationship to DSLs? Thanks in advance.
- CreRecombinase 6y ago
  R some very lispy meta-programming features. Very briefly, if I'm a function (go with me here) `foo <- function(x,y){whatever}`,and someone calls me, `foo(bar,baz)` I get to know that `baz` is "bound" to `y`. The caller doesn't even have to have a `baz` in scope; substituting `y` for `baz` is done lazily, if the function never asks for the actual value of `y`, there's no problem. You can read more about this here http://adv-r.had.co.nz/Computing-on-the-language.html http://adv-r.had.co.nz/Computing-on-the-language.html
  - throwaway29103 6y ago
    Great
- nerdponx 6y ago
  R gives you the ability to capture un-evaluated expressions, manipulate them, and then evaluate the manipulated versions. This ability lets you construct some fairly powerful DSLs and language extensions.
  - throwaway29103 6y ago
    Thanks for the succinct explanation
- tmoertel 6y ago
  I wrote a summary that goes into some detail: http://blog.moertel.com/posts/2006-01-20-wondrous-oddities-rs-function-call-semantics.html http://blog.moertel.com/posts/2006-01-20-wondrous-oddities-r...
  - throwaway29103 6y ago
    Fantastic
melling 6y ago
How about Julia? I’m watching that MIT class with Grant Sanderson that uses Julia. It seems like a reasonable language, and it’s fast. Of course, I reserve the right to change my opinion after I build a few things. Dynamic typing... hmmm...
ryndbfsrw 6y ago
> Python offers more intuitive memory management than R (which is really more a statement on how bad R is at memory efficiency). I haven't found that to be the case. Loading a 2gb CSV file when you have 8gb of ram is touch and go with pandas but with data.table it’s a breeze not to mention operations once loaded up are a fair-bit faster. The pythong version has recently been released and already provides a serious speed bump over pandas not to mention out the box memory-mapping for huge files. I recommend checking it out if you haven't heard of it https://h2oai.github.io/db-benchmark/ https://h2oai.github.io/db-benchmark/
jwilber 6y ago
Same and I have to disagree with basically everything you’ve said, save the DSL bit and the R package bit. In particular, I don’t see how the tidyverse is at all ‘magical’ - take the two most popular tidy libs, dplyr and ggplot2. The api for dplyr is very explicit, intuitive, and based on long-standing precedence (sql). The api for ggplot2 is admittedly less intuitive, but is itself an implementation of widely known framework (Grammar of Graphics). If by magical you mean in their abstractions, those are about as far from magical as one can get. The tidyverse does get iffy sometimes when you need to dig into the rlang/tidy eval area, but that’s all well-documented. I haven’t had to use R in a production environment for about a year now, but I always enjoyed the concise tidy api. Base R is a total mess and largely inconsistent, but I guess that’s what you get when statisticians from different uni’s patchwork a language in their spare time.
- civilized 6y ago
  The dplyr team is genuinely working hard to make the tabular data manipulation tool. IIRC versions 0.5, 0.7, and 1.0 all brought major enhancements. The basic structure has remained consistent but the advanced API keeps getting refactored, usually for the better I think. (I know some people swear by data.table, but I find it gets ugly fast if you want to do anything more than simple joins or split-apply-combine)
  - Tarq0n 6y ago
    Data.table's join syntax is literal insanity to me. That said I used dtplyr to generate data.table code recently and it worked well, except for one edge case.
- vharuck 6y ago
  >The tidyverse does get iffy sometimes when you need to dig into the rlang/tidy eval area, but that’s all well-documented. Non-standard evaluation, in my opinion, does fine in the top layer (analysis scripts, DSLs). But it's a pain to build on top of magic without a more concrete layer between. Base R uses character vectors and name attributes to hide the magic. The Tidyverse uses symbols. I find character vectors much easier to understand and manipulate. Formulas are the only scenarios in base R where symbol and expression manipulation is sometimes necessary. And I don't do that thinking, "Boy, I wish this was how I did 95% of my work." Personally, I use data.table. It's been great even for packages. If it's an internal package dataset, I don't need to manipulate symbols. And if it's a user-supplied argument, there's usually a simple and fast way to do it with a character vector. Or I define an S3 class, give a helper function that produces a dataset with known names, and just write functions around that. The user can handle combining data however they want. >Base R is a total mess and largely inconsistent I wouldn't go so far as to say "total mess." Not even an annoyingly big mess. The few functions that make up most of analysis or package code are dead simple. Inconsistent argument names is hardly a problem with an IDE.
clircle 6y ago
> Needing to download a million random packages from grad students of the internet is a recipe for disaster. This is baloney. You don't need to rely on a million R packages. You really only need a good plotting library (the built-in one is fine, or use ggplot), a good data frame library (dplyr, data.table, or use build-in one), and a few extras to handle some weird rough edges (lubridate, forcats). This takes you to 90% to your analysis goals. Only novices are 'using a million random packages', which is probably the case for python as well.
- antipaul 6y ago
  Indeed. `library(tidyverse)` and I’m set for like 70-80% of use cases for data manipulation, exploratory data analysis, and statistics. Though to be complete, any ML project means python and scikit learn.
- phillc73 6y ago
  I also believe this is now being addressed by the pharmar[1] group with their riskmetric[2] package. [1] https://www.pharmar.org/ https://www.pharmar.org/ [2] https://github.com/pharmaR/riskmetric https://github.com/pharmaR/riskmetric
throwaway4007 6y ago
But I want ggplot to do magic. Rendering things is extremely tedious and boring. I don't want to spend hours on stackoverflow searching for the special obscure incantations that will shift the legend and margins this way because matplotlib did an ugly job while constantly adapting boilerplate back and forth between the declarative and object-oriented API, I want it to just do what I mean! Is there any coincidence that there are literally tens of different (and concurrent) rendering libraries for Python while the R world more or less settled for ggplot2? Writing matplotlib is such a pain in the ass that people go to great lengths to actually avoid using matplotlib (while still not having the expressivity, features and ease of use of ggplot2). Magic is bad when you're shipping production code that's shared among multiple people who have to then spend a lot of time assimilating the mental model of the magic. It's perfect when you just want to plot stuff and draw nice figures.
- phillc73 6y ago
  Base R plot() can also be pretty powerful[1] and look great.[2] As an alternative to ggplot2 I would also propose that Vega Lite[3] could be a contender, with an excellent cross language ecosystem. [1] http://karolis.koncevicius.lt/posts/r_base_plotting_without_wrappers/ http://karolis.koncevicius.lt/posts/r_base_plotting_without_... [2] https://github.com/KKPMW/basetheme https://github.com/KKPMW/basetheme [3] https://vegawidget.github.io/vlbuildr/ https://vegawidget.github.io/vlbuildr/
- deleted 6y ago
  [deleted]
abecode 6y ago
The Tidyverse is under development and it can be tough to keep up with the changes or know where to start (e.g., plyr vs dplyr, melt vs pivot_longer). However, ggplot2 is based on a pretty solid theory of graphics (Bertin's visual variables and Wilkinson's grammar of graphics). These can take some time to get into but it's not magic once you get the idea that you're mapping data variables to visual variables.
saeranv 6y ago
> R blows Python out of the water when it comes to expressing concise and readable linear algebra/stats computation. Done right, base R looks like mathematical pseudo-code. Are you including Numpy as part of Python in this statement? I've found Numpy to be as mathematically expressive as R, although I'm not a R power-user.

prionassembly 6y ago

Do you have any insights on Julia?

throwaway29103 6y ago
I am not the person you are asking, but having used octave, matlab, R, Python for many years and julia for about a month I think that for numerical/algebraic stuff, julia is the one to go; the only reason holding me back is that communities using octave-matlab-R need to be convinced and their codebases need to be ported.

throwaway29103 6y ago

Genuinely curious: what do you mean with respect to "R's non-standard evaluation" and its relationship to DSLs? Thanks in advance.

CreRecombinase 6y ago
R some very lispy meta-programming features. Very briefly, if I'm a function (go with me here) `foo <- function(x,y){whatever}`,and someone calls me, `foo(bar,baz)` I get to know that `baz` is "bound" to `y`. The caller doesn't even have to have a `baz` in scope; substituting `y` for `baz` is done lazily, if the function never asks for the actual value of `y`, there's no problem. You can read more about this here http://adv-r.had.co.nz/Computing-on-the-language.html http://adv-r.had.co.nz/Computing-on-the-language.html
- throwaway29103 6y ago
  Great
nerdponx 6y ago
R gives you the ability to capture un-evaluated expressions, manipulate them, and then evaluate the manipulated versions. This ability lets you construct some fairly powerful DSLs and language extensions.
- throwaway29103 6y ago
  Thanks for the succinct explanation
tmoertel 6y ago
I wrote a summary that goes into some detail: http://blog.moertel.com/posts/2006-01-20-wondrous-oddities-rs-function-call-semantics.html http://blog.moertel.com/posts/2006-01-20-wondrous-oddities-r...
- throwaway29103 6y ago
  Fantastic

melling 6y ago

How about Julia? I’m watching that MIT class with Grant Sanderson that uses Julia. It seems like a reasonable language, and it’s fast. Of course, I reserve the right to change my opinion after I build a few things. Dynamic typing... hmmm...

ryndbfsrw 6y ago

> Python offers more intuitive memory management than R (which is really more a statement on how bad R is at memory efficiency). I haven't found that to be the case. Loading a 2gb CSV file when you have 8gb of ram is touch and go with pandas but with data.table it’s a breeze not to mention operations once loaded up are a fair-bit faster. The pythong version has recently been released and already provides a serious speed bump over pandas not to mention out the box memory-mapping for huge files. I recommend checking it out if you haven't heard of it https://h2oai.github.io/db-benchmark/ https://h2oai.github.io/db-benchmark/

jwilber 6y ago

Same and I have to disagree with basically everything you’ve said, save the DSL bit and the R package bit. In particular, I don’t see how the tidyverse is at all ‘magical’ - take the two most popular tidy libs, dplyr and ggplot2. The api for dplyr is very explicit, intuitive, and based on long-standing precedence (sql). The api for ggplot2 is admittedly less intuitive, but is itself an implementation of widely known framework (Grammar of Graphics). If by magical you mean in their abstractions, those are about as far from magical as one can get. The tidyverse does get iffy sometimes when you need to dig into the rlang/tidy eval area, but that’s all well-documented. I haven’t had to use R in a production environment for about a year now, but I always enjoyed the concise tidy api. Base R is a total mess and largely inconsistent, but I guess that’s what you get when statisticians from different uni’s patchwork a language in their spare time.

civilized 6y ago
The dplyr team is genuinely working hard to make the tabular data manipulation tool. IIRC versions 0.5, 0.7, and 1.0 all brought major enhancements. The basic structure has remained consistent but the advanced API keeps getting refactored, usually for the better I think. (I know some people swear by data.table, but I find it gets ugly fast if you want to do anything more than simple joins or split-apply-combine)
- Tarq0n 6y ago
  Data.table's join syntax is literal insanity to me. That said I used dtplyr to generate data.table code recently and it worked well, except for one edge case.
vharuck 6y ago
>The tidyverse does get iffy sometimes when you need to dig into the rlang/tidy eval area, but that’s all well-documented. Non-standard evaluation, in my opinion, does fine in the top layer (analysis scripts, DSLs). But it's a pain to build on top of magic without a more concrete layer between. Base R uses character vectors and name attributes to hide the magic. The Tidyverse uses symbols. I find character vectors much easier to understand and manipulate. Formulas are the only scenarios in base R where symbol and expression manipulation is sometimes necessary. And I don't do that thinking, "Boy, I wish this was how I did 95% of my work." Personally, I use data.table. It's been great even for packages. If it's an internal package dataset, I don't need to manipulate symbols. And if it's a user-supplied argument, there's usually a simple and fast way to do it with a character vector. Or I define an S3 class, give a helper function that produces a dataset with known names, and just write functions around that. The user can handle combining data however they want. >Base R is a total mess and largely inconsistent I wouldn't go so far as to say "total mess." Not even an annoyingly big mess. The few functions that make up most of analysis or package code are dead simple. Inconsistent argument names is hardly a problem with an IDE.

clircle 6y ago

> Needing to download a million random packages from grad students of the internet is a recipe for disaster. This is baloney. You don't need to rely on a million R packages. You really only need a good plotting library (the built-in one is fine, or use ggplot), a good data frame library (dplyr, data.table, or use build-in one), and a few extras to handle some weird rough edges (lubridate, forcats). This takes you to 90% to your analysis goals. Only novices are 'using a million random packages', which is probably the case for python as well.

antipaul 6y ago
Indeed. `library(tidyverse)` and I’m set for like 70-80% of use cases for data manipulation, exploratory data analysis, and statistics. Though to be complete, any ML project means python and scikit learn.
phillc73 6y ago
I also believe this is now being addressed by the pharmar[1] group with their riskmetric[2] package. [1] https://www.pharmar.org/ https://www.pharmar.org/ [2] https://github.com/pharmaR/riskmetric https://github.com/pharmaR/riskmetric

throwaway4007 6y ago

But I want ggplot to do magic. Rendering things is extremely tedious and boring. I don't want to spend hours on stackoverflow searching for the special obscure incantations that will shift the legend and margins this way because matplotlib did an ugly job while constantly adapting boilerplate back and forth between the declarative and object-oriented API, I want it to just do what I mean! Is there any coincidence that there are literally tens of different (and concurrent) rendering libraries for Python while the R world more or less settled for ggplot2? Writing matplotlib is such a pain in the ass that people go to great lengths to actually avoid using matplotlib (while still not having the expressivity, features and ease of use of ggplot2). Magic is bad when you're shipping production code that's shared among multiple people who have to then spend a lot of time assimilating the mental model of the magic. It's perfect when you just want to plot stuff and draw nice figures.

phillc73 6y ago
Base R plot() can also be pretty powerful[1] and look great.[2] As an alternative to ggplot2 I would also propose that Vega Lite[3] could be a contender, with an excellent cross language ecosystem. [1] http://karolis.koncevicius.lt/posts/r_base_plotting_without_wrappers/ http://karolis.koncevicius.lt/posts/r_base_plotting_without_... [2] https://github.com/KKPMW/basetheme https://github.com/KKPMW/basetheme [3] https://vegawidget.github.io/vlbuildr/ https://vegawidget.github.io/vlbuildr/
deleted 6y ago
[deleted]

abecode 6y ago

The Tidyverse is under development and it can be tough to keep up with the changes or know where to start (e.g., plyr vs dplyr, melt vs pivot_longer). However, ggplot2 is based on a pretty solid theory of graphics (Bertin's visual variables and Wilkinson's grammar of graphics). These can take some time to get into but it's not magic once you get the idea that you're mapping data variables to visual variables.

saeranv 6y ago

> R blows Python out of the water when it comes to expressing concise and readable linear algebra/stats computation. Done right, base R looks like mathematical pseudo-code. Are you including Numpy as part of Python in this statement? I've found Numpy to be as mathematically expressive as R, although I'm not a R power-user.