6 ms·

i'm an old R user, now migrated fully to python. For those of you who us R still what is your use case? We found R has a really hard time integrating into dat

i'm an old R user, now migrated fully to python. For those of you who us R still what is your use case? We found R has a really hard time integrating into data pipelines and was best used as a standalone tool by individuals, which doesn't really work in our particular professional setup where everyone works collaboratively together. What we found was that R had alot of packages but most haven't been touched in years and when you contact the owner you find they've often moved onto the python/pandas/scikit eco system

nojito 3y ago
>For those of you who us R still what is your use case? Still the best replacement for EDA and reproducible analysis that used to be done in Excel.
medstrom 3y ago
Basically with tidyverse, R can let you write less code and keep it readable: https://www.sumsar.net/blog/pandas-feels-clunky-when-coming-from-r/ https://www.sumsar.net/blog/pandas-feels-clunky-when-coming-... Can't speak to abandonment, but it seems a lot of recent devel is occurring inside the the tidyverse, which is deprecating a whole bunch of other stuff.
- chollida1 3y ago
  I will agree that I left just as tidyverse was coming of age and I'm sometimes jealous i never got to use it. What Hadley Wickham has done is very impressive.
guccigav 3y ago
Exactly what you said, R is easy to get started for individuals in social science fields. Most people I know who want to dive deeper end up learning Python anyway.
- countrymile 3y ago
  This is true, I've found teaching r to get things done is very fast and readable fully achievable in a semester compared to teaching pandas (and also having to teach how to program in python)
- omaranto 3y ago
  And people wishing to delve even deeper switch back to R? I don't use it, but I understood that most advanced techniques in statistics are implemented first, and sometimes only, as R packages, no?
- trts 3y ago
  this is kinda funny to me. practically every social scientist I’ve known that had to use R was appalled by how cryptic and unintuitive it was. I always understood that it was its open source nature being a successor to the S language that gave it traction
ildjarn 3y ago
R is much better for REPL style development and functional programming. Python could be so much better with some minor syntax extensions.
- chollida1 3y ago
  I find that with vscode and the immediate window I get a decent repl. What about R's language makes it better for Repl driven development?
  - pama 3y ago
    If you haven’t used Emacs ESS it may be hard to explain what you’re missing with a true REPL, but if you had used it in the past and add tidyverse to it, you basically have super smooth interactive editing with the ability to quickly pool text from other REPLs, past notes or scripts. Contrary to the similar Python repl, you can easily pull and edit chains of multiple commands via R’s piping.
  - lottin 3y ago
    For one thing, R code can be written more concisely, due to the fact that the language is vector-based and functionally-oriented.
pama 3y ago
My main use case is making high quality visualizations for quick data exploration and sharing with a team. It is easy to guarantee that fonts are large enough, style is minimalist and clean, and filtering, transforming views or facets iteratively is only a couple characters change.
rpier001 3y ago
R is the better EDA language by far. Python has caught up a lot. Notebook diffs are now readable in git with the right tooling, that's huge. The drum about not fitting into data pipelines... if you're literally using a bash pipe its true most R programmers have no idea how to do that. Otherwise, that is where Docker and k8s shine. On packaging. R's package authority runs tests and ensures that all packages work with the latest version of their peers. The dependency heck is much less deep as a result. We use R at my employer still because we put statistical data science into production. Our experts come to us comfortable with R. Reimplementation would be absurd.
yabbs 3y ago
Bayesian stats Traditional stats Very fast iteration for data exploration in REPL (vs code or R studio). Prefer pipeline workflows (Tidyverse/maggrittr). Prefer functional Prefer array based. Prefer 1-indexed arrays (yes there are some of us).
usgroup 3y ago
I think its more intuitive for statistical applications where Python is grossly under-represented. This includes things like the design and analysis of experiments but also lots of domain specific statistics and algorithms such as in bioinformatics, chemistry, and so on. Typically those applications are not the sort of line-of-business enhancements ML in Python is more tuned to. I.e. recommender systems, NN models, and so on.
- doodledoodahs 3y ago
  The consistency of model specification across multiple libraries is really helpful (base lm, lme4, brms etc). Even though the syntax is sometimes extended, it seems consistent enough to mostly be comprehensible/guessable.
transcriptase 3y ago
Bioinformatics, particularly genetic mapping and population genomics. There’s an entire ecosystem of very mature tools actively maintained by labs to add analyses pertaining to advancements in the field, without breaking pipelines or silently changing the results of a given analysis from version to version. Take something like adegenet, where the manual itself is approaching 200 pages: https://cran.r-project.org/web/packages/adegenet/adegenet.pdf https://cran.r-project.org/web/packages/adegenet/adegenet.pd...
nomilk 3y ago
About half our team can wrangle and plot as fast as we can think of ideas. It creates an incredibly tight cycle time between us having ideas and getting answers; sometimes many (e.g. 10-20+) of those cycles in a single meeting. Before we used R, it would require someone jotting down things to investigate and reporting back in the next meeting. But we can do ~80% of whatever people can think of on the spot (more involved research questions can take more time). The unique qualities of R that allow this are that it's so easy to use, extremely reliable for package installation (problems occur approximately never), and the tidyverse makes it incredible easy to translate ideas into code, not only in its broad, easy to understand and powerful vocabulary, but in there being little 'nesting' required; instead working left to right and top to bottom (via the magrittr pipe) - i.e. your code, for the most part, is like reading a page in a book.
- j7ake 3y ago
  For exploring data, plotting data, and fitting complex models like GLMs and GAMs, using R is essentially as fast as thought at this point. So if you have tabular data, it’s a no brainer to use R. Getting the data into table form is often better suited for python. Fitting models that leverage autograd are also better with python.
  - jgalt212 3y ago
    > Getting the data into table form is often better suited for python. Fitting models that leverage autograd are also better with python. Totally agree. If it's in a dataframe, I prefer R. If I get involved pre-dataframe, I prefer Python.
jslakro 3y ago
What python learning material you recommend focused on data science?
- downrightmike 3y ago
  I like this and they have a video course on o'reilly https://www.amazon.com/Python-Programmers-Artificial-Intelligence-Studies/dp/0135224330 https://www.amazon.com/Python-Programmers-Artificial-Intelli...
zhdc1 3y ago
I’ve transitioned a lot of my work over to Julia, but R is still the most intuitive language I’ve used for scripting out data collection, cleaning, aggregation, and analysis cases. The ecosystem is simply better. The folks who maintain CRAN do a fantastic job. I can’t remember the last time a library incompatibility led to a show stopper. This is a weekly occurrence in Python.
- klmr 3y ago
  > I can’t remember the last time a library incompatibility led to a show stopper. Oh, it’s very common unless you basically only use < 5 packages that are completely stable and no longer actively developed: packages break backwards compatibility all the time, in small and in big ways, and version pinning in R categorically does not work as well as in Python, despite all the issues with the latter. People joke about the complex packaging ecosystem in Python but at least there is such a thing. R has no equivalent. In Python, if you have a versioned lockfile, anybody can redeploy your code unless a system dependency broke. In R, even with an ‘renv’ lockfile, installing the correct packages version is a crapshoot, and will frequently fail. Don’t get me wrong, ‘renv’ has made things much better (and ‘rig’ and PPM also help in small but important ways). But it’s still dire. At work we are facing these issues every other week on some code base.
  - apwheele 3y ago
    Agree with this, I am pretty agnostic to the pandas vs R whatever stuff (I prefer base R to tidyverse, and I like pandas, but realize I am old and probably not in majority based on comments online). But many teams who are "R adherent" folks I talk to are not deploying software in varying environments so much as reporting shops doing ad-hoc analytics. For those whom want to use both R/python, I have notes on using conda for R environments, https://andrewpwheeler.com/2022/04/08/managing-r-environments-using-conda/ https://andrewpwheeler.com/2022/04/08/managing-r-environment....
  - wodenokoto 3y ago
    At my old job we snapshotted CRAN and pinned versions of package dependencies _against_ CRAN.
ekianjo 3y ago
tidymodels is miles ahead the toys you have in python for traditional machine learning. of course Python is much better in other areas but that is a big reason to use R, together with the super powerful tidyverse syntax. and package management is much, much more reliable in R than in python.
- disgruntledphd2 3y ago
  Is tidy models better than sklearn? As honestly sklearn is one of the few things I was jealous of from the python ecosystem, historically.
acc_297 3y ago
Pk/PD work for pharmaceutical data analysis I didn’t like using R at first but I’ve come to appreciate the speed that comes with months of experience. It’s a language which feels like it has a lot of magical incantations you need to remember - the default namespace is much more crowded. Functions like sapply vs mapply are tricky to reason about from the documentation alone. The values NA vs Null vs integer(0) are all used as standins for real thrown errors and knowing which one to check for after calling a function can be tough. But after using it for a few hundred hours to do data processing and statistical regression it’s hard to imagine python or Julia being faster to use. But in all honesty for the pharmaceutical industry it’s mostly momentum that keeps R on top same reason they use a lot of FORTRAN90.
- klmr 3y ago
  > But in all honesty for the pharmaceutical industry it’s mostly momentum that keeps R on top I can’t agree with this: especially in PK/PD, R is only just now taking over from the previous (closed-source) systems. Momentum would keep R out, not in.
- nutshell42 3y ago
  > Functions like sapply vs mapply are tricky to reason about from the documentation alone. Could you please expand on that? It's unclear what you're referring to. > The values NA vs Null vs integer(0) are all used as standins for real thrown errors and knowing which one to check for after calling a function can be tough. `checkmate::assert_numeric()` (or similar) with base R you want isTRUE(): `stopifnot(isTRUE(is.finite(x)))` (or is.na or anything else) will error on empty values.
  - acc_297 3y ago
    I’ll check out those assert functions and I only meant that having apply,lapply,sapply,mapply took some getting used to most programming languages I had used prior include only 1 map / apply function and it’s still not clicking for me when there are performance penalties choosing one over another although I’ve read in the documentation that there are pros and cons
rcbdev 3y ago
R's biggest moat in my opinion is its much saner package management system and lower propensity to curb stomp existing libraries and projects with breaking changes. As a SWE I much rather inherit and maintain R services than Python services.
hadley 3y ago
If you tell me what makes R hard to integrate into data pipelines I will do my best to fix it :)
- wodenokoto 3y ago
  It’s been more than a few years since I worked in an R shop. While I loved wrangling and plotting data in the tidy verse I did find that the dependency management story in R to be even worse than Python. Maybe that’s the problem?
- dash2 3y ago
  This guy is the man to ask ^^^^^^
- chollida1 3y ago
  Wow, I really appreciate the reply. As I said in another comment here, I wish tidyverse was big when I was using R. I was an R user from about 2003-2010. We didn't have DPlyr at the moment though ggplot2 was coming around about that time I think. That helped alot for easy to develop visualizations. But in our specific cases, the distributed libraries we used were written in python and integrated well with native python code. Pandas was just coming out around 2010, I think, and I think multi threading was also an issue then, but I can't really remember. So our issues was partially our infrastructure tooling was going to python, but also we had a far easier time hiring people who were proficient in python and harder to find the same for R. And once you start writing more code in python it starts to become harder to justify two separate code bases that can do the same thing so the R code got phased out and rewritten in python so we could have a single code base and not have to duplicate functionality in two languages. Also a slight push for python came from the programmers who thought python represented a better language to know for their careers. Which looking back it does seem like python is used more often these days in general. So I guess there isn't much you could have done in this case. And as a side note, thanks for all the work you've done with R!!
- mushufasa 3y ago
  A few of the main issues I see, as a R user who built his company on python - when we wanted to build a web app that processes data, it was a lot more straightforward to build both in python, so we can process data within the web servers instead of having to manage multiple stages of infrastructure and different languages. There's no Django for R. - R will often do something instead of explicitly failing. This is the wrong tradeoff when running a production system, as if you're returning the wrong results to users you may not realize it unless there's an error - R reproducible builds are worse than python. That's saying something because python is a pretty low bar. But running production systems you can't have builds suddenly fail week over week because one of a hundred packages was updated
civilized 3y ago
> We found R has a really hard time integrating into data pipelines and was best used as a standalone tool by individuals In my org we have several 100% R teams (including mine) that have been developing and maintaining business-critical, data-intensive applications for a decade now. We don't find R difficult to integrate into data pipelines. We write our data pipelines in R, and we find it very efficient to do so. They talk to databases, APIs, command line tools, etc without issue. Doing what we do in Python is unimaginable, especially if pandas is the tabular lingua franca in the team. I vehemently agree with this article on the clunkiness of pandas from a sister comment: https://www.sumsar.net/blog/pandas-feels-clunky-when-coming-from-r/ https://www.sumsar.net/blog/pandas-feels-clunky-when-coming-.... Compared to dplyr and the tidyverse, pandas very noticeably gets in your way rather than being a tool of thought. (For what it's worth, there are other teams in my org that use Python for entirely justified reasons, and they use polars these days, not pandas.) If I had to complain about anything in R these days, it would be the increasing complexity and illegibility of error messages. Tidyverse tracebacks are often dozens or hundreds of lines. This is made much worse if you have a web app in the Shiny framework, as Shiny seems to mangle and garble what little useful information you can get (my kingdom for an error with a file name and line number). Even outside of advanced packages like Shiny, the reporting of error messages suffers from some clunkiness and irregularity. As an expert user, I can usually squint at the error barrage and infer what is really going on, but it's probably quite confusing and off-putting to newer users. Overall though, I'm not seeing any competition for R in our space. My fondest hope is that in the coming decades there arises a new, thoughtfully designed language with the Lispy flexibility of R, but also optional type safety and static analysis affordances. I'm not sure if that's even possible, but I hope the computer science geniuses figure out a way.
- samstave 3y ago
  [flagged]
- hadley 3y ago
  If you have specific issues around error messages and tracebacks please feel free to let me know directly or to file issues on Github. We really do care about the legibility of errors and tracebacks and me and my team have put a lot of effort into them in the last few years. But there's always room to do better and I'd love to know where the pain points are. (The intersection of tidyverse and shiny tracbacks are a known pain point that's hard to resolve. Unfortunately shiny and tidyverse did a bunch of parallel work that took us in slightly different directions and now it's hard to re-align.) One thing we are missing is a guide to reading traceback for newer users. Often experts can get a good sense of where the problem is, but we've failed to teach newer users how to get the most value from a traceback.
samstave 3y ago
im somewhat of an armchair data scientist myself. However - I'd love to learn your ways; specifically - what are your best recommendations for python over R? Specifically, even though my R skills are weak - I think that RStudio is pretty darn amazing - what do you recommend over Rstudio? I'd truly like to hear what a good toolbox looks like from your perspective these days (especially now this little GPT toddler is bonking into everything in my domain)
richrichie 3y ago
I quit R a while ago - before data science became a thing - and switched to Julia for such tasks. R has lots of stats packages, but it is too esoteric and specialised a language to be useful IMO.
wodenokoto 3y ago
Did you work with Rstudio Server and still found it not collaborative enough?
tropical333 3y ago
> What we found was that R had alot of packages but most haven't been touched in years and when you contact the owner you find they've often moved onto the python/pandas/scikit eco system As a "bilingual" R & Python user, I've found this to be true for the latter language as well :) I don't have much to add on top of what other useRs have mentioned, except another testimonial that our company has successfully used R in production for 6+ years, from data "pipeline" stuff you mentioned to dozens upon dozens of predictive models of varying complexities. When faced with a new data analysis ask, 99%+ of the time I reach for R (although without the tidyverse, that number would be much lower). Like another commenter said, the ease by which you can plot in R blows Python away. Seaborn seems like a decent compromise in my limited experience, but plotting in "base" matplotlib makes me want to die.
- disgruntledphd2 3y ago
  Plotnine is a pretty rocking ggplot clone in Python. Just import star and you're golden.
  - hadley 3y ago
    We (Posit) have hired Hassan (the maintainer of plotnine) so this is great to hear :)
    - disgruntledphd2 3y ago
      It definitely is! If you could hurry up and destroy Jupiter notebooks that would be sweet ;)
vharuck 3y ago
Why I still use R for analysis at work: - R Markdown is just great for static reports. We use PowerBI or ArcGIS for interactive stuff. - GIS is a breeze. My work provides licenses for ArcGIS, which has a Python library for scripting. Despite that, it is so much easier to do stuff in R, which can read and create ArcGIS shapefiles. - Exploratory data analysis is easy. Often, before meetings, I'll connect to the database in R and make a few basic tables. Then I can query, aggregate, or plot data sitting the meeting. I have custom ggplot themes in a package, so even my happy hastily created plots look nice. - RStudio is amazing. What it lacks in editing tricks, it more than makes up for in simplifying R-specific tasks. Showing plots is automatic, rendering and viewing markdown reports (of any type) is two buttons, testing and building a package are each two buttons. - I spent a lot of time evangelizing R (team-wide presentations, being the "R guy" for troubleshooting, organizing an R User Group with members from different teams, creating an internal package repository). Some became happy converts, the rest begrudgingly accepted it as a tool we would use. I don't know if I could do it again with another language. I'll admit my work doesn't get incorporated into pipelines. We get the data, analyze it, create reports, and share the reports by email or on our public website. The statisticians are segregated from the developers here. State government resists change, especially role changes that don't match grants' or laws' wording.
- ekianjo 3y ago
  > R Markdown is just great for static reports. Quarto (also supporting R) is a good replacement for rmarkdown (with a saner syntax) and I say this as someone who has extensively used rmarkdown over the years.
sebastianavina 3y ago
RStudio is a nice tool for making some quick graphs on data, descriptive analyisis and quickly exploring a dataset. Building some reports, or manipulating small datasets for beautiful graphs. For anything else, we use Python.
ryndbfsrw 3y ago
[dead]
axpy906 3y ago
I stopped using it in 2015, when I began to learn how to code. At my FAANG company, there are teams that use it for econometrics. I think that’s Rs sweet spot, still in 2024.
minimaxir 3y ago
There is currently no Python equivalent for both the ease of use and output quality of ggplot2 for data visualization. Many have tried over the past decade, but none have gotten close. (Plotnine was the closest: per Hadley in another comment his company hired the maintainer)
trts 3y ago
two reasons for me 1) tidyverse makes prodding and plotting my data faster and more enjoyable. when I am prototyping a model I'll sometimes do the groundwork in R and then migrate the production version to python 2) I can't seem to write data wrangling code in py that is as aesthetically pleasing and easy to reinterpret later. could just be that I started in R, but while the methods in pandas "work" I don't always totally understand why they work the way they do. with tidy it works the way I expect and feels easier to read back and iterate on
winwang 3y ago
Was there any performance difference between R and Python in your case?
melondonkey 3y ago
[dead]
clatan 3y ago
I'm an old R user forced to mostly use python because that's what the team uses. R is so much better than python in many areas concerning data pipelines: connecting with external database systems through an unified API, superior data munging utilities, as well as plotting, a more comprehensive (obviously) statistical analysis toolset. I even find rmarkdown vastly superior to jupyter. But IMO the best reason to use R rather tha python is that its tools will make you approach the problem as a statistician rather than a programmer.
dxbydt 3y ago
>what is your use case? if you are doing Bayesian stats, fitting hierarchical models, or using Stan in any serious capacity, R/Stan is so much more ergonomic than Pystan. Here’s a long list of pros-cons: https://discourse.mc-stan.org/t/various-observations-on-rstan-cmdstanr-pystan-and-cmdstanpy-after-teaching-with-all-in-parallel/23455 https://discourse.mc-stan.org/t/various-observations-on-rsta...
twobitshifter 3y ago
I had thought this book was redone by someone in Python. Does anyone remember seeing that?

nojito 3y ago

>For those of you who us R still what is your use case? Still the best replacement for EDA and reproducible analysis that used to be done in Excel.

medstrom 3y ago

Basically with tidyverse, R can let you write less code and keep it readable: https://www.sumsar.net/blog/pandas-feels-clunky-when-coming-from-r/ https://www.sumsar.net/blog/pandas-feels-clunky-when-coming-... Can't speak to abandonment, but it seems a lot of recent devel is occurring inside the the tidyverse, which is deprecating a whole bunch of other stuff.

chollida1 3y ago
I will agree that I left just as tidyverse was coming of age and I'm sometimes jealous i never got to use it. What Hadley Wickham has done is very impressive.

guccigav 3y ago

Exactly what you said, R is easy to get started for individuals in social science fields. Most people I know who want to dive deeper end up learning Python anyway.

countrymile 3y ago
This is true, I've found teaching r to get things done is very fast and readable fully achievable in a semester compared to teaching pandas (and also having to teach how to program in python)
omaranto 3y ago
And people wishing to delve even deeper switch back to R? I don't use it, but I understood that most advanced techniques in statistics are implemented first, and sometimes only, as R packages, no?
trts 3y ago
this is kinda funny to me. practically every social scientist I’ve known that had to use R was appalled by how cryptic and unintuitive it was. I always understood that it was its open source nature being a successor to the S language that gave it traction

ildjarn 3y ago

R is much better for REPL style development and functional programming. Python could be so much better with some minor syntax extensions.

chollida1 3y ago
I find that with vscode and the immediate window I get a decent repl. What about R's language makes it better for Repl driven development?
- pama 3y ago
  If you haven’t used Emacs ESS it may be hard to explain what you’re missing with a true REPL, but if you had used it in the past and add tidyverse to it, you basically have super smooth interactive editing with the ability to quickly pool text from other REPLs, past notes or scripts. Contrary to the similar Python repl, you can easily pull and edit chains of multiple commands via R’s piping.
- lottin 3y ago
  For one thing, R code can be written more concisely, due to the fact that the language is vector-based and functionally-oriented.

pama 3y ago

My main use case is making high quality visualizations for quick data exploration and sharing with a team. It is easy to guarantee that fonts are large enough, style is minimalist and clean, and filtering, transforming views or facets iteratively is only a couple characters change.

rpier001 3y ago

R is the better EDA language by far. Python has caught up a lot. Notebook diffs are now readable in git with the right tooling, that's huge. The drum about not fitting into data pipelines... if you're literally using a bash pipe its true most R programmers have no idea how to do that. Otherwise, that is where Docker and k8s shine. On packaging. R's package authority runs tests and ensures that all packages work with the latest version of their peers. The dependency heck is much less deep as a result. We use R at my employer still because we put statistical data science into production. Our experts come to us comfortable with R. Reimplementation would be absurd.

yabbs 3y ago

Bayesian stats Traditional stats Very fast iteration for data exploration in REPL (vs code or R studio). Prefer pipeline workflows (Tidyverse/maggrittr). Prefer functional Prefer array based. Prefer 1-indexed arrays (yes there are some of us).

usgroup 3y ago

I think its more intuitive for statistical applications where Python is grossly under-represented. This includes things like the design and analysis of experiments but also lots of domain specific statistics and algorithms such as in bioinformatics, chemistry, and so on. Typically those applications are not the sort of line-of-business enhancements ML in Python is more tuned to. I.e. recommender systems, NN models, and so on.

doodledoodahs 3y ago
The consistency of model specification across multiple libraries is really helpful (base lm, lme4, brms etc). Even though the syntax is sometimes extended, it seems consistent enough to mostly be comprehensible/guessable.

transcriptase 3y ago

Bioinformatics, particularly genetic mapping and population genomics. There’s an entire ecosystem of very mature tools actively maintained by labs to add analyses pertaining to advancements in the field, without breaking pipelines or silently changing the results of a given analysis from version to version. Take something like adegenet, where the manual itself is approaching 200 pages: https://cran.r-project.org/web/packages/adegenet/adegenet.pdf https://cran.r-project.org/web/packages/adegenet/adegenet.pd...

nomilk 3y ago

About half our team can wrangle and plot as fast as we can think of ideas. It creates an incredibly tight cycle time between us having ideas and getting answers; sometimes many (e.g. 10-20+) of those cycles in a single meeting. Before we used R, it would require someone jotting down things to investigate and reporting back in the next meeting. But we can do ~80% of whatever people can think of on the spot (more involved research questions can take more time). The unique qualities of R that allow this are that it's so easy to use, extremely reliable for package installation (problems occur approximately never), and the tidyverse makes it incredible easy to translate ideas into code, not only in its broad, easy to understand and powerful vocabulary, but in there being little 'nesting' required; instead working left to right and top to bottom (via the magrittr pipe) - i.e. your code, for the most part, is like reading a page in a book.

j7ake 3y ago
For exploring data, plotting data, and fitting complex models like GLMs and GAMs, using R is essentially as fast as thought at this point. So if you have tabular data, it’s a no brainer to use R. Getting the data into table form is often better suited for python. Fitting models that leverage autograd are also better with python.
- jgalt212 3y ago
  > Getting the data into table form is often better suited for python. Fitting models that leverage autograd are also better with python. Totally agree. If it's in a dataframe, I prefer R. If I get involved pre-dataframe, I prefer Python.

jslakro 3y ago

What python learning material you recommend focused on data science?

downrightmike 3y ago
I like this and they have a video course on o'reilly https://www.amazon.com/Python-Programmers-Artificial-Intelligence-Studies/dp/0135224330 https://www.amazon.com/Python-Programmers-Artificial-Intelli...

zhdc1 3y ago

I’ve transitioned a lot of my work over to Julia, but R is still the most intuitive language I’ve used for scripting out data collection, cleaning, aggregation, and analysis cases. The ecosystem is simply better. The folks who maintain CRAN do a fantastic job. I can’t remember the last time a library incompatibility led to a show stopper. This is a weekly occurrence in Python.

klmr 3y ago
> I can’t remember the last time a library incompatibility led to a show stopper. Oh, it’s very common unless you basically only use < 5 packages that are completely stable and no longer actively developed: packages break backwards compatibility all the time, in small and in big ways, and version pinning in R categorically does not work as well as in Python, despite all the issues with the latter. People joke about the complex packaging ecosystem in Python but at least there is such a thing. R has no equivalent. In Python, if you have a versioned lockfile, anybody can redeploy your code unless a system dependency broke. In R, even with an ‘renv’ lockfile, installing the correct packages version is a crapshoot, and will frequently fail. Don’t get me wrong, ‘renv’ has made things much better (and ‘rig’ and PPM also help in small but important ways). But it’s still dire. At work we are facing these issues every other week on some code base.
- apwheele 3y ago
  Agree with this, I am pretty agnostic to the pandas vs R whatever stuff (I prefer base R to tidyverse, and I like pandas, but realize I am old and probably not in majority based on comments online). But many teams who are "R adherent" folks I talk to are not deploying software in varying environments so much as reporting shops doing ad-hoc analytics. For those whom want to use both R/python, I have notes on using conda for R environments, https://andrewpwheeler.com/2022/04/08/managing-r-environments-using-conda/ https://andrewpwheeler.com/2022/04/08/managing-r-environment....
- wodenokoto 3y ago
  At my old job we snapshotted CRAN and pinned versions of package dependencies _against_ CRAN.

ekianjo 3y ago

tidymodels is miles ahead the toys you have in python for traditional machine learning. of course Python is much better in other areas but that is a big reason to use R, together with the super powerful tidyverse syntax. and package management is much, much more reliable in R than in python.

disgruntledphd2 3y ago
Is tidy models better than sklearn? As honestly sklearn is one of the few things I was jealous of from the python ecosystem, historically.

acc_297 3y ago

Pk/PD work for pharmaceutical data analysis I didn’t like using R at first but I’ve come to appreciate the speed that comes with months of experience. It’s a language which feels like it has a lot of magical incantations you need to remember - the default namespace is much more crowded. Functions like sapply vs mapply are tricky to reason about from the documentation alone. The values NA vs Null vs integer(0) are all used as standins for real thrown errors and knowing which one to check for after calling a function can be tough. But after using it for a few hundred hours to do data processing and statistical regression it’s hard to imagine python or Julia being faster to use. But in all honesty for the pharmaceutical industry it’s mostly momentum that keeps R on top same reason they use a lot of FORTRAN90.

klmr 3y ago
> But in all honesty for the pharmaceutical industry it’s mostly momentum that keeps R on top I can’t agree with this: especially in PK/PD, R is only just now taking over from the previous (closed-source) systems. Momentum would keep R out, not in.
nutshell42 3y ago
> Functions like sapply vs mapply are tricky to reason about from the documentation alone. Could you please expand on that? It's unclear what you're referring to. > The values NA vs Null vs integer(0) are all used as standins for real thrown errors and knowing which one to check for after calling a function can be tough. `checkmate::assert_numeric()` (or similar) with base R you want isTRUE(): `stopifnot(isTRUE(is.finite(x)))` (or is.na or anything else) will error on empty values.
- acc_297 3y ago
  I’ll check out those assert functions and I only meant that having apply,lapply,sapply,mapply took some getting used to most programming languages I had used prior include only 1 map / apply function and it’s still not clicking for me when there are performance penalties choosing one over another although I’ve read in the documentation that there are pros and cons

rcbdev 3y ago

R's biggest moat in my opinion is its much saner package management system and lower propensity to curb stomp existing libraries and projects with breaking changes. As a SWE I much rather inherit and maintain R services than Python services.

hadley 3y ago

If you tell me what makes R hard to integrate into data pipelines I will do my best to fix it :)

wodenokoto 3y ago
It’s been more than a few years since I worked in an R shop. While I loved wrangling and plotting data in the tidy verse I did find that the dependency management story in R to be even worse than Python. Maybe that’s the problem?
dash2 3y ago
This guy is the man to ask ^^^^^^
chollida1 3y ago
Wow, I really appreciate the reply. As I said in another comment here, I wish tidyverse was big when I was using R. I was an R user from about 2003-2010. We didn't have DPlyr at the moment though ggplot2 was coming around about that time I think. That helped alot for easy to develop visualizations. But in our specific cases, the distributed libraries we used were written in python and integrated well with native python code. Pandas was just coming out around 2010, I think, and I think multi threading was also an issue then, but I can't really remember. So our issues was partially our infrastructure tooling was going to python, but also we had a far easier time hiring people who were proficient in python and harder to find the same for R. And once you start writing more code in python it starts to become harder to justify two separate code bases that can do the same thing so the R code got phased out and rewritten in python so we could have a single code base and not have to duplicate functionality in two languages. Also a slight push for python came from the programmers who thought python represented a better language to know for their careers. Which looking back it does seem like python is used more often these days in general. So I guess there isn't much you could have done in this case. And as a side note, thanks for all the work you've done with R!!
mushufasa 3y ago
A few of the main issues I see, as a R user who built his company on python - when we wanted to build a web app that processes data, it was a lot more straightforward to build both in python, so we can process data within the web servers instead of having to manage multiple stages of infrastructure and different languages. There's no Django for R. - R will often do something instead of explicitly failing. This is the wrong tradeoff when running a production system, as if you're returning the wrong results to users you may not realize it unless there's an error - R reproducible builds are worse than python. That's saying something because python is a pretty low bar. But running production systems you can't have builds suddenly fail week over week because one of a hundred packages was updated

civilized 3y ago

> We found R has a really hard time integrating into data pipelines and was best used as a standalone tool by individuals In my org we have several 100% R teams (including mine) that have been developing and maintaining business-critical, data-intensive applications for a decade now. We don't find R difficult to integrate into data pipelines. We write our data pipelines in R, and we find it very efficient to do so. They talk to databases, APIs, command line tools, etc without issue. Doing what we do in Python is unimaginable, especially if pandas is the tabular lingua franca in the team. I vehemently agree with this article on the clunkiness of pandas from a sister comment: https://www.sumsar.net/blog/pandas-feels-clunky-when-coming-from-r/ https://www.sumsar.net/blog/pandas-feels-clunky-when-coming-.... Compared to dplyr and the tidyverse, pandas very noticeably gets in your way rather than being a tool of thought. (For what it's worth, there are other teams in my org that use Python for entirely justified reasons, and they use polars these days, not pandas.) If I had to complain about anything in R these days, it would be the increasing complexity and illegibility of error messages. Tidyverse tracebacks are often dozens or hundreds of lines. This is made much worse if you have a web app in the Shiny framework, as Shiny seems to mangle and garble what little useful information you can get (my kingdom for an error with a file name and line number). Even outside of advanced packages like Shiny, the reporting of error messages suffers from some clunkiness and irregularity. As an expert user, I can usually squint at the error barrage and infer what is really going on, but it's probably quite confusing and off-putting to newer users. Overall though, I'm not seeing any competition for R in our space. My fondest hope is that in the coming decades there arises a new, thoughtfully designed language with the Lispy flexibility of R, but also optional type safety and static analysis affordances. I'm not sure if that's even possible, but I hope the computer science geniuses figure out a way.

samstave 3y ago
[flagged]
hadley 3y ago
If you have specific issues around error messages and tracebacks please feel free to let me know directly or to file issues on Github. We really do care about the legibility of errors and tracebacks and me and my team have put a lot of effort into them in the last few years. But there's always room to do better and I'd love to know where the pain points are. (The intersection of tidyverse and shiny tracbacks are a known pain point that's hard to resolve. Unfortunately shiny and tidyverse did a bunch of parallel work that took us in slightly different directions and now it's hard to re-align.) One thing we are missing is a guide to reading traceback for newer users. Often experts can get a good sense of where the problem is, but we've failed to teach newer users how to get the most value from a traceback.

samstave 3y ago

im somewhat of an armchair data scientist myself. However - I'd love to learn your ways; specifically - what are your best recommendations for python over R? Specifically, even though my R skills are weak - I think that RStudio is pretty darn amazing - what do you recommend over Rstudio? I'd truly like to hear what a good toolbox looks like from your perspective these days (especially now this little GPT toddler is bonking into everything in my domain)

richrichie 3y ago

I quit R a while ago - before data science became a thing - and switched to Julia for such tasks. R has lots of stats packages, but it is too esoteric and specialised a language to be useful IMO.

wodenokoto 3y ago

Did you work with Rstudio Server and still found it not collaborative enough?

tropical333 3y ago

> What we found was that R had alot of packages but most haven't been touched in years and when you contact the owner you find they've often moved onto the python/pandas/scikit eco system As a "bilingual" R & Python user, I've found this to be true for the latter language as well :) I don't have much to add on top of what other useRs have mentioned, except another testimonial that our company has successfully used R in production for 6+ years, from data "pipeline" stuff you mentioned to dozens upon dozens of predictive models of varying complexities. When faced with a new data analysis ask, 99%+ of the time I reach for R (although without the tidyverse, that number would be much lower). Like another commenter said, the ease by which you can plot in R blows Python away. Seaborn seems like a decent compromise in my limited experience, but plotting in "base" matplotlib makes me want to die.

disgruntledphd2 3y ago
Plotnine is a pretty rocking ggplot clone in Python. Just import star and you're golden.
- hadley 3y ago
  We (Posit) have hired Hassan (the maintainer of plotnine) so this is great to hear :)
  - disgruntledphd2 3y ago
    It definitely is! If you could hurry up and destroy Jupiter notebooks that would be sweet ;)

vharuck 3y ago

Why I still use R for analysis at work: - R Markdown is just great for static reports. We use PowerBI or ArcGIS for interactive stuff. - GIS is a breeze. My work provides licenses for ArcGIS, which has a Python library for scripting. Despite that, it is so much easier to do stuff in R, which can read and create ArcGIS shapefiles. - Exploratory data analysis is easy. Often, before meetings, I'll connect to the database in R and make a few basic tables. Then I can query, aggregate, or plot data sitting the meeting. I have custom ggplot themes in a package, so even my happy hastily created plots look nice. - RStudio is amazing. What it lacks in editing tricks, it more than makes up for in simplifying R-specific tasks. Showing plots is automatic, rendering and viewing markdown reports (of any type) is two buttons, testing and building a package are each two buttons. - I spent a lot of time evangelizing R (team-wide presentations, being the "R guy" for troubleshooting, organizing an R User Group with members from different teams, creating an internal package repository). Some became happy converts, the rest begrudgingly accepted it as a tool we would use. I don't know if I could do it again with another language. I'll admit my work doesn't get incorporated into pipelines. We get the data, analyze it, create reports, and share the reports by email or on our public website. The statisticians are segregated from the developers here. State government resists change, especially role changes that don't match grants' or laws' wording.

ekianjo 3y ago
> R Markdown is just great for static reports. Quarto (also supporting R) is a good replacement for rmarkdown (with a saner syntax) and I say this as someone who has extensively used rmarkdown over the years.

sebastianavina 3y ago

RStudio is a nice tool for making some quick graphs on data, descriptive analyisis and quickly exploring a dataset. Building some reports, or manipulating small datasets for beautiful graphs. For anything else, we use Python.

ryndbfsrw 3y ago

[dead]

axpy906 3y ago

I stopped using it in 2015, when I began to learn how to code. At my FAANG company, there are teams that use it for econometrics. I think that’s Rs sweet spot, still in 2024.

minimaxir 3y ago

There is currently no Python equivalent for both the ease of use and output quality of ggplot2 for data visualization. Many have tried over the past decade, but none have gotten close. (Plotnine was the closest: per Hadley in another comment his company hired the maintainer)

trts 3y ago

two reasons for me 1) tidyverse makes prodding and plotting my data faster and more enjoyable. when I am prototyping a model I'll sometimes do the groundwork in R and then migrate the production version to python 2) I can't seem to write data wrangling code in py that is as aesthetically pleasing and easy to reinterpret later. could just be that I started in R, but while the methods in pandas "work" I don't always totally understand why they work the way they do. with tidy it works the way I expect and feels easier to read back and iterate on

winwang 3y ago

Was there any performance difference between R and Python in your case?

melondonkey 3y ago

[dead]

clatan 3y ago

I'm an old R user forced to mostly use python because that's what the team uses. R is so much better than python in many areas concerning data pipelines: connecting with external database systems through an unified API, superior data munging utilities, as well as plotting, a more comprehensive (obviously) statistical analysis toolset. I even find rmarkdown vastly superior to jupyter. But IMO the best reason to use R rather tha python is that its tools will make you approach the problem as a statistician rather than a programmer.

dxbydt 3y ago

>what is your use case? if you are doing Bayesian stats, fitting hierarchical models, or using Stan in any serious capacity, R/Stan is so much more ergonomic than Pystan. Here’s a long list of pros-cons: https://discourse.mc-stan.org/t/various-observations-on-rstan-cmdstanr-pystan-and-cmdstanpy-after-teaching-with-all-in-parallel/23455 https://discourse.mc-stan.org/t/various-observations-on-rsta...

twobitshifter 3y ago

I had thought this book was redone by someone in Python. Does anyone remember seeing that?