13 ms·

An opinionated view of the Tidyverse “dialect” of the R language

137 points by porker 7y ago

zelda_1 7y ago
When I'm writing the book (www.anotherbookondatascience.com) I always try to avoid the usage of tidyverse and got some criticism. But now I feel supported.
kickout 7y ago
Decent critiques in my opinion. I was trained on/learned base-R myself ~10 years ago and love the "tidyverse" (although the author is correct, data.table is superior for big data and its not particularly close). Can't imagine people that only know tidyr/dplyr/etc without knowing base-R, seems like those people would get exposed quickly. I enjoy base-R solutions where I can. I use data.table when my data sets are medium-ism. But like most people, my data is small and people just call it big and thus the attraction to dplyr in particular. Those chained pipes are addicting. Always love seeing R discussion on HN, seems to be taboo though (not a 'real' language and all that).
- Bootvis 7y ago
  I don’t even understand the not a real language critique, R: - is based on scheme, a HN favourite; - integrates very well with C++ through RCpp allowing you to do whatever you want. 9/10 times someone already went through the trouble for you.
  - kickout 7y ago
    I agree; which is why I generally get confused when I see that criticism. This is only an assumption though; maybe its just a vocal minority that don't like R or I only notice the negative comments. Who knows
    - erikpukinskis 7y ago
      For what it’s worth, I am a JavaScript programmer and I feel like HN shits on JS All. The. Time. Some people just like to shit on languages. I think it’s just insecurity. They worry a lot about whether they’re a good enough developer, so they try hard to be on the “most advanced” platform they can figure out, and then need to talk down to the other platforms. Secure, talented, older developers, in my experience, tend to be open to whatever technology is most appropriate. They have confidence in their ability to learn new things. And see the good traits of any tech, as well as the bad one—a necessary skill if you’re going to be the kind of developer who can make traction in any situation. The platform fetishists can only really work from inside their special tower.
      - newen 7y ago
        Javascript is just a terrible language and programming environment though. Anyone can criticize people who disagree with them by calling them "insecure". It's a meaningless argument.
  - thom 7y ago
    I don't think there's any shame in R effectively being a good, productive DSL for a lot of stats and viz work, built around fast stuff written in C++. I also think this is largely true of a lot of Python as well, tbh. In the future my hope is that projects like Arrow end up with even more of the workload being taken off R's shoulders.
    - vincent-toups 7y ago
      I'm a data scientist and, frankly, I really strongly prefer R over Python. Its mostly the whitespace in Python, but the language is also slow and crufty as heck. R, if you use it responsibly, feels a lot more like your standard, lexically scoped, dynamically typed language.
  - atheriel 7y ago
    Absolutely. I usually respond to the "R-is-not-a-real-language" line with something like "R is syntactic sugar on a Lisp where the atoms are APL arrays". It's plenty interesting from a computer science perspective, if you bother to look. For instance, why does R use <- instead of = for assignment? Because initial versions (of S) predate C -- developed down the hall -- the language that first introduced = for assignment.
  - nerdponx 7y ago
    The R standard library is kind of weird and hard to use for "general purpose" programming. It's very clearly a domain specific language. Also, it's unbelievably slow for basic operations like looping, function calls, and variable assignment. It's literally orders of magnitude slower than Python (I've tested it). Unlike its fellow C-flavored-Lisp Javascript, R retains an extreme level of homoiconicity, Which apparently makes the language very difficult to compile or optimize. I don't know how e.g. SBCL does it, but over the lifetime of the R language no one has managed to implement a non-trivial subset of R that performs better than GNU R (fastR was never finished to my knowledge). So you are basically relegated to writing code in C, Fortran, or C++ if you need even decent performance. Otherwise you are stuck using "vectorized" operations like lapply(), which are fine but make for a jarring experience if you're coming from other languages. So of course it's a "real" language in the sense that Bash is a real language. But it's ultimately and fundamentally a domain-specific scripting language and I don't know if there is a way around that.
    - kgwgk 7y ago
      Are you aware of Radford Neal’s pqR? https://radfordneal.wordpress.com/ https://radfordneal.wordpress.com/
      - nerdponx 7y ago
        I had heard of it in the past but I didn't realize it was in a finished state and keeping pace with recent R versions. I'll read up on it, thank you. Edit: looks like it's still a full major version behind? To be fair, there weren't many significant changes between R 2.x and 3.x.
        kgwgk 7y ago
        Well, it may not be keeping up with the new versions but at least it does cover a non-trivial subset of the standard implementation. :-) I have not tried it in many years and I don't know how much faster or compatible it is. Nevertheless, I think it's a good thing that an "outsider" is looking at performance issues. As far as I know it's essentially a one-man's effort. One could imagine that RStudio or Microsoft would be able to improve R performance substantially if they wanted to.
  - epistasis 7y ago
    The "not a real language" critique often comes from those without any knowledge of Scheme; "real" languages are like C# or Java, and maybe Python, even though the lisp-like properties of Python are not so much appreciated. As C# and Java have gotten more functional aspects over the years, I've seen less and less of the "not a real language" criticism.
    - deleted 7y ago
      [deleted]
  - vincent-toups 7y ago
    "Based on Scheme" really, profoundly, oversells it. R's idea of Scheme is deeply bizarre and profoundly outdated.
  - GlennS 7y ago
    There's really not much Lisp/Scheme left in R these days. Function arguments/environments are still pairlists: a type of cons cell. You can manipulate the parameters to a function as symbols before you evaluate them. This is useful — many very good R libraries all make heavy use of it — and certainly very interesting. Lispers probably know it as quote/antiquote. I suppose we could also point that, much like most Lisps, R gives you plenty of object systems to choose from... And that's about it for R-as-a-Lisp. It's also a really bad language for working with tree-structured data, something Lisps normally excel at.
  - laichzeit0 7y ago
    The last time I tried putting a REST API in front of my predictive model, I used plumbr. I also then learned that the R runtime was single threaded, so that only 1 request could be processed at a time. I don’t know how you can overcome this, and it makes things significantly difficult to put into a real production environment. I’m sure someone will chime in that there are different runtimes, like Microsoft’s (does this run on MacOS? If not, how can I use it to develop and test on my laptop) or some expensive RStudio solution. The fact is you have none of those problems in e.g., Python. Although I would prefer to use Caret for many predictive modeling tasks, it is non-trivial to take the R runtime to production. That was what I walked away with.
- atheriel 7y ago
  I'm in a similar boat, using base R, data.table, or tidyverse as needed. But over time I've found that I respect, more and more, the good decisions that have been made in base R.
c3534l 7y ago
Tidyverse was nearly the only thing I really liked about R. The syntax is quirky, it's full of cruft, and unpleasant to make full programs out of. But if I used a new package, I knew instantly how to use it.
bagrow 7y ago
I think the same thing is happening in Python for data science due to Pandas. It’s a great package, but many data scientists are only able to manipulate data using Pandas, and know little “base” Python.
- jphoward 7y ago
  That sounds incredible to me as using pandas without falling into the traps of chained indexing seems so much harder than 90% of Python.
  - telchar 7y ago
    I've been planning to go from "can work it out with copious examples" to "knows when to use apply, transform, etc. off the cuff" level of knowledge in Pandas - do you have any suggestions on good resources on this? I'd like to understand better why the data structures work the way they do and thus have an intuition on what operations to use when. The O'Reilly Python Data Science Handbook[0] seems like it might be useful here, but I'm not sure if it is still up to date. [0] https://www.oreilly.com/library/view/python-data-science/9781491912126/ https://www.oreilly.com/library/view/python-data-science/978...
    - squaresmile 7y ago
      I find the pandas documentation pretty good [0]. Actually, I find the documentation, official tutorials of pandas, numpy, matplotlib all pretty good. Each package has their own idiosyncrasies but I think the documentation and code examples cover them well enough. The OReily book should not be that outdated if at all. It also covers other essential tools. [0] https://pandas.pydata.org/pandas-docs/stable/getting_started/index.html https://pandas.pydata.org/pandas-docs/stable/getting_started...
- mruts 7y ago
  You can’t really do any data analysis without pandas in python though. In fact python is a terrible language and the only reason anyone should use it is for access to sklearn, scikit, pytorth, and pandas. Hopefully Julia will be able to unseat python for data analysis in the future.
dracodoc 7y ago
One reason dplyr was promoted is that you can connect to different server backend like sparkr, sql etc, which is the enterprise direction RStudio aiming at. In the other hand, data.table is your friend when you are using your own machine. With more and more memory and cpu power available, its benefits are actually increasing. I have been using data.table from almost day one, it does worth more recognition. The syntax is a little bit hard in the beginning but can be grasped after some efforts. I often feel that my exploration code will be much more tedious look if written in dplyr style.
- vasili111 7y ago
  If my university uses Enterprise RStutio on grid it is better to stick with tidyverse rather than data.table?
  - Tarq0n 7y ago
    It doesn't make a difference. Rstudio's products aren't integrated with their libraries in particular.
thom 7y ago
Wow, the idea that the Tidyverse is somehow burdened with an overabundance of computer science or functional programming is just hilarious to this programmer who has had to maintain, debug and optimise other people's R code.
- jrumbut 7y ago
  I think this is what the author is getting at, that correct tidyverse usage requires greater experience and knowledge and creates difficult to debug and optimize code when used incorrectly (and is easy to use incorrectly). I tend to agree. I've found tidyverse code has a write-only quality to it. Since I'm going to see more of it I plan to dive into the inner workings of at least dyplr and purr. That said it is hard to deny what Hadley Wickham has done for the R community and I can't write off the idea that he sees a bigger picture I'm missing.
  - hardboiled 7y ago
    Tidyverse code is not write-only; it is designed to be mostly read-only where the level of abstraction is at the level of the domain, which in this case is data frames and data pipelines akin to the same constructs in relational algebra/SQL or data processing (map/reduce). Correct usage requires learning the right abstractions, but fortunately these abstractions are shared across language communities and frameworks. If you are doing any data processing work and you do not know about foundational functional concepts ie map/reduce then I would argue the code you write is less readable, overly focused on the idiosyncracies of how to do something rather than how entities and data are related to each other - their logic and structure. The main optimization is to be correct and expressive of one's intention and purpose. If you need performance use Julia.
    - jrumbut 7y ago
      I understand that's the goal, but it hasn't been what I've seen from user's code. I think the issue, which the article describes, is that the tidyverse has too large a surface area for something aimed at very smart people who are not software engineers and don't have time to become one. So they end up with a subset of functionality they can get stuff done with, but then they need to collaborate with someone who uses a different subset. That can be messy. I think in an environment where everyone learns R from the same course/book/MOOC/whatever (or have done lots of different sorts of programming) and the organization can impose a style guide the tidyverse approach would be great, but when you have people coming from all sorts of places and backgrounds I don't think it's a good fit.
jhbadger 7y ago
At least in my case, although I do like some of the tidyverse (ggplot is tidyverse after all, and probably my favorite plotting system in any language), I find the "tibble" data structure and its disdain for row names really annoying. Many, many, packages which I use expect a data frame with row names and likewise output such. So I am constantly converting back and forth between tibbles and data frames.
- Bootvis 7y ago
  The claim that ggplot2 is tidyverse can be disputed. It existed before and works fine outside of it.
  - minimaxir 7y ago
    ggplot2 is under the tidyverse umbrella, and will be installed/loaded with library(tidyverse). The original article is more about dplyr though.
  - hadley 7y ago
    You are of course free to dispute what is in the tidyverse, but I pretty strongly believe that it is part of the tidyverse
    - RosanaAnaDana 7y ago
      Seeing as the tidyverse is pretty much your invention, its well within your purvue to define what is in and what is out. I do think, respectfully, that both the op and the general sentiment article have the right of it. Data.table and data.table-esque notation represent such an improvement of tibble/dplyr, that within my company, we're making a concerted effort to purge all tidyverse packages from general use (less ggplot). When new developers come on, if they are coming from tidyverse, their first task will be something involving pipes and data.table. Tidyverse was fine in school. It doesn't pass muster in production, at least not in our work. Data.table syntax is simpler, easier to read, easier to teach, and orders of magnitude faster. It plays nicer with other packages than the tidyverse (if it fits into a DF, it almost always fits into a DT, and i've never met a tibble that I didn't wish was a data.table), and since almost all of our datasets are 10's to 1000's of millions of lines long, the decision was really made for us.
throw20102010 7y ago
I think he’s exaggerating the effect of some of these claims. People who “grew up” learning R+Tidyverse will discriminate against non-Tidyverse users? Give me a break. You are either an advanced or competent or novice R user. You either understand how pipes (%>%) work or you don’t. I doubt that anyone is going to be denied a job because they are an amazing R programmer but they just don’t have the experience with a particular set of packages, especially those that are as well implemented and easy to learn as the Tidyverse. I can totally imagine someone not getting a job because they are a crap programmer, and then blaming it on something else. Or I can also imagine someone saying that the Tidyverse packages suck, and then be denied a job because of their attitude. I could make a similar argument for most of the substantive claims in his post. Some of these examples (dplyr vs. data.table) are cherry picked. I have several of my own examples where read_csv is way faster than read.csv, so maybe, like all good programmers, we should be testing and profiling our code and implementing the parts that make the most sense for our needs. The bottom line is that the Tidyverse is a good set of packages and you are free to use them or not. There isn’t some blood feud (like vi vs. emacs) between users and non-users, we all get along just fine. There are dozens of great tutorials on how to learn R that don’t use the Tidyverse, and RStudio is under no obligation to offer a full course on every possible way to learn R. Any R user is also a competent Google user. It’s fine to be opinionated. But, a professor as respected as Norm Matloff should be careful of how they say things, or they risk souring their students on a set of packages that might be very useful in the future.
- educationdata 7y ago
  "cherry picked"? There is no doubt that data.table is generally way faster than dplyr, unless you cherry pick a few use cases. The problem is that dplyr is much slower than data.table and RStudio is promoting tidyverse too much to make the slower choice a default for many users. The article is 100% correct on this issue.
  - cwyers 7y ago
    Depending on the size of your data, you might not care that dplyr is slower than data.table. If you're better at writing/composing dplyr, you can often make up the speed difference between the two in terms of the savings in time spent writing and reading code. And if your data is that large, there are solutions like dbplyr out there to run dplyr code on various backends and offload the computation outside of R.
antipaul 7y ago
dplyr is an "advanced" topic? It's, like, the easiest data manipulation library in any language.
- cwyers 7y ago
  People who were big into R pre-Tidyverse have this tendency to view the way they manipulated data frames pre-Tidyverse as the "basics," and Tidyverse as the "advanced" method. I think it does anyone learning R for the first time an incredible disservice, for the reason you state: dplyr is fantastic. But the old guard is (rightly) afraid that if people start with dplyr, they'll never get around to learning what base R provides for manipulating data frames. Which is good in my view, and nobody should have to learn that.
pgcudahy 7y ago
In this article and others I've read, people complain about the tidyverse's lack of performance, but I think that places too much emphasis on speed of execution versus speed of development. As an academic, most of the R users I know only code as a portion of their scientific projects. Besides data analysis we're doing data collection, manuscript writing and grant writing. The tidyverse's more english-like syntax (eg select() versus `[`) and following a series of pipes rather than unnesting ten sets of brackets makes it so much easier to come back to my code after a few weeks or months and pick up where I left off, or to work with other people's code. My time is more valuable than computer time so the tidyverse is my choice.
- FranzFerdiNaN 7y ago
  This is my experience as a data engineer/analyst. I work in the healthcare space and my analyses are run on datasets that are at the uppermost half a million rows. Yes, data.table is faster than dplyr/tibbles, but i dont care. Like you said, most of my time is spent on expressing my thoughts into code, not running the code. Tidyverse really simplifies it compard to base R. Its good to know tools like data.table exist though, people shouldnt think the tidyverse is the only way to do things.
sghi 7y ago
As someone who uses and teaches R extensively (and loves the tidyverse) the tidyverse is so much easier to teach to people without any programming experience and who have very little faith in their tech or maths skills (which was me as well). The tidyverse just 'made sense' to me when I started using R for the first time a few years ago, and now I love using R and programming. On the other hand, some of my ex-classmates learnt base R (because that's what we were taught) and found it hard, didn't learn anything properly, and now still think R or other programming languages are opaque and hard. I'm not particularly fussed if Statistics Profs prefer data.table to dplyr or base R to tidyr, I know what is easier to teach, understand and use for me and a lot of other ecology/bio students and people.
- hardboiled 7y ago
  I concur with your sentiments having cultivated data science teams from the ground up with diverse educational backgrounds. Programming in base R is more akin to assembly language and has accreted a babel of inconsistencies that make it difficult to teach and learn. Learning base R isolates you into a Galapagos island of academics who are either ignorant of the needs of data workers or too elitist to engage with those not in their priesthood. Learning Tidyverse is a considerably better transition for learning other languages, frameworks, and libraries. Functional programming is closer to algebra than indexing into data structures with magic numbers. I've found more success teaching functional pipelines of data structures using the idioms in Tidyverse as a general framework for data work than base R. Abstraction has a cost but for learning it is the appropriate cost. I sense that much of this `monopolistic` fear mongering is really about feeling out of date.
- mespe 7y ago
  "I know what is easier to teach, understand and use" I think this really depends on the end point. If you want to learn to read data into R and do basic manipulations, plotting, modeling, etc., the Tidyverse absolutely has a lower bar to entry. Once you get into writing functions, it gets a little trickier. Knowing some of the base R programming concepts and skills will make you much more efficient. If you start debugging and profiling code, only knowing the Tidyverse becomes a liability because you fundamentally do not understand R's computational model (the Tidyverse does not follow it). Hence, if your end goal is to write and debug functions in R, the steeper learning curve of base R can more than pay off. If not, then the Tidyverse's low bar to entry can be more attractive.
atheriel 7y ago
Most of these tidyverse vs. data.table arguments end up sounding a bit irrelevant to me. They tend to focus on syntax, which is largely down to preference, or performance, which is really only important once in a while. And they seem oddly focused on the need to choose one or the other, when it seems obvious that all are good choices with various tradeoffs. I tried my hand at saying something more original about this debate recently: https://unconj.ca/blog/for-and-against-data-table.html https://unconj.ca/blog/for-and-against-data-table.html On the other hand, it's nice to see some criticism of RStudio and Hadley, even if it does stray into the conspiratorial. They've done a lot for the R community, but they do have some obvious blind spots.
- vharuck 7y ago
  It's not that RStudio has nefarious motives that cause harm, it's that they have neutral and sometimes benevolent motives with unforeseen consequences.
cwyers 7y ago
A friend pointed out to me that the author a few weeks ago put out a Python versus R comparison where he makes the rather incredible claim that the addition of some opinionated packages in the R ecosystem is more disunifying than the 2.7 to 3.x split in Python: https://github.com/matloff/R-vs.-Python-for-Data-Science https://github.com/matloff/R-vs.-Python-for-Data-Science
firstinstinct 7y ago
A related post is http://varianceexplained.org/r/teach-tidyverse/ http://varianceexplained.org/r/teach-tidyverse/
firstinstinct 7y ago
What I see clearly is that is not a good option to begin with R inferno, except when you are a fighter prepared to learn at any cost.
uptownfunk 7y ago
I guess I am what you would call an R power user. What’s really crazy is I never realized how fast and easy excel pivot tables were to work with. I know I know it sounds ridiculous. But if you do a lot data splicing and dicing, excel can actually get your cuts out way faster through pivot tables than writing R code. So if you use R for almost everything, give Excel a try as well. And the nice thing is this is even more powerful if you work with non data science folks, because then you can say I just did this in excel (where excel skills should be required by everyone in the company, and R for the data scientists) and they’ll probably just leave you alone the next time they need something since they will try to do it in excel first.
- inputcoffee 7y ago
  Agree 100%, but only if it is a one time activity. If you have to automatically pull files and do the operation several times, it is better to go with R (or python, or awk sed or whatever).
roenxi 7y ago
He is glossing over the major issue with base R, which is its tendancy to switch data types in a way that appears random to new users. As an undergrad I spent nights literally on the verge of tears debugging R code where the types had been mucked up by R's bizarre semantics. Claiming that data[foo] is the same as filter(data, foo) is not correct - the [] and [[]] operators have a lot of strange side effects depending on what argument is passed in. It is like saying map() and a for loop are the same because everything a map can do a for loop can do. That is true, but the single-mindedness of map is a major selling point because you know it is only going to do certain specific things. The issues here will disproportionately affect beginners who are likely to accidentally pass in the wrong thing - it needs to fail with an error or in a predictable way to best help them. Hadley's code fails sensibly. One of the major innovations of the tibble package is that it puts the types of each column into the headings when it prints the table out. Dr. Matloff is grossly underestimating how painful it is for beginners to, eg, figure out when they are dealing with a column of strings vs a column of factors. If a beginner copies code to someone else for help, the other person can actually tell what types are involved! It is like magic compared to base R. Given how ad-hoc the R community is and the paucity of people with programmer backgrounds, these minor usability improvements on controlling types are important. And anybody who cares about efficiency is using the wrong language. EDIT Also, this comment that RStudio is bypassing an Open Source project's leadership - that is one fo the major features of open source. It is usually a healthy sign that the project leadership are lost in the weeds. I recall a situation with broad parallels in the history of GCC for example; it didn't kill the project although I believe a bad codebase died.
- nonbel 7y ago
  > "Dr. Matloff is grossly underestimating how painful it is for beginners to, eg, figure out when they are dealing with a column of strings vs a column of factors." This is just sapply(df, class)... Is this not found in R 101 material that beginners are likely to find?
  - ACow_Adonis 7y ago
    Pay no attention to the myriad of other apply functions hiding behind the curtain and the difficulty in remembering what each of them does and why you can't just use map, eh? :) So to answer your question, no, its not just simply that.
firstinstinct 7y ago
at the end of the comments of the post http://varianceexplained.org/r/teach-tidyverse/ http://varianceexplained.org/r/teach-tidyverse/ compare base R: var1 <- "pounds" var2 <- "wt" mtcars[[var1]] <- mtcars[[var2]]/1000 Now hadley solution comment: You can now express that sort of thing fairly elegantly with dplyr: var1 <- sym("pounds") var2 <- sym("wt") mtcars <- mutate(mtcars,!!var1 := !!var/100) Watch your steps: !!, :=, sym. Thats a lot of added complexity to be able to use column names, and to top it all, the use of non standard evaluation make things very complex when one go a little away from basic EDA.
- roenxi 7y ago
  Well, strictly literately to this case the tidyverse command would be: mtcars %<>% mutate(pounds = wt / 1000) Which is obviously simpler, and quite likely what a beginner is actually trying to do. The real complaint - "tidyverse doesn't let us use variables to name columns!" - is fair enough, following quasiquotation is hard work. If you need that, drop back to base R. However, it is harder to teach because all the common operations (select, filter, etc) will involve some combination of []/[[]] and relatively hard to figure out which one a piece of code is trying to do. Except for the lucky few who have brains wired to think in terms of arrays and database relations, someone learning the language is unlikely to thank you for that.
mespe 7y ago
I teach and consult on R and data science. I had the privilege of learning R from one of R's core developers. My students often ask why I don't use the Tidyverse. The answer is because I don't need to - I can do everything the Tidyverse does and so much more in base R. This article only briefly touches on what I think is the biggest issue with the Tidyverse. The Tidyverse is incredibly limiting. The "tidy" workflow hides many details of the language, which leads many users to think all data has to be in a tidy "data.frame" (or now tibble) and organized just so. The functions the Tidyverse authors have chosen to implement are seen as the limits of R's capabilities. Every data problem has to be forced into the Tidyverse box (or it is impossible). I cannot tell you how many Tidyverse scripts I have seen where 90% of the operations are munging the data into the correct format for the Tidyverse functions, when the original data can be handled with a single base R function (e.g. lapply). Most Tidyverse users accept this as just the way R is. Most R users who only learn the Tidyverse never hit its limits, and for them the Tidyverse is perfectly OK. Those that do either resign themselves to the perceived limits of R, or (hopefully) start learning some base R. One friend who took the latter path once exclaimed to me "Logical subsetting is amazing!?!" - this is a foundation piece of the language. That she went 2+ years in the "Tidyverse" without even knowing it was an option was eye opening to me. To its credit, the Tidyverse is empowering - with very little programming knowledge a beginner can do a lot of data science. However, the majority of these people get stuck as "expert beginners". Some of these become fierce advocates of the Tidyverse without truly understanding base R. Meanwhile, none of the truly advanced R users I encounter use the Tidyverse. ed.typo
- FranzFerdiNaN 7y ago
  "Meanwhile, none of the truly advanced R users I encounter use the Tidyverse." This is most likely because truly advanced R users have been using R since far before the Tidyverse existed. A whole new generation of R users is being brought up with the Tidyverse, so im curious to see how the situation will be in 10 years time.
  - mespe 7y ago
    This is not entirely true - many are of the same "generation" as me. To be clear, I don't actually consider myself an advanced user. To me, many of these advanced users are pushing against the limits of the language in a way you don't see with run of the mill data science. That said, I have heard my mentor say "I don't get the point of <insert tidyverse function> - I did this 20 years ago."
phonebucket 7y ago
I initially viewed this article with skepticism, rather liking the functional style of the Tidyverse. However, the claims the author makes are entirely valid. Data table is a fantastic package, and easy to learn. Tidyverse really has split the R community in disparate sets of R users. To my mind, one of R’s greatest strengths is its meta programming capabilities. It is this that allows such broad paradigms to exist within the same language. In that, the challenges that R is presented with is similar to languages like Lisp dialects and Scala. I don’t have solutions, but the author has convinced me that a problem is indeed there.
minimaxir 7y ago
From personal experience, I learned R during college as a part of my stats courses, before tidyverse even existed. After doing a few personal projects using base R with complex manipulations, I would have quit using R entirely if it were not for tidyverse/dplyr. I'm a pragmatic software engineer: I prefer to get a data analysis job as fast as necessary, and IMO there isn't much merit in making things more complicated by using base R just for the sake of base R. It's definitely not faster, both in cognitive load and in speed. In my data science work there hasn't been a single case where I've needed to fall back to base R, or wanted to. Pipes are being undersold here: the examples only show a single piped action, whereas pipes shine when you have to do 5+ operations in a single manipulation (I've seen base R code that does that by just having a lot of redundant df assignments and it's a pain to read).
- tvladeck 7y ago
  > a lot of redundant df assignments either that or a profusion of variables going into your global environment, which is a massive pain to track!
- kgwgk 7y ago
  This also looks painful: https://community.rstudio.com/t/magrittr-inside-a-package/2033 https://community.rstudio.com/t/magrittr-inside-a-package/20...
  - minimaxir 7y ago
    That's an interesting caveat with pipes/magrittr (I didn't know of it), although it's out of scope of this debate since this discussion is just about how tidyverse uses pipes. In my experience if there's a package that's required in a script/R Notebook outside of tidyverse, I just import the entire library at the start of the file Python-style (to make the dependency obvious), which would avoid this issue.
    - kgwgk 7y ago
      If you claim that something like the_data <- read.csv('/path/to/data/file.csv') the_data <- subset(the_data, variable_a > x) the_data <- transform(the_data, variable_c = variable_a/variable_b) the_data <- head(the_data, 100) is a pain to read compared to the_data <- read.csv('/path/to/data/file.csv') %>% subset(variable_a > x) %>% transform(variable_c = variable_a/variable_b) %>% head(100) I thinks it’s fair to mention how by making things more complicated by using pipes you expose yourself to other issues. (In my opinion the first variant is not less readable, and it has the advantage of allowing any of the operations to be removed by commenting the corresponding line, while in the second case to remove the last operation you also need to remove the piping operator in the preceding line.) [Edit: I’m not sure I understand your “out of scope” remark. You don’t mean that magrittr is outside of tidyverse, do you?]
inputcoffee 7y ago
I was waiting for the critique... but I never quite saw it. Imho, the data problem Tidyverse is trying to solve is basically the ones we face in a database. So, select, join, inner join and so forth. Show me all the rows in this datatable where the 4th columm is larger than the 6th column and the number itself is odd. Something like that. There might be other ways to do it, but you want your select, filter, summarize, mutate etc functions to all work with each other, pipe to each other and be compatible. Maybe there is a better way to do all this -- I haven't seen it but I am not an expert -- but you have to show that to me. So, in base R, walk through a set of example of mutating, joining, filtering and so forth, and show me how they are all easier. Then I'll say, wow there is an alternative to this Tidyverse thing. But in lieu of that demo, this felt more like an intro to a complaint than an actual complaint. Edit: Also, its funny that Wickham is (apparently) such a nice fellow that people go out of the way to be nice to him in critiques.
- barcadad 7y ago
  His critique is more about the impact of the full ecosystem effect of the Tidyverse, not what you are referring to, which is just the dplyr semantics. The Tidyverse demands that it's many related packages use tidy data principles and lock users into that approach, which differs from base-R. Much of this discussion is really just a debate about dplyr and magrittr rather than the fragmentation that the broader tidyverse has brought on. All that said, I agree with many commenters that the Tidyverse's improvements to speed of development can more than offset the speed of execution issues, at least for small-to-medium datasets.
  - RosanaAnaDana 7y ago
    I think any one making the point of speed of development have seriously missed the boat. Lets be real: tibbles suck. Once you get the hang of data.table syntax for matrix operations, its superiority becomes impeccably clear. I run a data science group a large geospatial company and we develop day in and day out in R and python. We've purged tidyverse as much as possible from all of our code base. We've moved completely over to data.tables, which make the vast majority of the tidyverse irrelevant.
kgwgk 7y ago
> Ironically, though consistency of interface is a goal, new versions of Tidy libraries are often incompatible with older ones, a very serious problem in the software engineering world. I was (sadly) amused by a recent comment on HN linking to some tweets celebrating that some R code still ran four years later.
- tylermw 7y ago
  That was my comment, and the point wasn't that the R code still ran four years later--the point was that R Core's focus on backwards compatibility (in reference to R being hamstrung via its focus on "compatibility with S") isn't a bad thing. You're never going to get the 2.x - 3.x schism that python had with future version of R, assuming R core continues with that philosophy.
  - deleted 7y ago
    [deleted]
  - kgwgk 7y ago
    I completely agree with your point and I think saying that R is keeping compatibility with S doesn’t make much sense. R is keeping compatibility with R, as it should. Maybe I lack context, but those tweets looked to me as if someone said “I can run this four year old game in the latest release of Windows!” or “Three versions of Office later I can still open my excel files!”
stilley2 7y ago
I recently tried to do some stuff in R with tidyverse, and was not a fan. I'm no expert, but the tidyverse's frequent use of nonstandard evaluation drove me crazy. It makes it so much more difficult to write functions encapsulating tidy functions. However, I never see people complain about this, so maybe I'm doing something wrong or not grokking something...
- vincent-toups 7y ago
  I really hate this. And its so unnecessary!
- _Wintermute 7y ago
  It's because RStudio are pushing R/tidyverse as an interactive datascience environment, so most of the tutorials you see will be aimed at users sending a few lines a time to the REPL. Programming with the tidyverse and its non-standard evaluation really is a pain.
- jointpdf 7y ago
  There is a new operator, {{ or “curly curly”, aimed at dealing with this. It’s part of the rlang (0.4) package and looks snazzy / easy to grasp. Check it out: https://www.tidyverse.org/articles/2019/06/rlang-0-4-0/ https://www.tidyverse.org/articles/2019/06/rlang-0-4-0/
  - RosanaAnaDana 7y ago
    Yuck.
    - jointpdf 7y ago
      Have anything more constructive to say?
- malshe 7y ago
  No, you are not doing anything wrong. It's a real pain.
zhdc1 7y ago
I am a feverent user of data.table and try to avoid dplyr when ever possible. I have colleagues who are exactly the opposite. No one cares. If you want efficient code, try not to use dplyr. If you want readable code, data.table isn't the best answer. If you want speed, there are better languages than R to choose from, even with Rcpp. If you want to get something done, go with whatever you know best and will do the job. Avoid Stata.
ellisv 7y ago
As an R user of some 10+ years, I think some of this is spot on and some of it is just silly. Pretty much everyone knows the Tidyverse is slow to run, but many of us find it faster to write and very readable because it avoids one-time-use assignments (i.e. in `a <- f(x); g(a)` the variable `a` is only used once). Naming things is hard. Prof. Matloff's point is well taken though - you don't need the Tidyverse and it doesn't run faster. Fair criticism. > The "star" of the Tidyverse, dplyr, consists of 263 functions. While a user initially need not use more than a small fraction of them, the high complexity is clear. Every time a user needs some variant of an operation, she must sift through those hundreds of functions for one suited to her current need. > By contrast, if she knows base-R (not difficult), she can handle any situation. Yet base R has some 1,346 functions. Another 455 functions in `stats`. I'm not sure the quantity of functions has as much bearing on learning as the quality and consistency. So this seems like a silly argument to me. Mostly I think Prof. Matloff is forgetting that base R is kinda terrible and no one else has done much about it aside from RStudio.
- cwyers 7y ago
  > Mostly I think Prof. Matloff is forgetting that base R is kinda terrible and no one else has done much about it aside from RStudio. I think he'd disagree with the contention that base R is kinda terrible. Like, I'm pretty sure he's wrong. But this isn't something he's forgotten, it's something he's never believed. The point about the Tidyverse and RStudio is that RStudio doesn't make any of its revenue from the Tidyverse. They distribute it free. There are no upsells for any of it -- I can't go out and buy ggplot2 add ons, I can't buy dplyr support contracts. There's no money to be made on the Tidyverse, and plenty of resources thrown at it. So what is RStudio selling? They sell an IDE, they sell servers to host things built in R, they sell a package manager solution, they offer most of these things as a SaaS offering as well (although rstudio.cloud is still in trial and they haven't started charging yet). In other words, RStudio offers tooling for people who write R. They don't really care what kind of R you're writing, from a financial perspective. They get paid the same if you're using RStudio to write stuff using data.table as they get paid if you use dplyr. RStudio is making a bet that more people will want to write R code if they make R a more pleasant language to work in. And anecdotally, they're absolutely right -- I tried to learn R before the Tidyverse, and I hated every moment of it. I like post-Tidyverse R (although I got into it when people still called it the Hadleyverse). I know several other people with similar experiences to mine.
ahurmazda 7y ago
With respect, its laughable to think that anyone working on big-data with R has _not_ heard of `data.table`. When your livelihood depends on it, you tend to find it out one way or the other. However, I'd claim most folks don't deal with big data or even medium data! [Aside: I chuckled at the 1e5 columns on the plot]. Personally, at the small-to-medium range, all languages are fast enough. It boils down to ergonomics for me. I _detest_ the subscript notation. It slows me down -- not just productivity-wise but also how I read or reason-about my code. Not surprisingly, I prefer `dplyr` over `data.table` (base R or `pandas`)

137 points by porker 7y ago

When I'm writing the book (www.anotherbookondatascience.com) I always try to avoid the usage of tidyverse and got some criticism. But now I feel supported.

kickout 7y ago

Decent critiques in my opinion. I was trained on/learned base-R myself ~10 years ago and love the "tidyverse" (although the author is correct, data.table is superior for big data and its not particularly close). Can't imagine people that only know tidyr/dplyr/etc without knowing base-R, seems like those people would get exposed quickly. I enjoy base-R solutions where I can. I use data.table when my data sets are medium-ism. But like most people, my data is small and people just call it big and thus the attraction to dplyr in particular. Those chained pipes are addicting. Always love seeing R discussion on HN, seems to be taboo though (not a 'real' language and all that).

Bootvis 7y ago
I don’t even understand the not a real language critique, R: - is based on scheme, a HN favourite; - integrates very well with C++ through RCpp allowing you to do whatever you want. 9/10 times someone already went through the trouble for you.
- kickout 7y ago
  I agree; which is why I generally get confused when I see that criticism. This is only an assumption though; maybe its just a vocal minority that don't like R or I only notice the negative comments. Who knows
  - erikpukinskis 7y ago
    For what it’s worth, I am a JavaScript programmer and I feel like HN shits on JS All. The. Time. Some people just like to shit on languages. I think it’s just insecurity. They worry a lot about whether they’re a good enough developer, so they try hard to be on the “most advanced” platform they can figure out, and then need to talk down to the other platforms. Secure, talented, older developers, in my experience, tend to be open to whatever technology is most appropriate. They have confidence in their ability to learn new things. And see the good traits of any tech, as well as the bad one—a necessary skill if you’re going to be the kind of developer who can make traction in any situation. The platform fetishists can only really work from inside their special tower.
    - newen 7y ago
      Javascript is just a terrible language and programming environment though. Anyone can criticize people who disagree with them by calling them "insecure". It's a meaningless argument.
- thom 7y ago
  I don't think there's any shame in R effectively being a good, productive DSL for a lot of stats and viz work, built around fast stuff written in C++. I also think this is largely true of a lot of Python as well, tbh. In the future my hope is that projects like Arrow end up with even more of the workload being taken off R's shoulders.
  - vincent-toups 7y ago
    I'm a data scientist and, frankly, I really strongly prefer R over Python. Its mostly the whitespace in Python, but the language is also slow and crufty as heck. R, if you use it responsibly, feels a lot more like your standard, lexically scoped, dynamically typed language.
- atheriel 7y ago
  Absolutely. I usually respond to the "R-is-not-a-real-language" line with something like "R is syntactic sugar on a Lisp where the atoms are APL arrays". It's plenty interesting from a computer science perspective, if you bother to look. For instance, why does R use <- instead of = for assignment? Because initial versions (of S) predate C -- developed down the hall -- the language that first introduced = for assignment.
- nerdponx 7y ago
  The R standard library is kind of weird and hard to use for "general purpose" programming. It's very clearly a domain specific language. Also, it's unbelievably slow for basic operations like looping, function calls, and variable assignment. It's literally orders of magnitude slower than Python (I've tested it). Unlike its fellow C-flavored-Lisp Javascript, R retains an extreme level of homoiconicity, Which apparently makes the language very difficult to compile or optimize. I don't know how e.g. SBCL does it, but over the lifetime of the R language no one has managed to implement a non-trivial subset of R that performs better than GNU R (fastR was never finished to my knowledge). So you are basically relegated to writing code in C, Fortran, or C++ if you need even decent performance. Otherwise you are stuck using "vectorized" operations like lapply(), which are fine but make for a jarring experience if you're coming from other languages. So of course it's a "real" language in the sense that Bash is a real language. But it's ultimately and fundamentally a domain-specific scripting language and I don't know if there is a way around that.
  - kgwgk 7y ago
    Are you aware of Radford Neal’s pqR? https://radfordneal.wordpress.com/ https://radfordneal.wordpress.com/
    - nerdponx 7y ago
      I had heard of it in the past but I didn't realize it was in a finished state and keeping pace with recent R versions. I'll read up on it, thank you. Edit: looks like it's still a full major version behind? To be fair, there weren't many significant changes between R 2.x and 3.x.
      - kgwgk 7y ago
        Well, it may not be keeping up with the new versions but at least it does cover a non-trivial subset of the standard implementation. :-) I have not tried it in many years and I don't know how much faster or compatible it is. Nevertheless, I think it's a good thing that an "outsider" is looking at performance issues. As far as I know it's essentially a one-man's effort. One could imagine that RStudio or Microsoft would be able to improve R performance substantially if they wanted to.
- epistasis 7y ago
  The "not a real language" critique often comes from those without any knowledge of Scheme; "real" languages are like C# or Java, and maybe Python, even though the lisp-like properties of Python are not so much appreciated. As C# and Java have gotten more functional aspects over the years, I've seen less and less of the "not a real language" criticism.
  - deleted 7y ago
    [deleted]
- vincent-toups 7y ago
  "Based on Scheme" really, profoundly, oversells it. R's idea of Scheme is deeply bizarre and profoundly outdated.
- GlennS 7y ago
  There's really not much Lisp/Scheme left in R these days. Function arguments/environments are still pairlists: a type of cons cell. You can manipulate the parameters to a function as symbols before you evaluate them. This is useful — many very good R libraries all make heavy use of it — and certainly very interesting. Lispers probably know it as quote/antiquote. I suppose we could also point that, much like most Lisps, R gives you plenty of object systems to choose from... And that's about it for R-as-a-Lisp. It's also a really bad language for working with tree-structured data, something Lisps normally excel at.
- laichzeit0 7y ago
  The last time I tried putting a REST API in front of my predictive model, I used plumbr. I also then learned that the R runtime was single threaded, so that only 1 request could be processed at a time. I don’t know how you can overcome this, and it makes things significantly difficult to put into a real production environment. I’m sure someone will chime in that there are different runtimes, like Microsoft’s (does this run on MacOS? If not, how can I use it to develop and test on my laptop) or some expensive RStudio solution. The fact is you have none of those problems in e.g., Python. Although I would prefer to use Caret for many predictive modeling tasks, it is non-trivial to take the R runtime to production. That was what I walked away with.
atheriel 7y ago
I'm in a similar boat, using base R, data.table, or tidyverse as needed. But over time I've found that I respect, more and more, the good decisions that have been made in base R.

c3534l 7y ago

Tidyverse was nearly the only thing I really liked about R. The syntax is quirky, it's full of cruft, and unpleasant to make full programs out of. But if I used a new package, I knew instantly how to use it.

bagrow 7y ago

I think the same thing is happening in Python for data science due to Pandas. It’s a great package, but many data scientists are only able to manipulate data using Pandas, and know little “base” Python.

jphoward 7y ago
That sounds incredible to me as using pandas without falling into the traps of chained indexing seems so much harder than 90% of Python.
- telchar 7y ago
  I've been planning to go from "can work it out with copious examples" to "knows when to use apply, transform, etc. off the cuff" level of knowledge in Pandas - do you have any suggestions on good resources on this? I'd like to understand better why the data structures work the way they do and thus have an intuition on what operations to use when. The O'Reilly Python Data Science Handbook[0] seems like it might be useful here, but I'm not sure if it is still up to date. [0] https://www.oreilly.com/library/view/python-data-science/9781491912126/ https://www.oreilly.com/library/view/python-data-science/978...
  - squaresmile 7y ago
    I find the pandas documentation pretty good [0]. Actually, I find the documentation, official tutorials of pandas, numpy, matplotlib all pretty good. Each package has their own idiosyncrasies but I think the documentation and code examples cover them well enough. The OReily book should not be that outdated if at all. It also covers other essential tools. [0] https://pandas.pydata.org/pandas-docs/stable/getting_started/index.html https://pandas.pydata.org/pandas-docs/stable/getting_started...
mruts 7y ago
You can’t really do any data analysis without pandas in python though. In fact python is a terrible language and the only reason anyone should use it is for access to sklearn, scikit, pytorth, and pandas. Hopefully Julia will be able to unseat python for data analysis in the future.

dracodoc 7y ago

One reason dplyr was promoted is that you can connect to different server backend like sparkr, sql etc, which is the enterprise direction RStudio aiming at. In the other hand, data.table is your friend when you are using your own machine. With more and more memory and cpu power available, its benefits are actually increasing. I have been using data.table from almost day one, it does worth more recognition. The syntax is a little bit hard in the beginning but can be grasped after some efforts. I often feel that my exploration code will be much more tedious look if written in dplyr style.

vasili111 7y ago
If my university uses Enterprise RStutio on grid it is better to stick with tidyverse rather than data.table?
- Tarq0n 7y ago
  It doesn't make a difference. Rstudio's products aren't integrated with their libraries in particular.

thom 7y ago

Wow, the idea that the Tidyverse is somehow burdened with an overabundance of computer science or functional programming is just hilarious to this programmer who has had to maintain, debug and optimise other people's R code.

jrumbut 7y ago
I think this is what the author is getting at, that correct tidyverse usage requires greater experience and knowledge and creates difficult to debug and optimize code when used incorrectly (and is easy to use incorrectly). I tend to agree. I've found tidyverse code has a write-only quality to it. Since I'm going to see more of it I plan to dive into the inner workings of at least dyplr and purr. That said it is hard to deny what Hadley Wickham has done for the R community and I can't write off the idea that he sees a bigger picture I'm missing.
- hardboiled 7y ago
  Tidyverse code is not write-only; it is designed to be mostly read-only where the level of abstraction is at the level of the domain, which in this case is data frames and data pipelines akin to the same constructs in relational algebra/SQL or data processing (map/reduce). Correct usage requires learning the right abstractions, but fortunately these abstractions are shared across language communities and frameworks. If you are doing any data processing work and you do not know about foundational functional concepts ie map/reduce then I would argue the code you write is less readable, overly focused on the idiosyncracies of how to do something rather than how entities and data are related to each other - their logic and structure. The main optimization is to be correct and expressive of one's intention and purpose. If you need performance use Julia.
  - jrumbut 7y ago
    I understand that's the goal, but it hasn't been what I've seen from user's code. I think the issue, which the article describes, is that the tidyverse has too large a surface area for something aimed at very smart people who are not software engineers and don't have time to become one. So they end up with a subset of functionality they can get stuff done with, but then they need to collaborate with someone who uses a different subset. That can be messy. I think in an environment where everyone learns R from the same course/book/MOOC/whatever (or have done lots of different sorts of programming) and the organization can impose a style guide the tidyverse approach would be great, but when you have people coming from all sorts of places and backgrounds I don't think it's a good fit.

jhbadger 7y ago

At least in my case, although I do like some of the tidyverse (ggplot is tidyverse after all, and probably my favorite plotting system in any language), I find the "tibble" data structure and its disdain for row names really annoying. Many, many, packages which I use expect a data frame with row names and likewise output such. So I am constantly converting back and forth between tibbles and data frames.

Bootvis 7y ago
The claim that ggplot2 is tidyverse can be disputed. It existed before and works fine outside of it.
- minimaxir 7y ago
  ggplot2 is under the tidyverse umbrella, and will be installed/loaded with library(tidyverse). The original article is more about dplyr though.
- hadley 7y ago
  You are of course free to dispute what is in the tidyverse, but I pretty strongly believe that it is part of the tidyverse
  - RosanaAnaDana 7y ago
    Seeing as the tidyverse is pretty much your invention, its well within your purvue to define what is in and what is out. I do think, respectfully, that both the op and the general sentiment article have the right of it. Data.table and data.table-esque notation represent such an improvement of tibble/dplyr, that within my company, we're making a concerted effort to purge all tidyverse packages from general use (less ggplot). When new developers come on, if they are coming from tidyverse, their first task will be something involving pipes and data.table. Tidyverse was fine in school. It doesn't pass muster in production, at least not in our work. Data.table syntax is simpler, easier to read, easier to teach, and orders of magnitude faster. It plays nicer with other packages than the tidyverse (if it fits into a DF, it almost always fits into a DT, and i've never met a tibble that I didn't wish was a data.table), and since almost all of our datasets are 10's to 1000's of millions of lines long, the decision was really made for us.

throw20102010 7y ago

I think he’s exaggerating the effect of some of these claims. People who “grew up” learning R+Tidyverse will discriminate against non-Tidyverse users? Give me a break. You are either an advanced or competent or novice R user. You either understand how pipes (%>%) work or you don’t. I doubt that anyone is going to be denied a job because they are an amazing R programmer but they just don’t have the experience with a particular set of packages, especially those that are as well implemented and easy to learn as the Tidyverse. I can totally imagine someone not getting a job because they are a crap programmer, and then blaming it on something else. Or I can also imagine someone saying that the Tidyverse packages suck, and then be denied a job because of their attitude. I could make a similar argument for most of the substantive claims in his post. Some of these examples (dplyr vs. data.table) are cherry picked. I have several of my own examples where read_csv is way faster than read.csv, so maybe, like all good programmers, we should be testing and profiling our code and implementing the parts that make the most sense for our needs. The bottom line is that the Tidyverse is a good set of packages and you are free to use them or not. There isn’t some blood feud (like vi vs. emacs) between users and non-users, we all get along just fine. There are dozens of great tutorials on how to learn R that don’t use the Tidyverse, and RStudio is under no obligation to offer a full course on every possible way to learn R. Any R user is also a competent Google user. It’s fine to be opinionated. But, a professor as respected as Norm Matloff should be careful of how they say things, or they risk souring their students on a set of packages that might be very useful in the future.

educationdata 7y ago
"cherry picked"? There is no doubt that data.table is generally way faster than dplyr, unless you cherry pick a few use cases. The problem is that dplyr is much slower than data.table and RStudio is promoting tidyverse too much to make the slower choice a default for many users. The article is 100% correct on this issue.
- cwyers 7y ago
  Depending on the size of your data, you might not care that dplyr is slower than data.table. If you're better at writing/composing dplyr, you can often make up the speed difference between the two in terms of the savings in time spent writing and reading code. And if your data is that large, there are solutions like dbplyr out there to run dplyr code on various backends and offload the computation outside of R.

antipaul 7y ago

dplyr is an "advanced" topic? It's, like, the easiest data manipulation library in any language.

cwyers 7y ago
People who were big into R pre-Tidyverse have this tendency to view the way they manipulated data frames pre-Tidyverse as the "basics," and Tidyverse as the "advanced" method. I think it does anyone learning R for the first time an incredible disservice, for the reason you state: dplyr is fantastic. But the old guard is (rightly) afraid that if people start with dplyr, they'll never get around to learning what base R provides for manipulating data frames. Which is good in my view, and nobody should have to learn that.

pgcudahy 7y ago

In this article and others I've read, people complain about the tidyverse's lack of performance, but I think that places too much emphasis on speed of execution versus speed of development. As an academic, most of the R users I know only code as a portion of their scientific projects. Besides data analysis we're doing data collection, manuscript writing and grant writing. The tidyverse's more english-like syntax (eg select() versus `[`) and following a series of pipes rather than unnesting ten sets of brackets makes it so much easier to come back to my code after a few weeks or months and pick up where I left off, or to work with other people's code. My time is more valuable than computer time so the tidyverse is my choice.

FranzFerdiNaN 7y ago
This is my experience as a data engineer/analyst. I work in the healthcare space and my analyses are run on datasets that are at the uppermost half a million rows. Yes, data.table is faster than dplyr/tibbles, but i dont care. Like you said, most of my time is spent on expressing my thoughts into code, not running the code. Tidyverse really simplifies it compard to base R. Its good to know tools like data.table exist though, people shouldnt think the tidyverse is the only way to do things.

sghi 7y ago

As someone who uses and teaches R extensively (and loves the tidyverse) the tidyverse is so much easier to teach to people without any programming experience and who have very little faith in their tech or maths skills (which was me as well). The tidyverse just 'made sense' to me when I started using R for the first time a few years ago, and now I love using R and programming. On the other hand, some of my ex-classmates learnt base R (because that's what we were taught) and found it hard, didn't learn anything properly, and now still think R or other programming languages are opaque and hard. I'm not particularly fussed if Statistics Profs prefer data.table to dplyr or base R to tidyr, I know what is easier to teach, understand and use for me and a lot of other ecology/bio students and people.

hardboiled 7y ago
I concur with your sentiments having cultivated data science teams from the ground up with diverse educational backgrounds. Programming in base R is more akin to assembly language and has accreted a babel of inconsistencies that make it difficult to teach and learn. Learning base R isolates you into a Galapagos island of academics who are either ignorant of the needs of data workers or too elitist to engage with those not in their priesthood. Learning Tidyverse is a considerably better transition for learning other languages, frameworks, and libraries. Functional programming is closer to algebra than indexing into data structures with magic numbers. I've found more success teaching functional pipelines of data structures using the idioms in Tidyverse as a general framework for data work than base R. Abstraction has a cost but for learning it is the appropriate cost. I sense that much of this `monopolistic` fear mongering is really about feeling out of date.
mespe 7y ago
"I know what is easier to teach, understand and use" I think this really depends on the end point. If you want to learn to read data into R and do basic manipulations, plotting, modeling, etc., the Tidyverse absolutely has a lower bar to entry. Once you get into writing functions, it gets a little trickier. Knowing some of the base R programming concepts and skills will make you much more efficient. If you start debugging and profiling code, only knowing the Tidyverse becomes a liability because you fundamentally do not understand R's computational model (the Tidyverse does not follow it). Hence, if your end goal is to write and debug functions in R, the steeper learning curve of base R can more than pay off. If not, then the Tidyverse's low bar to entry can be more attractive.

atheriel 7y ago

Most of these tidyverse vs. data.table arguments end up sounding a bit irrelevant to me. They tend to focus on syntax, which is largely down to preference, or performance, which is really only important once in a while. And they seem oddly focused on the need to choose one or the other, when it seems obvious that all are good choices with various tradeoffs. I tried my hand at saying something more original about this debate recently: https://unconj.ca/blog/for-and-against-data-table.html https://unconj.ca/blog/for-and-against-data-table.html On the other hand, it's nice to see some criticism of RStudio and Hadley, even if it does stray into the conspiratorial. They've done a lot for the R community, but they do have some obvious blind spots.

vharuck 7y ago
It's not that RStudio has nefarious motives that cause harm, it's that they have neutral and sometimes benevolent motives with unforeseen consequences.

cwyers 7y ago

A friend pointed out to me that the author a few weeks ago put out a Python versus R comparison where he makes the rather incredible claim that the addition of some opinionated packages in the R ecosystem is more disunifying than the 2.7 to 3.x split in Python: https://github.com/matloff/R-vs.-Python-for-Data-Science https://github.com/matloff/R-vs.-Python-for-Data-Science

firstinstinct 7y ago

A related post is http://varianceexplained.org/r/teach-tidyverse/ http://varianceexplained.org/r/teach-tidyverse/

firstinstinct 7y ago

What I see clearly is that is not a good option to begin with R inferno, except when you are a fighter prepared to learn at any cost.

uptownfunk 7y ago

I guess I am what you would call an R power user. What’s really crazy is I never realized how fast and easy excel pivot tables were to work with. I know I know it sounds ridiculous. But if you do a lot data splicing and dicing, excel can actually get your cuts out way faster through pivot tables than writing R code. So if you use R for almost everything, give Excel a try as well. And the nice thing is this is even more powerful if you work with non data science folks, because then you can say I just did this in excel (where excel skills should be required by everyone in the company, and R for the data scientists) and they’ll probably just leave you alone the next time they need something since they will try to do it in excel first.

inputcoffee 7y ago
Agree 100%, but only if it is a one time activity. If you have to automatically pull files and do the operation several times, it is better to go with R (or python, or awk sed or whatever).

roenxi 7y ago

He is glossing over the major issue with base R, which is its tendancy to switch data types in a way that appears random to new users. As an undergrad I spent nights literally on the verge of tears debugging R code where the types had been mucked up by R's bizarre semantics. Claiming that data[foo] is the same as filter(data, foo) is not correct - the [] and [[]] operators have a lot of strange side effects depending on what argument is passed in. It is like saying map() and a for loop are the same because everything a map can do a for loop can do. That is true, but the single-mindedness of map is a major selling point because you know it is only going to do certain specific things. The issues here will disproportionately affect beginners who are likely to accidentally pass in the wrong thing - it needs to fail with an error or in a predictable way to best help them. Hadley's code fails sensibly. One of the major innovations of the tibble package is that it puts the types of each column into the headings when it prints the table out. Dr. Matloff is grossly underestimating how painful it is for beginners to, eg, figure out when they are dealing with a column of strings vs a column of factors. If a beginner copies code to someone else for help, the other person can actually tell what types are involved! It is like magic compared to base R. Given how ad-hoc the R community is and the paucity of people with programmer backgrounds, these minor usability improvements on controlling types are important. And anybody who cares about efficiency is using the wrong language. EDIT Also, this comment that RStudio is bypassing an Open Source project's leadership - that is one fo the major features of open source. It is usually a healthy sign that the project leadership are lost in the weeds. I recall a situation with broad parallels in the history of GCC for example; it didn't kill the project although I believe a bad codebase died.

nonbel 7y ago
> "Dr. Matloff is grossly underestimating how painful it is for beginners to, eg, figure out when they are dealing with a column of strings vs a column of factors." This is just sapply(df, class)... Is this not found in R 101 material that beginners are likely to find?
- ACow_Adonis 7y ago
  Pay no attention to the myriad of other apply functions hiding behind the curtain and the difficulty in remembering what each of them does and why you can't just use map, eh? :) So to answer your question, no, its not just simply that.

firstinstinct 7y ago

at the end of the comments of the post http://varianceexplained.org/r/teach-tidyverse/ http://varianceexplained.org/r/teach-tidyverse/ compare base R: var1 <- "pounds" var2 <- "wt" mtcars[[var1]] <- mtcars[[var2]]/1000 Now hadley solution comment: You can now express that sort of thing fairly elegantly with dplyr: var1 <- sym("pounds") var2 <- sym("wt") mtcars <- mutate(mtcars,!!var1 := !!var/100) Watch your steps: !!, :=, sym. Thats a lot of added complexity to be able to use column names, and to top it all, the use of non standard evaluation make things very complex when one go a little away from basic EDA.

roenxi 7y ago
Well, strictly literately to this case the tidyverse command would be: mtcars %<>% mutate(pounds = wt / 1000) Which is obviously simpler, and quite likely what a beginner is actually trying to do. The real complaint - "tidyverse doesn't let us use variables to name columns!" - is fair enough, following quasiquotation is hard work. If you need that, drop back to base R. However, it is harder to teach because all the common operations (select, filter, etc) will involve some combination of []/[[]] and relatively hard to figure out which one a piece of code is trying to do. Except for the lucky few who have brains wired to think in terms of arrays and database relations, someone learning the language is unlikely to thank you for that.

mespe 7y ago

I teach and consult on R and data science. I had the privilege of learning R from one of R's core developers. My students often ask why I don't use the Tidyverse. The answer is because I don't need to - I can do everything the Tidyverse does and so much more in base R. This article only briefly touches on what I think is the biggest issue with the Tidyverse. The Tidyverse is incredibly limiting. The "tidy" workflow hides many details of the language, which leads many users to think all data has to be in a tidy "data.frame" (or now tibble) and organized just so. The functions the Tidyverse authors have chosen to implement are seen as the limits of R's capabilities. Every data problem has to be forced into the Tidyverse box (or it is impossible). I cannot tell you how many Tidyverse scripts I have seen where 90% of the operations are munging the data into the correct format for the Tidyverse functions, when the original data can be handled with a single base R function (e.g. lapply). Most Tidyverse users accept this as just the way R is. Most R users who only learn the Tidyverse never hit its limits, and for them the Tidyverse is perfectly OK. Those that do either resign themselves to the perceived limits of R, or (hopefully) start learning some base R. One friend who took the latter path once exclaimed to me "Logical subsetting is amazing!?!" - this is a foundation piece of the language. That she went 2+ years in the "Tidyverse" without even knowing it was an option was eye opening to me. To its credit, the Tidyverse is empowering - with very little programming knowledge a beginner can do a lot of data science. However, the majority of these people get stuck as "expert beginners". Some of these become fierce advocates of the Tidyverse without truly understanding base R. Meanwhile, none of the truly advanced R users I encounter use the Tidyverse. ed.typo

FranzFerdiNaN 7y ago
"Meanwhile, none of the truly advanced R users I encounter use the Tidyverse." This is most likely because truly advanced R users have been using R since far before the Tidyverse existed. A whole new generation of R users is being brought up with the Tidyverse, so im curious to see how the situation will be in 10 years time.
- mespe 7y ago
  This is not entirely true - many are of the same "generation" as me. To be clear, I don't actually consider myself an advanced user. To me, many of these advanced users are pushing against the limits of the language in a way you don't see with run of the mill data science. That said, I have heard my mentor say "I don't get the point of <insert tidyverse function> - I did this 20 years ago."

phonebucket 7y ago

I initially viewed this article with skepticism, rather liking the functional style of the Tidyverse. However, the claims the author makes are entirely valid. Data table is a fantastic package, and easy to learn. Tidyverse really has split the R community in disparate sets of R users. To my mind, one of R’s greatest strengths is its meta programming capabilities. It is this that allows such broad paradigms to exist within the same language. In that, the challenges that R is presented with is similar to languages like Lisp dialects and Scala. I don’t have solutions, but the author has convinced me that a problem is indeed there.

minimaxir 7y ago

From personal experience, I learned R during college as a part of my stats courses, before tidyverse even existed. After doing a few personal projects using base R with complex manipulations, I would have quit using R entirely if it were not for tidyverse/dplyr. I'm a pragmatic software engineer: I prefer to get a data analysis job as fast as necessary, and IMO there isn't much merit in making things more complicated by using base R just for the sake of base R. It's definitely not faster, both in cognitive load and in speed. In my data science work there hasn't been a single case where I've needed to fall back to base R, or wanted to. Pipes are being undersold here: the examples only show a single piped action, whereas pipes shine when you have to do 5+ operations in a single manipulation (I've seen base R code that does that by just having a lot of redundant df assignments and it's a pain to read).

tvladeck 7y ago
> a lot of redundant df assignments either that or a profusion of variables going into your global environment, which is a massive pain to track!
kgwgk 7y ago
This also looks painful: https://community.rstudio.com/t/magrittr-inside-a-package/2033 https://community.rstudio.com/t/magrittr-inside-a-package/20...
- minimaxir 7y ago
  That's an interesting caveat with pipes/magrittr (I didn't know of it), although it's out of scope of this debate since this discussion is just about how tidyverse uses pipes. In my experience if there's a package that's required in a script/R Notebook outside of tidyverse, I just import the entire library at the start of the file Python-style (to make the dependency obvious), which would avoid this issue.
  - kgwgk 7y ago
    If you claim that something like the_data <- read.csv('/path/to/data/file.csv') the_data <- subset(the_data, variable_a > x) the_data <- transform(the_data, variable_c = variable_a/variable_b) the_data <- head(the_data, 100) is a pain to read compared to the_data <- read.csv('/path/to/data/file.csv') %>% subset(variable_a > x) %>% transform(variable_c = variable_a/variable_b) %>% head(100) I thinks it’s fair to mention how by making things more complicated by using pipes you expose yourself to other issues. (In my opinion the first variant is not less readable, and it has the advantage of allowing any of the operations to be removed by commenting the corresponding line, while in the second case to remove the last operation you also need to remove the piping operator in the preceding line.) [Edit: I’m not sure I understand your “out of scope” remark. You don’t mean that magrittr is outside of tidyverse, do you?]

inputcoffee 7y ago

I was waiting for the critique... but I never quite saw it. Imho, the data problem Tidyverse is trying to solve is basically the ones we face in a database. So, select, join, inner join and so forth. Show me all the rows in this datatable where the 4th columm is larger than the 6th column and the number itself is odd. Something like that. There might be other ways to do it, but you want your select, filter, summarize, mutate etc functions to all work with each other, pipe to each other and be compatible. Maybe there is a better way to do all this -- I haven't seen it but I am not an expert -- but you have to show that to me. So, in base R, walk through a set of example of mutating, joining, filtering and so forth, and show me how they are all easier. Then I'll say, wow there is an alternative to this Tidyverse thing. But in lieu of that demo, this felt more like an intro to a complaint than an actual complaint. Edit: Also, its funny that Wickham is (apparently) such a nice fellow that people go out of the way to be nice to him in critiques.

barcadad 7y ago
His critique is more about the impact of the full ecosystem effect of the Tidyverse, not what you are referring to, which is just the dplyr semantics. The Tidyverse demands that it's many related packages use tidy data principles and lock users into that approach, which differs from base-R. Much of this discussion is really just a debate about dplyr and magrittr rather than the fragmentation that the broader tidyverse has brought on. All that said, I agree with many commenters that the Tidyverse's improvements to speed of development can more than offset the speed of execution issues, at least for small-to-medium datasets.
- RosanaAnaDana 7y ago
  I think any one making the point of speed of development have seriously missed the boat. Lets be real: tibbles suck. Once you get the hang of data.table syntax for matrix operations, its superiority becomes impeccably clear. I run a data science group a large geospatial company and we develop day in and day out in R and python. We've purged tidyverse as much as possible from all of our code base. We've moved completely over to data.tables, which make the vast majority of the tidyverse irrelevant.

kgwgk 7y ago

> Ironically, though consistency of interface is a goal, new versions of Tidy libraries are often incompatible with older ones, a very serious problem in the software engineering world. I was (sadly) amused by a recent comment on HN linking to some tweets celebrating that some R code still ran four years later.

tylermw 7y ago
That was my comment, and the point wasn't that the R code still ran four years later--the point was that R Core's focus on backwards compatibility (in reference to R being hamstrung via its focus on "compatibility with S") isn't a bad thing. You're never going to get the 2.x - 3.x schism that python had with future version of R, assuming R core continues with that philosophy.
- deleted 7y ago
  [deleted]
- kgwgk 7y ago
  I completely agree with your point and I think saying that R is keeping compatibility with S doesn’t make much sense. R is keeping compatibility with R, as it should. Maybe I lack context, but those tweets looked to me as if someone said “I can run this four year old game in the latest release of Windows!” or “Three versions of Office later I can still open my excel files!”

stilley2 7y ago

I recently tried to do some stuff in R with tidyverse, and was not a fan. I'm no expert, but the tidyverse's frequent use of nonstandard evaluation drove me crazy. It makes it so much more difficult to write functions encapsulating tidy functions. However, I never see people complain about this, so maybe I'm doing something wrong or not grokking something...

vincent-toups 7y ago
I really hate this. And its so unnecessary!
_Wintermute 7y ago
It's because RStudio are pushing R/tidyverse as an interactive datascience environment, so most of the tutorials you see will be aimed at users sending a few lines a time to the REPL. Programming with the tidyverse and its non-standard evaluation really is a pain.
jointpdf 7y ago
There is a new operator, {{ or “curly curly”, aimed at dealing with this. It’s part of the rlang (0.4) package and looks snazzy / easy to grasp. Check it out: https://www.tidyverse.org/articles/2019/06/rlang-0-4-0/ https://www.tidyverse.org/articles/2019/06/rlang-0-4-0/
- RosanaAnaDana 7y ago
  Yuck.
  - jointpdf 7y ago
    Have anything more constructive to say?
malshe 7y ago
No, you are not doing anything wrong. It's a real pain.

zhdc1 7y ago

I am a feverent user of data.table and try to avoid dplyr when ever possible. I have colleagues who are exactly the opposite. No one cares. If you want efficient code, try not to use dplyr. If you want readable code, data.table isn't the best answer. If you want speed, there are better languages than R to choose from, even with Rcpp. If you want to get something done, go with whatever you know best and will do the job. Avoid Stata.

ellisv 7y ago

As an R user of some 10+ years, I think some of this is spot on and some of it is just silly. Pretty much everyone knows the Tidyverse is slow to run, but many of us find it faster to write and very readable because it avoids one-time-use assignments (i.e. in `a <- f(x); g(a)` the variable `a` is only used once). Naming things is hard. Prof. Matloff's point is well taken though - you don't need the Tidyverse and it doesn't run faster. Fair criticism. > The "star" of the Tidyverse, dplyr, consists of 263 functions. While a user initially need not use more than a small fraction of them, the high complexity is clear. Every time a user needs some variant of an operation, she must sift through those hundreds of functions for one suited to her current need. > By contrast, if she knows base-R (not difficult), she can handle any situation. Yet base R has some 1,346 functions. Another 455 functions in `stats`. I'm not sure the quantity of functions has as much bearing on learning as the quality and consistency. So this seems like a silly argument to me. Mostly I think Prof. Matloff is forgetting that base R is kinda terrible and no one else has done much about it aside from RStudio.

cwyers 7y ago
> Mostly I think Prof. Matloff is forgetting that base R is kinda terrible and no one else has done much about it aside from RStudio. I think he'd disagree with the contention that base R is kinda terrible. Like, I'm pretty sure he's wrong. But this isn't something he's forgotten, it's something he's never believed. The point about the Tidyverse and RStudio is that RStudio doesn't make any of its revenue from the Tidyverse. They distribute it free. There are no upsells for any of it -- I can't go out and buy ggplot2 add ons, I can't buy dplyr support contracts. There's no money to be made on the Tidyverse, and plenty of resources thrown at it. So what is RStudio selling? They sell an IDE, they sell servers to host things built in R, they sell a package manager solution, they offer most of these things as a SaaS offering as well (although rstudio.cloud is still in trial and they haven't started charging yet). In other words, RStudio offers tooling for people who write R. They don't really care what kind of R you're writing, from a financial perspective. They get paid the same if you're using RStudio to write stuff using data.table as they get paid if you use dplyr. RStudio is making a bet that more people will want to write R code if they make R a more pleasant language to work in. And anecdotally, they're absolutely right -- I tried to learn R before the Tidyverse, and I hated every moment of it. I like post-Tidyverse R (although I got into it when people still called it the Hadleyverse). I know several other people with similar experiences to mine.

ahurmazda 7y ago

With respect, its laughable to think that anyone working on big-data with R has _not_ heard of `data.table`. When your livelihood depends on it, you tend to find it out one way or the other. However, I'd claim most folks don't deal with big data or even medium data! [Aside: I chuckled at the 1e5 columns on the plot]. Personally, at the small-to-medium range, all languages are fast enough. It boils down to ergonomics for me. I _detest_ the subscript notation. It slows me down -- not just productivity-wise but also how I read or reason-about my code. Not surprisingly, I prefer `dplyr` over `data.table` (base R or `pandas`)