19 ms·
The problem is people using R without trying to learn about the language itself, just assuming it works like their favourite language. For example complaining
by mziel 11y ago
The problem is people using R without trying to learn about the language itself, just assuming it works like their favourite language.
For example complaining that R is slow and then writing iterative solution instead of using vectorization. When I saw the example the author gave my first thought was "sapply/lapply". Lapply is essential to the R use, and is being taught early on in every book/course on R I've ever saw.
"In 2012, I’m the kind of person who uses apply() a dozen times a day, and is vaguely aware that R has a million related built-in functions like sapply(), tapply(), lapply(), and vapply(), yet still has absolutely no idea what all of those actually do. "
- x0x0 11y agoIt's kind of amazing to see someone admit to spending hundreds or thousands of hours using R, yet refuse to spend a couple hours learning the language a little better. Whining that your tools are hard without investing any effort in them is just dumb. The R help even comes with code samples that you can run!
- Fomite 11y agoR is, I think, an interesting language because it's heavily used by people who would not otherwise learn a programming language. If you compare R not with other programming languages, but with other ways of working with statistical data, this makes far more sense. I don't actually "know" SAS in the way I know a programming language - I know the commands I invoke to do what I want it to do. Similarly, I encounter lots of people using R who don't actually know what a function is, just that lm(x~y) gets them what they want.
- cvwright 11y agoI see this as a failure of our educational system. Speaking as an academic in CS, it's our job to teach people skills that they need for dealing with computers in the course of their career. The Math department does this for basic calculus and probability; the English department does this for literature and composition. Why don't more CS departments offer the service courses that scientists and engineers need to really learn how to manipulate their data and make sense of it? At least part of the problem is probably that the other departments won't require their students to take such a course...
- dmichulke 11y agoMy personal experience is similar because I know quite a few people in social sciences. Conceptually, this is similar to rats pulling a lever or monkeys being reinforced to type the right characters. It also explains p-hacking and many other problems of interpretation. Now one question I always have is - if you consider R just a tool - what is the difference between things I should fully understand (R?) and things I should only know how to use (e.g., my cell phone)? How can I justify saying that people should understand R while I myself don't understand quite a few aspects of my cell phone?
- simonbyrne 11y agoAlso, many people use it only intermittently, maybe once every six months or so when they have some data to look at. Rather than try to relearn the language and its quirks yet again, it's much easier to take what you did last time and tweak it until you get what you need.
- Fomite 11y agoThis too. Between collecting data, writing grant proposals, writing papers, etc. I don't spend time day-in, day-out using R. Often the first hour of that is thinking "Shit, how do I do that again? Has Hadley written a package to do this better by now? What did I do last time - why did I do that last time?"
- copperx 11y agoSurprisingly, this is true for many programmers of general-purpose languages.
- TheLogothete 11y agoIn my experience a lot of the claims that R is slow are greatly exaggerated and made by people who don't actually use it. Kind of an echo chamber. Every time I see someone say chose Python instead because of speed, I roll my eyes.
- existencebox 11y agoIt's funny you bring up python; I say this not as a comment on your thesis, but related, since I often hear the "python is slow" trope but that's only half true, you can typically write python that is plenty "fast enough" (As a day-jobbing data pipeline engineer) if you're implementing with an understanding of what things will drive you into the mud. This goes beyond just understanding the tool you're using, fundamentally writing an O(N^N) or something is going to hurt even if you're in C#/C. Have seen that plenty, frankly, more than I've seen "legitimately slow python" Anyway, this was just a thought rolling around in my head given the discussion.
- GFK_of_xmaspast 11y agoI've translated plenty of numerical code from (pure-ish) python to c and c++, and usually get about a 100x speedup, sometimes as high as 800x, implementing the same algorithms.
- Cogito 11y agoAny idea where the speedups came from? Is it that the problems weren't algorithmically limited in the first place (lots of io for example), reduction of overhead etc (what kind of python was the code running on before?), or just that the speedup on low level operations added up cumulatively and cam e to dominate the other timing factors? Also, did you change the data structures or use the same ones as in python? Was any of the speed boost data structure related?
- klmr 11y agoPython and similar dynamic languages suffer from the fact that every name access (variable, function, etc) incurs a dynamic lookup of that name in a (nested) dictionary. Statically compiled languages don’t have this. There are fairly recent, clever optimisations that can avoid many of these lookups but (a) they are not implemented in any of the common implementations of Python, R, etc (JavaScript has them though). But even with these optimisations in place we cannot get rid of such lookups altogether, and they kill cache locality and branch prediction. There are other reasons for slowdown (automatically managed garbage collection is a big one, and so is any kind of indirection, e.g. callbacks). But usually the big one is name lookup.
- Thriptic 11y agoYea I feel like sapply, lapply, and mapply will cover most of what people need to do. Hell I've personally only really ever needed sapply as I don't work much with lists.
- chrisseaton 11y ago> For example complaining that R is slow and then writing iterative solution instead of using vectorisation. But if someone prefers iterative solutions, or that's all they know, why can't R make them just as fast as the vectorised versions?
- deleted 11y ago[deleted]
- YeGoblynQueenne 11y ago>> But if someone prefers iterative solutions, or that's all they know, why can't R make them just as fast as the vectorised versions? R is interpreted and dynamically typed, so when you declare a variable, the interpreter has to do some bookkeeping to figure out the type of the variable, allocate memory for it and so on. If you write a loop by hand, the interpreter has to do this bookkeeping once for each iteration. If you write your code in vectorised form, the interpreter can sort out the bookkeeping once and then hand over to the lower-level code (C or Fortran) the vectorised functions are interpreted in. This can also be further optimised to take advantage of processor vector instructions, parallel processing etc. So I'm afraid we can't have our pie and eat it. If we want an interpreted language with somewhat intuitive notation, then it has to have crappy slow loops. If we want a language with fast loops we have to rely on C or Fortran and forget about vectorised notation.
- chrisseaton 11y agoWhy can't a JIT solve this? It shouldn't need to do the bookkeeping for every iteration if it has JIT compiled it. A JIT should be able to take advantage of processor vector instructions etc.
- x0x0 11y agoThere's some movement in that direction. However, the R core committers are essentially not only volunteers, but they're all (afaik) academic statisticians. One of the people who made strides in this direction is primarily an computational statistician at Iowa (Luke Tierney / compiler package). Building a high performance runtime/jit is wildly out of their scope of expertise. In retrospect, and I think many of them would agree, building and maintaining their own runtime was a giant mistake. Yet here we are. Serious compiler people (Jan Vitek, others) have made strides towards a faster implementation (his in java / fastr IIRC), but it suffers from the same problem as cpython: there are millions of lines of C code in packages or internal functions that have the details of the R interpreter / C interface deeply embedded in them. In fact, there's probably far more "R" code written in C than in R. Undoing this mess is not easy, and probably not possible. Oh, reading Evaluating the Design of the R Language [1] will shed some more light on why it's hard to make R run fast. [1] http://r.cs.purdue.edu/pub/ecoop12.pdf http://r.cs.purdue.edu/pub/ecoop12.pdf edited to correctly describe Luke as per gbrown
- kbenson 11y ago> The problem is people using R without trying to learn about the language itself, just assuming it works like their favourite language. I think you just explained Perl in a nutshell (err, the idiom, not the book). It seems whenever a language supports enough idioms of the usual C-like languages people will gravitate towards those, likely due to the high population of people that know those idioms and can fall-back on them without having to think too hard. I doubt Lisp has as much of a problem of people trying to write C in Lisp.
- bachmeier 11y ago> The problem is people using R without trying to learn about the language itself I see a lot of blub when I read posts about R. So much so that I start with the assumption that any post about R is a blub post.
- todd8 11y agoIt's been a few years since I really looked at R, but I don't think the problems with R are simply that people don't learn the language. Some languages are simply not as good as others. We can all learn more about the tools we use when programming, I know that I certainly could. But this doesn't make it our fault that a language is tricky or hard to debug or hard to understand. If we worked at it, I suppose we could all write more efficient programs by using assembler, but that doesn't mean that assembler is the best possible programming language for, say, statistical programming. Someone, Ross Ihaka, that knows a thing or two about R wrote a short post 6 years ago and said "simply start over and build something better". Take a look: http://www.r-bloggers.com/“simply-start-over-and-build-something-better”/ http://www.r-bloggers.com/“simply-start-over-and-build-somet... My hope is that Julia will eventually be adopted as a basis for a future statistical programming language.
- TheLogothete 11y agoThe push for R is enormous. It's not getting replaced.
- sdenton4 11y agoFrom your link, hilariously relevant to the blog post at hand: "First, scalar computations in R are very slow. This in part because the R interpreter is very slow, but also because there are a no scalar types. By introducing scalars and using compilation it looks like its possible to get a speedup by a factor of several hundred for scalar computations. This is important because it means that many ghastly uses of array operations and the apply functions could be replaced by simple loops. The cost of these improvements is that scope declarations become mandatory and (optional) type declarations are necessary to help the compiler."
- todd8 11y agoI should point out that Ross Ihaka along with Robert Gentleman created R.
- jlebar 11y ago> The problem is people using R without trying to learn about the language itself It's not the user's fault. Like, congratulations on being better at R than the author of TFA. Maybe you're smarter than him, maybe you've put in more time learning, maybe you've just spent your time more intelligently, maybe you lucked out and bought better books...who knows. But this line of reasoning completely misses the author's point, which is that despite having used the language for years, he still finds it inscrutable. "It would be easier if you were better at R" is a tautology, and unhelpful. The issue is that the author finds it hard to become better at R. We can disagree as to whether or not it's objectively hard to become better at R, but this is a perfectly valid criticism to make. It's not the user's fault.
- jcizzle 11y agoI think the person you are responding to is simply saying some things are more complex than others and require more understanding and experience. R apparently falls into that category. If the author wants to gain that experience, I think the time spent on this blog post may be better used to reading a book on R. Fault might be a strong word, but the author has certainly made a decision on what they spend their time on and the results are as expected.
- RockyMcNuts 11y agosome things require more understanding and experience, for no obvious benefit...in which case they're anti-patterns, or not best practice in language design. there are a lot of things in R that are good but it's an old language and there's a lot of cruft. like "R has three object oriented systems (plus the base types), so it can be a bit intimidating." http://adv-r.had.co.nz/OO-essentials.html http://adv-r.had.co.nz/OO-essentials.html and believe it or not, there are things that require looping through a data frame and when I had to do that a few years ago it was unbelievably slow... going multithreaded was non-trivial, writing that section in C was non-trivial...ended up rewriting the whole thing in python and was a lot happier.
- mziel 11y ago
- crispyambulance 11y agoI sympathize with the OP and also feel frustrated with R (and I say that as a regular R "practitioner"). Part of the problem, I think, is the built-in documentation. The typical R user is a domain-expert just trying to get some work done. Occasionally, they'll get stuck and try something like "?sapply". What appears is usually a terse, confusing mess that takes a VERY LONG TIME to digest and is the LAST THING you want to read when you're trying to make a living solving a problem other than understanding R documentation. Below is the "Description" for Apply (which you get when you try ?sapply). Does it _really_ explain the essentials of what you need to use "apply"? "... lapply returns a list of the same length as X, each element of which is the result of applying FUN to the corresponding element of X. sapply is a user-friendly version and wrapper of lapply by default returning a vector, matrix or, if simplify = "array", an array if appropriate, by applying simplify2array(). sapply(x, f, simplify = FALSE, USE.NAMES = FALSE) is the same as lapply(x, f). vapply is similar to sapply, but has a pre-specified type of return value, so it can be safer (and sometimes faster) to use. ..."
- mziel 11y agoYMMV, but I think this is pretty clear: "lapply returns a list of the same length as X, each element of which is the result of applying FUN to the corresponding element of X" == lapply is a map() construct that takes a list and a function "sapply is a user-friendly version and wrapper of lapply by default returning a vector, matrix or, if simplify = "array", an array if appropriate, by applying simplify2array(). sapply(x, f, simplify = FALSE, USE.NAMES = FALSE) is the same as lapply(x, f)." == "sapply(x, f, simplify = FALSE, USE.NAMES = FALSE) is the same as lapply(x, f)"
- crispyambulance 11y agoIt could be A LOT more clear. The first sentence is, of course, obvious to everyone and probably already known by people searching for help on sapply. If you have a family of functions that "sort-of" do similar things, the most critical thing to communicate is some clear sense of when to use one or another of the functions. This doc degenerates into unhelpful jibberish instead. Perhaps a close reading of it would have helped the OP, but there is a unnecessarily high cost in frustration.