9 ms·

Don't Enforce R as a Standard

166 points by doctoboggan 12y ago

zzleeper 12y ago
These were the same types of people that said to me "don't use R, no one uses it" several years ago. I know there must be a balance between bleeding-edge and stability, but if in research you cannot use the new tools, then there will not be any progress at all.
- gerty 12y ago
  That's true. In my field, there's plenty of Matlab and it's arguably not even the best tool for the job: someone started using it, people improved on the work and now, no one cares about rewriting that.
weinzierl 12y ago
"...it can be written in lisp or cobold for all I care" The typo is too good. Cobold is the sprite that plays its tricks on us even 56 years after its first appearance.
- rcthompson 12y ago
  I assumed it was intentional.
  - weinzierl 12y ago
    That makes it even better.
guelo 12y ago
My non-academist opinion is that the biggest problem in science is that there is little incentive for scientists to reproduce each other's results. Anything that makes it easier for other labs to rerun an experiment is a good thing for your field overall. The goal of a paper should be to communicate as plainly as possible how to reproduce the experiment. That probably means a good-enough standard programming language that everyone learns in school and uses in their work.
- sz4kerto 12y ago
  Look at the Reproducibility Project: https://osf.io/ezcuj/ https://osf.io/ezcuj/
- Gimpei 12y ago
  I agree, unless it's something that can't be done well in the current language. Or, if like me, you're an economist and the default language used by academics is horrible, horrible Stata, which needs to be replaced by R.
  - zzleeper 12y ago
    For me, both are horrible. I use Stata a lot and thought a lot about switching to R (there is even a nice tutorial by Matthieu Gomez at http://www.princeton.edu/~mattg/statar/ http://www.princeton.edu/~mattg/statar/ ). However, I believe that moving to R is not the answer, as you will still be riddled with a lot of the problems that you faced in Stata such as a quirky syntax (except backticks. I hate those backticks). Maybe Python will catch up (I doubt it, what we need is a DSL, not something too general that requires loong commands to do a simple regress). My best bet would be Julia, but it's still a long way to go regarding things like missing values.
  - grayclhn 12y ago
    If you're a macroeconomist, the default is Matlab. I'm not sure which is worse.
- locokoko 12y ago
  Well I think it should communicate how to reproduce the experiment in a lab with the same or similar setup, not how to reproduce it with all kinds of different setups. If a lab does not have the equipment to reproduce a certain experiment, it's not the problem of the original authors. Also, I think you should choose equipment which best suits the experiment, not other people.
  - claytonjy 12y ago
    >If a lab does not have the equipment to reproduce a certain experiment, it's not the problem of the original authors. While I think that's true, a paper can potentially be much more impactful if its' results can be easily understood by others, and isn't that really the point of science? In that sense, it's better for the author to stay away from niche methods and tools unless absolutely necessary.
Mikeb85 12y ago
I think it's just that R has become the de-facto standard since the previous tools used were either terrible, or closed source and expensive. And it's probably taken them so long to adjust to R, that they don't want to change again any time soon. And you're right, the tool shouldn't matter, and I definitely agree with your stance on open source, but keep in mind those attitudes do exist. Speaking of attitudes concerning open source, here's one of my favourite blog posts about the subject: http://www.catuhe.com/post/Le-syndrome-du-Puppy.aspx http://www.catuhe.com/post/Le-syndrome-du-Puppy.aspx
apalmer 12y ago
Why did R take over the world of statistics anyways? I remember about 8 years ago I had an interest in statistics. Everything was SAS, SPSS etc which i just didnt have the budget for... I happened to find R as it was a freely available for linux, impression I got was no one used it, well some limited use in academia, but no professional usage. Fast forward it seems like the most widely used tool in computer data analysis. Did something fundamentally change?
- stillsut 12y ago
  Packages. They can be built by a single prof/grad student. They can be used by anyone based on R's package-manager GUI. Research in most fields is going to move more quickly than any software team can add functionality to monolithic programs like SAS. Therefore, most functionality becomes available only in R. Users - people who want to run specialized-domain-function-X on their dataset are best suited by learning R, then using their colleagues packages.
  - rcthompson 12y ago
    I don't know much about SAS or SPSS, but do they really not have a usable package/module system for code reuse? Or is it just not as good as R's?
    - claytonjy 12y ago
      I don't know about the package system for SAS or SPSS (though I am aware that papers get published with code for said platforms), but I think the bigger barrier is availability; R is free and open.
    - _deh 12y ago
      SAS a bit, but as another comment notes, nothing like CRAN. SPSS not at all, at best you could publish some sort of script.
  - _deh 12y ago
    Agree entirely. Academics led and industry followed.
deleted 12y ago
[deleted]
davidmr 12y ago
I'm torn on this. I've spent nearly the last 20 years in some form or another in academic/research computing. On the one hand, attitudes like the third reviewer's are a primary reason for the state of HPC today, where new advances in research are just as often finding a way to run 30 year old code on a modern supercomputer as they are writing new software to take advantage of the fantastic array of new hardware. The number of times you hear "sorry, we can't change that. Livermore wrote that 15 years ago, and nobody knows how to change it anymore" is enough to drive a rational person over the edge. On the other hand, the state of academic peer review being what it is, I don't fault the reviewer for suggesting the paper be resubmitted using more common methodology. A conscientious reviewer has a lot of papers to review, and spends a good amount of time on the ones that aren't written by crackpots. While the author was of course fully in their rights (and may even have advanced the field) to use their own software written in a relatively uncommon language, for the results and methodology to be understood, that's asking a lot of a reviewer. I'm genuinely unsure as to how I would have responded to the review request. I have much sympathy for both people involved.
- deleted 12y ago
  [deleted]
- Blahah 12y ago
  The correct response from the reviewer would have been "I am not capable of reviewing code written in Julia, therefore I must decline this review request" rather than "I am not capable of reviewing code written in Julia, therefore I must recommend rejecting the paper".
  - blumkvist 12y ago
    Are you some sort of supreme authority on such matters?
    - rjurney 12y ago
      No, but he does sound right. Rejecting a paper because of a made up standard based on brand preference is stupid.
jdietrich 12y ago
To give the other side of the argument: Most scientists have little or no quality training in software development, but scientific research is increasingly reliant on software. At present, software is a glaring black box in a great deal of research, because very few reviewers have the skills to thoroughly scrutinise it. A software monoculture always has negative consequences, but fragmentation can be equally problematic in many cases. A reviewer saying "This would be better in R" usually means "I have no chance of understanding your code, because it's not in R". For better or worse, R is currently the statistical computing lingua franca in most fields. I believe that the scientific method is in real trouble, due largely to the immense complexity of much modern-day research. Scientists have access to immensely powerful analytical and statistical tools, but most lack the training and infrastructural support to use them in a rigorous manner. Bad practices in software and statistics are the norm, rather than the exception; I'm sure most of this is just an honest shortcoming, but I'm equally sure that the lack of CS and stats experience amongst reviewers is a gift to would-be Bogdanovs and Obokatas. Science has always been a collaborative effort, but I think most fields are in desperate need of greater support from computer scientists and statisticians. Ultimately I would like to see those professions become deeply integrated into all scientific fields, with the expectation that all papers should credit a statistician and (where applicable) a computer scientist. Likewise, editors and reviewers need far closer ties to CS and statistics professionals. Of course, these issues are intertwined with many other problems in funding and peer review. Until then, the hegemony of R may simply be a price we have to pay for better research in the short-term. A software monoculture at least gives reviewers a fighting chance of spotting issues with software that might affect the validity of results.
- rjurney 12y ago
  I think its more like, "I have no chance of understanding your code, BECAUSE IT'S IN R." The vast majority of R users and I bet 95% of ecologists, can't read and understand R modules. Its the PHP of Data Science. The point, then is to supply code with simple instructions so that anyone can run it. Secondary goal: write it in a language a human can actually read. If you're doing science and you want people to understand your code, you use Python, not R.
minimaxir 12y ago
The only reason I still use R is because of the hadley-verse packages. (Namely dplyr and ggplot2). R's base packages are incredibly terrible and counterintuitive, especially compared to Python's pandas and scipy.
Pinatubo 12y ago
I think the author is being overly sensitive here, and the reviewers are actually giving good advice. Most methods papers in most fields are ignored by the vast majority of researchers. Just like the author doesn't have the time to rewrite his code in R, his prospective audience doesn't have the time to learn a new software package just to try out his proposed method. If the author actually wants to change the way research is conducted in his field, he needs to make it easy for others to try his method out and possibly change the way they do research. As of today, that means an R package. Or, the author can dig in his heels, refuse to write an R package, and be ignored. Maybe that isn't fair, but as I've learned through my own experiences, that's the way it is.
mattexx 12y ago
Peer review is a huge part of our culture here at Climate. It's often unreasonable to expect a single reviewer to review a complex paper, so we sometimes address this by assigning each reviewer a domain. Something like: Reviewer 1: Review scientific theory (domain expert) Reviewer 2: Review scientific methods and conclusions (senior scientist) Reviewer 3: Review code for scientific accuracy (senior programmer)
moron4hire 12y ago
Even the language we use to talk about arithmetic, algebra, and calculus had to--at one time--be agreed to and standardized, so that mathematicians could have a common ground on which to speak to each other about their ideas and how they reached them. The program is not just "a tool". It's your proof. You may have written a proof in the paper, but that was just a practice run. It's the one you wrote in the code that is the real proof, because there is little garaunteeing that seasoned software developers write code that matches the spec, say nothing about" amateur" programmers in the sciences. Now, that does not preclude innovation. People still invent new mathematical notation systems. It's usually with the express purpose of solving problems that can't easily or at all be stolved in current systems. It is the bleeding edge of the science of math. But physicists are still expected to use the math their colleagues will understand, or spend a lot of time explaining their new system (QM anyone?). There is, of course, the issue that math is significantly less rigorous and specific than code. Math doesn't compile and doesn't run. Or rather, it gets compiled and ran by people. Part of avoiding new, arbitrary notations is so that your work is accessible for verification. That gives running code, especially coffee with a comprehensive set of unit tests, a distinct advantage over math. But that doesn't obviate the bed for verification. If you are using any nontrivial piece of custom software in your science, it needs to be scrutinized just as much as the rest of the math in your paper--if not more so, since it does the actual work. The first reviewer is almost certainly right. If they can't understand your code, then they aren't the equivalent of lay-users complaining about open source software. First of all, it is still an issue of much debate whether or not releasing source code to user's for software products is necessary. If you are writing code for your science, you have an absolute duty to open source that code (even if in a non-extendable way, we still need to see the code). You wouldn't make a claim without providing the math. And you wouldn't provide math other people couldn't understand, arguing "some math is better than no math". I don't think that standard should be R, or that it necessarily has to always be the same thing. It can adapt over time. But until you manage to release a few papers on how Julia improves over R, in explicit detail... "when in Rome, do as the Romans."
bachmeier 12y ago
Does this story tell us more about R as a standard, or more about the refereeing process? I've dealt with referees for a long time and I'd have to go with the latter. This is to me yet another case of a referee viewing his/her preferences as the only correct way to do things.
scottfr 12y ago
This is really just an example of Bike Shedding. Reviewing research papers is really, really hard work to do well. It takes a massive time commitment to truly digest and understand novel work to the level where you can provide a quality critique and review. It's much easier to throw out facile critiques like the ones these reviewers provided. It shows that you know your stuff and did your job as a reviewer, without requiring you to actually understand the authors' innovations or lack-there-of.
- tmalsburg2 12y ago
  You are implying that the rest of the review, which we haven't seen, was useless and shallow. However, the author himself said that the reviews were actually "helpful and constructive". I agree that cases of bike shedding exist, but there is no evidence that this was the case here.
tmalsburg2 12y ago
The author writes that the software wasn't the focus of the manuscript and that the reviewers should not have commented on the software at all but that it actually not true. If you look at the preprint manuscript, you'll find that the software is prominently advertised in the abstract of the paper. In this situation, it is perfectly reasonable for the reviewer to comment on the implementation of the software. And as much as I like Julia, I think he is right when he says that an R package would be more useful.

166 points by doctoboggan 12y ago

These were the same types of people that said to me "don't use R, no one uses it" several years ago. I know there must be a balance between bleeding-edge and stability, but if in research you cannot use the new tools, then there will not be any progress at all.

gerty 12y ago
That's true. In my field, there's plenty of Matlab and it's arguably not even the best tool for the job: someone started using it, people improved on the work and now, no one cares about rewriting that.

weinzierl 12y ago

"...it can be written in lisp or cobold for all I care" The typo is too good. Cobold is the sprite that plays its tricks on us even 56 years after its first appearance.

rcthompson 12y ago
I assumed it was intentional.
- weinzierl 12y ago
  That makes it even better.

guelo 12y ago

My non-academist opinion is that the biggest problem in science is that there is little incentive for scientists to reproduce each other's results. Anything that makes it easier for other labs to rerun an experiment is a good thing for your field overall. The goal of a paper should be to communicate as plainly as possible how to reproduce the experiment. That probably means a good-enough standard programming language that everyone learns in school and uses in their work.

sz4kerto 12y ago
Look at the Reproducibility Project: https://osf.io/ezcuj/ https://osf.io/ezcuj/
Gimpei 12y ago
I agree, unless it's something that can't be done well in the current language. Or, if like me, you're an economist and the default language used by academics is horrible, horrible Stata, which needs to be replaced by R.
- zzleeper 12y ago
  For me, both are horrible. I use Stata a lot and thought a lot about switching to R (there is even a nice tutorial by Matthieu Gomez at http://www.princeton.edu/~mattg/statar/ http://www.princeton.edu/~mattg/statar/ ). However, I believe that moving to R is not the answer, as you will still be riddled with a lot of the problems that you faced in Stata such as a quirky syntax (except backticks. I hate those backticks). Maybe Python will catch up (I doubt it, what we need is a DSL, not something too general that requires loong commands to do a simple regress). My best bet would be Julia, but it's still a long way to go regarding things like missing values.
- grayclhn 12y ago
  If you're a macroeconomist, the default is Matlab. I'm not sure which is worse.
locokoko 12y ago
Well I think it should communicate how to reproduce the experiment in a lab with the same or similar setup, not how to reproduce it with all kinds of different setups. If a lab does not have the equipment to reproduce a certain experiment, it's not the problem of the original authors. Also, I think you should choose equipment which best suits the experiment, not other people.
- claytonjy 12y ago
  >If a lab does not have the equipment to reproduce a certain experiment, it's not the problem of the original authors. While I think that's true, a paper can potentially be much more impactful if its' results can be easily understood by others, and isn't that really the point of science? In that sense, it's better for the author to stay away from niche methods and tools unless absolutely necessary.

Mikeb85 12y ago

I think it's just that R has become the de-facto standard since the previous tools used were either terrible, or closed source and expensive. And it's probably taken them so long to adjust to R, that they don't want to change again any time soon. And you're right, the tool shouldn't matter, and I definitely agree with your stance on open source, but keep in mind those attitudes do exist. Speaking of attitudes concerning open source, here's one of my favourite blog posts about the subject: http://www.catuhe.com/post/Le-syndrome-du-Puppy.aspx http://www.catuhe.com/post/Le-syndrome-du-Puppy.aspx

apalmer 12y ago

Why did R take over the world of statistics anyways? I remember about 8 years ago I had an interest in statistics. Everything was SAS, SPSS etc which i just didnt have the budget for... I happened to find R as it was a freely available for linux, impression I got was no one used it, well some limited use in academia, but no professional usage. Fast forward it seems like the most widely used tool in computer data analysis. Did something fundamentally change?

stillsut 12y ago
Packages. They can be built by a single prof/grad student. They can be used by anyone based on R's package-manager GUI. Research in most fields is going to move more quickly than any software team can add functionality to monolithic programs like SAS. Therefore, most functionality becomes available only in R. Users - people who want to run specialized-domain-function-X on their dataset are best suited by learning R, then using their colleagues packages.
- rcthompson 12y ago
  I don't know much about SAS or SPSS, but do they really not have a usable package/module system for code reuse? Or is it just not as good as R's?
  - claytonjy 12y ago
    I don't know about the package system for SAS or SPSS (though I am aware that papers get published with code for said platforms), but I think the bigger barrier is availability; R is free and open.
  - _deh 12y ago
    SAS a bit, but as another comment notes, nothing like CRAN. SPSS not at all, at best you could publish some sort of script.
- _deh 12y ago
  Agree entirely. Academics led and industry followed.

deleted 12y ago

[deleted]

davidmr 12y ago

I'm torn on this. I've spent nearly the last 20 years in some form or another in academic/research computing. On the one hand, attitudes like the third reviewer's are a primary reason for the state of HPC today, where new advances in research are just as often finding a way to run 30 year old code on a modern supercomputer as they are writing new software to take advantage of the fantastic array of new hardware. The number of times you hear "sorry, we can't change that. Livermore wrote that 15 years ago, and nobody knows how to change it anymore" is enough to drive a rational person over the edge. On the other hand, the state of academic peer review being what it is, I don't fault the reviewer for suggesting the paper be resubmitted using more common methodology. A conscientious reviewer has a lot of papers to review, and spends a good amount of time on the ones that aren't written by crackpots. While the author was of course fully in their rights (and may even have advanced the field) to use their own software written in a relatively uncommon language, for the results and methodology to be understood, that's asking a lot of a reviewer. I'm genuinely unsure as to how I would have responded to the review request. I have much sympathy for both people involved.

deleted 12y ago
[deleted]
Blahah 12y ago
The correct response from the reviewer would have been "I am not capable of reviewing code written in Julia, therefore I must decline this review request" rather than "I am not capable of reviewing code written in Julia, therefore I must recommend rejecting the paper".
- blumkvist 12y ago
  Are you some sort of supreme authority on such matters?
  - rjurney 12y ago
    No, but he does sound right. Rejecting a paper because of a made up standard based on brand preference is stupid.

jdietrich 12y ago

To give the other side of the argument: Most scientists have little or no quality training in software development, but scientific research is increasingly reliant on software. At present, software is a glaring black box in a great deal of research, because very few reviewers have the skills to thoroughly scrutinise it. A software monoculture always has negative consequences, but fragmentation can be equally problematic in many cases. A reviewer saying "This would be better in R" usually means "I have no chance of understanding your code, because it's not in R". For better or worse, R is currently the statistical computing lingua franca in most fields. I believe that the scientific method is in real trouble, due largely to the immense complexity of much modern-day research. Scientists have access to immensely powerful analytical and statistical tools, but most lack the training and infrastructural support to use them in a rigorous manner. Bad practices in software and statistics are the norm, rather than the exception; I'm sure most of this is just an honest shortcoming, but I'm equally sure that the lack of CS and stats experience amongst reviewers is a gift to would-be Bogdanovs and Obokatas. Science has always been a collaborative effort, but I think most fields are in desperate need of greater support from computer scientists and statisticians. Ultimately I would like to see those professions become deeply integrated into all scientific fields, with the expectation that all papers should credit a statistician and (where applicable) a computer scientist. Likewise, editors and reviewers need far closer ties to CS and statistics professionals. Of course, these issues are intertwined with many other problems in funding and peer review. Until then, the hegemony of R may simply be a price we have to pay for better research in the short-term. A software monoculture at least gives reviewers a fighting chance of spotting issues with software that might affect the validity of results.

rjurney 12y ago
I think its more like, "I have no chance of understanding your code, BECAUSE IT'S IN R." The vast majority of R users and I bet 95% of ecologists, can't read and understand R modules. Its the PHP of Data Science. The point, then is to supply code with simple instructions so that anyone can run it. Secondary goal: write it in a language a human can actually read. If you're doing science and you want people to understand your code, you use Python, not R.

minimaxir 12y ago

The only reason I still use R is because of the hadley-verse packages. (Namely dplyr and ggplot2). R's base packages are incredibly terrible and counterintuitive, especially compared to Python's pandas and scipy.

Pinatubo 12y ago

I think the author is being overly sensitive here, and the reviewers are actually giving good advice. Most methods papers in most fields are ignored by the vast majority of researchers. Just like the author doesn't have the time to rewrite his code in R, his prospective audience doesn't have the time to learn a new software package just to try out his proposed method. If the author actually wants to change the way research is conducted in his field, he needs to make it easy for others to try his method out and possibly change the way they do research. As of today, that means an R package. Or, the author can dig in his heels, refuse to write an R package, and be ignored. Maybe that isn't fair, but as I've learned through my own experiences, that's the way it is.

mattexx 12y ago

Peer review is a huge part of our culture here at Climate. It's often unreasonable to expect a single reviewer to review a complex paper, so we sometimes address this by assigning each reviewer a domain. Something like: Reviewer 1: Review scientific theory (domain expert) Reviewer 2: Review scientific methods and conclusions (senior scientist) Reviewer 3: Review code for scientific accuracy (senior programmer)

moron4hire 12y ago

Even the language we use to talk about arithmetic, algebra, and calculus had to--at one time--be agreed to and standardized, so that mathematicians could have a common ground on which to speak to each other about their ideas and how they reached them. The program is not just "a tool". It's your proof. You may have written a proof in the paper, but that was just a practice run. It's the one you wrote in the code that is the real proof, because there is little garaunteeing that seasoned software developers write code that matches the spec, say nothing about" amateur" programmers in the sciences. Now, that does not preclude innovation. People still invent new mathematical notation systems. It's usually with the express purpose of solving problems that can't easily or at all be stolved in current systems. It is the bleeding edge of the science of math. But physicists are still expected to use the math their colleagues will understand, or spend a lot of time explaining their new system (QM anyone?). There is, of course, the issue that math is significantly less rigorous and specific than code. Math doesn't compile and doesn't run. Or rather, it gets compiled and ran by people. Part of avoiding new, arbitrary notations is so that your work is accessible for verification. That gives running code, especially coffee with a comprehensive set of unit tests, a distinct advantage over math. But that doesn't obviate the bed for verification. If you are using any nontrivial piece of custom software in your science, it needs to be scrutinized just as much as the rest of the math in your paper--if not more so, since it does the actual work. The first reviewer is almost certainly right. If they can't understand your code, then they aren't the equivalent of lay-users complaining about open source software. First of all, it is still an issue of much debate whether or not releasing source code to user's for software products is necessary. If you are writing code for your science, you have an absolute duty to open source that code (even if in a non-extendable way, we still need to see the code). You wouldn't make a claim without providing the math. And you wouldn't provide math other people couldn't understand, arguing "some math is better than no math". I don't think that standard should be R, or that it necessarily has to always be the same thing. It can adapt over time. But until you manage to release a few papers on how Julia improves over R, in explicit detail... "when in Rome, do as the Romans."

bachmeier 12y ago

Does this story tell us more about R as a standard, or more about the refereeing process? I've dealt with referees for a long time and I'd have to go with the latter. This is to me yet another case of a referee viewing his/her preferences as the only correct way to do things.

scottfr 12y ago

This is really just an example of Bike Shedding. Reviewing research papers is really, really hard work to do well. It takes a massive time commitment to truly digest and understand novel work to the level where you can provide a quality critique and review. It's much easier to throw out facile critiques like the ones these reviewers provided. It shows that you know your stuff and did your job as a reviewer, without requiring you to actually understand the authors' innovations or lack-there-of.

tmalsburg2 12y ago
You are implying that the rest of the review, which we haven't seen, was useless and shallow. However, the author himself said that the reviews were actually "helpful and constructive". I agree that cases of bike shedding exist, but there is no evidence that this was the case here.

tmalsburg2 12y ago

The author writes that the software wasn't the focus of the manuscript and that the reviewers should not have commented on the software at all but that it actually not true. If you look at the preprint manuscript, you'll find that the software is prominently advertised in the abstract of the paper. In this situation, it is perfectly reasonable for the reviewer to comment on the implementation of the software. And as much as I like Julia, I think he is right when he says that an R package would be more useful.