9 ms·

Groundhog: Addressing the Threat That R Poses to Reproducible Research

161 points by snakeboy 6y ago

cauthon 6y ago
This title is an exceedingly hot take for someone who wrote a new package manager. Also, it appears that Groundhog is itself a CRAN package and the author recommends installing with install.packages(). So is the author committing to never making any backwards incompatible updates to their new package?
- resonantjacket5 6y ago
  Your take seems a bit 'hot' too? How else would you install the cran packages without using install.packages? Unless if you want them to recursively install it using groundhog but that seems unnecessary. As long as you have the timestamp it should work, though I assume there will be some edge case. What you're saying is like don't use pip because you don't install it using pip? Or don't use package-lock.json because you can't install npm through npm?
  - cat199 6y ago
    > Your take seems a bit 'hot' too? OP is not claiming that Groundhog itself is a threat to the R language ecosystem itself, whereas the author is claiming that the R language is itself a threat to Science itself...
  - scottmcdot 6y ago
    Someone correct me if I'm wrong, but can't you copy and paste the package folder into your libpath directory and R can load it that way with actually running install.packages()?
    - vharuck 6y ago
      Usually, yes. However, it is possible for a package to have code that only runs when it is installed. If you just copy-paste, it won't be run.
  - cauthon 6y ago
    No, I’m saying don’t call CRAN a “threat to reproducible science” and then make your solution a CRAN package
- coolreader18 6y ago
  > So is the author committing to never making any backwards incompatible updates to their new package? Well, yes, probably. It's not all that hard, and groundhog seems to have a fairly simple API anyways. And groundhog still uses CRAN packages, it just brings a method of pinning them to a specific version.
- bsza 6y ago
  I think it's more like a Wayback Machine for R programs, since the author of a science paper isn't required to use groundhog. You can just provide it the date the article was published, which you already know, and it reconstructs how the program worked on that day. Also, because groundhog isn't made for the author to use, whether or not the interface changes is irrelevant. You'll never encounter library(groundhog) in a paper.
  - st1x7 6y ago
    > and it reconstructs how the program worked on that day. It reconstructs how the fully updated version of everything worked that day which isn't necessarily the same as the researcher's environment. It's a horrible idea to use dates instead of package versions for this. The author's library doesn't solve the problem it claims to solve.
    - SCLeo 6y ago
      If I am understanding this correctly, the problem is that the paper authors do not provide a specific version or a package.json equivalent. In that case, using dates seem to be the only choice.
      - st1x7 6y ago
        Even if that's the case, using dates isn't a solution because dates don't give you the build that the researcher used. Date of publication is different from the date when the code ran and there is no guarantee that the researcher ran the latest version of every dependency that was available to them anyway. In fact that's very unlikely considering that some their libraries might require older versions. It might not even be possible to take the latest version of every package and use them in the same environment.
        SCLeo 6y ago
        So, what is your better alternative then? I honestly believe using version available at that date is better than using the latest version.
- jsmith99 6y ago
  That's the problem. This package is very similar to Microsoft's checkpoint package which is based on Microsoft's MRAN snapshots, and this package also uses MRAN. The article explains the difference is that this package allows you to specify the date in the code itself, whereas checkpoint is used to set a whole installation to a specific date. But this is no advantage as it means code will stop working if the groundhog package changes, whereas with checkpoint a paper could just say 'use packages as of date x'.
f6v 6y ago
The only working solution I’ve seen is using Docker container with Jupyter Lab and all the dependencies installed. I hate pulling those huge images on my 256GB MBP, but it works. Of course, only bigger labs do that, since individual researchers are often unfamiliar with Docker. However, if I run my software on HPC cluster, that’s no longer an option. The HPC at my university doesn’t allow running Docker, only Singularity containers(which isn’t supported on Mac).
qrohlf 6y ago
I’ve had some brief run-ins with R, and it doesn’t surprise me that it doesn’t have a versioning story for packages, and that the patched-in system described here is based on dates rather than something like a SHA or version number... My favorite description of the language comes from http://arrgh.tim-smith.us/ http://arrgh.tim-smith.us/: > R is a shockingly dreadful language for an exceptionally useful data analysis environment. I feel like this is just one more data point to support that statement.
- jcheng 6y ago
  Packrat and its successor renv are the most popular package management systems for R, and they are based on versions/SHAs and lockfiles, like most other languages today. https://rstudio.github.io/packrat/ https://rstudio.github.io/packrat/ https://rstudio.github.io/renv/articles/renv.html https://rstudio.github.io/renv/articles/renv.html
GlennS 6y ago
I know of two other existing solutions to this, although I don't know enough to compare. I don't think either of these tick all the author's boxes. Microsoft MRAN https://mran.microsoft.com/ https://mran.microsoft.com/ > For the purpose of reproducibility, MRAN hosts daily snapshots of the CRAN R packages and R releases as far back as Sept. 17, 2014. MRAN doesn't seem to be very well known or used in the R community, but I don't really know why? Separately, Nix https://nixos.org/ https://nixos.org/ also solves this problem for lots of different languages, but is difficult to get started with and still a bit rough around the edges. Probably not a good recommendation for a typical analyst or academic at this point.
- chalst 6y ago
  The article discusses MRAN in footnote 5, when arguing against the MRAN-based 'checkpoint' approach. Nixpkg/Nixos is obviously a useful technology for reproducibility, but note that the output of Nix scripts can depend on the time the system was built, the contents of URLs and the system architecture unless care is taken.
  - GlennS 6y ago
    So it does, I missed that!
  - myWindoonn 6y ago
    This is misleading; empirically, nixpkgs is about 99% [0] reproducible already. We know that the main variance is between language-specific behaviors; Python, Rust, and C all are prone to reproducibility problems. In general, we want the output to depend on the system architecture and the contents of URLs. Nix uses hashes to require that URL contents don't change over time, which protects from those contents changing arbitrarily. [0] https://r13y.com/ https://r13y.com/
    - chalst 6y ago
      The current community around NixOS and Nixpkgs handles these issues just fine, but if 'just use Nix' was regarded as a magic bullet for reproducibility in science, I'm guessing it wouldn't work out so well.
threeseed 6y ago
Nothing about this is specific to R. If you want to guarantee reproducible results you have to use a container/image with libraries added at build time. Anytime you are relying on floating versions or downloaded libraries you will have issues.
- jdc 6y ago
  Yeah or even just vendorize your dependencies.
- tempay 6y ago
  Even this isn’t enough to be reproducible for complex numeric code as switching CPU can make a big difference with small differences being amplified. Hopefully none of those cases matter but it’s hard to definitively prove that.
  - andi999 6y ago
    If the research results depend on small differences being amplified you have a much much bigger problem. (but if course this could happen unnoticed/sloppy work)
    - bonoboTP 6y ago
      That's true but not an excuse! It's still extremely important when assessing an anomaly. If you can say "okay this is a known-good config that gets me the numbers from the paper", it's an enormous help in uncovering what leads to issues. If you can't even get those numbers, then you can suspect any number of things. Maybe you're not using the right data, maybe there was a typo, maybe someone fraudulently manually tweaked the numbers, maybe you forgot to do a step in the processing chain etc etc. There's no way to know what's going on if you can't even be sure how the original numbers were created.
winrid 6y ago
Does R not allow you to lock into versions? Is this person suggesting we never improve anything? :)
anderscarling 6y ago
I’ve yet to use it personally, but renv [1] seems to try to solve the reproducible builds problem in a way more similar to other modern package managers (e.g. by generating a lockfile). This approach enables stricter validations against tampering with the package repositories as a hash of the package can be stored in the lockfile, however it is obviously a bit more complex to use than the groundhog approach. [1]: https://github.com/rstudio/renv https://github.com/rstudio/renv
- notafraudster 6y ago
  Agreed that renv is a better solution here. Even the example code for Groundhog is not written in idiomatic R which does not inspire confidence. Simonssohn is a legend in transparent research but not primarily a coder or software tool contributor (take a look at the source for p-curve if you want to see what I mean) and I think a secondary threat to reproducibility is relying on tools that end up abandoned or deprecated or for which bugs never get fixed.
  - vharuck 6y ago
    >Even the example code for Groundhog is not written in idiomatic R which does not inspire confidence. Not to mention the given example for irreproducibility in base R looks at code that would be a bug in the script for 3.6. It's only useful to keep this reproducible if I'm debugging the script. And, in this case, anyone who's proficient with R would recognize this problem from personal experience or the many warnings in tutorials. I usually wouldn't shoot down a given example as though it disproved the existence of any example, but I don't know if there is another example. Unless old code relied on undocumented or contrary-to-documented behavior.
- prepend 6y ago
  I came here to say this. This seems like a non-issue given renv. And renv gives a more reproducible, I think, solution as it pins to versions, not dates.
timsneath 6y ago
I'm curious whether this actually solves the problem. I understand how this assists with reproducibility of packages, but the R software itself is updated frequently, as is briefly noted in the preamble to this document. Indeed, the release notes [0] are fairly transparent about the relatively long list of changes. Given this, it almost seems more dangerous to imply through this package that a particular date's results are reproducible, since unless the user has the same version of R, they may see different results anyway. [0]: https://stat.ethz.ch/pipermail/r-announce/2020/000653.html https://stat.ethz.ch/pipermail/r-announce/2020/000653.html
asperous 6y ago
Reproducible? Or deterministic? There's certainly benefits to being able to pull down research source code, and bug checking it. That's how programmers check code: tests and audits. However I think reproducing research is more often then not done "from scratch", taking a new sample, treating it, checking results. "independent verification". Re-using source code saves time, but I would argue not being able to shouldn't threaten reproducibility.
- davnn 6y ago
  > Re-using source code saves time, but I would argue not being able to shouldn't threaten reproducibility. More often than not it‘s not clear from a paper what exactly the authors did to a achieve a specific result. Being able to exactly reproduce what previous authors did should improve reproducibility; also for new samples.
  - asdff 6y ago
    It's also standard fare for typical lab work. A good paper's methods section would contain enough detail for you to go into your lab and repeat the experiment yourself, even down to the catalog number for the reagents to order from the lab supplier. Code should be no different, that's why it's encouraged that authors submit all code used in analysis and generation of figures.
    - davnn 6y ago
      The fun thing is that there are approaches that want to go beyond this kind of methodological description of a scientific process to code [0, 1]. In general I would say that the more we can remove the human aspect and inherent ambiguity of science, the better for reproducibility. See [2] for a couple of examples. [0] https://www.emeraldcloudlab.com/ https://www.emeraldcloudlab.com/ [1] https://nextjournal.com/ https://nextjournal.com/ [2] https://www.youtube.com/watch?v=L1UgdoP2aeg https://www.youtube.com/watch?v=L1UgdoP2aeg
- coliveira 6y ago
  That's something that eludes software people: reproducibility in science is the ability to create independent tests. Making software available, while useful, does very little for reproducibility from the scientific point of view.
snicker7 6y ago
Adding dates to source code? No thanks. If you want reproducibility, invest in guix. Everything else is a hack.
stewbrew 6y ago
I wish the (default) utils::install.packages function could take a version number of the requested library. I also wish library() would automatically install libraries not available on the system. (Both can be achieved with custom functions that shadow the default ones but I would like to see this functionality in the base packages.) Other than that, I think all alternatives to this "threat called R" are worse. It's telling the author has to cite a bug from 2016 for an example of a breaking change.
vhhn 6y ago
There are two camps in the R world - tidyverse and base-R (tiniverse). Its not a coincidence that the author gives an example from the tidyverse ecosystem. Authors and users of tidyverse value other things like consistency and new features over API stability and backward compatility. The base-R ecosystem is actually very stable and so the original package manager is very simple. With R spreading out from the academic environment and with many new authors breaking their packages' APIs we observe new attempts to solve the issues with dependencies (such as renv or https://rsuite.io https://rsuite.io)
st1x7 6y ago
Either I'm misunderstanding or this is a non-problem. You can specify older versions of a package when you install it. You can also manage them with packrat. As long as researchers share their language and package versions, you can fully reproduce their environment. (And the base language is really stable, almost to a fault.) This is just a bad way for the author to promote their own library for dealing with this. The way their library seems to approach this (using dates instead of versions) seems horrible too - on any given date I can have a random selection of packages in my environment, some of them up-to-date, some of them not. So unless all researchers start using the author's library (and update to the latest versions of everything just before they publish), it's only making things worse and not really solving the problem it claims to solve.
- kwertzzz 6y ago
  I full agree. With version number you can have a good sense whether a package update breaks your code or not (as long the package authors following semantic versioning). I think in julia this problem is solved quite nicely with the Project.toml (list of package that you directly dependent) and Manifest.toml file (the version numbers of the complete dependency tree which is automatically generated). It seems that in groundhog you declare only direct dependencies. Is there a way to store the full dependency tree in R ?
  - Hasnep 6y ago
    Using renv (or packrat) generates a renv.lock file with the version of every dependency.
- ekianjo 6y ago
  Packrat is deprecated now. It is recommended to use renv instead.
  - st1x7 6y ago
    Thanks, I just checked renv out (I haven't had to work with R in over a year). Renv looks much better than packrat at a first glance.
roel_v 6y ago
Apart from all the other considerations and problems with various types of package management, consider this: "Update January 6th, 2021 A reader alerted me to a bug with the current groundhog (version 1.1.0) where you cannot set the groundhog library to be a folder containing spaces in the name." So we are talking about software here that somehow made it to version 1.1 *without anyone ever using a directory with spaces in it with it". This can be interpreted in two ways: either very few people have spaces in their paths, or very few people have actually ever even tried (not even really used, I'm only talking about the most basic trial use) this package. I'm not a betting man, but if I were, I know where I'd put my money...
- deleted 6y ago
  [deleted]
- bayindirh 6y ago
  As I can see from the researchers in our cluster and my own academic research, most people still avoid spaces in paths and files like the plague. YMMV of course.
  - roel_v 6y ago
    Of course, and so do I. But nobody ever even encountering the situation and/or bothering to report it, that's a whole different matter.
    - bayindirh 6y ago
      My guess is people are encountering the situation, working around it and calling a day. Maybe a little note here and there but, I don't think someone would report it due to a couple of reasons. First of all, I don't think people report this type of stuff because they don't know how to report it, and secondly think it doesn't need to support this use case anyway since space is a latecomer to naming and path game.
  - dstick 6y ago
    If my own hobby python projects are anything to go by, there aren’t even folders ;-) I have a friend who taught herself R for her research and it was basically one big procedural codebase.
samch93 6y ago
Can recommend the paper "A Reproducible Data Analysis Workflow with R Markdown, Git, Make, and Docker" by Peikert and Brandmaier [1], which shows a much more robust approach to reproducibility. [1] https://psyarxiv.com/8xzqy/ https://psyarxiv.com/8xzqy/
- mslip 6y ago
  Thanks!
tpoacher 6y ago
A very nice and useful library indeed; though I don't think the article really needed to sound so doomsdayish and apocalyptic about it.
wodenokoto 6y ago
Wow, that's a lot of pessimism for a fairly elegant solution to the fact that almost no R code has package versioning defined. I think the major sales point here is: > A nice feature of groundhog is that it makes 'retrofitting' existing code quite easy. If you come across a script that no longer works, you can change its library() statements for groundhog.library() ones, using as the groundhog.day the date the code was probably written (say when it was posted on the internet), and it may work again. I don't know how good ratpack is now a days. I've never met an R application that uses it, but at my old work, we would take a dated snapshot of CRAN at the beginning of every new project. If we needed to update a package we could then "update CRAN" for that project. When productionising a project it would be frozen to a date in CRAN.
- nojito 6y ago
  >Wow, that's a lot of pessimism for a fairly elegant solution to the fact that almost no R code has package versioning defined. This isn't true. https://mran.microsoft.com/documents/rro/reproducibility https://mran.microsoft.com/documents/rro/reproducibility https://rstudio.github.io/packrat/ https://rstudio.github.io/packrat/
ppod 6y ago
Very clickbaity headline. The problem described is real, but just as real or worse in other statistical software, so it's not 'R' as a whole that poses a threat to reproducibility.
AuthorizedCust 6y ago
His example is poor. dplyr wasn’t even at 1.0 in 2016. It was only 0.50: https://blog.rstudio.com/2016/06/27/dplyr-0-5-0/ https://blog.rstudio.com/2016/06/27/dplyr-0-5-0/. Of course one might expect breaking changes in a maturing package.
torcete 6y ago
I hope all these efforts for reproducibility and avoiding library resolution conflicts consolidate into one solution.
hermitcrab 6y ago
Being able to assemble a solution from parts (as in R packages) is super flexible. But complex and potentially brittle. Reproducability is a big problem all around. When I create releases I put the binaries as well as the source in version control, because changes in tools/libraries etc mean that I probably won't be able to create the exact same binary several years later from the same source. There is always a tradeoff between flexibility and simplicity. Clearly software needs to be able to change, or you are never going to be able to improve it or fix bugs. And an assembly of constantly changing parts is clearly going to come with its own challenges. My own software product, Easy Data Transform (which competes with R to some extent) trades off some flexibility for simplicity by having a single set of binaries for each platform. You can't add any components (without hacking). So the same version of software should always give the same result.
hprotagonist 6y ago
seems like a fairly esoteric way to spell “lockfile with hashes”, but hey, R seems fairly esoteric to me anyway.
oli5679 6y ago
I find the miniconda docker image quite useful for making reproducible R environments. You can install specific package versions recorded in environment.yml file. There are probably many ways to do this but this is an approach I like. https://docs.anaconda.com/anaconda/user-guide/tasks/using-r-language/ https://docs.anaconda.com/anaconda/user-guide/tasks/using-r-... https://hub.docker.com/r/continuumio/miniconda https://hub.docker.com/r/continuumio/miniconda
paultopia 6y ago
This language about "threat" seems a bit overblown. Especially when we ask: compared to what? Some commercial package where different versions might have different and poorly documented data storage formats? (Have you ever tried to read an old SPSS or SAS or STATA data file in any reasonable environment? It is a nightmare.) Excel??
Railsify 6y ago
This is not a problem the R poses, this is a problem that people pose. If you want to run the same code you need to bundle the source code of all packages and archive it with the data. This is not an R problem at all.
dracodoc 6y ago
Title aside, the purposed solution just - use Microsoft MRAN which did the heavy lifting of hosting archives - use date instead of version - install package automatically in first time (which pacman::p_load has been doing for ages) and easier to use in script level. It's not coincidence that most package manager solutions used version instead of date to control the environment: - A paper published on 2017 may used a date in 2017.10.01, but there is a high possibility that some of the dependency packages might be of earlier date, unless the author update packages every day/week, which is not a good habit anyway because updating too frequently will break things more frequently. - Then how can you reproduce the environment using a date? The underlying assumption that all packages will be latest till that date simply doesn't hold. That's why packrat/renv etc will use a lock file to record all package versions, and why you will need a project to manage libraries, because you will need to maintain different library environments and cannot install to same location. Yet the author take installing all packages to a single location as a feature since you don't need to install same package again, and try to avoid project and prefer script as much as possible when doing reproducible research?
SoSoRoCoCo 6y ago
> The problem is that packages are constantly being updated, and sometimes those updates are not backwards compatible. Uh oh, someone just discovered the modern programming landscape! Python, Node, R, Rust, and other langs/OSes with package managers are at the mercy of volunteers who keep important packages healthy. Once issues stop being fixed, y'all better have local copies. This used to be predominantly an OS issue, now it is a language issue, too.
- swsieber 6y ago
  > The problem is that packages are constantly being updated, and sometimes those updates are not backwards compatible. > Python, Node, R, Rust, Correct me if I'm wrong, but for binary programs, a lock file easily mitigates these issue. I know Node and Rust both support lock files.
  - SoSoRoCoCo 6y ago
    Yes, you're right. That's what makes lock files so important. I think we're past worrying about those wheels/npms/pkgs disappearing from the internet. My concern is more about packages going stale and don't peer-match with other packages that evolve, or major versions that change results: not so much for R pkgs but there have been cases of major versions breaking existing projects, or requiring significant effort to update. (One example that zinged me is the FFI interface for Node. The "official" package hasn't been touched in years, and the "replacement", FFI-NAPI, is still has lots of open issues. We were using in-house fixes for some time.)
mespe 6y ago
While I agree with many of the negative comments here about issues with how this is implemented, the tone of some comments is... not great. To the point that I would be reluctant to share work I do in R on Hacker News, which is not helping anyone. Just a reminder: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- h2odragon 6y ago
  No kidding. Seems like an elegant solution to a potential problem to me. I've only used R at the "poke stick at numbers" level, but this would have been a useful addition to that trivial use.

161 points by snakeboy 6y ago

This title is an exceedingly hot take for someone who wrote a new package manager. Also, it appears that Groundhog is itself a CRAN package and the author recommends installing with install.packages(). So is the author committing to never making any backwards incompatible updates to their new package?

resonantjacket5 6y ago
Your take seems a bit 'hot' too? How else would you install the cran packages without using install.packages? Unless if you want them to recursively install it using groundhog but that seems unnecessary. As long as you have the timestamp it should work, though I assume there will be some edge case. What you're saying is like don't use pip because you don't install it using pip? Or don't use package-lock.json because you can't install npm through npm?
- cat199 6y ago
  > Your take seems a bit 'hot' too? OP is not claiming that Groundhog itself is a threat to the R language ecosystem itself, whereas the author is claiming that the R language is itself a threat to Science itself...
- scottmcdot 6y ago
  Someone correct me if I'm wrong, but can't you copy and paste the package folder into your libpath directory and R can load it that way with actually running install.packages()?
  - vharuck 6y ago
    Usually, yes. However, it is possible for a package to have code that only runs when it is installed. If you just copy-paste, it won't be run.
- cauthon 6y ago
  No, I’m saying don’t call CRAN a “threat to reproducible science” and then make your solution a CRAN package
coolreader18 6y ago
> So is the author committing to never making any backwards incompatible updates to their new package? Well, yes, probably. It's not all that hard, and groundhog seems to have a fairly simple API anyways. And groundhog still uses CRAN packages, it just brings a method of pinning them to a specific version.
bsza 6y ago
I think it's more like a Wayback Machine for R programs, since the author of a science paper isn't required to use groundhog. You can just provide it the date the article was published, which you already know, and it reconstructs how the program worked on that day. Also, because groundhog isn't made for the author to use, whether or not the interface changes is irrelevant. You'll never encounter library(groundhog) in a paper.
- st1x7 6y ago
  > and it reconstructs how the program worked on that day. It reconstructs how the fully updated version of everything worked that day which isn't necessarily the same as the researcher's environment. It's a horrible idea to use dates instead of package versions for this. The author's library doesn't solve the problem it claims to solve.
  - SCLeo 6y ago
    If I am understanding this correctly, the problem is that the paper authors do not provide a specific version or a package.json equivalent. In that case, using dates seem to be the only choice.
    - st1x7 6y ago
      Even if that's the case, using dates isn't a solution because dates don't give you the build that the researcher used. Date of publication is different from the date when the code ran and there is no guarantee that the researcher ran the latest version of every dependency that was available to them anyway. In fact that's very unlikely considering that some their libraries might require older versions. It might not even be possible to take the latest version of every package and use them in the same environment.
      - SCLeo 6y ago
        So, what is your better alternative then? I honestly believe using version available at that date is better than using the latest version.
jsmith99 6y ago
That's the problem. This package is very similar to Microsoft's checkpoint package which is based on Microsoft's MRAN snapshots, and this package also uses MRAN. The article explains the difference is that this package allows you to specify the date in the code itself, whereas checkpoint is used to set a whole installation to a specific date. But this is no advantage as it means code will stop working if the groundhog package changes, whereas with checkpoint a paper could just say 'use packages as of date x'.

f6v 6y ago

The only working solution I’ve seen is using Docker container with Jupyter Lab and all the dependencies installed. I hate pulling those huge images on my 256GB MBP, but it works. Of course, only bigger labs do that, since individual researchers are often unfamiliar with Docker. However, if I run my software on HPC cluster, that’s no longer an option. The HPC at my university doesn’t allow running Docker, only Singularity containers(which isn’t supported on Mac).

qrohlf 6y ago

I’ve had some brief run-ins with R, and it doesn’t surprise me that it doesn’t have a versioning story for packages, and that the patched-in system described here is based on dates rather than something like a SHA or version number... My favorite description of the language comes from http://arrgh.tim-smith.us/ http://arrgh.tim-smith.us/: > R is a shockingly dreadful language for an exceptionally useful data analysis environment. I feel like this is just one more data point to support that statement.

jcheng 6y ago
Packrat and its successor renv are the most popular package management systems for R, and they are based on versions/SHAs and lockfiles, like most other languages today. https://rstudio.github.io/packrat/ https://rstudio.github.io/packrat/ https://rstudio.github.io/renv/articles/renv.html https://rstudio.github.io/renv/articles/renv.html

GlennS 6y ago

I know of two other existing solutions to this, although I don't know enough to compare. I don't think either of these tick all the author's boxes. Microsoft MRAN https://mran.microsoft.com/ https://mran.microsoft.com/ > For the purpose of reproducibility, MRAN hosts daily snapshots of the CRAN R packages and R releases as far back as Sept. 17, 2014. MRAN doesn't seem to be very well known or used in the R community, but I don't really know why? Separately, Nix https://nixos.org/ https://nixos.org/ also solves this problem for lots of different languages, but is difficult to get started with and still a bit rough around the edges. Probably not a good recommendation for a typical analyst or academic at this point.

chalst 6y ago
The article discusses MRAN in footnote 5, when arguing against the MRAN-based 'checkpoint' approach. Nixpkg/Nixos is obviously a useful technology for reproducibility, but note that the output of Nix scripts can depend on the time the system was built, the contents of URLs and the system architecture unless care is taken.
- GlennS 6y ago
  So it does, I missed that!
- myWindoonn 6y ago
  This is misleading; empirically, nixpkgs is about 99% [0] reproducible already. We know that the main variance is between language-specific behaviors; Python, Rust, and C all are prone to reproducibility problems. In general, we want the output to depend on the system architecture and the contents of URLs. Nix uses hashes to require that URL contents don't change over time, which protects from those contents changing arbitrarily. [0] https://r13y.com/ https://r13y.com/
  - chalst 6y ago
    The current community around NixOS and Nixpkgs handles these issues just fine, but if 'just use Nix' was regarded as a magic bullet for reproducibility in science, I'm guessing it wouldn't work out so well.

threeseed 6y ago

Nothing about this is specific to R. If you want to guarantee reproducible results you have to use a container/image with libraries added at build time. Anytime you are relying on floating versions or downloaded libraries you will have issues.

jdc 6y ago
Yeah or even just vendorize your dependencies.
tempay 6y ago
Even this isn’t enough to be reproducible for complex numeric code as switching CPU can make a big difference with small differences being amplified. Hopefully none of those cases matter but it’s hard to definitively prove that.
- andi999 6y ago
  If the research results depend on small differences being amplified you have a much much bigger problem. (but if course this could happen unnoticed/sloppy work)
  - bonoboTP 6y ago
    That's true but not an excuse! It's still extremely important when assessing an anomaly. If you can say "okay this is a known-good config that gets me the numbers from the paper", it's an enormous help in uncovering what leads to issues. If you can't even get those numbers, then you can suspect any number of things. Maybe you're not using the right data, maybe there was a typo, maybe someone fraudulently manually tweaked the numbers, maybe you forgot to do a step in the processing chain etc etc. There's no way to know what's going on if you can't even be sure how the original numbers were created.

winrid 6y ago

Does R not allow you to lock into versions? Is this person suggesting we never improve anything? :)

anderscarling 6y ago

I’ve yet to use it personally, but renv [1] seems to try to solve the reproducible builds problem in a way more similar to other modern package managers (e.g. by generating a lockfile). This approach enables stricter validations against tampering with the package repositories as a hash of the package can be stored in the lockfile, however it is obviously a bit more complex to use than the groundhog approach. [1]: https://github.com/rstudio/renv https://github.com/rstudio/renv

notafraudster 6y ago
Agreed that renv is a better solution here. Even the example code for Groundhog is not written in idiomatic R which does not inspire confidence. Simonssohn is a legend in transparent research but not primarily a coder or software tool contributor (take a look at the source for p-curve if you want to see what I mean) and I think a secondary threat to reproducibility is relying on tools that end up abandoned or deprecated or for which bugs never get fixed.
- vharuck 6y ago
  >Even the example code for Groundhog is not written in idiomatic R which does not inspire confidence. Not to mention the given example for irreproducibility in base R looks at code that would be a bug in the script for 3.6. It's only useful to keep this reproducible if I'm debugging the script. And, in this case, anyone who's proficient with R would recognize this problem from personal experience or the many warnings in tutorials. I usually wouldn't shoot down a given example as though it disproved the existence of any example, but I don't know if there is another example. Unless old code relied on undocumented or contrary-to-documented behavior.
prepend 6y ago
I came here to say this. This seems like a non-issue given renv. And renv gives a more reproducible, I think, solution as it pins to versions, not dates.

timsneath 6y ago

I'm curious whether this actually solves the problem. I understand how this assists with reproducibility of packages, but the R software itself is updated frequently, as is briefly noted in the preamble to this document. Indeed, the release notes [0] are fairly transparent about the relatively long list of changes. Given this, it almost seems more dangerous to imply through this package that a particular date's results are reproducible, since unless the user has the same version of R, they may see different results anyway. [0]: https://stat.ethz.ch/pipermail/r-announce/2020/000653.html https://stat.ethz.ch/pipermail/r-announce/2020/000653.html

asperous 6y ago

Reproducible? Or deterministic? There's certainly benefits to being able to pull down research source code, and bug checking it. That's how programmers check code: tests and audits. However I think reproducing research is more often then not done "from scratch", taking a new sample, treating it, checking results. "independent verification". Re-using source code saves time, but I would argue not being able to shouldn't threaten reproducibility.

davnn 6y ago
> Re-using source code saves time, but I would argue not being able to shouldn't threaten reproducibility. More often than not it‘s not clear from a paper what exactly the authors did to a achieve a specific result. Being able to exactly reproduce what previous authors did should improve reproducibility; also for new samples.
- asdff 6y ago
  It's also standard fare for typical lab work. A good paper's methods section would contain enough detail for you to go into your lab and repeat the experiment yourself, even down to the catalog number for the reagents to order from the lab supplier. Code should be no different, that's why it's encouraged that authors submit all code used in analysis and generation of figures.
  - davnn 6y ago
    The fun thing is that there are approaches that want to go beyond this kind of methodological description of a scientific process to code [0, 1]. In general I would say that the more we can remove the human aspect and inherent ambiguity of science, the better for reproducibility. See [2] for a couple of examples. [0] https://www.emeraldcloudlab.com/ https://www.emeraldcloudlab.com/ [1] https://nextjournal.com/ https://nextjournal.com/ [2] https://www.youtube.com/watch?v=L1UgdoP2aeg https://www.youtube.com/watch?v=L1UgdoP2aeg
coliveira 6y ago
That's something that eludes software people: reproducibility in science is the ability to create independent tests. Making software available, while useful, does very little for reproducibility from the scientific point of view.

snicker7 6y ago

Adding dates to source code? No thanks. If you want reproducibility, invest in guix. Everything else is a hack.

stewbrew 6y ago

I wish the (default) utils::install.packages function could take a version number of the requested library. I also wish library() would automatically install libraries not available on the system. (Both can be achieved with custom functions that shadow the default ones but I would like to see this functionality in the base packages.) Other than that, I think all alternatives to this "threat called R" are worse. It's telling the author has to cite a bug from 2016 for an example of a breaking change.

vhhn 6y ago

There are two camps in the R world - tidyverse and base-R (tiniverse). Its not a coincidence that the author gives an example from the tidyverse ecosystem. Authors and users of tidyverse value other things like consistency and new features over API stability and backward compatility. The base-R ecosystem is actually very stable and so the original package manager is very simple. With R spreading out from the academic environment and with many new authors breaking their packages' APIs we observe new attempts to solve the issues with dependencies (such as renv or https://rsuite.io https://rsuite.io)

st1x7 6y ago

Either I'm misunderstanding or this is a non-problem. You can specify older versions of a package when you install it. You can also manage them with packrat. As long as researchers share their language and package versions, you can fully reproduce their environment. (And the base language is really stable, almost to a fault.) This is just a bad way for the author to promote their own library for dealing with this. The way their library seems to approach this (using dates instead of versions) seems horrible too - on any given date I can have a random selection of packages in my environment, some of them up-to-date, some of them not. So unless all researchers start using the author's library (and update to the latest versions of everything just before they publish), it's only making things worse and not really solving the problem it claims to solve.

kwertzzz 6y ago
I full agree. With version number you can have a good sense whether a package update breaks your code or not (as long the package authors following semantic versioning). I think in julia this problem is solved quite nicely with the Project.toml (list of package that you directly dependent) and Manifest.toml file (the version numbers of the complete dependency tree which is automatically generated). It seems that in groundhog you declare only direct dependencies. Is there a way to store the full dependency tree in R ?
- Hasnep 6y ago
  Using renv (or packrat) generates a renv.lock file with the version of every dependency.
ekianjo 6y ago
Packrat is deprecated now. It is recommended to use renv instead.
- st1x7 6y ago
  Thanks, I just checked renv out (I haven't had to work with R in over a year). Renv looks much better than packrat at a first glance.

roel_v 6y ago

Apart from all the other considerations and problems with various types of package management, consider this: "Update January 6th, 2021 A reader alerted me to a bug with the current groundhog (version 1.1.0) where you cannot set the groundhog library to be a folder containing spaces in the name." So we are talking about software here that somehow made it to version 1.1 *without anyone ever using a directory with spaces in it with it". This can be interpreted in two ways: either very few people have spaces in their paths, or very few people have actually ever even tried (not even really used, I'm only talking about the most basic trial use) this package. I'm not a betting man, but if I were, I know where I'd put my money...

deleted 6y ago
[deleted]
bayindirh 6y ago
As I can see from the researchers in our cluster and my own academic research, most people still avoid spaces in paths and files like the plague. YMMV of course.
- roel_v 6y ago
  Of course, and so do I. But nobody ever even encountering the situation and/or bothering to report it, that's a whole different matter.
  - bayindirh 6y ago
    My guess is people are encountering the situation, working around it and calling a day. Maybe a little note here and there but, I don't think someone would report it due to a couple of reasons. First of all, I don't think people report this type of stuff because they don't know how to report it, and secondly think it doesn't need to support this use case anyway since space is a latecomer to naming and path game.
- dstick 6y ago
  If my own hobby python projects are anything to go by, there aren’t even folders ;-) I have a friend who taught herself R for her research and it was basically one big procedural codebase.

samch93 6y ago

Can recommend the paper "A Reproducible Data Analysis Workflow with R Markdown, Git, Make, and Docker" by Peikert and Brandmaier [1], which shows a much more robust approach to reproducibility. [1] https://psyarxiv.com/8xzqy/ https://psyarxiv.com/8xzqy/

mslip 6y ago
Thanks!

tpoacher 6y ago

A very nice and useful library indeed; though I don't think the article really needed to sound so doomsdayish and apocalyptic about it.

wodenokoto 6y ago

Wow, that's a lot of pessimism for a fairly elegant solution to the fact that almost no R code has package versioning defined. I think the major sales point here is: > A nice feature of groundhog is that it makes 'retrofitting' existing code quite easy. If you come across a script that no longer works, you can change its library() statements for groundhog.library() ones, using as the groundhog.day the date the code was probably written (say when it was posted on the internet), and it may work again. I don't know how good ratpack is now a days. I've never met an R application that uses it, but at my old work, we would take a dated snapshot of CRAN at the beginning of every new project. If we needed to update a package we could then "update CRAN" for that project. When productionising a project it would be frozen to a date in CRAN.

nojito 6y ago
>Wow, that's a lot of pessimism for a fairly elegant solution to the fact that almost no R code has package versioning defined. This isn't true. https://mran.microsoft.com/documents/rro/reproducibility https://mran.microsoft.com/documents/rro/reproducibility https://rstudio.github.io/packrat/ https://rstudio.github.io/packrat/

ppod 6y ago

Very clickbaity headline. The problem described is real, but just as real or worse in other statistical software, so it's not 'R' as a whole that poses a threat to reproducibility.

AuthorizedCust 6y ago

His example is poor. dplyr wasn’t even at 1.0 in 2016. It was only 0.50: https://blog.rstudio.com/2016/06/27/dplyr-0-5-0/ https://blog.rstudio.com/2016/06/27/dplyr-0-5-0/. Of course one might expect breaking changes in a maturing package.

torcete 6y ago

I hope all these efforts for reproducibility and avoiding library resolution conflicts consolidate into one solution.

hermitcrab 6y ago

Being able to assemble a solution from parts (as in R packages) is super flexible. But complex and potentially brittle. Reproducability is a big problem all around. When I create releases I put the binaries as well as the source in version control, because changes in tools/libraries etc mean that I probably won't be able to create the exact same binary several years later from the same source. There is always a tradeoff between flexibility and simplicity. Clearly software needs to be able to change, or you are never going to be able to improve it or fix bugs. And an assembly of constantly changing parts is clearly going to come with its own challenges. My own software product, Easy Data Transform (which competes with R to some extent) trades off some flexibility for simplicity by having a single set of binaries for each platform. You can't add any components (without hacking). So the same version of software should always give the same result.

hprotagonist 6y ago

seems like a fairly esoteric way to spell “lockfile with hashes”, but hey, R seems fairly esoteric to me anyway.

oli5679 6y ago

I find the miniconda docker image quite useful for making reproducible R environments. You can install specific package versions recorded in environment.yml file. There are probably many ways to do this but this is an approach I like. https://docs.anaconda.com/anaconda/user-guide/tasks/using-r-language/ https://docs.anaconda.com/anaconda/user-guide/tasks/using-r-... https://hub.docker.com/r/continuumio/miniconda https://hub.docker.com/r/continuumio/miniconda

paultopia 6y ago

This language about "threat" seems a bit overblown. Especially when we ask: compared to what? Some commercial package where different versions might have different and poorly documented data storage formats? (Have you ever tried to read an old SPSS or SAS or STATA data file in any reasonable environment? It is a nightmare.) Excel??

Railsify 6y ago

This is not a problem the R poses, this is a problem that people pose. If you want to run the same code you need to bundle the source code of all packages and archive it with the data. This is not an R problem at all.

dracodoc 6y ago

Title aside, the purposed solution just - use Microsoft MRAN which did the heavy lifting of hosting archives - use date instead of version - install package automatically in first time (which pacman::p_load has been doing for ages) and easier to use in script level. It's not coincidence that most package manager solutions used version instead of date to control the environment: - A paper published on 2017 may used a date in 2017.10.01, but there is a high possibility that some of the dependency packages might be of earlier date, unless the author update packages every day/week, which is not a good habit anyway because updating too frequently will break things more frequently. - Then how can you reproduce the environment using a date? The underlying assumption that all packages will be latest till that date simply doesn't hold. That's why packrat/renv etc will use a lock file to record all package versions, and why you will need a project to manage libraries, because you will need to maintain different library environments and cannot install to same location. Yet the author take installing all packages to a single location as a feature since you don't need to install same package again, and try to avoid project and prefer script as much as possible when doing reproducible research?

SoSoRoCoCo 6y ago

> The problem is that packages are constantly being updated, and sometimes those updates are not backwards compatible. Uh oh, someone just discovered the modern programming landscape! Python, Node, R, Rust, and other langs/OSes with package managers are at the mercy of volunteers who keep important packages healthy. Once issues stop being fixed, y'all better have local copies. This used to be predominantly an OS issue, now it is a language issue, too.

swsieber 6y ago
> The problem is that packages are constantly being updated, and sometimes those updates are not backwards compatible. > Python, Node, R, Rust, Correct me if I'm wrong, but for binary programs, a lock file easily mitigates these issue. I know Node and Rust both support lock files.
- SoSoRoCoCo 6y ago
  Yes, you're right. That's what makes lock files so important. I think we're past worrying about those wheels/npms/pkgs disappearing from the internet. My concern is more about packages going stale and don't peer-match with other packages that evolve, or major versions that change results: not so much for R pkgs but there have been cases of major versions breaking existing projects, or requiring significant effort to update. (One example that zinged me is the FFI interface for Node. The "official" package hasn't been touched in years, and the "replacement", FFI-NAPI, is still has lots of open issues. We were using in-house fixes for some time.)

mespe 6y ago

While I agree with many of the negative comments here about issues with how this is implemented, the tone of some comments is... not great. To the point that I would be reluctant to share work I do in R on Hacker News, which is not helping anyone. Just a reminder: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

h2odragon 6y ago
No kidding. Seems like an elegant solution to a potential problem to me. I've only used R at the "poke stick at numbers" level, but this would have been a useful addition to that trivial use.