4 ms·

I'm sure this will be useful for some folks, but I'll stick with 'git' to track changes and 'make' to organize my work. It seems as though this project is a so

I'm sure this will be useful for some folks, but I'll stick with 'git' to track changes and 'make' to organize my work. It seems as though this project is a sort of system for creating Makefiles, and that would be great for folks who are unfamiliar with them. I'm not sure of the audience, though. At least in my research area, there are mainly two groups. 1. People who use latex (and scripting languages) are comfortable with writing Makefiles. 2. People who work in excel (etc) and incorporate results into msword. Neither group seems a likely candidate for this (admittedly intriguing) software.

conformist 2y ago
There are many people in group 1 in academia eg in physics and maths that are comfortable with latex and scripting languages but mostly use email to share files. Anything that helps them organise their collaborative work better without having to deal with git helps (eg see eg success of overleaf).
- ska 2y ago
  Part of the problem is that git is a fairly poor fit for these workflows. I spent time getting some mathematicians working together via version control rather than email, it was a bit of a mixed bag even using something simpler (e.g. svn). Eventual we moved back to email, except the rule was email me your update as a reply to the version you edited, and I scripted something to put it all into a repo on my end to manage merges etc. Worked ok. Better than the version where we locked access for edit but people forgot to unlock and went off to a conference... If I was doing the same now, I'd probably set up on github, give each person a branch off main, and give them scripts for "send my changes" and "update other changes" - then manage all the merges behind the scenes for anyone who didn't want to bother. I think expecting everyone in a working group to acquire the skills to deal with merge issues properly etc. is too far if they don't do any significant software work already. In the latter case., teach them.
__MatrixMan__ 2y ago
It's easy to collect and verify metadata involving the hashes of intermediate artifacts such that readers can observe it and trust that the charts correspond with the data because they trust whoever published the metadata. This could be automatic, just part of the reader. The trouble with make is that unless you're very disciplined or very lucky, if you build the images and documents on your machine and I do the same on mine, we're going to get artifacts that look similar but hash differently, if for no other reason than that there's a timestamp showing up somewhere and throwing it off (though often for more concerning reasons involving the versions of whatever your Makefile is calling). That prevents any kind of automated consensus finding about the functional dependency between the artifacts. Now reviewers must rebuild the artifacts themselves and compare the outputs visually before they can be assured that the data and visualizations are indeed connected. So if we want to get to a place where papers can be more readily trusted--a place where the parts of research that can be replicated automatically, are replicated automatically, then we're going to need something that provides a bit more structure than make (something like nix, with a front end like Jacquard lab notebook). The idea that we could take some verifiable computational step and represent it in a UI such that the status of that verification is accessible, rather than treating the makefile as an authoritative black box... I think it's rather exciting. Even if I don't really care about the UI so much, having the computational structure be accessible is important.
- XorNot 2y ago
  Here's the thing though: you're trying to solve a problem here which doesn't exist. In physical science, no one commits academic fraud by manipulating a difference between the graphs they publish and the data they collected...they just enter bad data to start with. Or apply extremely invalid statistical methods or the like. You can't fix this by trying to attest the data pipeline.
  - __MatrixMan__ 2y ago
    I'm not really trying to address fraud. Most of the time when I try to recreate a computational result from a paper, things go poorly. I want to fix that. Recently I found one where the authors must've mislabeled something because the data for mutant A actually corresponded with the plot for mutant B. Other times it'll take days of tinkering just to figure out which versions of the dependencies are necessary to make it work at all. None of that sort of sleuthing should've required a human in the loop at all. I should be able to hold one thing constant (be it the data or the pipeline), change the other, and rebuild the paper to determine whether the conclusions are invariant to the change I made. Human patience for applying skepticism to complex things is scarce. I want to automate as much of replication as possible so that what skepticism is available is applied more effectively. It would just be a nicer world to live in.
jpeloquin 2y ago
Even in group 1, when I go back to a project that I haven't worked on in years, it would be helpful to be able to query the build system to list the dependencies of a particular artifact, including data dependencies. I.e., reverse dependency lookup. Also list which files could change as a consequence of changing another artifact. And return results based on what the build actually did, not just the rules as specified. I think make can't do this because it has no ability to hash & cache results. Newer build systems like Bazel, Please, and Pants should be able to do this but I haven't used them much yet.

conformist 2y ago

There are many people in group 1 in academia eg in physics and maths that are comfortable with latex and scripting languages but mostly use email to share files. Anything that helps them organise their collaborative work better without having to deal with git helps (eg see eg success of overleaf).

ska 2y ago
Part of the problem is that git is a fairly poor fit for these workflows. I spent time getting some mathematicians working together via version control rather than email, it was a bit of a mixed bag even using something simpler (e.g. svn). Eventual we moved back to email, except the rule was email me your update as a reply to the version you edited, and I scripted something to put it all into a repo on my end to manage merges etc. Worked ok. Better than the version where we locked access for edit but people forgot to unlock and went off to a conference... If I was doing the same now, I'd probably set up on github, give each person a branch off main, and give them scripts for "send my changes" and "update other changes" - then manage all the merges behind the scenes for anyone who didn't want to bother. I think expecting everyone in a working group to acquire the skills to deal with merge issues properly etc. is too far if they don't do any significant software work already. In the latter case., teach them.

__MatrixMan__ 2y ago

It's easy to collect and verify metadata involving the hashes of intermediate artifacts such that readers can observe it and trust that the charts correspond with the data because they trust whoever published the metadata. This could be automatic, just part of the reader. The trouble with make is that unless you're very disciplined or very lucky, if you build the images and documents on your machine and I do the same on mine, we're going to get artifacts that look similar but hash differently, if for no other reason than that there's a timestamp showing up somewhere and throwing it off (though often for more concerning reasons involving the versions of whatever your Makefile is calling). That prevents any kind of automated consensus finding about the functional dependency between the artifacts. Now reviewers must rebuild the artifacts themselves and compare the outputs visually before they can be assured that the data and visualizations are indeed connected. So if we want to get to a place where papers can be more readily trusted--a place where the parts of research that can be replicated automatically, are replicated automatically, then we're going to need something that provides a bit more structure than make (something like nix, with a front end like Jacquard lab notebook). The idea that we could take some verifiable computational step and represent it in a UI such that the status of that verification is accessible, rather than treating the makefile as an authoritative black box... I think it's rather exciting. Even if I don't really care about the UI so much, having the computational structure be accessible is important.

XorNot 2y ago
Here's the thing though: you're trying to solve a problem here which doesn't exist. In physical science, no one commits academic fraud by manipulating a difference between the graphs they publish and the data they collected...they just enter bad data to start with. Or apply extremely invalid statistical methods or the like. You can't fix this by trying to attest the data pipeline.
- __MatrixMan__ 2y ago
  I'm not really trying to address fraud. Most of the time when I try to recreate a computational result from a paper, things go poorly. I want to fix that. Recently I found one where the authors must've mislabeled something because the data for mutant A actually corresponded with the plot for mutant B. Other times it'll take days of tinkering just to figure out which versions of the dependencies are necessary to make it work at all. None of that sort of sleuthing should've required a human in the loop at all. I should be able to hold one thing constant (be it the data or the pipeline), change the other, and rebuild the paper to determine whether the conclusions are invariant to the change I made. Human patience for applying skepticism to complex things is scarce. I want to automate as much of replication as possible so that what skepticism is available is applied more effectively. It would just be a nicer world to live in.

jpeloquin 2y ago

Even in group 1, when I go back to a project that I haven't worked on in years, it would be helpful to be able to query the build system to list the dependencies of a particular artifact, including data dependencies. I.e., reverse dependency lookup. Also list which files could change as a consequence of changing another artifact. And return results based on what the build actually did, not just the rules as specified. I think make can't do this because it has no ability to hash & cache results. Newer build systems like Bazel, Please, and Pants should be able to do this but I haven't used them much yet.