4 ms·
I really wish that I had a tool like this back in grad school. I was doing bioinformatics work and merging, chopping, and processing various datasets over many
by jboggan 14y ago
I really wish that I had a tool like this back in grad school. I was doing bioinformatics work and merging, chopping, and processing various datasets over many months. When a new version of the underlying data came out it was not an easy task to go back and re-process it through dozens of steps in Perl and R. Having a tool like this would have made it a single command to do so and also ensured repeatability and transparency in my data, something which is often sorely lacking in an academic setting.
I am one of the data engineers at Factual and though I didn't have a role in creating it I definitely enjoy using it on a day to day basis. You begin to see the utility of it when you have a dozen people working up and down a data pipeline and need to coordinate as product specs evolve or schemas change.
I also really like the tagging features - you can add specific tags to different steps in the build and run different "flavors" of your workflow depending upon what is needed. For example, you might build a workflow that collects, cleans, filters, and performs calculations on data from all over the world - but you might also want alternative versions of the build that only work on specific regions or smaller debug datasets. Tags make that really simple to do, even when many steps are shared by the different versions or the dependencies are complicated.
- xaa 14y agoAs a fellow bioinformatician I can agree that this looks quite useful. Although (since you mention R), I wonder why there's no love for R in Drake, given that R is perhaps the quintessential data processing language.
- dirtyvagabond 14y agoThere is love for R in Drake! As of about an hour ago: https://github.com/Factual/drake/commit/f63dd2630ca3e5e4a6a6baa4296d62dcd078690e https://github.com/Factual/drake/commit/f63dd2630ca3e5e4a6a6...