I counted to at least 10 awk components in a recent bioinformatics/cheminformatics workflow we wrote (Search for 'awk' in these lines: https://github.com/pharmb

I counted to at least 10 awk components in a recent bioinformatics/cheminformatics workflow we wrote (Search for 'awk' in these lines: https://github.com/pharmbio/ptp-project/blob/aa91f1/exp/20180426-wo-drugbank/wo_drugbank_wf.go#L177-L329 https://github.com/pharmbio/ptp-project/blob/aa91f1/exp/2018... ). For the curious, the workflow builds ML models to predict binding to unwanted proteins for new drug compounds, based on drug molecule-protein target interaction data, and is implemented with out Go-based SciPipe workflow library [0]. The rawdata is a 18GB tsv file (ExcapeDB [1]), and AWK really really shines for this. I sometimes wish we had an SQL-like language for expressing all these computations, because we ended up doing some pretty complex join-like filtering and stuff. But from my tests with SQLite, I would not get a query answer until after like ~5 minutes with SQLite, so I gave up. With AWK, I can do a "head -n 10" on the result of the AWK operation (or on the input data file), and immediately verify that it works as expected. This seems to be one of the strongest points of the unix tool philosophy: Ability to check partial results in no time, and so iterate quicker on developing long "pipelines" of chained commands. It also fits perfectly as a way to build scipipe workflow components, since having the component code in a separate language (AWK), rather than as inline Go, which is also possible in SciPipe, it is super-easy to send this component of to the HPC resource manager (given that we have a shared parallel filesystem): Just prepend the SLURM [2] salloc command with its parameter, before the command. [0] http://scipipe.org http://scipipe.org [1] https://zenodo.org/record/173258 https://zenodo.org/record/173258 [2] https://slurm.schedmd.com/ https://slurm.schedmd.com/