4 ms·

For bioinformatics, take a look at Nextflow. I personally think it is miles ahead of the competition having reviewed about a dozen options out there. This look

For bioinformatics, take a look at Nextflow. I personally think it is miles ahead of the competition having reviewed about a dozen options out there. This looks useful, but can it submit jobs to cloud compute clusters or HPC systems and operate locally? Maybe I’m missing the point in terms of the purpose.

zekrioca 7y ago
It seems to be used mainly for generic operations (locally available in localhost). However, commands like 'sbatch' (HPC manager system) could theoretically be used to manage the infrastructure.. however, again, I dunno if there would have an integration of such commands natively with guix, though it would be interesting..
- fwip 7y ago
  I'm not sure if you're talking about nextflow, but it has support for many different job managers (slurm plus like ten more), AWS batch, and Google cloud pipelines.
  - zekrioca 7y ago
    No, I was indeed talking about guix.
snackematician 7y ago
Nextflow is indeed the best option today. It's sad that it's based on Groovy though, which seems past its heyday as a language/community. I'm very excited that this is based on Scheme -- I've long wanted a lispy workflow language! I'm a little concerned about the coverage of the Guix package manager though. I guess instead of writing Dockerfiles the user would have to learn to write Guix packages. The "Getting Started" example uses samtools so I guess this is oriented towards a similar bioinformatics audience. However without HPC/Cloud support it's probably not too practical, yet. Addendum: Listened to the FOSDEM 2019 talk, seems like it does support Docker and HPC. However I need AWS Batch support for it to be really useful to me, hopefully that will be implemented at some point.
- zekrioca 7y ago
  If there is Docker/HPC/AWS support in the system through its local commands, there will be support.. Check the cluster mode setup in guix documentation (https://www.gnu.org/software/guix/manual/en/guix.html https://www.gnu.org/software/guix/manual/en/guix.html) Edit: adding gnu guix manual link
- zmmmmm 7y ago
  > s sad that it's based on Groovy though It's actually very cool since you can drop in any Java library you like, which is particularly nice in the bioinformatics space where HTSJDK, Picard and co. give you enormous power in that space.
totalperspectiv 7y ago
I've been migrating to nextflow over the last month. It really is fantastic. I even use it in place of bash scripts for little things now just because resume is so nice.
vk3wtf 7y ago
Nextflow does not capture the software required, which is important for reproducibility. Containers are not a solution for this as it only shifts the problem up a level. My efforts with BioNix (https://github.com/PapenfussLab/bionix https://github.com/PapenfussLab/bionix) achieve reproducible pipelines by using Nix to capture the software, workflow, and handle execution either locally, on a compute cluster, or HPC. Guixwl looks similar to BioNix, though BioNix is a thin layer of Nix expressions and Guixwl seems to be more then that and could be more general. BioNix is targeted at bioinformatics and just builds on nixpkgs.

zekrioca 7y ago

It seems to be used mainly for generic operations (locally available in localhost). However, commands like 'sbatch' (HPC manager system) could theoretically be used to manage the infrastructure.. however, again, I dunno if there would have an integration of such commands natively with guix, though it would be interesting..

fwip 7y ago
I'm not sure if you're talking about nextflow, but it has support for many different job managers (slurm plus like ten more), AWS batch, and Google cloud pipelines.
- zekrioca 7y ago
  No, I was indeed talking about guix.

snackematician 7y ago

Nextflow is indeed the best option today. It's sad that it's based on Groovy though, which seems past its heyday as a language/community. I'm very excited that this is based on Scheme -- I've long wanted a lispy workflow language! I'm a little concerned about the coverage of the Guix package manager though. I guess instead of writing Dockerfiles the user would have to learn to write Guix packages. The "Getting Started" example uses samtools so I guess this is oriented towards a similar bioinformatics audience. However without HPC/Cloud support it's probably not too practical, yet. Addendum: Listened to the FOSDEM 2019 talk, seems like it does support Docker and HPC. However I need AWS Batch support for it to be really useful to me, hopefully that will be implemented at some point.

zekrioca 7y ago
If there is Docker/HPC/AWS support in the system through its local commands, there will be support.. Check the cluster mode setup in guix documentation (https://www.gnu.org/software/guix/manual/en/guix.html https://www.gnu.org/software/guix/manual/en/guix.html) Edit: adding gnu guix manual link
zmmmmm 7y ago
> s sad that it's based on Groovy though It's actually very cool since you can drop in any Java library you like, which is particularly nice in the bioinformatics space where HTSJDK, Picard and co. give you enormous power in that space.

totalperspectiv 7y ago

I've been migrating to nextflow over the last month. It really is fantastic. I even use it in place of bash scripts for little things now just because resume is so nice.

vk3wtf 7y ago

Nextflow does not capture the software required, which is important for reproducibility. Containers are not a solution for this as it only shifts the problem up a level. My efforts with BioNix (https://github.com/PapenfussLab/bionix https://github.com/PapenfussLab/bionix) achieve reproducible pipelines by using Nix to capture the software, workflow, and handle execution either locally, on a compute cluster, or HPC. Guixwl looks similar to BioNix, though BioNix is a thin layer of Nix expressions and Guixwl seems to be more then that and could be more general. BioNix is targeted at bioinformatics and just builds on nixpkgs.