4 ms·

Genuine curious of the use case of command line by biology students. Can you elaborate?

deleted 7y ago
[deleted]
joycian 7y ago
When you have DNA sequencing data, you often have to use bash / command line to glue together the various tooling. As an example: https://bedtools.readthedocs.io/en/latest/ https://bedtools.readthedocs.io/en/latest/
- cure 7y ago
  Yes, this is very common. But there are also many people who use Common Workflow Language (https://www.commonwl.org/ https://www.commonwl.org/) for this purpose. CWL allows for workflow portability between execution engines. The open source engine my team creates (Arvados - https://arvados.org https://arvados.org) is ideal for large-scale processing work. We store, process and manage petabytes of genome data with it, and execute workflows that span hundreds of simultaneous compute nodes.
  - joycian 7y ago
    This looks very cool! Thanks for sharing it. I'm definitely going to mention this in the next technical meeting at the research group.
JdeBP 7y ago
There is an entire field called bioinformatics. There is a fair amount of text and flat file database processing in it. * https://bioinformatics.stackexchange.com/ https://bioinformatics.stackexchange.com/ * https://en.wikipedia.org/wiki/Bioinformatics https://en.wikipedia.org/wiki/Bioinformatics
asdff 7y ago
A raw sequencing file can be massive, like 80-100gb per sample, and if you have a sample size of 1000, storage alone is going to be a huge requirement. Then you need to load these huge files into memory, so you are gonna want like 128gb of ram. Then you need to process your files, and if you want to get it done this week, you want at least a dozen cores working on your job. These requirements are far too expensive for an individual or even a lab to own a capable workstation, so everyone gets familiar with the command line and cluster computing instead, and they use whichever computer is cheap or comfortable (or even a phone if you wanted to be cheeky) and just run the jobs on the cluster.

deleted 7y ago

[deleted]

When you have DNA sequencing data, you often have to use bash / command line to glue together the various tooling. As an example: https://bedtools.readthedocs.io/en/latest/ https://bedtools.readthedocs.io/en/latest/

cure 7y ago
Yes, this is very common. But there are also many people who use Common Workflow Language (https://www.commonwl.org/ https://www.commonwl.org/) for this purpose. CWL allows for workflow portability between execution engines. The open source engine my team creates (Arvados - https://arvados.org https://arvados.org) is ideal for large-scale processing work. We store, process and manage petabytes of genome data with it, and execute workflows that span hundreds of simultaneous compute nodes.
- joycian 7y ago
  This looks very cool! Thanks for sharing it. I'm definitely going to mention this in the next technical meeting at the research group.

JdeBP 7y ago

There is an entire field called bioinformatics. There is a fair amount of text and flat file database processing in it. * https://bioinformatics.stackexchange.com/ https://bioinformatics.stackexchange.com/ * https://en.wikipedia.org/wiki/Bioinformatics https://en.wikipedia.org/wiki/Bioinformatics

asdff 7y ago

A raw sequencing file can be massive, like 80-100gb per sample, and if you have a sample size of 1000, storage alone is going to be a huge requirement. Then you need to load these huge files into memory, so you are gonna want like 128gb of ram. Then you need to process your files, and if you want to get it done this week, you want at least a dozen cores working on your job. These requirements are far too expensive for an individual or even a lab to own a capable workstation, so everyone gets familiar with the command line and cluster computing instead, and they use whichever computer is cheap or comfortable (or even a phone if you wanted to be cheeky) and just run the jobs on the cluster.