4 ms·

Slight tangent, but forcing biology undergrads to work their way through this would probably help them a lot when they suddenly get saddled with some sequencing

Slight tangent, but forcing biology undergrads to work their way through this would probably help them a lot when they suddenly get saddled with some sequencing output and have literally never heard of Bash let alone used it. My lab of 40+ people has maybe 2 or 3 people who know how to use the command line.

yitchelle 7y ago
Genuine curious of the use case of command line by biology students. Can you elaborate?
- deleted 7y ago
  [deleted]
- joycian 7y ago
  When you have DNA sequencing data, you often have to use bash / command line to glue together the various tooling. As an example: https://bedtools.readthedocs.io/en/latest/ https://bedtools.readthedocs.io/en/latest/
  - cure 7y ago
    Yes, this is very common. But there are also many people who use Common Workflow Language (https://www.commonwl.org/ https://www.commonwl.org/) for this purpose. CWL allows for workflow portability between execution engines. The open source engine my team creates (Arvados - https://arvados.org https://arvados.org) is ideal for large-scale processing work. We store, process and manage petabytes of genome data with it, and execute workflows that span hundreds of simultaneous compute nodes.
    - joycian 7y ago
      This looks very cool! Thanks for sharing it. I'm definitely going to mention this in the next technical meeting at the research group.
- JdeBP 7y ago
  There is an entire field called bioinformatics. There is a fair amount of text and flat file database processing in it. * https://bioinformatics.stackexchange.com/ https://bioinformatics.stackexchange.com/ * https://en.wikipedia.org/wiki/Bioinformatics https://en.wikipedia.org/wiki/Bioinformatics
- asdff 7y ago
  A raw sequencing file can be massive, like 80-100gb per sample, and if you have a sample size of 1000, storage alone is going to be a huge requirement. Then you need to load these huge files into memory, so you are gonna want like 128gb of ram. Then you need to process your files, and if you want to get it done this week, you want at least a dozen cores working on your job. These requirements are far too expensive for an individual or even a lab to own a capable workstation, so everyone gets familiar with the command line and cluster computing instead, and they use whichever computer is cheap or comfortable (or even a phone if you wanted to be cheeky) and just run the jobs on the cluster.
UweSchmidt 7y ago
As a rule, for any given topic, experts on the topic propose to add this topic to the curriculum. While this topic would indeed be useful, many other topics that are also not in the curriculum would be equally useful. Does your lab work well with 2-3 people being able to set up the data processing for the rest of the team? Learning the unix commandline is considerable effort and requires regular practice. Instead of learning bash, biologists could spend their time reading relevant papers in their specialization or brush up on statistics.
- ImaCake 7y ago
  >Does your lab work well with 2-3 people being able to set up the data processing for the rest of the team? Those people who know their stuff aren't doing that job because they have their own data to deal with. There are people with 10,000 line proteomics outputs who manually search for missing entries. There is at least one student with single cell RNAseq data which she can't even look at (never heard of fastqc or the LESS command). It is frankly disasterous.
acidburnNSA 7y ago
I'm a nuclear engineer and I noticed that there was no official venue where people were learning the particular computer basics that I consider essential for daily operation of a computer as an engineering tool. I figured maybe other fields have the issue too and I got it together enough to out out my first and only ebook on the topic. It's higher level than OP here, but still focuses mostly on the command line. Anyway, it's called Digital Superpowers and is $10. If anyone wants a preview chapter or anything I can send you one. https://digitalsuperpowers.com https://digitalsuperpowers.com
- sjellis 7y ago
  I really like the concept of this book. Your choice of topics is interesting, because it's so ecletic.
  - acidburnNSA 7y ago
    Thanks. Yeah it's an odd set, totally derived from the things I personally have found really useful in my computational physics career as well as my computer-related hobbies.

yitchelle 7y ago

Genuine curious of the use case of command line by biology students. Can you elaborate?

deleted 7y ago
[deleted]
joycian 7y ago
When you have DNA sequencing data, you often have to use bash / command line to glue together the various tooling. As an example: https://bedtools.readthedocs.io/en/latest/ https://bedtools.readthedocs.io/en/latest/
- cure 7y ago
  Yes, this is very common. But there are also many people who use Common Workflow Language (https://www.commonwl.org/ https://www.commonwl.org/) for this purpose. CWL allows for workflow portability between execution engines. The open source engine my team creates (Arvados - https://arvados.org https://arvados.org) is ideal for large-scale processing work. We store, process and manage petabytes of genome data with it, and execute workflows that span hundreds of simultaneous compute nodes.
  - joycian 7y ago
    This looks very cool! Thanks for sharing it. I'm definitely going to mention this in the next technical meeting at the research group.
JdeBP 7y ago
There is an entire field called bioinformatics. There is a fair amount of text and flat file database processing in it. * https://bioinformatics.stackexchange.com/ https://bioinformatics.stackexchange.com/ * https://en.wikipedia.org/wiki/Bioinformatics https://en.wikipedia.org/wiki/Bioinformatics
asdff 7y ago
A raw sequencing file can be massive, like 80-100gb per sample, and if you have a sample size of 1000, storage alone is going to be a huge requirement. Then you need to load these huge files into memory, so you are gonna want like 128gb of ram. Then you need to process your files, and if you want to get it done this week, you want at least a dozen cores working on your job. These requirements are far too expensive for an individual or even a lab to own a capable workstation, so everyone gets familiar with the command line and cluster computing instead, and they use whichever computer is cheap or comfortable (or even a phone if you wanted to be cheeky) and just run the jobs on the cluster.

UweSchmidt 7y ago

As a rule, for any given topic, experts on the topic propose to add this topic to the curriculum. While this topic would indeed be useful, many other topics that are also not in the curriculum would be equally useful. Does your lab work well with 2-3 people being able to set up the data processing for the rest of the team? Learning the unix commandline is considerable effort and requires regular practice. Instead of learning bash, biologists could spend their time reading relevant papers in their specialization or brush up on statistics.

ImaCake 7y ago
>Does your lab work well with 2-3 people being able to set up the data processing for the rest of the team? Those people who know their stuff aren't doing that job because they have their own data to deal with. There are people with 10,000 line proteomics outputs who manually search for missing entries. There is at least one student with single cell RNAseq data which she can't even look at (never heard of fastqc or the LESS command). It is frankly disasterous.

acidburnNSA 7y ago

I'm a nuclear engineer and I noticed that there was no official venue where people were learning the particular computer basics that I consider essential for daily operation of a computer as an engineering tool. I figured maybe other fields have the issue too and I got it together enough to out out my first and only ebook on the topic. It's higher level than OP here, but still focuses mostly on the command line. Anyway, it's called Digital Superpowers and is $10. If anyone wants a preview chapter or anything I can send you one. https://digitalsuperpowers.com https://digitalsuperpowers.com

sjellis 7y ago
I really like the concept of this book. Your choice of topics is interesting, because it's so ecletic.
- acidburnNSA 7y ago
  Thanks. Yeah it's an odd set, totally derived from the things I personally have found really useful in my computational physics career as well as my computer-related hobbies.