4 ms·

The question, why many scientific communities haven't been racing to adopt Big Data technology is a tough one. Whether or not utilization is a major factor I ca

The question, why many scientific communities haven't been racing to adopt Big Data technology is a tough one. Whether or not utilization is a major factor I can't really say. For Cuneiform, we have assumed that the major obstacle for using platforms like Hadoop or Spark directly is that these systems require a library to have a Java or Scala interface. If your software is written in, say, Python you need to either - write a wrapper (an approach taken by Crossbow [1], SeqPig [2], or BioPig [3]) - reimplement (an approach taken by CloudBurst [4] or ADAM [5]) Reimplementing a whole pipeline represents a huge upfront cost. Writing wrappers is more feasible and, accordingly, a huge amount of custom tailored pipelines and ad-hoc worklfow systems have emerged recently. So if it boils down to wrapping libraries and running them on Hadoop, then we need to make this as easy as somehow possible. Cuneiform is a step in this direction. [1] http://bowtie-bio.sourceforge.net/crossbow/index.shtml http://bowtie-bio.sourceforge.net/crossbow/index.shtml [2] http://bioinformatics.oxfordjournals.org/content/30/1/119.long http://bioinformatics.oxfordjournals.org/content/30/1/119.lo... [3] http://bioinformatics.oxfordjournals.org/content/early/2013/09/10/bioinformatics.btt528 http://bioinformatics.oxfordjournals.org/content/early/2013/... [4] http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2682523/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2682523/ [5] http://bdgenomics.org/projects/adam/ http://bdgenomics.org/projects/adam/

chubot 11y ago
The Java/JVM bias is a deficiency of open source Big Data, not bioinformatics!!! FWIW, Google doesn't have a JVM bias in terms of its big data stack, but it still has langauge biases. Bioinformatics is built on Unix tools and Unix composition AFAICT, and that philosophy needs to make its way back to the big data stack IMO. My question is what's wrong with what they're doing? If it's working, why not keep doing the same thing, rather than trying to move to Hadoop, which sounds like the wrong tool?
- joergen7 11y ago
  > The Java/JVM bias is a deficiency of open source Big Data, not bioinformatics!!! I agree. > Bioinformatics is built on Unix tools and Unix composition AFAICT That is true for most bioinformatics tools with a few important exceptions. CummeRbund, e.g., is a must for RNA-Seq and you need to drive it from R. It's hard to tell whether supporting only command line would actually be enough to conveniently describe workflows. For Cuneiform, we assume that it isn't. > what's wrong with what they're doing? It's not that moving to Big Data stacks is the ultimate answer to everything. If your workflow runs fast enough on one machine, probably it's best to change nothing. Nevertheless, we're observing a dramatic drop in sequencing costs right now. And the more this trend continues, the bigger will be the demand for scalable infrastructures in Next Generation Sequencing.

chubot 11y ago

The Java/JVM bias is a deficiency of open source Big Data, not bioinformatics!!! FWIW, Google doesn't have a JVM bias in terms of its big data stack, but it still has langauge biases. Bioinformatics is built on Unix tools and Unix composition AFAICT, and that philosophy needs to make its way back to the big data stack IMO. My question is what's wrong with what they're doing? If it's working, why not keep doing the same thing, rather than trying to move to Hadoop, which sounds like the wrong tool?

joergen7 11y ago
> The Java/JVM bias is a deficiency of open source Big Data, not bioinformatics!!! I agree. > Bioinformatics is built on Unix tools and Unix composition AFAICT That is true for most bioinformatics tools with a few important exceptions. CummeRbund, e.g., is a must for RNA-Seq and you need to drive it from R. It's hard to tell whether supporting only command line would actually be enough to conveniently describe workflows. For Cuneiform, we assume that it isn't. > what's wrong with what they're doing? It's not that moving to Big Data stacks is the ultimate answer to everything. If your workflow runs fast enough on one machine, probably it's best to change nothing. Nevertheless, we're observing a dramatic drop in sequencing costs right now. And the more this trend continues, the bigger will be the demand for scalable infrastructures in Next Generation Sequencing.