2 ms·
Sure, a 30x human WGS resequencing analysis has gotten pretty trivial over the past decade, but now we also have thousands or millions, plus expression data set
by epistasis 2y ago
Sure, a 30x human WGS resequencing analysis has gotten pretty trivial over the past decade, but now we also have thousands or millions, plus expression data sets, HI-C, etc. etc. and how to combine them. There may not be compute clusters in genomics labs anymore, because funding agencies will only pay for cloud and not hardware, but there are lots of people working on large scale computation that doesn't fit on a single node.
- jltsiren 2y agoWe actually got a new cluster recently. ~25 nodes with 128 or 192 physical cores, 2 TB RAM, and >10 TB local scratch space each. And most issues arise from the old-school practice of running many small jobs to make scheduling easier. But if you restructure your work to copy the data to local storage, run for a number of hours without accessing the network, and copy the results back to network storage, the issues tend to go away. Compared to the cluster I was using a decade ago, individual nodes are an order of magnitude faster, they can run an order of magnitude bigger jobs, and local storage is two orders of magnitude faster. Meanwhile, increases in network bandwidth have been modest. I/O has become cheap relative to compute, while network has become a scarce resource.