3 ms·
I've followed Hail and applaud the Broad Institute's work wrt establishing better bioinformatics software and toolkits so I hope this doesn't come as rude, but
by jessfyi 4y ago
I've followed Hail and applaud the Broad Institute's work wrt establishing better bioinformatics software and toolkits so I hope this doesn't come as rude, but I can't imagine an instance in a real industry or academic workflow where you need 300ms feedback from an experiment to "maintain flow" considering how long experiments on data that large (especially exome sequencing!) take overall? My (likely lacking) imagination aside I guess what I'm really saying is that I don't know what's preventing the usecase you've described from being performed locally considering there'd be even less latency?
- danking00 4y ago300ms is my ideal latency, but we don’t achieve that under all circumstances. Even for blob storage, I see as much as 100ms latency. That said, my laptop has maybe 8 cores. Even if I had 0ms reads from an SSD, I’m compute bound for some tasks. Moreover, I think we have differing definitions of “experiment”. In the context of a sequencing study, I think an “experiment” can be as simple as answering the hypothesis: does the missingness of a genotype correlate with any sample metadata (e.g. sequencing platform). You might try to test that hypothesis by looking at a PC1-PC2 plot with points colored by sequencing platform where the PCA is conducted on the 0/1 indicator matrix of missingness. In the dry lab, that is what I mean by experiment. By that definition, a scientist does many experiments a day. Particularly for sequencing studies, these experiments are data-intensive, I need to run a simple computation on a lot of data to confirm the hypothesis.