3 ms·

> Are you saying this can be optimised to fit inside a single 10 core server in terms of compute loads? I'm currently employed to write software which does DN

> Are you saying this can be optimised to fit inside a single 10 core server in terms of compute loads? I'm currently employed to write software which does DNA analysis. DNA is known for being Big Data. Some applications are very compute intensive and others are very data-relation intensive. The very compute intensive applications process about 1GB of data in about 30 minutes on a 32 core Xeon 6xxx with 32GB of RAM assigned to it. The data-relation intensive application processes about 350GB of data in 15 minutes on a single core of the same CPU but with 700GB of RAM assigned to it. Both are heavily optimized but in different ways. So without knowing more about what you're doing with that data, it's hard to say.

vagab0nd 7y ago
That's very interesting. Do you mind sharing some more details on what those computations are? Where can I learn more about these DNA analysis use cases?
- inetknght 7y ago
  I've found that Wikipedia has really good high-level information for a lot of the subject matter. For use cases, the business sells direct-to-consumer DNA tests. I've worked on several of the analysis products' softwares. The compute-heavy workload calculates edit distance [0] of short paired-end sequenced DNA [1] vs the human genome [2]. There is open source software to manipulate the FASTA/FASTQ [3] and SAM files [4] and run the calculations [5]. The aligned file is processed in a couple of minutes to genetic variation report [6] which is used for some of the analysis products that were purchased. One popular product will give you a haplogroup [7] which basically tells you where you are in a genetic tree. The relationship estimator uses a different sequencing technology and basically consumes a CSV file from the sequencer's manufacturer. It uses a proprietary algorithm to calculate centimorgans [8]. That then gives relationship estimates between you and other people who've purchased the product. [0]: https://en.wikipedia.org/wiki/Needleman%E2%80%93Wunsch_algorithm https://en.wikipedia.org/wiki/Needleman%E2%80%93Wunsch_algor... [1]: https://en.wikipedia.org/wiki/Next-generation_sequencing https://en.wikipedia.org/wiki/Next-generation_sequencing [2]: https://en.wikipedia.org/wiki/Reference_genome https://en.wikipedia.org/wiki/Reference_genome [3]: https://en.wikipedia.org/wiki/FASTQ_format https://en.wikipedia.org/wiki/FASTQ_format [4]: https://github.com/samtools/hts-specs https://github.com/samtools/hts-specs [5]: https://en.wikipedia.org/wiki/List_of_sequence_alignment_software https://en.wikipedia.org/wiki/List_of_sequence_alignment_sof... [6]: https://en.wikipedia.org/wiki/Variant_Call_Format https://en.wikipedia.org/wiki/Variant_Call_Format [7]: https://en.wikipedia.org/wiki/Haplogroup https://en.wikipedia.org/wiki/Haplogroup [8]: https://en.wikipedia.org/wiki/Centimorgan https://en.wikipedia.org/wiki/Centimorgan
hnews_account_1 7y ago
So my entire dataset is ~24 x 250GB files. That 24 number can be larger if I can find an efficient way of processing each 250GB chunk. Each 250GB chunk is actually stock tick data so it has 500 stocks inside it. A heavily traded stock takes up ~10 GB of memory while a very thinly traded stock can top out at just 700 MB. While I hope that each 250GB chunk has everything in order and I can separate it cleanly, I don't trust it. So I broke up all the files by stock in an embarrassingly parallelised code I wrote and got 500 separate files. However, the problem is, I'd want to process these 500 files parallely so each stock gets only so much memory (and hence, my original constraint remains). And for each stock, I want the data to be queried on timestamps, so I need a way to quickly say "I have data at 9am on Thursday, I want data from 4pm Wednesday to 8am Thursday to create a model". I figured the best way to do this efficiently was to have the code create a massive database for all the stocks and query it efficiently in SQL. But I'm stuck there due to a lack of tools.
- inetknght 7y ago
  Your problem sounds quite similar to a lot of FinTech interview questions for software engineers. They're solved problems but solutions aren't easy. I have no doubt I could solve your problems. I honestly don't care to do so here though. I imagine there's a software development/engineering team at your work or school you could ask for guidance though.
  - hnews_account_1 7y ago
    > I have no doubt I could solve your problems. I honestly don't care to do so here though. That's fair. I did not think it'd be such a difficult problem when I first set out to do it myself. But every single turn leading to a dead end kinda bummed me out. I'm going to finally resort to the database method of storing all the data in one query-able file and work off of that.

vagab0nd 7y ago

That's very interesting. Do you mind sharing some more details on what those computations are? Where can I learn more about these DNA analysis use cases?

inetknght 7y ago
I've found that Wikipedia has really good high-level information for a lot of the subject matter. For use cases, the business sells direct-to-consumer DNA tests. I've worked on several of the analysis products' softwares. The compute-heavy workload calculates edit distance [0] of short paired-end sequenced DNA [1] vs the human genome [2]. There is open source software to manipulate the FASTA/FASTQ [3] and SAM files [4] and run the calculations [5]. The aligned file is processed in a couple of minutes to genetic variation report [6] which is used for some of the analysis products that were purchased. One popular product will give you a haplogroup [7] which basically tells you where you are in a genetic tree. The relationship estimator uses a different sequencing technology and basically consumes a CSV file from the sequencer's manufacturer. It uses a proprietary algorithm to calculate centimorgans [8]. That then gives relationship estimates between you and other people who've purchased the product. [0]: https://en.wikipedia.org/wiki/Needleman%E2%80%93Wunsch_algorithm https://en.wikipedia.org/wiki/Needleman%E2%80%93Wunsch_algor... [1]: https://en.wikipedia.org/wiki/Next-generation_sequencing https://en.wikipedia.org/wiki/Next-generation_sequencing [2]: https://en.wikipedia.org/wiki/Reference_genome https://en.wikipedia.org/wiki/Reference_genome [3]: https://en.wikipedia.org/wiki/FASTQ_format https://en.wikipedia.org/wiki/FASTQ_format [4]: https://github.com/samtools/hts-specs https://github.com/samtools/hts-specs [5]: https://en.wikipedia.org/wiki/List_of_sequence_alignment_software https://en.wikipedia.org/wiki/List_of_sequence_alignment_sof... [6]: https://en.wikipedia.org/wiki/Variant_Call_Format https://en.wikipedia.org/wiki/Variant_Call_Format [7]: https://en.wikipedia.org/wiki/Haplogroup https://en.wikipedia.org/wiki/Haplogroup [8]: https://en.wikipedia.org/wiki/Centimorgan https://en.wikipedia.org/wiki/Centimorgan

hnews_account_1 7y ago

So my entire dataset is ~24 x 250GB files. That 24 number can be larger if I can find an efficient way of processing each 250GB chunk. Each 250GB chunk is actually stock tick data so it has 500 stocks inside it. A heavily traded stock takes up ~10 GB of memory while a very thinly traded stock can top out at just 700 MB. While I hope that each 250GB chunk has everything in order and I can separate it cleanly, I don't trust it. So I broke up all the files by stock in an embarrassingly parallelised code I wrote and got 500 separate files. However, the problem is, I'd want to process these 500 files parallely so each stock gets only so much memory (and hence, my original constraint remains). And for each stock, I want the data to be queried on timestamps, so I need a way to quickly say "I have data at 9am on Thursday, I want data from 4pm Wednesday to 8am Thursday to create a model". I figured the best way to do this efficiently was to have the code create a massive database for all the stocks and query it efficiently in SQL. But I'm stuck there due to a lack of tools.

inetknght 7y ago
Your problem sounds quite similar to a lot of FinTech interview questions for software engineers. They're solved problems but solutions aren't easy. I have no doubt I could solve your problems. I honestly don't care to do so here though. I imagine there's a software development/engineering team at your work or school you could ask for guidance though.
- hnews_account_1 7y ago
  > I have no doubt I could solve your problems. I honestly don't care to do so here though. That's fair. I did not think it'd be such a difficult problem when I first set out to do it myself. But every single turn leading to a dead end kinda bummed me out. I'm going to finally resort to the database method of storing all the data in one query-able file and work off of that.