4 ms·

Question at the meta-level: what software/platforms are producing, analyzing, modifying & consuming this data? Is it possible some legacy systems that produce

Question at the meta-level: what software/platforms are producing, analyzing, modifying & consuming this data? Is it possible some legacy systems that produce and consume these massive files on-site would more sensibly run in the cloud, directly & selectively accessing the data chunks they need over fast backbone connections? Also, is there any room for an rsync type approach, sending compressed deltas rather than naively sending huge files that may be redundant? Not to say disintermediating a BigCO expensive patented vendor-locked-in MLPOS (Market Leading Piece Of Shit) doesn't sound exciting -- it does. I'm just curious and a bit skeptical that it is always necessary to mass-copy all this data over and over.

rcthompson 7y ago
The systems that produce massive DNA sequence files are machines reading the sequences of actual physical DNA molecules. The cloud can't sequence DNA, as far as I know.
- ncmncm 7y ago
  Most DNA sequences are only slightly different from sequences you already have. There ought to be mature differs by now that would enable you to send just differences. If not, there is a huge opportunity for somebody.
  - vmilner 7y ago
    There are such differs. https://en.wikipedia.org/wiki/Compression_of_Genomic_Sequencing_Data https://en.wikipedia.org/wiki/Compression_of_Genomic_Sequenc...
    - ncmncm 7y ago
      Thank you. I see that the state of the art remains very primitive, 18kx results notwithstanding.
  - rcthompson 7y ago
    Sure, you can compress the sequences by diffing against the published genome sequence, and many tools exist to do just that. But that alone maxes out at 50% compression, because the other half of the file is the quality scores that quantify the confidence level of each base call, for which there is no reference. Of course there are patterns in the quality scores, and you can compress these as well using the usual techniques for general compression, but it's going to limit how much you can compress without information loss. In addition, the sequencer assigns an unique ID to each sequence read. It's often ok to discard this information, but occasionally you need it in order to trace a specific problematic sequence through the entire analysis pipeline. So if you want lossless compression, you have to compress these as well. They should be rather compressible, since they are typically a prefix global to the file followed by a number that increments by 1 for each sequence. But for a general compressor you can't just automatically assume that, because according to the (lack of a) file format spec, each ID could be anything as long as they're unique within the file. So if you want guaranteed lossless compression of any FASTQ file, your compressor needs to be sufficiently general. Beyond that, compression of sequences by encoding only the differences from a specific reference sequence somewhat ties you down to using a specific reference, since that reference sequence is required for decompression. This is inconvenient because the reference sequence for a species is updated over time, and if you want to use the new reference, you'll need to decompress with the old reference and then re-compress with the new one. Do you want to do that for all your data, every time the reference is updated? Reference-based compression also means the compressed files are no longer self-contained, which may not be acceptable for certain use cases. None of these issues are fundamentally impossible to address, but the point is that it's not nearly as simple as it sounds, and anything fancy you do to try and compress a sequence file will generally impose some additional requirements on what the receiver needs to do in order to read the file. And many receivers are not very technically inclined researchers whose plates are already full of other things they need to be doing besides figuring out how to decompress your new unfamiliar sequence compression format,
    - ncmncm 7y ago
      Thank you for this enlightening explanation. I should have guessed that it, like most things, is not as easy as it looks.
- danbmil99 7y ago
  > The cloud can't sequence DNA, as far as I know. Not yet. Famous last words?
ncmncm 7y ago
Cloud vendors make it cheap to move data in, expensive to move it out, but also expensive (and curiously inconvenient) to keep there.
burntwater 7y ago
Aspera is the standard in broadcast video for transferring video files. Typically you wouldn't want just a small piece, you would want the entire hour-long, 3 terabyte video. Also, you would still need to get the data into the cloud in the first place.

rcthompson 7y ago

The systems that produce massive DNA sequence files are machines reading the sequences of actual physical DNA molecules. The cloud can't sequence DNA, as far as I know.

ncmncm 7y ago
Most DNA sequences are only slightly different from sequences you already have. There ought to be mature differs by now that would enable you to send just differences. If not, there is a huge opportunity for somebody.
- vmilner 7y ago
  There are such differs. https://en.wikipedia.org/wiki/Compression_of_Genomic_Sequencing_Data https://en.wikipedia.org/wiki/Compression_of_Genomic_Sequenc...
  - ncmncm 7y ago
    Thank you. I see that the state of the art remains very primitive, 18kx results notwithstanding.
- rcthompson 7y ago
  Sure, you can compress the sequences by diffing against the published genome sequence, and many tools exist to do just that. But that alone maxes out at 50% compression, because the other half of the file is the quality scores that quantify the confidence level of each base call, for which there is no reference. Of course there are patterns in the quality scores, and you can compress these as well using the usual techniques for general compression, but it's going to limit how much you can compress without information loss. In addition, the sequencer assigns an unique ID to each sequence read. It's often ok to discard this information, but occasionally you need it in order to trace a specific problematic sequence through the entire analysis pipeline. So if you want lossless compression, you have to compress these as well. They should be rather compressible, since they are typically a prefix global to the file followed by a number that increments by 1 for each sequence. But for a general compressor you can't just automatically assume that, because according to the (lack of a) file format spec, each ID could be anything as long as they're unique within the file. So if you want guaranteed lossless compression of any FASTQ file, your compressor needs to be sufficiently general. Beyond that, compression of sequences by encoding only the differences from a specific reference sequence somewhat ties you down to using a specific reference, since that reference sequence is required for decompression. This is inconvenient because the reference sequence for a species is updated over time, and if you want to use the new reference, you'll need to decompress with the old reference and then re-compress with the new one. Do you want to do that for all your data, every time the reference is updated? Reference-based compression also means the compressed files are no longer self-contained, which may not be acceptable for certain use cases. None of these issues are fundamentally impossible to address, but the point is that it's not nearly as simple as it sounds, and anything fancy you do to try and compress a sequence file will generally impose some additional requirements on what the receiver needs to do in order to read the file. And many receivers are not very technically inclined researchers whose plates are already full of other things they need to be doing besides figuring out how to decompress your new unfamiliar sequence compression format,
  - ncmncm 7y ago
    Thank you for this enlightening explanation. I should have guessed that it, like most things, is not as easy as it looks.
danbmil99 7y ago
> The cloud can't sequence DNA, as far as I know. Not yet. Famous last words?

ncmncm 7y ago

Cloud vendors make it cheap to move data in, expensive to move it out, but also expensive (and curiously inconvenient) to keep there.

burntwater 7y ago

Aspera is the standard in broadcast video for transferring video files. Typically you wouldn't want just a small piece, you would want the entire hour-long, 3 terabyte video. Also, you would still need to get the data into the cloud in the first place.