3 ms·
> one quirk of that space is that they work with enormous text files. Why not use some binary format like BSON? Compressing with gzip works but then you can't
by nextaccountic 4y ago
> one quirk of that space is that they work with enormous text files.
Why not use some binary format like BSON? Compressing with gzip works but then you can't query it compressed
- hoseja 4y agoBecause scientists are woefully undereducated on the finer points of computer science. Does it (eventually) produce publishable papers? is all they care about.
- exyi 4y agoIt was probably fasta of fastq format, which is pretty much line separated strings of the DNA letters. BSON won't help with that. You could try to squeeze the each of the 4 letters into 2 bits, but they use few more letters to indicate "unknown", so it's not that easy. And even the simplest compression algorithms just do that for you. And text is easier to work with in any hacked up script. There are some binary formats (BAM, I think), but people often prefer the text format anyway. When compressed, the size is pretty much the same
- nextaccountic 4y ago> And even the simplest compression algorithms just do that for you. They do, but you need to uncompress to read the N-th letter