6 ms·

What timing! I'm just experimenting with some bioinformatics codes, and wow... the formats are terrible. They're mostly sequential text files, some tens of giga

What timing! I'm just experimenting with some bioinformatics codes, and wow... the formats are terrible. They're mostly sequential text files, some tens of gigabytes in size. Simply uncompressing a file can be a significant bottleneck, as this can inherently use only 1 CPU core. Keep in mind that these files are intended to be processed on monstrously huge 128 core and 2TB memory machines! What happens is that the system uses 0.5% of its capacity to manipulate the files while the other 99.5% is heating the data center air. I'm looking at the CPU usage metric graph now, and the machine is spending half the time at 100% load and half the time at 1% load. That's half the total capacity wasted. If you find yourself in 2022 or later designing a file format intended for bulk data and you use any of the words "stream", "serialization", or "text", stop. Rethink what you have done, and this time consider that normal machines you can buy for normal money have 256 hardware threads. Soon, 512, and then 1024 in just a few years! Everything at this scale should be split into blocks for parallel processing. Shoving text files into a .tar.gz archive is not acceptable any more. I don't care what 1970s "standard" it adheres to, that just doesn't matter any more, the hardware has moved on. I think it's high time that the industry standardised on a generic "container" format to replace legacy archive file formats. Something akin to a random-access archive file like zip, but designed so that even a single file can be decoded in parallel, even if compressed. As an example of "parallel thinking", CRC or SHA style checksums are automatically a no-go. They're inherently sequential. Instead, Merkle trees would have to be used. Compression efficiency of small files with random access could be improved by using something like zstd as the internal compression algorithm, but with a shared dictionary stored separately. This would retain the advantages of 'tar' without requiring sequential decoding of files. Etc...

iafiaf 4y ago
The "bioinformatics formats" might be terrible, but they work. In fact, they are meant to be Excel-readable which keeps my collaborators (and me) happy. Coming from a CS /programming background, it is natural to feel the urge to "fix" the formats (<insert relevant XKCD>), until you realise that there are a libraries that easily handle serialization/parsing. Besides, "bioinformatic formats" is a meaningless word anyway. FASTQ, VCFs, BCL, AIRR-seq -- all different and it just works.
- jiggawatts 4y ago
  "Just works" and "I've been waiting 15 minutes now for this file to un-gzip" aren't compatible in my book, especially on a computer that should be able to process that file in seconds. Also, I'd love to see someone open a 75 GB FASTQ file in Excel.
  - Ultimatt 4y ago
    VCF at least typically uses bgzip which is essentially gzipped sections concatenated, but parallel unzipable for random access, cram is also parallelisable in the same way. Maybe you just dont know the formats and tooling so well? Im not sure anyone opens a fastq directly for viewing anymore, but they will want pile ups from a bam. The problem with bio formats isnt that they're text its that they are shit text formats too.
    - jghn 4y ago
      CRAM is a great example for some of the other people in the thread who say "just get a better format". There's been slow uptake in the larger community despite the benefits. For anyone looking to Solve Bioinformatics File Formats, it's important to understand why this is the case.
  - iafiaf 4y ago
    FASTQ is not for Excel, obviously - although you can still explore it in the shell. Nonetheless operating directly on FASTA/FASTQ files is often a "one-time" preprocessing task. You then serialize the preprocessed data and continue on from there. FASTA (and its various incantations) are not going anywhere anytime soon.
  - zmmmmm 4y ago
    Nearly all bioinfo tools operate in streaming mode which means line based gzipped formats work great as you can parallelise the processing with reading the file. Nobody ever unzips the whole file before starting to process it.
- rrwo 4y ago
maCDzP 4y ago
Have you tried pigz? It’s an implementation of gzip that enables parallel processing.
defrost 4y ago
These files are all about persistent storage for many many decades, not sub second processing. It's not that hard to pre process prior to feeding into whatever the latest processing architecture of the decade is .. but it's hell week when you're tasked with decoding some decades old non standard "only used for 18 months" "best practice at the time" obscure compression format data. Field seperated line orientated ASCII data doesn't require extensive out of band notes to understand or decipher years after you've passed on.
- cycomanic 4y ago
  > It's not that hard to pre process prior to feeding into whatever the latest processing architecture of the decade is .. but it's hell week when you're tasked with decoding some decades old non standard "only used for 18 months" "best practice at the time" obscure compression format data. That's quite disingenuous, there are very well documented data formats that have been around for decades (netCDF, HDF) those are not "obscure" formats and are infinitesimally better than text. > Field seperated line orientated ASCII data doesn't require extensive out of band notes to understand or decipher years after you've passed on. Field separated ACII data is terrible because it does not contain relevant metadata (except for column names), you quite possibly loose precision (also what even was the precision?), are slow to read, need to be additionally compressed ...
  - nanis 4y ago
    > (netCDF, HDF) those are not "obscure" formats and are infinitesimally better than text. so, not much better?
  - geysersam 4y ago
    > infinitesimally better Sounds like they better stick with the TSV format then ;)
ssivark 4y ago
Echoing my frustration... A few months ago I went through a similar search. IIRC my needs were: binary data storage (not ascii), index-based access on arrays without loading the whole file in memory, being able to attach metadata with the file contents, with interfaces for multiple scientific programming languages (Python, Julia, etc). I recall going through ASDF, BSDF and a handful of other formats, finally ending up with HDFS -- which was okay, but not fully satisfactory (I don't recall all my gripes right now).
- hanzel227 4y ago
  you're looking for zarr.
jltsiren 4y ago
Bioinformatics file formats are broken, but it's better to try to understand why they are broken before rushing to fix them. One key problem is that technology changes quickly. There are always new instruments generating new kinds of data with new properties and new features. People are using that data in new applications. Software comes many years behind the state of the art. First you need to figure out what is the exact problem the software is supposed to solve. Then you have to solve the problem and turn the prototype into a useful tool. This work is mostly done by researchers who may know a little about software engineering. By the time the situation is stable enough that software engineers who are not active researchers in the field could be useful, it's often too late to change the file formats. There is already too much legacy data and too many tools supporting the established formats. Another key problem is that the "broken" file formats are often good enough. When you have tabular data where the fields can be reasonably understood as text, a simple TSV-based format often gets the job done. Especially if individual datasets are only tens of gigabytes. By using a custom format, you avoid having to choose from many existing formats that all have their own issues. And that often guarantee you version conflicts and breaking changes in the future. Also, when it comes to parallelization, it's hard to beat running many independent jobs in parallel. While computers are getting bigger, individual problems are often not, as the underlying biological problems remain the same. In the work I do, a reasonable target system had 32 cores and 256 GB memory in 2015. That's still a reasonable target in 2022. The computers I use have become cheaper and faster, but they have not really changed.
- jiggawatts 4y ago
  > When you have tabular data where the fields can be reasonably understood as text, a simple TSV-based format often gets the job done. Especially if individual datasets are only tens of gigabytes. In what other industry is less than 0.5% utilisation accepted as "gets the job done?" TSV is a terrible format for multi-gigabyte files, because it uses line breaks. Technically it's possible to parse them in parallel, but if the format enables quoted strings with newlines, then it isn't possible to do this safely. A simple binary row-based or columnar format could be read orders of magnitude faster. No character-by-character processing needed, just "memcpy" and go.
noobermin 4y ago
>Simply uncompressing a file can be a significant bottleneck, as this can inherently use only 1 CPU core. Look this up: gnu parallel Also, with building on what the top reply to what you said, if "uncompressing is a bottleneck" and you didn't know you could uncompress multiple files at once suggests you should spend time learning about tools that exist before you jump and try to force your colleagues (who likely have much more experience than you) to adopt "new practices" that the js community will move on from in 6 months.
- Symbiote 4y ago
  TAR files can't be uncompressed in parallel. ZIP archives containing a single large file can't be uncompressed in parallel. The resulting CSV file can't be processed in parallel, as there might be quoted record delimiters. There are compressed file formats where a single file can be processed in parallel. Apache Avro and Parquet are examples I'm familiar with, but these handle columnar data (i.e. replacing CSV), not large numeric matrices etc.
  - noobermin 4y ago
    Of course...I'm assuming (perhaps incorrectly) that they are decompressing many files sequentially without knowing that you can easily run gunzip in parallel using tools like gnu parallel. The OP says they get to "ten gigabytes in size." It does not take 15 minutes to decompress a 10GB file on a modern workstation, as they complain in a child comment, unless you decompress multiple files sequentially. I've routinely compressed and decompressed 100GB+ files on 5 year old workstation class machines and it takes at most 5-ish minutes for one direction, (stress on "at most").
mnw21cam 4y ago
I'll argue the corner of bioinformatic file formats here. The main ones are FASTQ, SAM, BAM, VCF, and that format that GATK's DepthOfCoverage spews out. FASTQ is a plain text format, and is usually gzipped. This is usually not a problem, as the only thing that's going to happen to a FASTQ file is you're going to shove it through an aligner, and a single thread can un-gzip that file fast enough to keep a lot of cores busy doing the alignment. SAM is basically used for nothing, except very briefly as an output from the aligner before it is promptly converted into BAM. The BAM format is actually sensible. It's compressed and indexed. There's an alternative format out there called CRAM, which can be a little more efficient, but you need to ensure that some external files are still available in order to decompress it. VCF (and gVCF) files are text, and they are usually gzipped. Whether they are gzipped or not, they usually have an accompanying index file that allows any section to be accessed without having to sequentially read through. This is possible with the gzipped version because a variant of gzip called bgzip is used that compresses blocks of data, and locations of the start positions of those blocks are stored in the index. The DepthOfCoverage file format - OK, I don't have any defence of it. It's just huge. If you're storing the read depth for a single sample, it uses about 24 bytes per location, to store a single number that's usually less than 256. So, a typical file for a whole genome sequencing sample would be around 75GB. It also has no index. A few months ago I decided to write an alternative file format, which delta-encodes then huffman-encodes the number in blocks, and uses about 1.3 bits per location, and includes an index, so that 75GB file is now 0.5GB and is a heck of a lot faster to read. As an aside the GATK DepthOfCoverage is a fairly dire example of slow software. About 7 years ago I wrote my own DepthOfCoverage, which produces the same results but runs about 50 times faster. It wasn't hard. And because the BAM file format is sensible, yes it does access the files in parallel as you suggest. There are advantages of text file formats. The format is unlikely to be non-readable in 10 years. You can just load it up in less and have a read. The text format doesn't stop it being compressed and indexed and accessed in parallel. But yes, the data could often be stored in a more efficient manner. Finally, if you're finding that your server is regularly under-utilised, then you aren't doing load-management properly. You should use a queuing system that knows for each job how much RAM and how many CPU threads are used, and therefore how many can be run simultaneously.
IanCal 4y ago
For tabular data, parquet with zstd compression is very straight forward and quick. Columnar and split into row groups with stats on the columns within each, easily processed by many tools as standard. You can go further into arrow but parquet alone solves the majority of key issues IMO for tabular data.
- kzrdude 4y ago
  Parquet looks good but HDF5 is good just because it's hierarchical and can group together multiple related datasets. Maybe data taken in the same session on completely different axes/coordinates this is what I've missed with parquet.
jcelerier 4y ago
I had started a little bit of work towards that recently: https://github.com/celtera/uvfs https://github.com/celtera/uvfs It's very optimized towards my specific needs but could be a basis for what you mention
kortex 4y ago
> If you find yourself in 2022 or later designing a file format intended for bulk data and you use any of the words "stream", "serialization", Nit: I think I know what you are getting at, with blocks streams vs byte streams, but it's kinda hard to design a file format without serialization or byte streams. Not sure how that would work. > I think it's high time that the industry standardised on a generic "container" format to replace legacy archive file formats. I have a side project chipping away at just such a thing. It's quite daunting, so if this at all interests anyone, please comment/reach out. I'd love more of an excuse to work on this. SITO in a nutshell: - It's all based on msgpack, which does most of the heavy lift for serialization and datatype encoding - a sito stream comprises blocks, each block is a self-contained, independently decodeable msgpack array object. - each block an array of the form (type: smallint, header: optional(hashmap), data: any) - the type is either a single-byte int, or a packed int indicating a substream id - the sito primary stream comprises multiple independent substreams - there's no raw plaintext fields, but there is a plain unicode block which can be used to embed whatever plaintext metadata - substreams can each have whatever compression/codec they want - since each stream is a block stream, it's trivial to de/interleave - for data integrity, I want to do something like block-level CRC/FEC along with per-stream merkle trees but I haven't worked out the details yet - there are periodic "sync-blocks" which have a magic 8-byte sequence for starting a file, but also throughout the stream, to facilitate re-alignment of read heads - I've also been toying with the idea of using sqlite as stream indexes and as a general glue to keep track of what's going on (right now, you can arbitrarily start a new substream at any point in the primary stream, so it's hard to tell at the start of a file what's in it, sqlite pre-allocates pages so write heads can go back and update a prior index block) - nd-arrays are a particularly interesting datatype so there's an emphasis on ergonomics around handling them I plan on doing a simple PoC at some point soon showcasing SITAR, the sito archive format, with a python tarfile-like interface.
- kortex 4y ago
  I just updated the github page with my current state of notes, in case folks are curious to some of the details. https://github.com/xkortex/sito https://github.com/xkortex/sito
osigurdson 4y ago
I do feel that there is a "missing" simple/basic text format which is somewhere in between csv and json/yaml. csv has no hierarchy while json/yaml are horribly unreadable for tabular data as it has no concept of columns (see https://csvjson.com/csv2json https://csvjson.com/csv2json as an example). Maybe such a thing exists but I haven't seen it.
CodesInChaos 4y ago
> CRC style checksums are automatically a no-go. They're inherently sequential. AFAIK the linearity of CRCs allows them to be split, parallelized, and combined. This is also the case for polynomial MACs, like GHash or Poly1305.
- AlotOfReading 4y ago
  Yep, the only downside for CRC being that the recombination involves some expensive multiplications that are proportional to the total input length.
  - CodesInChaos 4y ago
    For Poly1305 you can use square-and-multiply to compute the exponentiation required for recombination, which has logarithmic cost, not linear. Are you sure a similar algorithm can't be applied to CRCs?
    - AlotOfReading 4y ago
      Sorry, I didn't phrase it well. It's also log(n) for CRCs via the same algorithm.
shpongled 4y ago
I work in proteomics, where the field decided to standardize on XML based file formats. It can take longer to parse the data out of the XML container than it does to actually run the analysis.
optimiz3 4y ago
> CRC or SHA style checksums are automatically a no-go CRC is 100% parallelizable FYI. Both in SIMD and also on a block level which can be merged.
elmolino89 4y ago
"bonus" points: * text/TSV data files made public as a requirement from the published articles may have spaces or dots in column names, missing line ends, non-unique row IDs, etc. * there is no limit as how many records can be stored in a single file. Latest dbSNP has more than 1000 millions of rows. * bunch of formats (GFF, GTF, VCF) has several TSV delimited "proper" columns (as: 1 value) and then special column where optional fields are piled in a different format, with another separator, field names etc. Real fun to parse...

iafiaf 4y ago

The "bioinformatics formats" might be terrible, but they work. In fact, they are meant to be Excel-readable which keeps my collaborators (and me) happy. Coming from a CS /programming background, it is natural to feel the urge to "fix" the formats (<insert relevant XKCD>), until you realise that there are a libraries that easily handle serialization/parsing. Besides, "bioinformatic formats" is a meaningless word anyway. FASTQ, VCFs, BCL, AIRR-seq -- all different and it just works.

jiggawatts 4y ago
"Just works" and "I've been waiting 15 minutes now for this file to un-gzip" aren't compatible in my book, especially on a computer that should be able to process that file in seconds. Also, I'd love to see someone open a 75 GB FASTQ file in Excel.
- Ultimatt 4y ago
  VCF at least typically uses bgzip which is essentially gzipped sections concatenated, but parallel unzipable for random access, cram is also parallelisable in the same way. Maybe you just dont know the formats and tooling so well? Im not sure anyone opens a fastq directly for viewing anymore, but they will want pile ups from a bam. The problem with bio formats isnt that they're text its that they are shit text formats too.
  - jghn 4y ago
    CRAM is a great example for some of the other people in the thread who say "just get a better format". There's been slow uptake in the larger community despite the benefits. For anyone looking to Solve Bioinformatics File Formats, it's important to understand why this is the case.
- iafiaf 4y ago
  FASTQ is not for Excel, obviously - although you can still explore it in the shell. Nonetheless operating directly on FASTA/FASTQ files is often a "one-time" preprocessing task. You then serialize the preprocessed data and continue on from there. FASTA (and its various incantations) are not going anywhere anytime soon.
- zmmmmm 4y ago
  Nearly all bioinfo tools operate in streaming mode which means line based gzipped formats work great as you can parallelise the processing with reading the file. Nobody ever unzips the whole file before starting to process it.
rrwo 4y ago

maCDzP 4y ago

Have you tried pigz? It’s an implementation of gzip that enables parallel processing.

defrost 4y ago

These files are all about persistent storage for many many decades, not sub second processing. It's not that hard to pre process prior to feeding into whatever the latest processing architecture of the decade is .. but it's hell week when you're tasked with decoding some decades old non standard "only used for 18 months" "best practice at the time" obscure compression format data. Field seperated line orientated ASCII data doesn't require extensive out of band notes to understand or decipher years after you've passed on.

cycomanic 4y ago
> It's not that hard to pre process prior to feeding into whatever the latest processing architecture of the decade is .. but it's hell week when you're tasked with decoding some decades old non standard "only used for 18 months" "best practice at the time" obscure compression format data. That's quite disingenuous, there are very well documented data formats that have been around for decades (netCDF, HDF) those are not "obscure" formats and are infinitesimally better than text. > Field seperated line orientated ASCII data doesn't require extensive out of band notes to understand or decipher years after you've passed on. Field separated ACII data is terrible because it does not contain relevant metadata (except for column names), you quite possibly loose precision (also what even was the precision?), are slow to read, need to be additionally compressed ...
- nanis 4y ago
  > (netCDF, HDF) those are not "obscure" formats and are infinitesimally better than text. so, not much better?
- geysersam 4y ago
  > infinitesimally better Sounds like they better stick with the TSV format then ;)

ssivark 4y ago

Echoing my frustration... A few months ago I went through a similar search. IIRC my needs were: binary data storage (not ascii), index-based access on arrays without loading the whole file in memory, being able to attach metadata with the file contents, with interfaces for multiple scientific programming languages (Python, Julia, etc). I recall going through ASDF, BSDF and a handful of other formats, finally ending up with HDFS -- which was okay, but not fully satisfactory (I don't recall all my gripes right now).

hanzel227 4y ago
you're looking for zarr.

jltsiren 4y ago

Bioinformatics file formats are broken, but it's better to try to understand why they are broken before rushing to fix them. One key problem is that technology changes quickly. There are always new instruments generating new kinds of data with new properties and new features. People are using that data in new applications. Software comes many years behind the state of the art. First you need to figure out what is the exact problem the software is supposed to solve. Then you have to solve the problem and turn the prototype into a useful tool. This work is mostly done by researchers who may know a little about software engineering. By the time the situation is stable enough that software engineers who are not active researchers in the field could be useful, it's often too late to change the file formats. There is already too much legacy data and too many tools supporting the established formats. Another key problem is that the "broken" file formats are often good enough. When you have tabular data where the fields can be reasonably understood as text, a simple TSV-based format often gets the job done. Especially if individual datasets are only tens of gigabytes. By using a custom format, you avoid having to choose from many existing formats that all have their own issues. And that often guarantee you version conflicts and breaking changes in the future. Also, when it comes to parallelization, it's hard to beat running many independent jobs in parallel. While computers are getting bigger, individual problems are often not, as the underlying biological problems remain the same. In the work I do, a reasonable target system had 32 cores and 256 GB memory in 2015. That's still a reasonable target in 2022. The computers I use have become cheaper and faster, but they have not really changed.

jiggawatts 4y ago
> When you have tabular data where the fields can be reasonably understood as text, a simple TSV-based format often gets the job done. Especially if individual datasets are only tens of gigabytes. In what other industry is less than 0.5% utilisation accepted as "gets the job done?" TSV is a terrible format for multi-gigabyte files, because it uses line breaks. Technically it's possible to parse them in parallel, but if the format enables quoted strings with newlines, then it isn't possible to do this safely. A simple binary row-based or columnar format could be read orders of magnitude faster. No character-by-character processing needed, just "memcpy" and go.

noobermin 4y ago

>Simply uncompressing a file can be a significant bottleneck, as this can inherently use only 1 CPU core. Look this up: gnu parallel Also, with building on what the top reply to what you said, if "uncompressing is a bottleneck" and you didn't know you could uncompress multiple files at once suggests you should spend time learning about tools that exist before you jump and try to force your colleagues (who likely have much more experience than you) to adopt "new practices" that the js community will move on from in 6 months.

Symbiote 4y ago
TAR files can't be uncompressed in parallel. ZIP archives containing a single large file can't be uncompressed in parallel. The resulting CSV file can't be processed in parallel, as there might be quoted record delimiters. There are compressed file formats where a single file can be processed in parallel. Apache Avro and Parquet are examples I'm familiar with, but these handle columnar data (i.e. replacing CSV), not large numeric matrices etc.
- noobermin 4y ago
  Of course...I'm assuming (perhaps incorrectly) that they are decompressing many files sequentially without knowing that you can easily run gunzip in parallel using tools like gnu parallel. The OP says they get to "ten gigabytes in size." It does not take 15 minutes to decompress a 10GB file on a modern workstation, as they complain in a child comment, unless you decompress multiple files sequentially. I've routinely compressed and decompressed 100GB+ files on 5 year old workstation class machines and it takes at most 5-ish minutes for one direction, (stress on "at most").

mnw21cam 4y ago

I'll argue the corner of bioinformatic file formats here. The main ones are FASTQ, SAM, BAM, VCF, and that format that GATK's DepthOfCoverage spews out. FASTQ is a plain text format, and is usually gzipped. This is usually not a problem, as the only thing that's going to happen to a FASTQ file is you're going to shove it through an aligner, and a single thread can un-gzip that file fast enough to keep a lot of cores busy doing the alignment. SAM is basically used for nothing, except very briefly as an output from the aligner before it is promptly converted into BAM. The BAM format is actually sensible. It's compressed and indexed. There's an alternative format out there called CRAM, which can be a little more efficient, but you need to ensure that some external files are still available in order to decompress it. VCF (and gVCF) files are text, and they are usually gzipped. Whether they are gzipped or not, they usually have an accompanying index file that allows any section to be accessed without having to sequentially read through. This is possible with the gzipped version because a variant of gzip called bgzip is used that compresses blocks of data, and locations of the start positions of those blocks are stored in the index. The DepthOfCoverage file format - OK, I don't have any defence of it. It's just huge. If you're storing the read depth for a single sample, it uses about 24 bytes per location, to store a single number that's usually less than 256. So, a typical file for a whole genome sequencing sample would be around 75GB. It also has no index. A few months ago I decided to write an alternative file format, which delta-encodes then huffman-encodes the number in blocks, and uses about 1.3 bits per location, and includes an index, so that 75GB file is now 0.5GB and is a heck of a lot faster to read. As an aside the GATK DepthOfCoverage is a fairly dire example of slow software. About 7 years ago I wrote my own DepthOfCoverage, which produces the same results but runs about 50 times faster. It wasn't hard. And because the BAM file format is sensible, yes it does access the files in parallel as you suggest. There are advantages of text file formats. The format is unlikely to be non-readable in 10 years. You can just load it up in less and have a read. The text format doesn't stop it being compressed and indexed and accessed in parallel. But yes, the data could often be stored in a more efficient manner. Finally, if you're finding that your server is regularly under-utilised, then you aren't doing load-management properly. You should use a queuing system that knows for each job how much RAM and how many CPU threads are used, and therefore how many can be run simultaneously.

IanCal 4y ago

For tabular data, parquet with zstd compression is very straight forward and quick. Columnar and split into row groups with stats on the columns within each, easily processed by many tools as standard. You can go further into arrow but parquet alone solves the majority of key issues IMO for tabular data.

kzrdude 4y ago
Parquet looks good but HDF5 is good just because it's hierarchical and can group together multiple related datasets. Maybe data taken in the same session on completely different axes/coordinates this is what I've missed with parquet.

jcelerier 4y ago

I had started a little bit of work towards that recently: https://github.com/celtera/uvfs https://github.com/celtera/uvfs It's very optimized towards my specific needs but could be a basis for what you mention

kortex 4y ago

> If you find yourself in 2022 or later designing a file format intended for bulk data and you use any of the words "stream", "serialization", Nit: I think I know what you are getting at, with blocks streams vs byte streams, but it's kinda hard to design a file format without serialization or byte streams. Not sure how that would work. > I think it's high time that the industry standardised on a generic "container" format to replace legacy archive file formats. I have a side project chipping away at just such a thing. It's quite daunting, so if this at all interests anyone, please comment/reach out. I'd love more of an excuse to work on this. SITO in a nutshell: - It's all based on msgpack, which does most of the heavy lift for serialization and datatype encoding - a sito stream comprises blocks, each block is a self-contained, independently decodeable msgpack array object. - each block an array of the form (type: smallint, header: optional(hashmap), data: any) - the type is either a single-byte int, or a packed int indicating a substream id - the sito primary stream comprises multiple independent substreams - there's no raw plaintext fields, but there is a plain unicode block which can be used to embed whatever plaintext metadata - substreams can each have whatever compression/codec they want - since each stream is a block stream, it's trivial to de/interleave - for data integrity, I want to do something like block-level CRC/FEC along with per-stream merkle trees but I haven't worked out the details yet - there are periodic "sync-blocks" which have a magic 8-byte sequence for starting a file, but also throughout the stream, to facilitate re-alignment of read heads - I've also been toying with the idea of using sqlite as stream indexes and as a general glue to keep track of what's going on (right now, you can arbitrarily start a new substream at any point in the primary stream, so it's hard to tell at the start of a file what's in it, sqlite pre-allocates pages so write heads can go back and update a prior index block) - nd-arrays are a particularly interesting datatype so there's an emphasis on ergonomics around handling them I plan on doing a simple PoC at some point soon showcasing SITAR, the sito archive format, with a python tarfile-like interface.

kortex 4y ago
I just updated the github page with my current state of notes, in case folks are curious to some of the details. https://github.com/xkortex/sito https://github.com/xkortex/sito

osigurdson 4y ago

I do feel that there is a "missing" simple/basic text format which is somewhere in between csv and json/yaml. csv has no hierarchy while json/yaml are horribly unreadable for tabular data as it has no concept of columns (see https://csvjson.com/csv2json https://csvjson.com/csv2json as an example). Maybe such a thing exists but I haven't seen it.

CodesInChaos 4y ago

> CRC style checksums are automatically a no-go. They're inherently sequential. AFAIK the linearity of CRCs allows them to be split, parallelized, and combined. This is also the case for polynomial MACs, like GHash or Poly1305.

AlotOfReading 4y ago
Yep, the only downside for CRC being that the recombination involves some expensive multiplications that are proportional to the total input length.
- CodesInChaos 4y ago
  For Poly1305 you can use square-and-multiply to compute the exponentiation required for recombination, which has logarithmic cost, not linear. Are you sure a similar algorithm can't be applied to CRCs?
  - AlotOfReading 4y ago
    Sorry, I didn't phrase it well. It's also log(n) for CRCs via the same algorithm.

shpongled 4y ago

I work in proteomics, where the field decided to standardize on XML based file formats. It can take longer to parse the data out of the XML container than it does to actually run the analysis.

optimiz3 4y ago

> CRC or SHA style checksums are automatically a no-go CRC is 100% parallelizable FYI. Both in SIMD and also on a block level which can be merged.

elmolino89 4y ago

"bonus" points: * text/TSV data files made public as a requirement from the published articles may have spaces or dots in column names, missing line ends, non-unique row IDs, etc. * there is no limit as how many records can be stored in a single file. Latest dbSNP has more than 1000 millions of rows. * bunch of formats (GFF, GTF, VCF) has several TSV delimited "proper" columns (as: 1 value) and then special column where optional fields are piled in a different format, with another separator, field names etc. Real fun to parse...