4 ms·

Bioinformatics file formats are broken, but it's better to try to understand why they are broken before rushing to fix them. One key problem is that technology

Bioinformatics file formats are broken, but it's better to try to understand why they are broken before rushing to fix them. One key problem is that technology changes quickly. There are always new instruments generating new kinds of data with new properties and new features. People are using that data in new applications. Software comes many years behind the state of the art. First you need to figure out what is the exact problem the software is supposed to solve. Then you have to solve the problem and turn the prototype into a useful tool. This work is mostly done by researchers who may know a little about software engineering. By the time the situation is stable enough that software engineers who are not active researchers in the field could be useful, it's often too late to change the file formats. There is already too much legacy data and too many tools supporting the established formats. Another key problem is that the "broken" file formats are often good enough. When you have tabular data where the fields can be reasonably understood as text, a simple TSV-based format often gets the job done. Especially if individual datasets are only tens of gigabytes. By using a custom format, you avoid having to choose from many existing formats that all have their own issues. And that often guarantee you version conflicts and breaking changes in the future. Also, when it comes to parallelization, it's hard to beat running many independent jobs in parallel. While computers are getting bigger, individual problems are often not, as the underlying biological problems remain the same. In the work I do, a reasonable target system had 32 cores and 256 GB memory in 2015. That's still a reasonable target in 2022. The computers I use have become cheaper and faster, but they have not really changed.

jiggawatts 4y ago
> When you have tabular data where the fields can be reasonably understood as text, a simple TSV-based format often gets the job done. Especially if individual datasets are only tens of gigabytes. In what other industry is less than 0.5% utilisation accepted as "gets the job done?" TSV is a terrible format for multi-gigabyte files, because it uses line breaks. Technically it's possible to parse them in parallel, but if the format enables quoted strings with newlines, then it isn't possible to do this safely. A simple binary row-based or columnar format could be read orders of magnitude faster. No character-by-character processing needed, just "memcpy" and go.
- sideshowb 4y ago
  You did say yourself that overall utilization was more like 50 percent. Interesting discussion though. I write some academic code in which I anticipate file load will be a bottleneck when we scale up one day, so I appreciate your thoughts
- exmadscientist 4y ago
  What I think jltsiren is trying to say is that if you're trying to parallelize within one file, you've already lost. Or your job is tiny, but then who cares. This matches my experience a decade ago as a rare scientist who could program. When I had to run a job on the giant cluster against our ~100TB dataset, I did not put hundreds of threads to work against one file. We had everything broken up into hundreds of files, so I could run one thread against each file (with a tiny bit of boundary patch-up), and it all just flew. Trying to get it "all in one" would have been an exercise in misery and pain. This can also be done by simply running many jobs at once. Many (but not all! never all!) scientific analyses naturally work well with that model, so don't fight it on technical purity grounds. This was all done rather straightforwardly with the horrid piece of radioactive software garbage that is (was? please say was? I dare not check) CERN's ROOT. It had few redeeming characteristics... except for being fast and efficient on massive datasets, once you got it running at all. And that counts for something!
  - SiempreViernes 4y ago
    The glory of ROOT prevails! But yes, it is better now than back in the 4.X days or whatever ungodly version it sounds like you used.
  - IanCal 4y ago
    > What I think jltsiren is trying to say is that if you're trying to parallelize within one file, you've already lost. I disagree, you don't need to split up a file to parallelize things if you just use a moderately recent format. Put it in parquet, have sensible row groups, and turn on zstd compression. Split it into multiple files if you want but fast access to subsets of files which are neatly compressed is very easy to get now. You also get the ability to store things losslessly, which you don't get so much without custom work with TSV (main example here is floats) and things like schemas. I've just tested out moving from one of my zstd parquet test files that's 190M, and it's 4.8G as a CSV file for a single datapoint.
zelphirkalt 4y ago
That's why we need more cooperation between researchers and actual software engineers and less "I can also code!" thinking of researchers. Yes they can, but please in their free time or publicly, so that actual software engineers can chime in and stop bad design from happening. Well coding in public is of course also no guarantee and neither is hiring a software engineer, but at least the chances are better for something useful to come out of it.
- melagonster 4y ago
  yes, we need replace all scientists with software developer.
  - zelphirkalt 4y ago
    You do not seem to grasp the meaning of "cooperation". Or were you trying to comment on another comment?
    - melagonster 4y ago
      sorry for this comment, I just frustrated.
- glomgril 4y ago
  not all coding is software engineering man
cycomanic 4y ago
So what prevented the use HDF or netcdf in bioinformatics? I mean these are not new format by any definition of new and I disagree the broken formats are often not good enough, it seems more that often the bio/med fields (probably/hopefully not bioinformatics) are still used to process data using excel (speaking from my limited interaction with mainly med researchers).
- jltsiren 4y ago
  Relative obscurity, most likely. If the formats are not used in bioinformatics, people developing bioinformatics tools are usually not familiar with them. And if developers are not familiar with the formats, they can't make informed decisions about using them. Yet another consequence of researcher-driven software development.
  - jghn 4y ago
    HDF has been around in some capacity in bioinformatics for at least 20 years. For instance the Bioconductor project had a package for it in the early aughts. But as to why it never took off, I suspect the reasons you cite are correct.
  - heuermh 4y ago
    We presented using Parquet formats for bioinformatics 2012/13-ish at the Bioinformatics Open Source Conference (BOSC) and got laughed out of the place. While using Apache Spark for bioinformatics [0] never really took off, I still think Parquet formats for bioinformatics [1] is a good idea, especially with DuckDB, Apache Arrow, etc. supporting Parquet out of the box. 0 - https://github.com/bigdatagenomics/adam https://github.com/bigdatagenomics/adam 1 - https://github.com/bigdatagenomics/bdg-formats https://github.com/bigdatagenomics/bdg-formats
    - jltsiren 4y ago
      Maybe column-oriented formats like Parquet never became popular in bioinformatics because new file formats usually come from people developing tools for upstream tasks such as read mapping, variant calling, and genome assembly. They are the ones who work with new kinds of data first. Those upstream tasks tend to be row-oriented. You often iterate over all rows, do something with them, and output new rows in another format. Alternatively, you read the entire input into in-memory data structures, do something, and later serialize the data structures. Using column-oriented formats for such tasks does not feel natural.
jrumbut 4y ago
It's probably a useful reminder that biologists do all kinds of other things besides loading sequence files. For instance, they generate the sequence files and grow the organisms that are getting sequenced.

jiggawatts 4y ago

> When you have tabular data where the fields can be reasonably understood as text, a simple TSV-based format often gets the job done. Especially if individual datasets are only tens of gigabytes. In what other industry is less than 0.5% utilisation accepted as "gets the job done?" TSV is a terrible format for multi-gigabyte files, because it uses line breaks. Technically it's possible to parse them in parallel, but if the format enables quoted strings with newlines, then it isn't possible to do this safely. A simple binary row-based or columnar format could be read orders of magnitude faster. No character-by-character processing needed, just "memcpy" and go.

sideshowb 4y ago
You did say yourself that overall utilization was more like 50 percent. Interesting discussion though. I write some academic code in which I anticipate file load will be a bottleneck when we scale up one day, so I appreciate your thoughts
exmadscientist 4y ago
What I think jltsiren is trying to say is that if you're trying to parallelize within one file, you've already lost. Or your job is tiny, but then who cares. This matches my experience a decade ago as a rare scientist who could program. When I had to run a job on the giant cluster against our ~100TB dataset, I did not put hundreds of threads to work against one file. We had everything broken up into hundreds of files, so I could run one thread against each file (with a tiny bit of boundary patch-up), and it all just flew. Trying to get it "all in one" would have been an exercise in misery and pain. This can also be done by simply running many jobs at once. Many (but not all! never all!) scientific analyses naturally work well with that model, so don't fight it on technical purity grounds. This was all done rather straightforwardly with the horrid piece of radioactive software garbage that is (was? please say was? I dare not check) CERN's ROOT. It had few redeeming characteristics... except for being fast and efficient on massive datasets, once you got it running at all. And that counts for something!
- SiempreViernes 4y ago
  The glory of ROOT prevails! But yes, it is better now than back in the 4.X days or whatever ungodly version it sounds like you used.
- IanCal 4y ago
  > What I think jltsiren is trying to say is that if you're trying to parallelize within one file, you've already lost. I disagree, you don't need to split up a file to parallelize things if you just use a moderately recent format. Put it in parquet, have sensible row groups, and turn on zstd compression. Split it into multiple files if you want but fast access to subsets of files which are neatly compressed is very easy to get now. You also get the ability to store things losslessly, which you don't get so much without custom work with TSV (main example here is floats) and things like schemas. I've just tested out moving from one of my zstd parquet test files that's 190M, and it's 4.8G as a CSV file for a single datapoint.

zelphirkalt 4y ago

That's why we need more cooperation between researchers and actual software engineers and less "I can also code!" thinking of researchers. Yes they can, but please in their free time or publicly, so that actual software engineers can chime in and stop bad design from happening. Well coding in public is of course also no guarantee and neither is hiring a software engineer, but at least the chances are better for something useful to come out of it.

melagonster 4y ago
yes, we need replace all scientists with software developer.
- zelphirkalt 4y ago
  You do not seem to grasp the meaning of "cooperation". Or were you trying to comment on another comment?
  - melagonster 4y ago
    sorry for this comment, I just frustrated.
glomgril 4y ago
not all coding is software engineering man

cycomanic 4y ago

So what prevented the use HDF or netcdf in bioinformatics? I mean these are not new format by any definition of new and I disagree the broken formats are often not good enough, it seems more that often the bio/med fields (probably/hopefully not bioinformatics) are still used to process data using excel (speaking from my limited interaction with mainly med researchers).

jltsiren 4y ago
Relative obscurity, most likely. If the formats are not used in bioinformatics, people developing bioinformatics tools are usually not familiar with them. And if developers are not familiar with the formats, they can't make informed decisions about using them. Yet another consequence of researcher-driven software development.
- jghn 4y ago
  HDF has been around in some capacity in bioinformatics for at least 20 years. For instance the Bioconductor project had a package for it in the early aughts. But as to why it never took off, I suspect the reasons you cite are correct.
- heuermh 4y ago
  We presented using Parquet formats for bioinformatics 2012/13-ish at the Bioinformatics Open Source Conference (BOSC) and got laughed out of the place. While using Apache Spark for bioinformatics [0] never really took off, I still think Parquet formats for bioinformatics [1] is a good idea, especially with DuckDB, Apache Arrow, etc. supporting Parquet out of the box. 0 - https://github.com/bigdatagenomics/adam https://github.com/bigdatagenomics/adam 1 - https://github.com/bigdatagenomics/bdg-formats https://github.com/bigdatagenomics/bdg-formats
  - jltsiren 4y ago
    Maybe column-oriented formats like Parquet never became popular in bioinformatics because new file formats usually come from people developing tools for upstream tasks such as read mapping, variant calling, and genome assembly. They are the ones who work with new kinds of data first. Those upstream tasks tend to be row-oriented. You often iterate over all rows, do something with them, and output new rows in another format. Alternatively, you read the entire input into in-memory data structures, do something, and later serialize the data structures. Using column-oriented formats for such tasks does not feel natural.

jrumbut 4y ago

It's probably a useful reminder that biologists do all kinds of other things besides loading sequence files. For instance, they generate the sequence files and grow the organisms that are getting sequenced.