3 ms·

So what prevented the use HDF or netcdf in bioinformatics? I mean these are not new format by any definition of new and I disagree the broken formats are often

So what prevented the use HDF or netcdf in bioinformatics? I mean these are not new format by any definition of new and I disagree the broken formats are often not good enough, it seems more that often the bio/med fields (probably/hopefully not bioinformatics) are still used to process data using excel (speaking from my limited interaction with mainly med researchers).

jltsiren 4y ago
Relative obscurity, most likely. If the formats are not used in bioinformatics, people developing bioinformatics tools are usually not familiar with them. And if developers are not familiar with the formats, they can't make informed decisions about using them. Yet another consequence of researcher-driven software development.
- jghn 4y ago
  HDF has been around in some capacity in bioinformatics for at least 20 years. For instance the Bioconductor project had a package for it in the early aughts. But as to why it never took off, I suspect the reasons you cite are correct.
- heuermh 4y ago
  We presented using Parquet formats for bioinformatics 2012/13-ish at the Bioinformatics Open Source Conference (BOSC) and got laughed out of the place. While using Apache Spark for bioinformatics [0] never really took off, I still think Parquet formats for bioinformatics [1] is a good idea, especially with DuckDB, Apache Arrow, etc. supporting Parquet out of the box. 0 - https://github.com/bigdatagenomics/adam https://github.com/bigdatagenomics/adam 1 - https://github.com/bigdatagenomics/bdg-formats https://github.com/bigdatagenomics/bdg-formats
  - jltsiren 4y ago
    Maybe column-oriented formats like Parquet never became popular in bioinformatics because new file formats usually come from people developing tools for upstream tasks such as read mapping, variant calling, and genome assembly. They are the ones who work with new kinds of data first. Those upstream tasks tend to be row-oriented. You often iterate over all rows, do something with them, and output new rows in another format. Alternatively, you read the entire input into in-memory data structures, do something, and later serialize the data structures. Using column-oriented formats for such tasks does not feel natural.
sungam 4y ago
HDF is used in single cell genomics
fuzzfactor 4y ago
>One key problem is that technology changes quickly. Each of your plateaus of stability often need to become recognized before the next step can be taken. With scientific software at the end of the train, the file-type/file-system needs to be well established and more stable than any software could be, and for a lot longer than the whole scientific project itself takes. A scientific filetype needs to be well-documented (better than ordinary software) and unchanged for long enough so that all agree no further changes are intended. It needs to have already been virtually perfectly stable, for more years than most research projects are likely to have their data remain useful in the future. netcdf is in this category while still being extensible and it is very old (well established) if not well known. Public domain government "codec" which basically decompresses netcdf to structured text, and in reverse. Now this new ASDF filetype looks like it does have useful features of its own except one thing: >ASDF is under active development Which can still be a drawback in this situation.

jltsiren 4y ago

Relative obscurity, most likely. If the formats are not used in bioinformatics, people developing bioinformatics tools are usually not familiar with them. And if developers are not familiar with the formats, they can't make informed decisions about using them. Yet another consequence of researcher-driven software development.

jghn 4y ago
HDF has been around in some capacity in bioinformatics for at least 20 years. For instance the Bioconductor project had a package for it in the early aughts. But as to why it never took off, I suspect the reasons you cite are correct.
heuermh 4y ago
We presented using Parquet formats for bioinformatics 2012/13-ish at the Bioinformatics Open Source Conference (BOSC) and got laughed out of the place. While using Apache Spark for bioinformatics [0] never really took off, I still think Parquet formats for bioinformatics [1] is a good idea, especially with DuckDB, Apache Arrow, etc. supporting Parquet out of the box. 0 - https://github.com/bigdatagenomics/adam https://github.com/bigdatagenomics/adam 1 - https://github.com/bigdatagenomics/bdg-formats https://github.com/bigdatagenomics/bdg-formats
- jltsiren 4y ago
  Maybe column-oriented formats like Parquet never became popular in bioinformatics because new file formats usually come from people developing tools for upstream tasks such as read mapping, variant calling, and genome assembly. They are the ones who work with new kinds of data first. Those upstream tasks tend to be row-oriented. You often iterate over all rows, do something with them, and output new rows in another format. Alternatively, you read the entire input into in-memory data structures, do something, and later serialize the data structures. Using column-oriented formats for such tasks does not feel natural.

sungam 4y ago

HDF is used in single cell genomics

fuzzfactor 4y ago

>One key problem is that technology changes quickly. Each of your plateaus of stability often need to become recognized before the next step can be taken. With scientific software at the end of the train, the file-type/file-system needs to be well established and more stable than any software could be, and for a lot longer than the whole scientific project itself takes. A scientific filetype needs to be well-documented (better than ordinary software) and unchanged for long enough so that all agree no further changes are intended. It needs to have already been virtually perfectly stable, for more years than most research projects are likely to have their data remain useful in the future. netcdf is in this category while still being extensible and it is very old (well established) if not well known. Public domain government "codec" which basically decompresses netcdf to structured text, and in reverse. Now this new ASDF filetype looks like it does have useful features of its own except one thing: >ASDF is under active development Which can still be a drawback in this situation.