4 ms·
FASTQ, BAM, BED, VCF, etc are all standardized in terms of file format, but the semantics of the content are not. Harmonizing data across multiple semantics is
by dekhn 10y ago
FASTQ, BAM, BED, VCF, etc are all standardized in terms of file format, but the semantics of the content are not. Harmonizing data across multiple semantics is still a very challenging problem that few people take seriously.
- patall 10y agoLike the good old "chr1" == "1" problem? Well, yeah that’s true. But still I think we have those consortia in place, why do we need another authority. They just have to set (and somewhat enforce) the rules.
- dekhn 10y agoIt goes far beyond "chr1" = "1". There are many other fields with next-to-impossible to interpret data. The consortia have, at best, advisory roles, they cannot enforce any rules. And given an extensible file format, third parties are going to extend it with poorly defined/standardized data. Then the consumers have to harmonize the data. Data harmonization (or cleaning) it typically one of the bottleneck steps in applying machine learning at scale.