3 ms·
> just juggling various bioinformatics file formats Your other points are spot on. This one I want to address specifically. The file formats. Academics love t
by cratermoon 4y ago
> just juggling various bioinformatics file formats
Your other points are spot on. This one I want to address specifically. The file formats. Academics love their incredibly over-engineered file formats. MARC. SGML. DICOM. HL7. RDF. Those are just the ones I know. Universally, they try to cover every corner case that anyone could ever imagine. Academics absolutely love their ontologies. Just implementing one of them is a nightmare. Going from one to another is an exercise in the philosophies of ontologies.
- manv1 4y agoHL7 is't technically an academic file format, it's an industry standard interchange format for health data. DICOM is for radiology. RDF and SGML, well, they're from the same era as XML, so yeah.
- kweingar 4y ago> Just implementing one of them is a nightmare. Going from one to another is an exercise in the philosophies of ontologies. Good thing there are lots of competing implementations! It would be a shame if these files were actually portable.
- zmmmmm 4y agoActually I think genomics / bioinformatics is a counterpoint there. One of the things I like about the field is nearly every file format is under-engineered. It's TSV all the way down and if you need compression gzip it. If you need to index that, sort it (literally often with unix sort command) and block-gzip it. Anything more engineered arose specifically because the above failed and something more is actually needed. The downside is it's a giant hellscape of unstructured, poorly specified formats where data types are barely specified at all or if they are most of the schema is published on some rambling blog post by some rando scientist. You will spend most of your time understanding it by empirical reverse engineering of the data that you are trying to deal with.
- cratermoon 4y agoOh, then eventually they'll get a committee together and after a few years they'll produce a unified file format that somehow manages to cover all the cases in the different existing formats (or at least the ones used by well-funded PIs) and is a hellscape of optional properties and required elements so poorly specified that it's impossible for any two implementations to communicate.