The big issue is how many tools would need to be rewritten to accommodate the updated file format. For better or worse there is a lot of inertia to the terrible

The big issue is how many tools would need to be rewritten to accommodate the updated file format. For better or worse there is a lot of inertia to the terrible file formats, it's not as simple as "Hey, change". As an example, consider clinical use cases. It can be near impossible to update the software in the pipelines for these situations. How do they move on from the old tools? Ok, now suppose they need to interact with data from non-clinical sources? Things like you're proposing can be done. As others have intimated in this thread, solving the technical issue is the easiest part of the chain. More than one Tech Person has walked into the fray assuming the only thing holding stuff back is that no one ever thought to apply Good Tech Solutions. Instead what's necessary are people who can understand why things are the way they are, and the human dynamics at play. Building a better mouse trap requires taking these into account from the start.