If anyone is interested in similar challenges, but in cryo electron microscopy, where the datasets are dozens of TB, this is a good paper discussing the file fo

If anyone is interested in similar challenges, but in cryo electron microscopy, where the datasets are dozens of TB, this is a good paper discussing the file formats, header organization, and bit-depth required for proper resolution: https://pubmed.ncbi.nlm.nih.gov/35724904/ https://pubmed.ncbi.nlm.nih.gov/35724904/ What I can say is that TEM data management is unmitigated hell, and thank God for that author (Ludtke) and his software (EMAN2). Biologists are unsurprisingly awful at data management.