> > Additionally, it is a problem that can largely be mitigated by simple text compression. > > "Simple text compression"? Not sure what you are referring to t

> > Additionally, it is a problem that can largely be mitigated by simple text compression. > > "Simple text compression"? Not sure what you are referring to there. If you are speaking of generalized compression, there's a bunch of information theory pointing out the problem with that thinking. For most natural text-only data, .csv.gz is probably close to the limits of what we can do. For numerical or categorical text containing patterns or structure, I agree that better formats may exist (e.g. H.264 for CSV files containing video frames). Not sure how a more general format will solve anything you couldn't already do by converting to a more specific format and tarring everything afterwards.