3 ms·
> What I think jltsiren is trying to say is that if you're trying to parallelize within one file, you've already lost. I disagree, you don't need to split up a
by IanCal 4y ago
> What I think jltsiren is trying to say is that if you're trying to parallelize within one file, you've already lost.
I disagree, you don't need to split up a file to parallelize things if you just use a moderately recent format.
Put it in parquet, have sensible row groups, and turn on zstd compression. Split it into multiple files if you want but fast access to subsets of files which are neatly compressed is very easy to get now.
You also get the ability to store things losslessly, which you don't get so much without custom work with TSV (main example here is floats) and things like schemas.
I've just tested out moving from one of my zstd parquet test files that's 190M, and it's 4.8G as a CSV file for a single datapoint.
- exmadscientist 4y agoWhat we're trying to say is that that approach does not work so well if your individual files are 1-2 orders of magnitude larger and your whole analysis dataset (which must be processed in its entirety to answer any question!) is 4-5 orders of magnitude larger. Yes, not everyone has such large data sets. (And not everyone has such small data sets!) But it is imperative that scientific computing infrastructure developers understand what classes of experiments they are serving, and what classes they can not or should not serve.
- IanCal 4y agoI'm not sure I follow. Splitting the file is fine, and very common, but you can absolutely get parallelisation within a file now using common standards. Even your single threaded code can be faster.