3 ms·

Did you have to reformat the data to get that size saving?

• prepend 6y ago
  Not at all. It was just swapping out readcsv with readparquet in R and Python. It was painless. Granted my dataset was mostly categorical so that’s why it compressed down so much, but it was a real lifesaver. It was also nice to be able to read while bundles of parquet a into a single dataframe easily. So is nice for “sharding” really big parquets over multiple files. Or fitting under file size limits on git repos.
  • alisonkisk 6y ago
    gzip probably would have been fine for that case even without parquet?
    • prepend 6y ago
      Gzip did help, but not as much as parquet. I don’t remember the exact sizes but I think parquet was was 3GB->20MB and gzip was like 3GB->180MB or something. Also gzip was an extra step of unzip then read.