3 ms·
I recently found it useful for the dumbest reason. A dataset was about 3GB as a CSV and 20MB as a parquet file created and consumed by arrow. The file also work
by prepend 6y ago
I recently found it useful for the dumbest reason. A dataset was about 3GB as a CSV and 20MB as a parquet file created and consumed by arrow. The file also worked flawlessly across different environments and languages.
So it’s a good transport tool. It also happens to be fast to load and query, but I only used it because of the compact way it stores data without any hoops to jump through.
Of course one might say that it’s stupid to try to pass around a 2GB or 20MB file, but in my case I needed to do that.
- kristianp 6y agoDid you have to reformat the data to get that size saving?
- prepend 6y agoNot at all. It was just swapping out readcsv with readparquet in R and Python. It was painless. Granted my dataset was mostly categorical so that’s why it compressed down so much, but it was a real lifesaver. It was also nice to be able to read while bundles of parquet a into a single dataframe easily. So is nice for “sharding” really big parquets over multiple files. Or fitting under file size limits on git repos.
- alisonkisk 6y agogzip probably would have been fine for that case even without parquet?
- prepend 6y agoGzip did help, but not as much as parquet. I don’t remember the exact sizes but I think parquet was was 3GB->20MB and gzip was like 3GB->180MB or something. Also gzip was an extra step of unzip then read.
- CameronNemo 6y agoParquet is not Arrow. Parquet has optimizations for storage size at the expense of compute readiness. Arrow maximizes computation efficiency, at the expense of storage size. Read more here: https://stackoverflow.com/questions/56472727/difference-between-apache-parquet-and-arrow https://stackoverflow.com/questions/56472727/difference-betw...
- prepend 6y agoI used arrow to read and write parquets across environments. Thats my dumb reason for finding arrow so useful.