3 ms·
On what scale? Part of the problem is the scale, but another part is that writing partitioned parquets seems poorly documented (I would love corrections, I spe
by eximius 10y ago
On what scale?
Part of the problem is the scale, but another part is that writing partitioned parquets seems poorly documented (I would love corrections, I spent a decent amount of time last week looking for good information)
- nl 10y agoI was recently working with a 38GB compressed text dataset locally. I can't remember how big it was in parquet. It's literally one line of code. See http://labs.vistarmedia.com/2016/12/27/indexing-json-logs-with-parquet.html http://labs.vistarmedia.com/2016/12/27/indexing-json-logs-wi... for an example (except you write to a local file system rather than HDFS).
- eximius 10y agoThat looks very helpful, I'll have to check it out!