4 ms·
(Py)Spark? I run it locally and do that kind of operation pretty frequently.
by nl 10y ago
(Py)Spark? I run it locally and do that kind of operation pretty frequently.
- eximius 10y agoOn what scale? Part of the problem is the scale, but another part is that writing partitioned parquets seems poorly documented (I would love corrections, I spent a decent amount of time last week looking for good information)
- nl 10y agoI was recently working with a 38GB compressed text dataset locally. I can't remember how big it was in parquet. It's literally one line of code. See http://labs.vistarmedia.com/2016/12/27/indexing-json-logs-with-parquet.html http://labs.vistarmedia.com/2016/12/27/indexing-json-logs-wi... for an example (except you write to a local file system rather than HDFS).
- eximius 10y agoThat looks very helpful, I'll have to check it out!