4 ms·

I wish it supported large csv to partitioned parquet. THAT is something I need a good solution for.

• quasiben 10y ago
  Are you familiar with fastparquet (https://github.com/dask/fastparquet https://github.com/dask/fastparquet) and pyarrow (https://pyarrow.readthedocs.io/en/latest https://pyarrow.readthedocs.io/en/latest) ?
  • ColanR 10y ago
    Your link is malformed. https://github.com/dask/fastparquet https://github.com/dask/fastparquet
    • ColanR 10y ago
      ...and its fixed.
  • eximius 10y ago
    Not until you linked me. :) They look like good steps but not quite what I need, for different reasons. I'll have to check when I get back to the office next week.
• nl 10y ago
  (Py)Spark? I run it locally and do that kind of operation pretty frequently.
  • eximius 10y ago
    On what scale? Part of the problem is the scale, but another part is that writing partitioned parquets seems poorly documented (I would love corrections, I spent a decent amount of time last week looking for good information)
    • nl 10y ago
      I was recently working with a 38GB compressed text dataset locally. I can't remember how big it was in parquet. It's literally one line of code. See http://labs.vistarmedia.com/2016/12/27/indexing-json-logs-with-parquet.html http://labs.vistarmedia.com/2016/12/27/indexing-json-logs-wi... for an example (except you write to a local file system rather than HDFS).
      • eximius 10y ago
        That looks very helpful, I'll have to check it out!