5 ms·

I was thinking of using Apache Drill for converting CSVs to Parquet. What else do people use for that?

by kkwteh 6y ago

mochomocha 6y ago
pyarrow
bsg75 6y ago
Spark
- falaki 6y ago
  Spark community put a lot of effort into a feature-rich CSV data source. Often times the most challenging part of ingesting CSV is parsing it. There are many flavors each with different corner cases for NULLs, comments, headers, etc.
lurker458 6y ago
I've also been looking for that. In an ideal world there would be a small, fast, standalone cli tool that can convert csv to parquet. There is a (sadly, unfinished) parquet writer Rust library in the Arrow repository that looks promising. All approaches I've tried so far (spark, pyarrow, drill, ...) require everything and the kitchen sink. So far I've settled on a java cli tool that uses jackson + org.apache.parquet internally, but it's cpu bound and has a huge amount of maven dependencies.
- meritt 6y ago
  pandas + fastparquet fairly lightweight. but yes I would love to see a simple c++/golang binary that's just a simple csv2parq call.
  - MrPowers 6y ago
    Newer versions of Pandas don't even need fastparquet anymore. This code works: import pandas as pd df = pd.read_csv('data/us_presidents.csv') df.to_parquet('tmp/us_presidents.parquet')
    - meritt 6y ago
      Nice! Does that work alongside reading in via chunks and writing via row_groups? If I have a 500GB CSV will it work?
poorman 6y ago
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html https://pandas.pydata.org/pandas-docs/stable/reference/api/p...
- nerdponx 6y ago
  +1, in my experience the Pandas CSV parser seems very robust and more than sufficiently fast. The only faster/better CSV parser I've used with any frequency is the fread function in the R package data.table. When I used R in the past, Parquet was much less popular, but I think now the arrow package supports writing to Parquet.
ecnahc515 6y ago
Maybe Presto/Athena
MrPowers 6y ago
This article explains how to convert CSVs to Parquet with Pandas, PySpark, and Koalas: https://mungingdata.com/python/writing-parquet-pandas-pyspark-koalas/ https://mungingdata.com/python/writing-parquet-pandas-pyspar... This article explains how to convert CSVs to Parquet with Go and Scala: https://mungingdata.com/go/csv-to-parquet/ https://mungingdata.com/go/csv-to-parquet/ If you're converting hundreds/thousands of files, Spark/PySpark is probably the best tool for the job. For fewer files, Python or Go is just fine.
tspann 6y ago
Just use a Apache NiFi fast and does that on 1 step

I was thinking of using Apache Drill for converting CSVs to Parquet. What else do people use for that?

by kkwteh 6y ago

I was thinking of using Apache Drill for converting CSVs to Parquet. What else do people use for that?

mochomocha 6y ago

pyarrow

bsg75 6y ago

Spark

falaki 6y ago
Spark community put a lot of effort into a feature-rich CSV data source. Often times the most challenging part of ingesting CSV is parsing it. There are many flavors each with different corner cases for NULLs, comments, headers, etc.

lurker458 6y ago

I've also been looking for that. In an ideal world there would be a small, fast, standalone cli tool that can convert csv to parquet. There is a (sadly, unfinished) parquet writer Rust library in the Arrow repository that looks promising. All approaches I've tried so far (spark, pyarrow, drill, ...) require everything and the kitchen sink. So far I've settled on a java cli tool that uses jackson + org.apache.parquet internally, but it's cpu bound and has a huge amount of maven dependencies.

meritt 6y ago
pandas + fastparquet fairly lightweight. but yes I would love to see a simple c++/golang binary that's just a simple csv2parq call.
- MrPowers 6y ago
  Newer versions of Pandas don't even need fastparquet anymore. This code works: import pandas as pd df = pd.read_csv('data/us_presidents.csv') df.to_parquet('tmp/us_presidents.parquet')
  - meritt 6y ago
    Nice! Does that work alongside reading in via chunks and writing via row_groups? If I have a 500GB CSV will it work?

poorman 6y ago

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html https://pandas.pydata.org/pandas-docs/stable/reference/api/p...

nerdponx 6y ago
+1, in my experience the Pandas CSV parser seems very robust and more than sufficiently fast. The only faster/better CSV parser I've used with any frequency is the fread function in the R package data.table. When I used R in the past, Parquet was much less popular, but I think now the arrow package supports writing to Parquet.

ecnahc515 6y ago

Maybe Presto/Athena

MrPowers 6y ago

This article explains how to convert CSVs to Parquet with Pandas, PySpark, and Koalas: https://mungingdata.com/python/writing-parquet-pandas-pyspark-koalas/ https://mungingdata.com/python/writing-parquet-pandas-pyspar... This article explains how to convert CSVs to Parquet with Go and Scala: https://mungingdata.com/go/csv-to-parquet/ https://mungingdata.com/go/csv-to-parquet/ If you're converting hundreds/thousands of files, Spark/PySpark is probably the best tool for the job. For fewer files, Python or Go is just fine.

tspann 6y ago

Just use a Apache NiFi fast and does that on 1 step