4 ms·

I would say most of the tooling around Parquet sucks. A lot of simple use cases (head, sample, merge, etc) are locked behind rather heavyweight ecosystems (Hado

I would say most of the tooling around Parquet sucks. A lot of simple use cases (head, sample, merge, etc) are locked behind rather heavyweight ecosystems (Hadoop, Spark, or other JVM dependencies). Newer tools are starting to develop that make the situation better. Is Parquet better/faster/stronger than keeping everything in schemaless CSVs? 100%, but it has historically meant that I have to make trade-offs to benefit the tooling rather than how I want to interactively approach my data. Most people do not have "big data" problems where the performance differences of Parquet vs csv matter.

vanpythonista 4y ago
Shameless plug, my project exposes the common Parquet operations using a Rust CLI tool that utilizes the Rust API for Apache Arrow and can be used without any Java/Hadoop/Spark dependencies. Also available a static binary. https://github.com/manojkarthick/pqrs https://github.com/manojkarthick/pqrs
- 41b696ef1113 4y ago
  That looks great! Exactly the kind of tooling required to allow me to feel more comfortable locking up human readable data into a blob. Being a single Rust binary is just the cherry on top. I might submit a few feature requests, but one that immediately comes to mind: csv -> parquet. Perhaps out of scope for the original vision, but having a single utility that could roundtrip data would be fantastically useful.
  - vanpythonista 4y ago
    Glad you like it. A CSV->Parquet conversion command would certainly be really useful!
1egg0myegg0 4y ago
I hope that DuckDB can help here! You can use the CLI or Python client to read a parquet file, but my favorite for quick exploration is popping open DBeaver (a SQL IDE) and using the DuckDB JDBC driver. That way you can see some column metadata, and even open multiple files, etc. I wrote a short guide for that here: https://duckdb.org/docs/guides/sql_editors/dbeaver https://duckdb.org/docs/guides/sql_editors/dbeaver

vanpythonista 4y ago

Shameless plug, my project exposes the common Parquet operations using a Rust CLI tool that utilizes the Rust API for Apache Arrow and can be used without any Java/Hadoop/Spark dependencies. Also available a static binary. https://github.com/manojkarthick/pqrs https://github.com/manojkarthick/pqrs

41b696ef1113 4y ago
That looks great! Exactly the kind of tooling required to allow me to feel more comfortable locking up human readable data into a blob. Being a single Rust binary is just the cherry on top. I might submit a few feature requests, but one that immediately comes to mind: csv -> parquet. Perhaps out of scope for the original vision, but having a single utility that could roundtrip data would be fantastically useful.
- vanpythonista 4y ago
  Glad you like it. A CSV->Parquet conversion command would certainly be really useful!

1egg0myegg0 4y ago

I hope that DuckDB can help here! You can use the CLI or Python client to read a parquet file, but my favorite for quick exploration is popping open DBeaver (a SQL IDE) and using the DuckDB JDBC driver. That way you can see some column metadata, and even open multiple files, etc. I wrote a short guide for that here: https://duckdb.org/docs/guides/sql_editors/dbeaver https://duckdb.org/docs/guides/sql_editors/dbeaver