3 ms·

If you need a quick tool to convert your CSV files, you can use csv2parquet from https://github.com/domoritz/arrow-tools https://github.com/domoritz/arrow-tools

If you need a quick tool to convert your CSV files, you can use csv2parquet from https://github.com/domoritz/arrow-tools https://github.com/domoritz/arrow-tools.

0cf8612b2e1e 4y ago
I think this remains a usability problem for parquet: most of the tooling sucks. Starting to get better, but parquet has been out for so many years and there are still few tools that natively let someone interact with the format. For years, most of the utilities were exclusively JVM based. My favorite way to explore parquet is to make a virtual link through DuckDB where you can seamlessly treat it like a SQL table. Even browse it in DBeaver.
- simonw 4y ago
  I mostly ignored Parquet until I realized DuckDB could handle it - now I'm much more likely to poke around with a Parquet file because I've got a solid toolkit available in both the CLI and as a Python library. A couple of notes I've made about that: Using DuckDB in Python to access Parquet data: https://til.simonwillison.net/duckdb/parquet https://til.simonwillison.net/duckdb/parquet Use DuckDB to convert parquet to JSON and then open it in Datasette Lite: https://til.simonwillison.net/duckdb/parquet-to-json https://til.simonwillison.net/duckdb/parquet-to-json
  - hackan 4y ago
    you can also use pandas in Python to access parquet data, and then i.e. convert it to sqlite
    - 0cf8612b2e1e 4y ago
      Naive pandas is going to limit you to RAM sizes when parquet is deliberately meant for huge datasets.
      - FridgeSeal 4y ago
        Anyone reading who is currently using pandas and running into this issue-polars is generally faster, and now supports (still new though) out-of-core workloads.
chatmasta 4y ago
We support exporting any query from Splitgraph to Parquet, try it here [0] (click the dice for a random query). You can also upload CSV or SQLite file(s) [1]. Each one will load into a table that you can query with a Postgres client, or export it (or just some subset of it you select with a query) as Parquet. Then you can load it into Seafowl [2]. We also have an API to export from Splitgraph to Seafowl, just need to add the button (this week). [0] https://www.splitgraph.com/query https://www.splitgraph.com/query [1] https://www.splitgraph.com/upload https://www.splitgraph.com/upload [2] https://seafowl.io/docs/guides/uploading-csv-parquet https://seafowl.io/docs/guides/uploading-csv-parquet

If you need a quick tool to convert your CSV files, you can use csv2parquet from https://github.com/domoritz/arrow-tools https://github.com/domoritz/arrow-tools.

0cf8612b2e1e 4y ago

I think this remains a usability problem for parquet: most of the tooling sucks. Starting to get better, but parquet has been out for so many years and there are still few tools that natively let someone interact with the format. For years, most of the utilities were exclusively JVM based. My favorite way to explore parquet is to make a virtual link through DuckDB where you can seamlessly treat it like a SQL table. Even browse it in DBeaver.

simonw 4y ago
I mostly ignored Parquet until I realized DuckDB could handle it - now I'm much more likely to poke around with a Parquet file because I've got a solid toolkit available in both the CLI and as a Python library. A couple of notes I've made about that: Using DuckDB in Python to access Parquet data: https://til.simonwillison.net/duckdb/parquet https://til.simonwillison.net/duckdb/parquet Use DuckDB to convert parquet to JSON and then open it in Datasette Lite: https://til.simonwillison.net/duckdb/parquet-to-json https://til.simonwillison.net/duckdb/parquet-to-json
- hackan 4y ago
  you can also use pandas in Python to access parquet data, and then i.e. convert it to sqlite
  - 0cf8612b2e1e 4y ago
    Naive pandas is going to limit you to RAM sizes when parquet is deliberately meant for huge datasets.
    - FridgeSeal 4y ago
      Anyone reading who is currently using pandas and running into this issue-polars is generally faster, and now supports (still new though) out-of-core workloads.

chatmasta 4y ago

We support exporting any query from Splitgraph to Parquet, try it here [0] (click the dice for a random query). You can also upload CSV or SQLite file(s) [1]. Each one will load into a table that you can query with a Postgres client, or export it (or just some subset of it you select with a query) as Parquet. Then you can load it into Seafowl [2]. We also have an API to export from Splitgraph to Seafowl, just need to add the button (this week). [0] https://www.splitgraph.com/query https://www.splitgraph.com/query [1] https://www.splitgraph.com/upload https://www.splitgraph.com/upload [2] https://seafowl.io/docs/guides/uploading-csv-parquet https://seafowl.io/docs/guides/uploading-csv-parquet