3 ms·

If you’re dealing with non trivial data sizes and if you have an analytics use case and you’re using CSV, you’re already doing it wrong. You bring up partition

by wenc 2y ago

If you’re dealing with non trivial data sizes and if you have an analytics use case and you’re using CSV, you’re already doing it wrong. You bring up partitioning — that can help performance but with Parquet, you can get performance they is far better than CSV even without partitioning because it only has to read the headers. And no, no one really needs to know about row groups (sure you can eke out more performance but few people do). All this is to say all your points are non starters. There’s no need to know all this and even the least optimized parquet dataset is better than CSV in every way. The only use case CSV might be good for is ETL into an actual database like Postgres which is what you’re doing in your article but that’s actually only a very small part of the analytics pipeline. Parquet is actually not that complicated but I always meet people who only know CSV and they feel anything more complicated is beyond them. Don’t be that guy. Shoehorning everything into CSV creates costs downstream like high storage costs (CSVs are much bigger), and no type safety means you have to validate types with a bunch of non standard rules creating a lot of technical debt. I’ve always managed to work with parquet with either DuckDB or Visidata (and occasionally Pandas or Polars). Visidata lets you peek into parquet like Vim lets you look into CSVs. I don’t miss CSVs at all.

int_19h 2y ago
The article doesn't claim that CSV is superior, though. It only mentions CSV once, as a workaround for the problem that relational databases mostly don't support importing from Parquet directly. It does rather feel like at least that part of the rant would be better addressed by adding such support, though. If Parquet is already a de facto standard in the industry, I'm sure there already are Postgres extensions that add such support; how hard would it be to convince the maintainers to take the best-written one and use it to add support directly to core Postgres, just as it already has for CSV and JSON?

by wenc 2y ago

int_19h 2y ago

The article doesn't claim that CSV is superior, though. It only mentions CSV once, as a workaround for the problem that relational databases mostly don't support importing from Parquet directly. It does rather feel like at least that part of the rant would be better addressed by adding such support, though. If Parquet is already a de facto standard in the industry, I'm sure there already are Postgres extensions that add such support; how hard would it be to convince the maintainers to take the best-written one and use it to add support directly to core Postgres, just as it already has for CSV and JSON?