4 ms·
I would say on the contrary, please stop dumping CSV data just because that’s all you know all to deal with. CSV is full of gotchas which is why CSV readers ha
by wenc 2y ago
I would say on the contrary, please stop dumping CSV data just because that’s all you know all to deal with.
CSV is full of gotchas which is why CSV readers have so many switches.
They are huge (and hugely inefficient at scale), have no type safety, don’t support predicate push down and are only meant to be read linearly.
Most of the OPs complaints are about loading and memory, not about actually using the data. OP is more concerned about infra and memory usage than actually the end goal: using the the data well.
I’m not saying Parquet is the be all and end all of data formats but it’s a darned sight better than CSVs for almost all analytics use cases.
People who use CSV because it’s readable and understandable either have non analytics use cases or just don’t want to learn anything new.
- racional 2y agoPeople who [disagree with my analysis] just don’t want to learn anything new. CSV is a well-discussed topic on HN. If you'd like to explore other points of view in regard to these issues, you could start with this fine recent thread: https://news.ycombinator.com/item?id=39814334 https://news.ycombinator.com/item?id=39814334
- wenc 2y agoI appreciate the caveats. Notice I mentioned analytics use cases which is a subset of all the software engineering use cases. CSVs have their place as a simple interchange format. They’ve been overextended to use cases they were never mean to support — I’m speaking out against these. If you’re using CSVs for data work (outside of simple ETL) it a bit like saying let me write a relational database on top of flat text files as a data store. Meanwhile the world has moved on to engines like Postgres. But no I’m going stick with my flat text format.