3 ms·

Unfortunately everyone has moved to "Parquet" (Packet? Parket? Pacquet?) already and we've sailed even further. I absolutely HATE this Parcquage.

by dheera 3y ago

_kst_ 3y ago
https://www.databricks.com/glossary/what-is-parquet https://www.databricks.com/glossary/what-is-parquet (I don't think everyone has moved to it. I had never heard of it myself.)
- dheera 3y ago
  A lot of the machine learning world has started using it, it's annoying as hell, solves a problem that doesn't exist, has inadequate documentation, lacks a good GUI viewer, and lacks good command line converters to JSON, XML, CSV, and everything else.
jll29 3y ago
No binary format will ever kill CSV: plain-text based formats embody the UNIX philosophy of text files and text processing tool pipes to go with them, and nothing is more durable than keeping your data in text based exchange formats. You won't remember Parquet in 15 years, but you will have CSV files in 50 years.
- nyokodo 3y ago
  > You won't remember Parquet in 15 years, but you will have CSV files in 50 years. You're probably right about CSV but probably not parquet. Parquet is already 11 years old, there are vast data warehouses that store parquet, it's first class in the spark ecosystem, and a key component of iceberg. Crucially, formats like parquet are "good enough" for a use case that doesn't appear to be going away. There is a high probability in my estimation that enough places are still using them in 15 years to be memorable even if it isn't as common or as visible.
- dheera 3y ago
  CSV is actually a nice format if it weren't for literal newlines being allowed INSIDE values. That alone makes it much harder to parse correctly with simple code because you can't count on ASCII mode readline()-like functions to fetch 1 record in entirety. Considering it also separates records with newlines, they really should have replaced newlines with "\n" and require escaping "\" with "\\".

Unfortunately everyone has moved to "Parquet" (Packet? Parket? Pacquet?) already and we've sailed even further. I absolutely HATE this Parcquage.

by dheera 3y ago

Unfortunately everyone has moved to "Parquet" (Packet? Parket? Pacquet?) already and we've sailed even further. I absolutely HATE this Parcquage.

_kst_ 3y ago

https://www.databricks.com/glossary/what-is-parquet https://www.databricks.com/glossary/what-is-parquet (I don't think everyone has moved to it. I had never heard of it myself.)

dheera 3y ago
A lot of the machine learning world has started using it, it's annoying as hell, solves a problem that doesn't exist, has inadequate documentation, lacks a good GUI viewer, and lacks good command line converters to JSON, XML, CSV, and everything else.

jll29 3y ago

No binary format will ever kill CSV: plain-text based formats embody the UNIX philosophy of text files and text processing tool pipes to go with them, and nothing is more durable than keeping your data in text based exchange formats. You won't remember Parquet in 15 years, but you will have CSV files in 50 years.

nyokodo 3y ago
> You won't remember Parquet in 15 years, but you will have CSV files in 50 years. You're probably right about CSV but probably not parquet. Parquet is already 11 years old, there are vast data warehouses that store parquet, it's first class in the spark ecosystem, and a key component of iceberg. Crucially, formats like parquet are "good enough" for a use case that doesn't appear to be going away. There is a high probability in my estimation that enough places are still using them in 15 years to be memorable even if it isn't as common or as visible.
dheera 3y ago
CSV is actually a nice format if it weren't for literal newlines being allowed INSIDE values. That alone makes it much harder to parse correctly with simple code because you can't count on ASCII mode readline()-like functions to fetch 1 record in entirety. Considering it also separates records with newlines, they really should have replaced newlines with "\n" and require escaping "\" with "\\".