4 ms·
It's strange to me that people complain about some variety in CSV files while acting as if parquet was one specific file format that's set in stone. They can't
by eska 3y ago
It's strange to me that people complain about some variety in CSV files while acting as if parquet was one specific file format that's set in stone.
They can't even decide which features are core, and the file format has many massive backwards-incompatible changes already. If you give me a parquet file I cannot guarantee that I can read it, and if I produce one I cannot guarantee that you can.
I treat formats such as parquet as I generally do: I try to allow various different inputs, and produce standard outputs. Parquet is something I allow purely as an optimization. CSV is the common default all of my tools have (UTF-8 without BOM, international locale, comma separator, quoting at the start of the value optional, standards-compliant date format or unix timestamps). Users generally don't have any issue with adapting their files to that format if there's any difference.
- orthoxerox 3y agoThat's true. Parquet went through the weirdest changes between its various revisions and because it was used for Hadoop data lakes, there's a whole bunch of data that is being stored in legacy formats. Off the top of my head: - different physical types to store timestamps: INT96 vs INT64 - different ways to interpret timestamps before tzdb (current vs earliest tzdb record) - different ways to handle proleptic Gregorian dates and timestamps - different ways to handle time zones (since Parquet only has the equivalents of LocalDateTime and Instant, but no OffsetDateTime or ZonedDateTime and earlier versions of Hive 3 were terribly confused which is which) - decimal data type was written differently, as a byte array in older versions and as int/byte array/binary in the newer ones - Hadoop ecosystem doesn't support decimals longer than 38 digits, but the file format supports them