3 ms·

You make a great point and I'm sorry about that. To answer some of your questions and try to add more detail: Strings are UTF-8. Internally Parquet has the c

You make a great point and I'm sorry about that. To answer some of your questions and try to add more detail: Strings are UTF-8. Internally Parquet has the concept of "physical types" (ie how bytes are arranged) and "logical types". 64 bit ints are just physical with no logical type. DateTimes are 64 bit int physical types (millis since unix epoch - aka "Javascript time") with a logical type of "datetime" so you know to read it that way. Bool columns are bits in a row. The "data bearing" parts of a file can be compressed and usually are. Most are using snappy, a fastish, moderate compression codec. I believe some are also using zstd. One of the common questions people ask is: "how does it compare against .csv.gz" and the answer is favourably. Gzipped csv is even slower to read and write and usually about 30% bigger on disk.

Godel_unicode 4y ago
One important intuition about gzip vs snappy is that gzip isn’t generally parallelizable, while snappy is. If your Hadoop cluster is storing enormous files this shows up very quickly in read performance.
- mxmlnkn 4y ago
  Decompression of arbitrary gzip files can be parallelized with pragzip: https://github.com/mxmlnkn/pragzip https://github.com/mxmlnkn/pragzip
  - Godel_unicode 4y ago
    Thus the words “isn’t generally” as opposed to “cannot be”.
- glogla 4y ago
  That however only impacts CSVs or other text files, where the compression is the envelope. In parquet, it is the internal chunks that are compressed, so it is as parallelizable as ever. Also, generic CSV is not parallelizable anyway, because it allows you to have (quoted) newlines inside fields and then there is really now way to split it on newlines because you never know if you're on beginning or row or in the middle. So parallel reading of CSVs is more of a optimization for specific CSVs that you know don't have newlines inside fields.

Godel_unicode 4y ago

One important intuition about gzip vs snappy is that gzip isn’t generally parallelizable, while snappy is. If your Hadoop cluster is storing enormous files this shows up very quickly in read performance.

mxmlnkn 4y ago
Decompression of arbitrary gzip files can be parallelized with pragzip: https://github.com/mxmlnkn/pragzip https://github.com/mxmlnkn/pragzip
- Godel_unicode 4y ago
  Thus the words “isn’t generally” as opposed to “cannot be”.
glogla 4y ago
That however only impacts CSVs or other text files, where the compression is the envelope. In parquet, it is the internal chunks that are compressed, so it is as parallelizable as ever. Also, generic CSV is not parallelizable anyway, because it allows you to have (quoted) newlines inside fields and then there is really now way to split it on newlines because you never know if you're on beginning or row or in the middle. So parallel reading of CSVs is more of a optimization for specific CSVs that you know don't have newlines inside fields.