4 ms·

Deequ is the big data / Spark alternative for similar functionality in case anyone is interested: https://github.com/awslabs/deequ https://github.com/awslabs/de

Deequ is the big data / Spark alternative for similar functionality in case anyone is interested: https://github.com/awslabs/deequ https://github.com/awslabs/deequ Looks like a great project and wanted to highlight why projects like these are much more useful for file formats that don't have metadata (CSV) vs file formats with metadata (Parquet). Parquet file footers contain metadata that provides column-level metadata, file-level metadata, and schema information. Cluster computing technologies can use Parquet metadata to skip entire files (Parquet predicate pushdown filtering). If there is an age column in the data, then the min/max values will be stored in the Parquet footer. If you run a Spark query for df = spark.read.parquet(/some_folder).where(age > 90), it'll only read in the files that have a max_age greater than 90. Data skipping is one of the best performance benefits. I wrote a blog post on analyzing Parquet file metadata with PyArrow if you're interested in learning more: https://mungingdata.com/pyarrow/parquet-metadata-min-max-statistics/ https://mungingdata.com/pyarrow/parquet-metadata-min-max-sta... File formats that make you infer the schema (CSV) are on the way out. Enjoy Parquet and the other benefits it provides, like column pruning!

BugsJustFindMe 5y ago
> File formats that make you infer the schema (CSV) are on the way out. Lol, good luck. People will continue using delimited text files for as long as they have eyeballs.
- MrPowers 5y ago
  Yea, haha, that was worded too strongly. Should have said "more folks are using Parquet for data workflows and adoption seems to be increasing". CSV is better when you need something mutable / human readable, so they'll always be around.
  - citilife 5y ago
    Main issue with Parquet is actually the metadata. CSV is nice because I can send it to a friend and they can instantly understand it; you can print it and understand it. In contrast, Parquet is good for systems and I definitely recommend it in programming, but sharing.. not so much.
stevesimmons 5y ago
> it'll only read in the files that have a max_age greater than 90. Data skipping is one of the best performance benefits. Actually Parquet files have a chunked structure, with maybe 200 row groups in a large Parquet file. The metadata is stored per chunk in the file footer. So if that suggests you only need a few chunks with a file, your data reads can be orders of magnitude faster. The data is stored column wise too. So if you only need a few of the columns, you can get another 10x increase in performance. It's like magic, if your data is ordered in a way that can exploit this.
- MrPowers 5y ago
  Good to know, thanks for clarifying! Parquet predicate pushdown filtering is even more powerful than I thought!

BugsJustFindMe 5y ago

> File formats that make you infer the schema (CSV) are on the way out. Lol, good luck. People will continue using delimited text files for as long as they have eyeballs.

MrPowers 5y ago
Yea, haha, that was worded too strongly. Should have said "more folks are using Parquet for data workflows and adoption seems to be increasing". CSV is better when you need something mutable / human readable, so they'll always be around.
- citilife 5y ago
  Main issue with Parquet is actually the metadata. CSV is nice because I can send it to a friend and they can instantly understand it; you can print it and understand it. In contrast, Parquet is good for systems and I definitely recommend it in programming, but sharing.. not so much.

stevesimmons 5y ago

> it'll only read in the files that have a max_age greater than 90. Data skipping is one of the best performance benefits. Actually Parquet files have a chunked structure, with maybe 200 row groups in a large Parquet file. The metadata is stored per chunk in the file footer. So if that suggests you only need a few chunks with a file, your data reads can be orders of magnitude faster. The data is stored column wise too. So if you only need a few of the columns, you can get another 10x increase in performance. It's like magic, if your data is ordered in a way that can exploit this.

MrPowers 5y ago
Good to know, thanks for clarifying! Parquet predicate pushdown filtering is even more powerful than I thought!