4 ms·
> 7.2 millions row groups Why would you need 7.2 mil row groups? Row group size when stored in HDFS is usually equal to HDFS bock size by default, which is 12
by maxnevermind 2y ago
> 7.2 millions row groups
Why would you need 7.2 mil row groups?
Row group size when stored in HDFS is usually equal to HDFS bock size by default, which is 128MB
7.2 mil * 128MB ~ 1PB
You have a single parquet file 1PB in size?
- thesz 2y agoParquet is not HDFS. It is a static format, not a B-tree in disguise like HDFS. You can have compressed Parquet columns with 8192 entries being a couple of tens bytes in size. 600 columns in a row group is then 12K bytes or so, leading us to 100GB file, not a petabyte. Four orders of magnitude of difference between your assessment and mine.