4 ms·
My Hacker News items table in ClickHouse has 47,428,860 items, and it's 5.82 GB compressed and 18.18 GB uncompressed. What makes Parquet compression worse here,
by gkbrk 7mo ago
My Hacker News items table in ClickHouse has 47,428,860 items, and it's 5.82 GB compressed and 18.18 GB uncompressed. What makes Parquet compression worse here, when both formats are columnar?
- xnx 7mo agoParquet has a few compression option. Not sure which one they are using.
- hirako2000 7mo agoPlus isn't the least wasteful format, native duckdb for instance compacts better. That's not just down to the compression algorithm, which as you say got three main options for parquet.
- 0cf8612b2e1e 7mo agoSorting, compression algorithm +level, and data types can all have an impact. I noted elsewhere that a Boolean is getting represented as an integer. That’s one bit vs 1-4 bytes. There is also flexibility in what you define as the dataset. Skinnier, but more focused tables could be space saving vs a wide table that covers everything -will probably break compressible runs of data.
- boznz 7mo ago.. and Remove all the political shit-slop since COVID/AI and it's probably under a gig.
- mulmen 7mo agoYou could download the data and run that analysis yourself. I’d be interested to see it, especially your method of identifying “political shit-slop” and “AI” and the relationship to COVID. Sounds like an interesting project.