3 ms·
I just want to be able to append metadata to a Parquet file at the end without rewriting the whole file. Tombstones could be baked in the parquet file this way.
by simlevesque 1y ago
I just want to be able to append metadata to a Parquet file at the end without rewriting the whole file. Tombstones could be baked in the parquet file this way.
It does work with "one more file" but it's not good for performance.
- shayonj 1y agoYeah. Or just sub out the data with null bytes. Something like that could be nice too.
- simlevesque 1y agoAre you familiar with Parquet ? you can't do that at all, you need to rewrite the whole file.
- shayonj 1y agoYeah , I poorly phrased it - I meant in an ideal situation with the benefits of parquet like columnar file structure. I very much understand that it’s not possible on parquet today for the reasons you mentioned and others.
- cpard 1y agoThat’s the whole reason of existence of Iceberg, Delta and Hudi right? Not as easy as just appending metadata to a parquet file but in the other hand, parquet was never and probably shouldn’t be designed with that functionality in mind.