3 ms·
The challenge is converting primary key updates/deletes to row offsets in a columnar table. That requires maintaining an expensive mapping or doing expensive s
by mslot 2mo ago
The challenge is converting primary key updates/deletes to row offsets in a columnar table. That requires maintaining an expensive mapping or doing expensive scans, and is not something you want Postgres itself to do. It's also this bursty, memory-intensive workload that you'd rather not have a lot of dedicated infrastructure for.
At Snowflake we use Snowflake to do the apply work. Hence end-to-end mirroring has more pieces than just Postgres, but the capturing of changes is cheap enough to do in Postgres directly.
(Author)
- hasyimibhar 2mo agoYes I’m not suggesting to do this inside Postgres. I’m hoping that a Postgres provider can provide this mirroring capability out of the box, similar to how they provide a connection-pooled endpoint out of the box so I don’t have to self-host pgBouncer. I just want to be able to check a box somewhere and have a table in Postgres automatically mirrored to Iceberg, with guarantee that no data is lost. They can charge more for it, I will gladly pay.
- mslot 2mo agoMakes sense, we just shipped it https://www.linkedin.com/posts/craigkerstiens_barely-over-2-weeks-ago-we-announced-the-share-7492602462068170753-eYwz/ https://www.linkedin.com/posts/craigkerstiens_barely-over-2-...