3 ms·

It's interesting to watch how different companies that offer both Postgres and warehousing solution under 1 roof approach the same problem: - ClickHouse focuse

It's interesting to watch how different companies that offer both Postgres and warehousing solution under 1 roof approach the same problem: - ClickHouse focuses on traditional CDC (ClickPipes) and just make it blazingly fast - Databricks leans on their unified storage architecture (LTAP) to avoid copying data (though you can argue there is still a copy in the cache) - Snowflake uses a data mirroring CDC as extension so it runs directly on Postgres I'm still waiting for a Postgres provider to just let me mirror data directly to Iceberg, so I can plug in my own stateless query engine.

jbonatakis 2mo ago
> I'm still waiting for a Postgres provider to just let me mirror data directly to Iceberg, so I can plug in my own stateless query engine. The issue that each of those providers above has recently adopted Postgres as a secondary product aimed at supporting their main product, an OLAP database or engine, so they don’t want you plugging in your own query engine. I’d bet you’re likely to see this from a Postgres-specific provider first, like Supabase.
- hasyimibhar 2mo ago
  I’m waiting for Cloudflare R2 to eventually support mirroring Postgres into R2 catalog. It seems like a nice fit because they already have R2 SQL.
- kiwicopple 2mo ago
  > mirror data directly to Iceberg > I’d bet you’re likely to see this from a Postgres-specific provider first, like Supabase. we deprecated this feature in our ETL tool[0]. The functionality is still in there but we can't support some of the production features we'd need for data/schema guarantees Iceberg is still nascent - only supporting single-table transactions (at least when we tried). A lot of important CDC/transactional semantics were "a work in progress" upstream. We shifted our focus to ducklake, which stores the catalog in Postgres [0] https://github.com/supabase/etl https://github.com/supabase/etl
- wasifaleem 2mo ago
  > The issue that each of those providers above has recently adopted Postgres as a secondary product aimed at supporting their main product, an OLAP database or engine, so they don’t want you plugging in your own query engine. Disclosure: I work on Supermetal You don't need to wait for a provider, and the provider is arguably the wrong place for this. They all have an incentive to make their own OLAP engine the happy path. A dedicated CDC tool that writes Iceberg to your own storage and catalog keeps the tables and the engine choice yours. We built exactly that, a native Iceberg destination with Merge on Read. Since Snowflake and Databricks reject equality delete files, there's also a positional deletes only mode that writes deletion vectors instead, so the tables are readable from whatever engine you use. https://docs.supermetal.io/docs/main/targets/iceberg/ https://docs.supermetal.io/docs/main/targets/iceberg/
brightball 2mo ago
Snowflake/Crunchydata comes close to doing that with the pglake extension. There’s not a mirror function like what Snowflake offers directly but you can come close with a pgcron to upsert changes to the iceberg tables every so often. You can also purge the table put to the iceberg version every so often too depending on your data needs. Then you can create a query unions the results of both.
mslot 2mo ago
The challenge is converting primary key updates/deletes to row offsets in a columnar table. That requires maintaining an expensive mapping or doing expensive scans, and is not something you want Postgres itself to do. It's also this bursty, memory-intensive workload that you'd rather not have a lot of dedicated infrastructure for. At Snowflake we use Snowflake to do the apply work. Hence end-to-end mirroring has more pieces than just Postgres, but the capturing of changes is cheap enough to do in Postgres directly. (Author)
- hasyimibhar 2mo ago
  Yes I’m not suggesting to do this inside Postgres. I’m hoping that a Postgres provider can provide this mirroring capability out of the box, similar to how they provide a connection-pooled endpoint out of the box so I don’t have to self-host pgBouncer. I just want to be able to check a box somewhere and have a table in Postgres automatically mirrored to Iceberg, with guarantee that no data is lost. They can charge more for it, I will gladly pay.
  - mslot 2mo ago
    Makes sense, we just shipped it https://www.linkedin.com/posts/craigkerstiens_barely-over-2-weeks-ago-we-announced-the-share-7492602462068170753-eYwz/ https://www.linkedin.com/posts/craigkerstiens_barely-over-2-...
pepperoni_pizza 2mo ago
> I'm still waiting for a Postgres provider to just let me mirror data directly to Iceberg, so I can plug in my own stateless query engine. If you count AWS as Postgres provider, DMS into Kinesis into Firehose can do that. There was preview of just Firehose doing it directly, but AWS have pulled it because it was too unreliable. Maybe they rebuilt it since?
- hasyimibhar 2mo ago
  DMS is so unreliable though.
rockostrich 2mo ago
Google: Charges out the wazoo for a half baked product built on top of Dataflow

jbonatakis 2mo ago

> I'm still waiting for a Postgres provider to just let me mirror data directly to Iceberg, so I can plug in my own stateless query engine. The issue that each of those providers above has recently adopted Postgres as a secondary product aimed at supporting their main product, an OLAP database or engine, so they don’t want you plugging in your own query engine. I’d bet you’re likely to see this from a Postgres-specific provider first, like Supabase.

hasyimibhar 2mo ago
I’m waiting for Cloudflare R2 to eventually support mirroring Postgres into R2 catalog. It seems like a nice fit because they already have R2 SQL.
kiwicopple 2mo ago
> mirror data directly to Iceberg > I’d bet you’re likely to see this from a Postgres-specific provider first, like Supabase. we deprecated this feature in our ETL tool[0]. The functionality is still in there but we can't support some of the production features we'd need for data/schema guarantees Iceberg is still nascent - only supporting single-table transactions (at least when we tried). A lot of important CDC/transactional semantics were "a work in progress" upstream. We shifted our focus to ducklake, which stores the catalog in Postgres [0] https://github.com/supabase/etl https://github.com/supabase/etl
wasifaleem 2mo ago
> The issue that each of those providers above has recently adopted Postgres as a secondary product aimed at supporting their main product, an OLAP database or engine, so they don’t want you plugging in your own query engine. Disclosure: I work on Supermetal You don't need to wait for a provider, and the provider is arguably the wrong place for this. They all have an incentive to make their own OLAP engine the happy path. A dedicated CDC tool that writes Iceberg to your own storage and catalog keeps the tables and the engine choice yours. We built exactly that, a native Iceberg destination with Merge on Read. Since Snowflake and Databricks reject equality delete files, there's also a positional deletes only mode that writes deletion vectors instead, so the tables are readable from whatever engine you use. https://docs.supermetal.io/docs/main/targets/iceberg/ https://docs.supermetal.io/docs/main/targets/iceberg/

brightball 2mo ago

Snowflake/Crunchydata comes close to doing that with the pglake extension. There’s not a mirror function like what Snowflake offers directly but you can come close with a pgcron to upsert changes to the iceberg tables every so often. You can also purge the table put to the iceberg version every so often too depending on your data needs. Then you can create a query unions the results of both.

mslot 2mo ago

The challenge is converting primary key updates/deletes to row offsets in a columnar table. That requires maintaining an expensive mapping or doing expensive scans, and is not something you want Postgres itself to do. It's also this bursty, memory-intensive workload that you'd rather not have a lot of dedicated infrastructure for. At Snowflake we use Snowflake to do the apply work. Hence end-to-end mirroring has more pieces than just Postgres, but the capturing of changes is cheap enough to do in Postgres directly. (Author)

hasyimibhar 2mo ago
Yes I’m not suggesting to do this inside Postgres. I’m hoping that a Postgres provider can provide this mirroring capability out of the box, similar to how they provide a connection-pooled endpoint out of the box so I don’t have to self-host pgBouncer. I just want to be able to check a box somewhere and have a table in Postgres automatically mirrored to Iceberg, with guarantee that no data is lost. They can charge more for it, I will gladly pay.
- mslot 2mo ago
  Makes sense, we just shipped it https://www.linkedin.com/posts/craigkerstiens_barely-over-2-weeks-ago-we-announced-the-share-7492602462068170753-eYwz/ https://www.linkedin.com/posts/craigkerstiens_barely-over-2-...

pepperoni_pizza 2mo ago

> I'm still waiting for a Postgres provider to just let me mirror data directly to Iceberg, so I can plug in my own stateless query engine. If you count AWS as Postgres provider, DMS into Kinesis into Firehose can do that. There was preview of just Firehose doing it directly, but AWS have pulled it because it was too unreliable. Maybe they rebuilt it since?

hasyimibhar 2mo ago
DMS is so unreliable though.

rockostrich 2mo ago

Google: Charges out the wazoo for a half baked product built on top of Dataflow