3 ms·

> I'm still waiting for a Postgres provider to just let me mirror data directly to Iceberg, so I can plug in my own stateless query engine. The issue that each

> I'm still waiting for a Postgres provider to just let me mirror data directly to Iceberg, so I can plug in my own stateless query engine. The issue that each of those providers above has recently adopted Postgres as a secondary product aimed at supporting their main product, an OLAP database or engine, so they don’t want you plugging in your own query engine. I’d bet you’re likely to see this from a Postgres-specific provider first, like Supabase.

hasyimibhar 2mo ago
I’m waiting for Cloudflare R2 to eventually support mirroring Postgres into R2 catalog. It seems like a nice fit because they already have R2 SQL.
kiwicopple 2mo ago
> mirror data directly to Iceberg > I’d bet you’re likely to see this from a Postgres-specific provider first, like Supabase. we deprecated this feature in our ETL tool[0]. The functionality is still in there but we can't support some of the production features we'd need for data/schema guarantees Iceberg is still nascent - only supporting single-table transactions (at least when we tried). A lot of important CDC/transactional semantics were "a work in progress" upstream. We shifted our focus to ducklake, which stores the catalog in Postgres [0] https://github.com/supabase/etl https://github.com/supabase/etl
wasifaleem 2mo ago
> The issue that each of those providers above has recently adopted Postgres as a secondary product aimed at supporting their main product, an OLAP database or engine, so they don’t want you plugging in your own query engine. Disclosure: I work on Supermetal You don't need to wait for a provider, and the provider is arguably the wrong place for this. They all have an incentive to make their own OLAP engine the happy path. A dedicated CDC tool that writes Iceberg to your own storage and catalog keeps the tables and the engine choice yours. We built exactly that, a native Iceberg destination with Merge on Read. Since Snowflake and Databricks reject equality delete files, there's also a positional deletes only mode that writes deletion vectors instead, so the tables are readable from whatever engine you use. https://docs.supermetal.io/docs/main/targets/iceberg/ https://docs.supermetal.io/docs/main/targets/iceberg/

hasyimibhar 2mo ago

I’m waiting for Cloudflare R2 to eventually support mirroring Postgres into R2 catalog. It seems like a nice fit because they already have R2 SQL.

kiwicopple 2mo ago

> mirror data directly to Iceberg > I’d bet you’re likely to see this from a Postgres-specific provider first, like Supabase. we deprecated this feature in our ETL tool[0]. The functionality is still in there but we can't support some of the production features we'd need for data/schema guarantees Iceberg is still nascent - only supporting single-table transactions (at least when we tried). A lot of important CDC/transactional semantics were "a work in progress" upstream. We shifted our focus to ducklake, which stores the catalog in Postgres [0] https://github.com/supabase/etl https://github.com/supabase/etl

wasifaleem 2mo ago

> The issue that each of those providers above has recently adopted Postgres as a secondary product aimed at supporting their main product, an OLAP database or engine, so they don’t want you plugging in your own query engine. Disclosure: I work on Supermetal You don't need to wait for a provider, and the provider is arguably the wrong place for this. They all have an incentive to make their own OLAP engine the happy path. A dedicated CDC tool that writes Iceberg to your own storage and catalog keeps the tables and the engine choice yours. We built exactly that, a native Iceberg destination with Merge on Read. Since Snowflake and Databricks reject equality delete files, there's also a positional deletes only mode that writes deletion vectors instead, so the tables are readable from whatever engine you use. https://docs.supermetal.io/docs/main/targets/iceberg/ https://docs.supermetal.io/docs/main/targets/iceberg/