2 ms·
Postgres dump to Apache Iceberg, OLake got one step closer
- pkhodiyar 2y agoIt gives me immense joy to announce that we OLake’s (olake.io/docs) now supports Postgres as a data souce. Now you can: 1. Sync Postgres -> AWS S3 in parquet format [JSON dump and Level 1 flattened data support] 2. Sync Postgres -> Iceberg [in a few days, schema evolution support of Iceberg] 3. Sync Postgres -> Local Filestorage with everything we support for S3. Want to test it out locally? Sync Postgres via OLake -> MinIO (using JDBC catalog) -> Query using Engines that support Iceberg V2 tables [doc launching pretty soon] We talked with 100s of DE’s to understand their pain point and the two major issues for them were replicating Postgres and MySQL to a lakehouse format. They said, we delivered. Our source connectors and writers are independent, meaning that after Apache Iceberg writers are being supported (PR - https://github.com/datazip-inc/olake/pull/113 https://github.com/datazip-inc/olake/pull/113), any new connector will be able to dump to Iceberg with minimal changes!