4 ms·

Estuary (https://estuary.dev https://estuary.dev ; I'm CTO) gives you a real time data lake'd change log of all the changes happening in your database in your c

Estuary (https://estuary.dev https://estuary.dev ; I'm CTO) gives you a real time data lake'd change log of all the changes happening in your database in your cloud storage -- complete with log sequence number, database time, and even before/after states if you use REPLICA IDENTITY FULL -- with no extra setup in your production DB. By default, if you then go on to materialize your collections somewhere else (like Snowflake), you get synchronized tables that follow your source DB as they update. But! You can also transform or materialize the complete history of your tables for auditing purposes from that same underlying data-lake, without going back to your source DB for another capture / WAL reader.

DenisM 3y ago
Are you using debezium to capture changes?
- jgraettinger1 3y ago
  No. We implemented our own [1] for a few reasons: * Scaling well to multi-TB DBs without pinning the write-ahead log (potentially filling your DB's disk) while the backfill is happening. Instead, our connector constantly reads the WAL and works well in setups like Supabase that have very restrictive WAL sizes (1GB iirc). * Incremental fault-tolerant backfills that can be stopped and resumed at will. * Flowing TOAST columns through seamlessly to your materialized destination, without requiring that you resort to REPLICA IDENTITY FULL. * Being able to offer "precise" captures which are logically consistent in terms of the sequence of create/update/delete events. The last one becomes really interesting when paired with REPLICA IDENTITY FULL because you can feed the resulting before/after states into an incremental computation (perhaps differential dataflow) for streaming updates of a query. Our work is based off of the Netflix DBLog paper, which we took and ran with. [1] https://github.com/estuary/connectors/tree/main/source-postgres https://github.com/estuary/connectors/tree/main/source-postg...
jvz 3y ago
I'm evaluating Flow for CDC. Do you support logical decoding messages from `pg_logical_emit_message`? This would allow us to add audit metadata[^1]. [^1]: https://www.infoq.com/articles/wonders-of-postgres-logical-decoding-messages/ https://www.infoq.com/articles/wonders-of-postgres-logical-d...
- jgraettinger1 3y ago
  No. But this is neat, and at a glance it looks straight forward to add. Happy to discuss further!
slotrans 3y ago
That sure sounds cool but I can't tell from your website that it does any of that. Even giving up on the marketing copy and going straight to the docs... I can't follow them.
- jgraettinger1 3y ago
  Hi fellow Dark Tower friend. Yes, marketing and docs are not our strongest suits and we need to do better. To be fair, though, we're also trying not to scare off less technical users who see a bullet list like above and think "well this is clearly not for me". It's a hard balance

DenisM 3y ago

Are you using debezium to capture changes?

jgraettinger1 3y ago
No. We implemented our own [1] for a few reasons: * Scaling well to multi-TB DBs without pinning the write-ahead log (potentially filling your DB's disk) while the backfill is happening. Instead, our connector constantly reads the WAL and works well in setups like Supabase that have very restrictive WAL sizes (1GB iirc). * Incremental fault-tolerant backfills that can be stopped and resumed at will. * Flowing TOAST columns through seamlessly to your materialized destination, without requiring that you resort to REPLICA IDENTITY FULL. * Being able to offer "precise" captures which are logically consistent in terms of the sequence of create/update/delete events. The last one becomes really interesting when paired with REPLICA IDENTITY FULL because you can feed the resulting before/after states into an incremental computation (perhaps differential dataflow) for streaming updates of a query. Our work is based off of the Netflix DBLog paper, which we took and ran with. [1] https://github.com/estuary/connectors/tree/main/source-postgres https://github.com/estuary/connectors/tree/main/source-postg...

jvz 3y ago

I'm evaluating Flow for CDC. Do you support logical decoding messages from `pg_logical_emit_message`? This would allow us to add audit metadata[^1]. [^1]: https://www.infoq.com/articles/wonders-of-postgres-logical-decoding-messages/ https://www.infoq.com/articles/wonders-of-postgres-logical-d...

jgraettinger1 3y ago
No. But this is neat, and at a glance it looks straight forward to add. Happy to discuss further!

slotrans 3y ago

That sure sounds cool but I can't tell from your website that it does any of that. Even giving up on the marketing copy and going straight to the docs... I can't follow them.

jgraettinger1 3y ago
Hi fellow Dark Tower friend. Yes, marketing and docs are not our strongest suits and we need to do better. To be fair, though, we're also trying not to scare off less technical users who see a bullet list like above and think "well this is clearly not for me". It's a hard balance