3 ms·

I had a read through your docs but was unable to find any info on how you handle the sync of hard deletions from sources. We use Stitch at the moment and have

I had a read through your docs but was unable to find any info on how you handle the sync of hard deletions from sources. We use Stitch at the moment and have found this to be a surprisingly hard problem to solve without binary log replication, and without full refreshes. For the moment we've ruled out binlogs as we use Aurora MySQL which requires you to binlog from the master, and tying my data warehouse replication to the master node concerns me. In incremental mode, for common DBs e.g. MySQL / Postgres, will Airbyte pick up hard deletions ever?

mtricot 6y ago
We don’t today, we will tomorrow when we start working on CDC. You’re raising a good point with MySQL, we will need to take this limitation into account. Hopefully there is a workaround.
- francoisp 6y ago
  check this out, https://github.com/francoisp/rosettable https://github.com/francoisp/rosettable. Uses binlog a reader to call postgres triggers, upon which you can build 2 way sync realtime.
francoisp 6y ago
hello Matt, you might be interested in this: uses binlog to provide 2 way sync to postgres including hard delete detection, using postgres triggers. https://github.com/francoisp/rosettable https://github.com/francoisp/rosettable