4 ms·

Looks awesome! Maybe I missed it in the docs but I wonder how you solved the problem of a source table's initial snapshot when performing CDC (locking a large t

Looks awesome! Maybe I missed it in the docs but I wonder how you solved the problem of a source table's initial snapshot when performing CDC (locking a large table on a production DB can be problematic). Did you implement something like described here? https://netflixtechblog.com/dblog-a-generic-change-data-capture-framework-69351fb9099b https://netflixtechblog.com/dblog-a-generic-change-data-capt... Lastly, I wonder what plugin you're using for the Postgres logical replication? Congrats! Edit: re-read it and saw about the DuckDB blog's inspired ctid solution for the initial snapshot. Do you find querying by ctid okay performance wise? As far as I remember it will use sequential scans

saisrirampur 3y ago
Great question. It uses something called as Tid Scan, which kind of means directly reading from disk based on an address and Tid Scans are very efficient. The snapshot approach captured in the blog I shared, ensures consistency.
- jgraettinger1 3y ago
  It does involve pinning the WAL while the backfill is happening, though, which is quite different from the incremental backfill achieved by DBLog. The former can be faster, because you’re able to walk the table in physical storage order, but your DB cannot reclaim WAL segments while this happens (disk can fill). The latter allows for resumable backfills that can be tuned to the available spare capacity of the source DB, and doesn’t cause WAL segments to back up, but must walk tables in logical order which can sometimes be slower (for example, if you key using a random UUID).