4 ms·
Well what are the hard parts then?
by anonymousDan 4y ago
Well what are the hard parts then?
- mywittyname 4y agoKeeping track of new record without hammering the database. Keeping track of updates without a dedicated column. Ditto for deletes. Schema drift. Managing datatype differences. I.e., Postgres supports complex datatypes that other DBs do not. Stored procedures. Basically any features of a DB that are beyond the bare minimum necessary to call something a relation database is an edge case. I rolled a solution for Postgres -> BQ by hand and it was much more involved than I expected. I had to set up a read-only replica and completely copying over the databases each run, which is only possible because they are so tiny.
- carlineng 4y agoJust to name a few off the top of my head -- - Data type mismatches between systems - Differences in handling ambiguous or bad data (e.g., null characters) - Handling backfills - Handling table schema changes - Writing merge queries to handle deletes/updates in a cost-effective way - Scrubbing the binlog of PII or other information that shouldn't make its way into the data warehouse - Determining which tables to replicate, and which to leave behind - Ability to replay the log from a point-in-time in case of an outage or other incident And I'm sure there are a lot more I'm not thinking of. None of these are terribly difficult in isolation, but there's a long tail of issues like these that need to be solved.