3 ms·
Ah, I see, you're right. Well, one solution would involve a log table used to log/lookup the migration status: Anytime the real-time system touches data (eg. d
by smnc 10y ago
Ah, I see, you're right.
Well, one solution would involve a log table used to log/lookup the migration status: Anytime the real-time system touches data (eg. deletes the subscription in the source table and has nothing to do in the new table) the event is logged. The backfilling routine uses that data to decide wether its data or the current data is "fresher".
If the old subscription data is never (or for a certain period) deleted but merely marked inactive (eg. with fields "timestamp_end" and "timestamp_modified" updated) then that information can be used during the backfilling to decide on what data is more recent. [edit: I guess this is what haldean is reffering to above by "tombstone records."]
That makes sense, I think. Does it? Elsewhere in the thread i referred to the procedure used as "conceptually relatively simple". Most things are conceptually simple, but always with pitfalls and dark things lurking around the edges.