3 ms·

I'm curious about how this edge case was handled: 1. Lisa has old tv_service subscription, saved to snapshot, but not live-copied to Subscription table. 2. Li

by rattray 10y ago

I'm curious about how this edge case was handled: 1. Lisa has old tv_service subscription, saved to snapshot, but not live-copied to Subscription table. 2. Lisa removes tv_service subscription. 3. Before snapshot is updated, batch processing adds lisa_tv_service to the Subscription table. 4. Subscription now has lisa_tv_service, but Lisa.subscriptions does not include tv_service. My only idea: wait a day, then a batch-process job looking for data that exists in Subscription table but not in Customer.subscriptions

smnc 10y ago
In step two both collections/tables are updated. The backfilling routine is hopefully "smart" enough to leave stuff already in the new table alone.
- rattray 10y ago
  Right, but there's nothing to delete in the Subscription table at step 2. `lisa_tv_service` isn't added until the backfilling step.
  - smnc 10y ago
    Ah, I see, you're right. Well, one solution would involve a log table used to log/lookup the migration status: Anytime the real-time system touches data (eg. deletes the subscription in the source table and has nothing to do in the new table) the event is logged. The backfilling routine uses that data to decide wether its data or the current data is "fresher". If the old subscription data is never (or for a certain period) deleted but merely marked inactive (eg. with fields "timestamp_end" and "timestamp_modified" updated) then that information can be used during the backfilling to decide on what data is more recent. [edit: I guess this is what haldean is reffering to above by "tombstone records."] That makes sense, I think. Does it? Elsewhere in the thread i referred to the procedure used as "conceptually relatively simple". Most things are conceptually simple, but always with pitfalls and dark things lurking around the edges.
haldean 10y ago
You could use a tombstone record instead of deleting outright, and the batch update could skip copies that have corresponding tombstones.

by rattray 10y ago

smnc 10y ago

In step two both collections/tables are updated. The backfilling routine is hopefully "smart" enough to leave stuff already in the new table alone.

rattray 10y ago
Right, but there's nothing to delete in the Subscription table at step 2. `lisa_tv_service` isn't added until the backfilling step.
- smnc 10y ago
  Ah, I see, you're right. Well, one solution would involve a log table used to log/lookup the migration status: Anytime the real-time system touches data (eg. deletes the subscription in the source table and has nothing to do in the new table) the event is logged. The backfilling routine uses that data to decide wether its data or the current data is "fresher". If the old subscription data is never (or for a certain period) deleted but merely marked inactive (eg. with fields "timestamp_end" and "timestamp_modified" updated) then that information can be used during the backfilling to decide on what data is more recent. [edit: I guess this is what haldean is reffering to above by "tombstone records."] That makes sense, I think. Does it? Elsewhere in the thread i referred to the procedure used as "conceptually relatively simple". Most things are conceptually simple, but always with pitfalls and dark things lurking around the edges.

haldean 10y ago

You could use a tombstone record instead of deleting outright, and the batch update could skip copies that have corresponding tombstones.