3 ms·
> Without the ability to lock system A It can be done using watermarking. Let us take the database example. Assume we have a timestamp column in database A. Th
by indogooner 9y ago
> Without the ability to lock system A
It can be done using watermarking. Let us take the database example. Assume we have a timestamp column in database A. The column can be used for syncing records to other storage system.
Typical implementations use a query (select records where timestamp >= 'initial_val' and timestamp <= 'max_val/desired_val') and then get the data.
Next run typically starts from max_val of previous run.
For open source you can look at Sqoop[1] or Gobblin[2] which do this
[1] https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html
[2] https://gobblin.readthedocs.io/ https://gobblin.readthedocs.io/
- pbowyer 9y agoThanks for the info re watermarking and the links. Re watermarking, would it prevent the overwriting example? If you could do conditional updates via the API then yes, because you could say "WHERE the record timestamp = the timestamp we have". But if not, the change in System A will be overwritten because no one knows it has changed.