2 ms·
Ask HN: How would you keep 2 APIs in sync?
Since Mulesoft's acquisition I've been thinking how I would build something similar. A couple of projects I'm involved in have mentioned "We need bi-directional sync between Salesforce and X, Y and Z", which has always been followed by "but MuleSoft and PieSync cost serious $$$$$".
Have have/would you build it? The resources I've found are about syncing files between two endpoints, or databases between mobile devices and the server (which is similar, but often encapsulated in the storage).
I already have a page of edge-cases where data would be incorrectly overwritten. E.g.:
09:00:00 Data polled from systems A and B (for these are 'enterprise' systems without webhooks or notifications or changefeeds)
09:00:01 A record (which is in the polled batch) is changed in system A
09:00:07 The changes are written back to each system, OVERWRITING THE CHANGE IN SYSTEM A MADE AT 09:00:01
Without the ability to lock system A while the sync is done, I don't see how this can be prevented?
References appreciated; I'm on holiday soon and up for a lot of reading. StackExchange has been very silent; 'ESB' and 'bi-directional sync' aren't the right keywords?
Finally, any open source implementations I should look at?
- indogooner 9y ago> Without the ability to lock system A It can be done using watermarking. Let us take the database example. Assume we have a timestamp column in database A. The column can be used for syncing records to other storage system. Typical implementations use a query (select records where timestamp >= 'initial_val' and timestamp <= 'max_val/desired_val') and then get the data. Next run typically starts from max_val of previous run. For open source you can look at Sqoop[1] or Gobblin[2] which do this [1] https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html [2] https://gobblin.readthedocs.io/ https://gobblin.readthedocs.io/
- pbowyer 9y agoThanks for the info re watermarking and the links. Re watermarking, would it prevent the overwriting example? If you could do conditional updates via the API then yes, because you could say "WHERE the record timestamp = the timestamp we have". But if not, the change in System A will be overwritten because no one knows it has changed.