5 ms·

Very cool document. I kind of got stuck at delta queries, though. How do you implement that? I can't find any reference to delta/removed queries on Mongo, Postg

Very cool document. I kind of got stuck at delta queries, though. How do you implement that? I can't find any reference to delta/removed queries on Mongo, Postgres, or MySQL. Do you just keep all records in the database and add a "removed" field? How would that solution work with user privacy & users truly wanting to remove data?

dsp1234 10y ago
Delta queries can be serviced by any system that supports temporal tables[0][1], or another mechanism where data changes are tracked. [0] - https://en.wikipedia.org/wiki/SQL:2011 https://en.wikipedia.org/wiki/SQL:2011 (temporal support) [1] - https://msdn.microsoft.com/en-us/library/dn935015.aspx https://msdn.microsoft.com/en-us/library/dn935015.aspx [2] - http://clarkdave.net/2015/02/historical-records-with-postgresql-and-temporal-tables-and-sql-2011/ http://clarkdave.net/2015/02/historical-records-with-postgre...
ChrisAtWork 10y ago
Delta queries are pretty easy, assuming you have a rich data store behind your data. As others have said, your data store needs to be able to say, "Show me changes since XYZ". Most of the Big Apps can do that, and from there the problems is one of API Semantics. This doc addresses the API Semantics, rather than the application design. To try to solve the design problem would be impossible as every application is different.
- iheart2code 10y ago
  I understand what the doc is trying to convey, I just hadn't ever thought of the application-level logic for record deltas before, so I got incredibly distracted from the document. It made me wonder how other people, not using temporal tables, are accomplishing this right now.
  - ChrisAtWork 10y ago
    I don't think I've ever seen delta's done at-scale with actual temporal tables, but rather with transaction Id's of some sort. For example (based on the doc) when you register a delta link, there's a row written to Azure Tables saying, "owner:iheart2code, deltaToken:foo, lastRecordIdSeen:123". When you then make the Delta request, we look up "foo" for you, find that the last id you've seen is 123, and then only give you records from the transaction table with an id larger than that. Temporal is always a can of worms, as clocks are impossible to keep in sync and there are endless race conditions. Making the delta tokens causal, rather than temporal, is the way to go. Anything else is brutal in the distributed systems world...

dsp1234 10y ago

Delta queries can be serviced by any system that supports temporal tables[0][1], or another mechanism where data changes are tracked. [0] - https://en.wikipedia.org/wiki/SQL:2011 https://en.wikipedia.org/wiki/SQL:2011 (temporal support) [1] - https://msdn.microsoft.com/en-us/library/dn935015.aspx https://msdn.microsoft.com/en-us/library/dn935015.aspx [2] - http://clarkdave.net/2015/02/historical-records-with-postgresql-and-temporal-tables-and-sql-2011/ http://clarkdave.net/2015/02/historical-records-with-postgre...

ChrisAtWork 10y ago

Delta queries are pretty easy, assuming you have a rich data store behind your data. As others have said, your data store needs to be able to say, "Show me changes since XYZ". Most of the Big Apps can do that, and from there the problems is one of API Semantics. This doc addresses the API Semantics, rather than the application design. To try to solve the design problem would be impossible as every application is different.

iheart2code 10y ago
I understand what the doc is trying to convey, I just hadn't ever thought of the application-level logic for record deltas before, so I got incredibly distracted from the document. It made me wonder how other people, not using temporal tables, are accomplishing this right now.
- ChrisAtWork 10y ago
  I don't think I've ever seen delta's done at-scale with actual temporal tables, but rather with transaction Id's of some sort. For example (based on the doc) when you register a delta link, there's a row written to Azure Tables saying, "owner:iheart2code, deltaToken:foo, lastRecordIdSeen:123". When you then make the Delta request, we look up "foo" for you, find that the last id you've seen is 123, and then only give you records from the transaction table with an id larger than that. Temporal is always a can of worms, as clocks are impossible to keep in sync and there are endless race conditions. Making the delta tokens causal, rather than temporal, is the way to go. Anything else is brutal in the distributed systems world...