4 ms·

As a data engineer, who is regularly fighting - "these two databases have different SQL dialects" - "did we miss a few rows due to poor transaction-isolation

As a data engineer, who is regularly fighting - "these two databases have different SQL dialects" - "did we miss a few rows due to poor transaction-isolation when trying to query recently changed rows on the upstream database" - "is there some checksum of a region of cells that accepts any arrangement of rows and columns that doesn't require me to think about ordering?" - "is there a subtle data format conversion problem silently mangling my data?" ...I've toying with trying to find a way to serialize everything consistently into something that can be XOR'd, then compare the output of XOR for two tables in two different databases that should be identical, without having to do some giant order-by comparison. Ideally, if necessary, this could be done by eye with two query windows pointing at two different databases... Basically, Datafold's datadiff, but in a way that could plausibly be home-rolled for on-premise applications (especially quick spot-checks), and not be a total maintenance nightmare... https://github.com/datafold/data-diff https://github.com/datafold/data-diff Don't have anything working yet, but it just seems like one could at least xor a bunch of integers and get something useful... Somehow.

smcameron 2y ago
An algorithm I've heard of in multiplayer game programming, in which massive state must be transmitted to many clients ... xor the previous game state with the new game state on the server ... most of the result will be zeros because most of the new state is the same as the old state. Then compress it. All those zeros compress really really well which is great for network traffic. Then transmit the compressed xor'ed state, client decompresses and xors with current state to get new state. Don't remember where I first heard of this but it was probably 15 or 20 years ago. Really clever. Never tried to implement such a thing though. Sounds fiddly and hard to debug.
- gblargg 2y ago
  In this framing MPEG video compression is just a more sophisticated XOR.

smcameron 2y ago

An algorithm I've heard of in multiplayer game programming, in which massive state must be transmitted to many clients ... xor the previous game state with the new game state on the server ... most of the result will be zeros because most of the new state is the same as the old state. Then compress it. All those zeros compress really really well which is great for network traffic. Then transmit the compressed xor'ed state, client decompresses and xors with current state to get new state. Don't remember where I first heard of this but it was probably 15 or 20 years ago. Really clever. Never tried to implement such a thing though. Sounds fiddly and hard to debug.

gblargg 2y ago
In this framing MPEG video compression is just a more sophisticated XOR.