3 ms·
this is nice... but missing is a discussion around sync frames when using delta encoding. typically in video compression, every once in a while a full frame im
by a-dub 5y ago
this is nice... but missing is a discussion around sync frames when using delta encoding.
typically in video compression, every once in a while a full frame image is included in the stream (mpeg calls these "i frames"). this allows for seeking and restarting after lost/corrupted data where just pure deltas require a complete replay and no gaps or corruption to recover the values.
in the case of something like this, i could imagine sync frames looking like a sentinel sequence for finding the sync frames, full value(s) for each delta encoded variable, plus possibly mini skip lists for searching (although an external index could be used for this too)
- Olreich 5y agoIt seems they are relying on the partitions to act as sync frames for this implementation.
- a-dub 5y agotrue. and maybe that's my bias from cutting my teeth on video compression formats, but i still think it's an interesting discussion. by explicitly including sync frames in the data storage format itself (rather than the storage medium) a few really nice properties emerge: 1) the compression in the stream becomes independent of storage. it remains self contained both in the database and also out of the database when being sent over the network or stored in ram. mpeg very nicely can be used for storage and transmission of data in it's compressed format. 2) by decoupling it from storage, compression can also be applied to the data itself rather than just the timestamps. sure, the block/partition headers could be expanded for the data, but then this gets messier when it comes to addition and deletion of dimensions. all of that said, for a simple tutorial on time series storage, this implementation is nice and simple.