3 ms·
Interesting. I wonder how different that is from RecordIO. Also, whether there'll be a Go implementation. [Edit, after looking a bit.] Pretty different. If I
by zellyn 9y ago
Interesting. I wonder how different that is from RecordIO. Also, whether there'll be a Go implementation.
[Edit, after looking a bit.]
Pretty different. If I remember correctly, RecordIO is re-synchronizing, whereas Riegeli seems to break things up into 64KB chunks, splitting messages across chunks if necessary.
[Edit, after finding more information.]
Interesting… looks like Riegeli is intended to compress well, rather than just store sequentially.
https://encode.ru/threads/2895-Riegeli-%E2%80%94-a-new-compressor-for-structured-data-(protocol-buffers) https://encode.ru/threads/2895-Riegeli-%E2%80%94-a-new-compr...
- throwaway84742 9y agoIIRC (but memory has faded considerably), RecordIO also did support something to aid compression across records (rather than just offer per-record compression). There was some gnarly code in it to that effect where there could be a compressed subset of several records within the file. But I might be wrong.