3 ms·
It looks like https://github.com/google/riegeli https://github.com/google/riegeli might be what you're looking for? (from a search of "RecordIO")
by vore 6y ago
It looks like https://github.com/google/riegeli https://github.com/google/riegeli might be what you're looking for? (from a search of "RecordIO")
- m0zg 6y agoFar too complicated for what it's supposed to do. Google's RecordIO basically just concatenates messages together, with a few additional provisions for compression and handling small messages. In contrast, Riegeli depends on three different compressors and Abseil, and its file format specification is quite elaborate: https://github.com/google/riegeli/blob/master/doc/riegeli_records_file_format.md https://github.com/google/riegeli/blob/master/doc/riegeli_re.... This could explain its lack of market penetration (only 221 stars at the time of this writing). The authors apparently don't care that it's not popular, since they don't even provide a code example on how to use it. It also apparently uses HighwayHash for integrity checking, which is overkill - such things should use crc32c, for which most modern CPUs have a dedicated, fast instruction.
- davidwtbuxton 6y agoAlso this https://github.com/xeno14/recordio https://github.com/xeno14/recordio