3 ms·
I wish Google open sourced RecordIO instead (or in addition). People reinvent this particular bicycle, poorly, pretty much in every project where engineers are
by m0zg 6y ago
I wish Google open sourced RecordIO instead (or in addition). People reinvent this particular bicycle, poorly, pretty much in every project where engineers are smart enough to introduce a _structured_ application log.
- vore 6y agoIt looks like https://github.com/google/riegeli https://github.com/google/riegeli might be what you're looking for? (from a search of "RecordIO")
- m0zg 6y agoFar too complicated for what it's supposed to do. Google's RecordIO basically just concatenates messages together, with a few additional provisions for compression and handling small messages. In contrast, Riegeli depends on three different compressors and Abseil, and its file format specification is quite elaborate: https://github.com/google/riegeli/blob/master/doc/riegeli_records_file_format.md https://github.com/google/riegeli/blob/master/doc/riegeli_re.... This could explain its lack of market penetration (only 221 stars at the time of this writing). The authors apparently don't care that it's not popular, since they don't even provide a code example on how to use it. It also apparently uses HighwayHash for integrity checking, which is overkill - such things should use crc32c, for which most modern CPUs have a dedicated, fast instruction.
- davidwtbuxton 6y agoAlso this https://github.com/xeno14/recordio https://github.com/xeno14/recordio
- jeffbee 6y agoTensorFlow's TFRecord is RecordIO.