4 ms·
One important thing missing from the current criticism is Protobuf’s lack of a facility for serializing a sequence of messages to a file. There’s RecordIO inte
by choppaface 7y ago
One important thing missing from the current criticism is Protobuf’s lack of a facility for serializing a sequence of messages to a file. There’s RecordIO internally at Google, yet they markedly declined to open-source the C++ lib for it. There’s hints of it in Protobuf Java and then Amazon has open-sourced their own implementation of it with the same name.
Lack of public RecordIO is partially to blame for creation of TFRecords, which are in many ways inferior to (for example) tar archives of string-serialized protobuf messages. (tar supports index, streaming, compression, etc).
- dekhn 7y agoI requested that the RecordIO format (bytes on the disk and codfe implementation) be opensourced (for ease of interoperability between Google datasets and open source/scientific work). It wasn't because there were some 'flaws' in the design, but it was pointed out that leveldb open sourced a format very similar to it (but which never got used outside of leveldb).