3 ms·
Yes, and that's what I did. In my case, I was writing to a file. And my languages were C, Python, and Lua. So, I wrote a small package (plus bindings for Python
by neuland 7y ago
Yes, and that's what I did. In my case, I was writing to a file. And my languages were C, Python, and Lua. So, I wrote a small package (plus bindings for Python and Lua) that would write out the message length as a uint64, then a uint8 for which schema the message is (I was working with multiple different message types), and then the actual message bytes.
So, it's do-able. But, I'm just saying that other formats/libraries support this natively out of the box.
- thedance 7y agoIt sounds a bit like you invented RecordIO[1]. I didn't realize you meant decoding from a stream containing several protobufs. In that case you do indeed need some kind of framing, because there is no end-of-message delimiter. The positive tradeoff is you can easily concatenate several protobufs into another, valid protobuf. That is impossible if there is a terminating symbol. 1: https://www.tensorflow.org/tutorials/load_data/tfrecord#tfrecords_format_details https://www.tensorflow.org/tutorials/load_data/tfrecord#tfre...
- neuland 7y agoOh neat. I'm not familiar with the ML/AI ecosystem. So, I didn't think to look there for formats; a bit of a shame I reinvented the wheel here, but it worked great for my specific use case. Yeah, it's all about the tradeoffs with formats. Protobuf is a good choice a lot of the time and I always think about it as one of the first/preferred options. gRPC is great too.