3 ms·
Protobuf has caused me nothing but trouble once I have even moderately large data sets. If what you are storing is large matrices or rasters, honestly nothing
by dagw 3y ago
Protobuf has caused me nothing but trouble once I have even moderately large data sets. If what you are storing is large matrices or rasters, honestly nothing beats HDF5.
- cjbgkagh 3y agoI’m old school, from the Managing Gigabytes by Ian H Witton era so I have ways of tailoring the data and protobuf schema to suit my application. Large matrices are stored as contiguous chunks, similar to how tensorflow stores tensors in graphs. I work with TBs of data in GB chunks. I used to work with PBs of data but that was a long time ago. Derialization of protobuf messages is so fast that it hasn’t been an issue and I can use it from any language with protobuf support.