3 ms·
Some time ago I joined a project that was recording data logs in Protobuf format. I demonstrated that we could log the serialized Protobuf structures to a SQLit
by rgovostes 6y ago
Some time ago I joined a project that was recording data logs in Protobuf format. I demonstrated that we could log the serialized Protobuf structures to a SQLite database and then use a plugin to query across the message fields: https://github.com/rgov/sqlite_protobuf https://github.com/rgov/sqlite_protobuf
Learning how to use the SQLite extension API was a little mind-expanding but it worked like a charm.
- mikepurvis 6y agoI attempted something similar with large rosbag files, but the performance gain wasn’t enough to make the initial processing worth it. That said, I’d like to try again with duckdb as that seems like it might be better suited to time series analysis.
- liuliu 6y agoFlatbuffers would be better as it doesn't need to deserialize at all (I think protobuf supports to partially deserialization nowadays?). I did something similar with https://github.com/liuliu/dflat https://github.com/liuliu/dflat too!