4 ms·

Supporting this, I don't think protobuf is a good format for storing history files. Backward compatibility is tricky with it - I don't think the history format

Supporting this, I don't think protobuf is a good format for storing history files. Backward compatibility is tricky with it - I don't think the history format is going to change all that much, but I think there were changes in history of fish shell like adding time field. If the choice would be on a binary format, I would much, much, much more prefer SQLite for that. Also, B-trees are perfectly fine choice for searching history, as they make prefix searches trivial. TOML is definitely an interesting choice if a text format is deemed necessary however. Array of tables syntax should be a fine syntax choice for storing history.

fpoling 9y ago
In addition protobuf is designed for communications, not incremental updates in a local store. As such making things robust to store things on a development machine is non-trivial. SQLite is so much better for that. After Firefox switched to use it for its own history, most problems with history corruption disappeared.
FRex 9y ago
I was about to say SQLite, it seems perfect - binary, fast, easy to update cheaply in place, immune to escaping et al problems (if you use prepared statements, not glue SQL with + of course), reliable, available in most languages (even for C and C++ the integration is trivial - it's a single pair of files drop in) and on the CLI, stable, the history data is very row/item oriented and not totally heterogeneous or free form, etc. I totally drank the SQLite koolaid a long time ago.
- cben 9y ago
  History file may be read & updated simultaneously my multiple processes. Is that sufficiently safe from corruption with SQLite? (honest question, no FUD implied. same risks apply to alternatives too.)
  - timdierks 9y ago
    Yes, SQLite has robust multiprocess locking and transaction support. https://sqlite.org/faq.html#q5 https://sqlite.org/faq.html#q5
theshrike79 9y ago
You can ad extra fields to protobufs just fine, they just need to be added to the end of the definition.

fpoling 9y ago

In addition protobuf is designed for communications, not incremental updates in a local store. As such making things robust to store things on a development machine is non-trivial. SQLite is so much better for that. After Firefox switched to use it for its own history, most problems with history corruption disappeared.

FRex 9y ago

I was about to say SQLite, it seems perfect - binary, fast, easy to update cheaply in place, immune to escaping et al problems (if you use prepared statements, not glue SQL with + of course), reliable, available in most languages (even for C and C++ the integration is trivial - it's a single pair of files drop in) and on the CLI, stable, the history data is very row/item oriented and not totally heterogeneous or free form, etc. I totally drank the SQLite koolaid a long time ago.

cben 9y ago
History file may be read & updated simultaneously my multiple processes. Is that sufficiently safe from corruption with SQLite? (honest question, no FUD implied. same risks apply to alternatives too.)
- timdierks 9y ago
  Yes, SQLite has robust multiprocess locking and transaction support. https://sqlite.org/faq.html#q5 https://sqlite.org/faq.html#q5

theshrike79 9y ago

You can ad extra fields to protobufs just fine, they just need to be added to the end of the definition.