4 ms·

I see that https://github.com/fish-shell/fish-shell/issues/3341 https://github.com/fish-shell/fish-shell/issues/3341 is scheduled for 3.0. This is also marked a

I see that https://github.com/fish-shell/fish-shell/issues/3341 https://github.com/fish-shell/fish-shell/issues/3341 is scheduled for 3.0. This is also marked as the fix for the broken/invalid YAML format that is used for storing the shell history https://github.com/fish-shell/fish-shell/issues/2258 https://github.com/fish-shell/fish-shell/issues/2258 I implore you to NOT use JSON or ProtocolBuffers, neither are appropriate formats for configuring my shell, or storing long sets of textual data. Protocol Buffers is hardly the kind of format I want to parse and read my history in, and JSON is a less than ideal configuration format for many reasons. https://github.com/vstakhov/libucl https://github.com/vstakhov/libucl UCL (Universal Configuration Language) is a much more appropriate format for configuring programs than JSON, and is has been accepted as standard for use configuring all tools included in the FreeBSD operating system. I also suggest you consider using a different language to store shell history, since shell history and shell configuration are two very different jobs with different goals and trade offs. I personally would suggest a simple safe format like TOML https://github.com/toml-lang/toml https://github.com/toml-lang/toml for storing the history. But If you do want a single format for both configuration and history, I would implore you to pick TOML not JSON or Protocol Buffers.

ridiculous_fish 9y ago
Thanks for writing that up, I linked to it from the issue. FWIW I agree with you that shell history should be a simple format, and even ad-hoc parseable via Unix tools. zsh uses simple : and ; delimiters. JSON is more complex, and binary formats like protobuf are way out.
__sr__ 9y ago
Out of the mainstream formats, JSON is among the easiest to parse - thanks to the CLI tool jq. Does UCL have something equivalent? Protobuf (or any other binary format) has the benefit of being faster to load, but I agree - it should not be be used to store things like history. It should be easy to parse history without having to write code. And with Protobuf, you need the schema. If a binary format is going to be used, it should be optional - or perhaps there should be tools to convert the history file to/from a plaintext format.
- jhoechtl 9y ago
  Would like to see comments in configuration, Json doesn't support comments.
- deleted 9y ago
  [deleted]
- techdragon 9y ago
  There is uclcmd [https://github.com/allanjude/uclcmd https://github.com/allanjude/uclcmd] which could do with some more work but was created with the intent of being a UCL equivalent of JQ. Also since he’s the kind of person that a HN reader will probably Ike to know more about. The original author of the aforementioned uclcmd, Allan Jude [http://twitter.com/allanjude http://twitter.com/allanjude], is the former host of the TechSNAP podcast [http://www.jupiterbroadcasting.com/show/techsnap/ http://www.jupiterbroadcasting.com/show/techsnap/], current host of the BSD Now podcast [http://www.jupiterbroadcasting.com/show/bsdnow/ http://www.jupiterbroadcasting.com/show/bsdnow/], a current member of the FreeBSD Core team, and co-authored not one but two books on the ZFS file system FreeBSD Mastery: ZFS [https://www.amazon.com/FreeBSD-Mastery-ZFS-7/dp/0692452354/ https://www.amazon.com/FreeBSD-Mastery-ZFS-7/dp/0692452354/] and FreeBSD Mastery: Advanced ZFS [https://www.amazon.com/FreeBSD-Mastery-Advanced-ZFS/dp/0692688684/ https://www.amazon.com/FreeBSD-Mastery-Advanced-ZFS/dp/06926...]
- nine_k 9y ago
  UCL can be trivially transformed to JSON or YAML; ucl-tool supports this from CLI.
GlitchMr 9y ago
Supporting this, I don't think protobuf is a good format for storing history files. Backward compatibility is tricky with it - I don't think the history format is going to change all that much, but I think there were changes in history of fish shell like adding time field. If the choice would be on a binary format, I would much, much, much more prefer SQLite for that. Also, B-trees are perfectly fine choice for searching history, as they make prefix searches trivial. TOML is definitely an interesting choice if a text format is deemed necessary however. Array of tables syntax should be a fine syntax choice for storing history.
- fpoling 9y ago
  In addition protobuf is designed for communications, not incremental updates in a local store. As such making things robust to store things on a development machine is non-trivial. SQLite is so much better for that. After Firefox switched to use it for its own history, most problems with history corruption disappeared.
- FRex 9y ago
  I was about to say SQLite, it seems perfect - binary, fast, easy to update cheaply in place, immune to escaping et al problems (if you use prepared statements, not glue SQL with + of course), reliable, available in most languages (even for C and C++ the integration is trivial - it's a single pair of files drop in) and on the CLI, stable, the history data is very row/item oriented and not totally heterogeneous or free form, etc. I totally drank the SQLite koolaid a long time ago.
  - cben 9y ago
    History file may be read & updated simultaneously my multiple processes. Is that sufficiently safe from corruption with SQLite? (honest question, no FUD implied. same risks apply to alternatives too.)
    - timdierks 9y ago
      Yes, SQLite has robust multiprocess locking and transaction support. https://sqlite.org/faq.html#q5 https://sqlite.org/faq.html#q5
fpoling 9y ago
For configuration I would very much prefer to have something that can be properly grepped/patched/generated with shell scripts if necessary. What openssh uses for its config mostly fulfill this, but ini file also are not that bad. Anything hierarchical like JSON/TOML/UCL etc. becomes very unattractive the moment one has to patch them from automated setup scripts.
- merlincorey 9y ago
  Templates in configuration management can help with this a bit.
- webwurst 9y ago
  There pretty mighty tools around to grep/patch/generate JSON data with shell scripts and they are made to deal with hierarchical data. https://stedolan.github.io/jq/manual/ https://stedolan.github.io/jq/manual/ http://jsonnet.org/ http://jsonnet.org/ Both are single binaries, so easy to run them where you need it. I don't know comparable stuff for the other mentioned formats, whenever I need to deal with yaml in scripts i run https://github.com/bronze1man/yaml2json https://github.com/bronze1man/yaml2json I would be totally fine with any format that can be converted to JSON on the fly. Of course I would love to have something like these tools more integrated within a shell, but the representing text format on the disk is not that important for me.
Conan_Kudo 9y ago
Long-time fish user here, I'll second this comment and implore that you choose TOML for the configuration format. TOML format is easy for users to understand, and well-written parsers for it exist in every programming language.
daenney 9y ago
Thanks for bringing up UCL! Hadn't seen that before but I definitely need to try this in a few places.

ridiculous_fish 9y ago

Thanks for writing that up, I linked to it from the issue. FWIW I agree with you that shell history should be a simple format, and even ad-hoc parseable via Unix tools. zsh uses simple : and ; delimiters. JSON is more complex, and binary formats like protobuf are way out.

__sr__ 9y ago

Out of the mainstream formats, JSON is among the easiest to parse - thanks to the CLI tool jq. Does UCL have something equivalent? Protobuf (or any other binary format) has the benefit of being faster to load, but I agree - it should not be be used to store things like history. It should be easy to parse history without having to write code. And with Protobuf, you need the schema. If a binary format is going to be used, it should be optional - or perhaps there should be tools to convert the history file to/from a plaintext format.

jhoechtl 9y ago
Would like to see comments in configuration, Json doesn't support comments.
deleted 9y ago
[deleted]
techdragon 9y ago
There is uclcmd [https://github.com/allanjude/uclcmd https://github.com/allanjude/uclcmd] which could do with some more work but was created with the intent of being a UCL equivalent of JQ. Also since he’s the kind of person that a HN reader will probably Ike to know more about. The original author of the aforementioned uclcmd, Allan Jude [http://twitter.com/allanjude http://twitter.com/allanjude], is the former host of the TechSNAP podcast [http://www.jupiterbroadcasting.com/show/techsnap/ http://www.jupiterbroadcasting.com/show/techsnap/], current host of the BSD Now podcast [http://www.jupiterbroadcasting.com/show/bsdnow/ http://www.jupiterbroadcasting.com/show/bsdnow/], a current member of the FreeBSD Core team, and co-authored not one but two books on the ZFS file system FreeBSD Mastery: ZFS [https://www.amazon.com/FreeBSD-Mastery-ZFS-7/dp/0692452354/ https://www.amazon.com/FreeBSD-Mastery-ZFS-7/dp/0692452354/] and FreeBSD Mastery: Advanced ZFS [https://www.amazon.com/FreeBSD-Mastery-Advanced-ZFS/dp/0692688684/ https://www.amazon.com/FreeBSD-Mastery-Advanced-ZFS/dp/06926...]
nine_k 9y ago
UCL can be trivially transformed to JSON or YAML; ucl-tool supports this from CLI.

GlitchMr 9y ago

Supporting this, I don't think protobuf is a good format for storing history files. Backward compatibility is tricky with it - I don't think the history format is going to change all that much, but I think there were changes in history of fish shell like adding time field. If the choice would be on a binary format, I would much, much, much more prefer SQLite for that. Also, B-trees are perfectly fine choice for searching history, as they make prefix searches trivial. TOML is definitely an interesting choice if a text format is deemed necessary however. Array of tables syntax should be a fine syntax choice for storing history.

fpoling 9y ago
In addition protobuf is designed for communications, not incremental updates in a local store. As such making things robust to store things on a development machine is non-trivial. SQLite is so much better for that. After Firefox switched to use it for its own history, most problems with history corruption disappeared.
FRex 9y ago
I was about to say SQLite, it seems perfect - binary, fast, easy to update cheaply in place, immune to escaping et al problems (if you use prepared statements, not glue SQL with + of course), reliable, available in most languages (even for C and C++ the integration is trivial - it's a single pair of files drop in) and on the CLI, stable, the history data is very row/item oriented and not totally heterogeneous or free form, etc. I totally drank the SQLite koolaid a long time ago.
- cben 9y ago
  History file may be read & updated simultaneously my multiple processes. Is that sufficiently safe from corruption with SQLite? (honest question, no FUD implied. same risks apply to alternatives too.)
  - timdierks 9y ago
    Yes, SQLite has robust multiprocess locking and transaction support. https://sqlite.org/faq.html#q5 https://sqlite.org/faq.html#q5

fpoling 9y ago

For configuration I would very much prefer to have something that can be properly grepped/patched/generated with shell scripts if necessary. What openssh uses for its config mostly fulfill this, but ini file also are not that bad. Anything hierarchical like JSON/TOML/UCL etc. becomes very unattractive the moment one has to patch them from automated setup scripts.

merlincorey 9y ago
Templates in configuration management can help with this a bit.
webwurst 9y ago
There pretty mighty tools around to grep/patch/generate JSON data with shell scripts and they are made to deal with hierarchical data. https://stedolan.github.io/jq/manual/ https://stedolan.github.io/jq/manual/ http://jsonnet.org/ http://jsonnet.org/ Both are single binaries, so easy to run them where you need it. I don't know comparable stuff for the other mentioned formats, whenever I need to deal with yaml in scripts i run https://github.com/bronze1man/yaml2json https://github.com/bronze1man/yaml2json I would be totally fine with any format that can be converted to JSON on the fly. Of course I would love to have something like these tools more integrated within a shell, but the representing text format on the disk is not that important for me.

Conan_Kudo 9y ago

Long-time fish user here, I'll second this comment and implore that you choose TOML for the configuration format. TOML format is easy for users to understand, and well-written parsers for it exist in every programming language.

daenney 9y ago

Thanks for bringing up UCL! Hadn't seen that before but I definitely need to try this in a few places.