6 ms·

fish shell is looking for a new text serialization format for its history file (currently it uses an ad-hoc broken psuedo-YAML). Boxes to check: 1. Self descr

fish shell is looking for a new text serialization format for its history file (currently it uses an ad-hoc broken psuedo-YAML). Boxes to check: 1. Self describing format 2. SAX-style parser available to C++ 3. Easy for users to understand and ad-hoc parse using command-line tools 4. No document closing necessary, so appending is trivial YAML looks pretty good: - cmd: git checkout file.txt when: 1565133286 pwd: /home/me/dir/ paths: - file.txt protobuf is also an option: entry { cmd: "git checkout file.txt" when: 1565133286 paths: "file.txt" } though I am unsure of how well its text serialization is supported. Any suggestions?

akx 7y ago
How about JSONL (JSON Lines)? http://jsonlines.org/ http://jsonlines.org/ Ps. Thanks for (all the) fish, it's my daily driver shell and keeps me that much more sane c.f. the alternatives.
- j0057 7y ago
  That's really close to [RFC 7464](https://tools.ietf.org/html/rfc7464 https://tools.ietf.org/html/rfc7464), JSON Text Sequences. It uses U+001E RECORD SEPARATOR. The `jq` tool supports those if you pass a flag.
  - edraferi 7y ago
    So just add a record separator at the end of each line?
  - meowface 7y ago
    jq can also handle newline-separated JSON objects, with the -s flag.
mjevans 7y ago
(suggestion) Drop the 4th requirement. Having to close contexts is a VERY good 'sanity check' to see if something is malformed or not. If appending is necessary make the parser handle multiple copies of the namespace and merge them upon output. Unknown keys and sections should also always be copied from input to output (this is how you embed comments).
- ridiculous_fish 7y ago
  I'm interested, can you explain more about the merging idea? To clarify the requirement, history could be a JSON array of objects: [ {"cmd": "git checkout", "when": 1234 }, {"cmd": "vagrant up", "when": 4567 } ] To append an entry to this file and keep it valid, one must locate the closing square bracket and overwrite it. That work is what I hope to avoid.
  - ainar-g 7y ago
    Why not use a “stream” of objects? {"cmd": "git checkout", "when": 1234} {"cmd": "vagrant up", "when": 4567} Not sure about other languages and libraries, but Go supports this out of the box[1]. And while we're at it, why not CSV? That can be processed with awk. #cmd,when "git checkout","1234" "vagrant up","4567" [1]: https://play.golang.org/p/sTN9z4Kv3DB https://play.golang.org/p/sTN9z4Kv3DB
    - ridiculous_fish 7y ago
      CSV is fine for simple cases but has issues with versioning (adding new/optional fields) and nested data like arrays. The object stream idea is apparently supported widely and seems pretty strong. Thanks for the suggestion!
      - zenexer 7y ago
        I use JSON streams a lot with command line tools. Keep in mind that it’s limited in that each object must consume a single line. This allows you to recover from syntax errors in a single entry; each line is a fresh start. Have you considered SQLite? I know it’s not a friendly text format, but it alleviates a lot of the issues with the “append to a text file” approach, such as concurrency. It’s great for this sort of thing.
  - mjevans 7y ago
    Loosely running with your example: A file already exists with this content: [{"cmd": "git checkout", "when": 1234 }] Another tool wants to add a setting/element/etc, and simply creates a new config object with only the change in question included and APPENDS the existing config. [{"cmd": "git checkout", "when": 1234 }][{"cmd": "vagrant up", "when": 4567},{"comment": "Comments, notes, etc are kept even if they don't validate to the recognized configuration options."}] A configuration validator / etc loads the config and merges these in state-machine order, over-writing existing values with the latest ones from the end of the stream of objects and then determining if the result is a valid configuration. (Maybe file references fail to resolve / don't open / there's some combination of settings that's not supported...) [ {"cmd": "git checkout", "when": 1234 }, {"cmd": "vagrant up", "when": 4567}, {"comment": "... actually kept as above"} ]
pjc50 7y ago
I'm tempted to suggest CSV.
orf 7y ago
Obligatory "thanks for fish shell". Try just using line-delimited JSON objects (http://jsonlines.org/ http://jsonlines.org/). It ticks all of your boxes, especially 3: "jq -s '.cmd' fish_history | histogram". Neither YAML or Protobufs are quite as easy as that. All in all it's ridiculously simple, easy to parse in a variety of languages and each row is a single line that's simple to iteratively parse without loading the whole thing into memory.
- keithnz 7y ago
  this seems like it makes json useful for logging, but not too useful as configuration. For instance, it doesn't support commenting, and it seems like every line needs to have all its children compressed onto one line?
  - orf 7y ago
    Indeed, it's not suitable for configuration. But we are talking about logging shell history, not configuring it.
breck 7y ago
Disclaimer: I work on Tree Notation. (https://github.com/treenotation/jtree https://github.com/treenotation/jtree) Here's a proposal: use a Tree Language. I created a demo for you called "Fished": https://github.com/breck7/fished https://github.com/breck7/fished. Took me just a few minutes but already get type check, autocomplete, syntax highlighting, and more. Tree Notation is early, and there will be kinks until the community is bigger, but I think it may be useful for you. http://treenotation.org/designer/#grammar%0A%20fishedNode%0A%20%20description%20A%20text%20serialization%20format%20for%20Fish%20Shell's%20history%20file.%0A%20%20root%0A%20%20inScope%20commandNode%0A%20anyCell%0A%20%20highlightScope%20string%0A%20intCell%0A%20%20highlightScope%20constant.numeric.integer%0A%20keywordCell%0A%20%20highlightScope%20keyword%0A%20abstractKeywordNode%0A%20%20firstCellType%20keywordCell%0A%20%20abstract%0A%20commandNode%0A%20%20extends%20abstractKeywordNode%0A%20%20inScope%20whenNode%20pwdNode%20pathsNode%0A%20%20cells%20anyCell%20anyCell%20anyCell%0A%20whenNode%0A%20%20extends%20abstractKeywordNode%0A%20%20cells%20intCell%0A%20pwdNode%0A%20%20extends%20abstractKeywordNode%0A%20%20cells%20anyCell%0A%20pathsNode%0A%20%20extends%20abstractKeywordNode%0A%20%20cells%20anyCell%0Asample%0A%20command%20git%20checkout%20file.txt%0A%20%20when%201565133286%0A%20%20pwd%20%2Fhome%2Fme%2Fdir%2F%0A%20%20paths%20file.txt http://treenotation.org/designer/#grammar%0A%20fishedNode%0A...
- dfee 7y ago
  Wow. This looks really cool. Is there a sort of design defense on how this was designed (tree notation)?
  - breck 7y ago
    Not sure if I’m familiar with the term “design defense”. Can you explain? Stumbled into the idea. Basically just brute forced it. Tried thousands of things, built a huge database of languages, and tried to keep it simple.
    - dfee 7y ago
      I take the idea of “design defense” from Pyramid (Python Web Framework) [0], and have incorporated it into documentation on my projects. Basically, it’s a narrative discussion of how this solution came to be, the trade offs involved, and perhaps its relationships with prior art. [0] https://docs.pylonsproject.org/projects/pyramid/en/1.10-branch/designdefense.html https://docs.pylonsproject.org/projects/pyramid/en/1.10-bran...
kevin_thibedeau 7y ago
Tcl with control structure commands disabled and infix assignment for convenience. Jim Tcl is a lightweight implementation if the main line isn't workable.
theli0nheart 7y ago
TOML?
- pitaj 7y ago
  Can't recommend TOML enough. I use it for everything. Super simple and easy to edit. It fulfills all of the requirements. There are several available C++ TOML parsers, including one from Boost.
  - jakear 7y ago
    Do you ever have the limitation that arrays must be of a single type come up as an issue?
    - pitaj 7y ago
      I haven't run into that. I think in the contexts for which TOML was meant, it isn't that big of an issue though. Also, I'm not entirely sure how many parsers enforce that restriction.
- nine_k 7y ago
  TOML is great when your data are mostly flat.
  - meowface 7y ago
    It's great at representing nested data in a flat way, as well.
- ziotom78 7y ago
  TOML would be great, if not for an annoying obscure detail in the specification that makes it hard to use for my typical use cases (scientific computation) [1]. Moreover, I find quite unintuitive how you are supposed to specify array of tables [2]: this kind of is much easier in JSON (which is the format I am currently using, although it is far from perfect). [1] https://github.com/toml-lang/toml/issues/356 https://github.com/toml-lang/toml/issues/356 [2] https://github.com/toml-lang/toml#user-content-array-of-tables https://github.com/toml-lang/toml#user-content-array-of-tabl...
Mathnerd314 7y ago
S-Expressions are quite simple, there are some parsers floating around in well-known projects, although I'm not sure they're SAX-style: https://leon.bottou.org/projects/minilisp https://leon.bottou.org/projects/minilisp I also wonder if you need a text format, or if SQLite or systemd's journal API would work.
bhawks 7y ago
I love proto, but the textformat was an after thought. The binary format is rigorously defined, portable, extensible and optimized. The text format was reverse engineered from the c++ implementation after the fact when folks found textproto useful. Unfortunately there are discrepancies between languages around the corner cases of the textformat and that's the sad world we live in. Avoid letting textproto be part of your user exposed interface.

akx 7y ago

How about JSONL (JSON Lines)? http://jsonlines.org/ http://jsonlines.org/ Ps. Thanks for (all the) fish, it's my daily driver shell and keeps me that much more sane c.f. the alternatives.

j0057 7y ago
That's really close to [RFC 7464](https://tools.ietf.org/html/rfc7464 https://tools.ietf.org/html/rfc7464), JSON Text Sequences. It uses U+001E RECORD SEPARATOR. The `jq` tool supports those if you pass a flag.
- edraferi 7y ago
  So just add a record separator at the end of each line?
- meowface 7y ago
  jq can also handle newline-separated JSON objects, with the -s flag.

mjevans 7y ago

(suggestion) Drop the 4th requirement. Having to close contexts is a VERY good 'sanity check' to see if something is malformed or not. If appending is necessary make the parser handle multiple copies of the namespace and merge them upon output. Unknown keys and sections should also always be copied from input to output (this is how you embed comments).

ridiculous_fish 7y ago
I'm interested, can you explain more about the merging idea? To clarify the requirement, history could be a JSON array of objects: [ {"cmd": "git checkout", "when": 1234 }, {"cmd": "vagrant up", "when": 4567 } ] To append an entry to this file and keep it valid, one must locate the closing square bracket and overwrite it. That work is what I hope to avoid.
- ainar-g 7y ago
  Why not use a “stream” of objects? {"cmd": "git checkout", "when": 1234} {"cmd": "vagrant up", "when": 4567} Not sure about other languages and libraries, but Go supports this out of the box[1]. And while we're at it, why not CSV? That can be processed with awk. #cmd,when "git checkout","1234" "vagrant up","4567" [1]: https://play.golang.org/p/sTN9z4Kv3DB https://play.golang.org/p/sTN9z4Kv3DB
  - ridiculous_fish 7y ago
    CSV is fine for simple cases but has issues with versioning (adding new/optional fields) and nested data like arrays. The object stream idea is apparently supported widely and seems pretty strong. Thanks for the suggestion!
    - zenexer 7y ago
      I use JSON streams a lot with command line tools. Keep in mind that it’s limited in that each object must consume a single line. This allows you to recover from syntax errors in a single entry; each line is a fresh start. Have you considered SQLite? I know it’s not a friendly text format, but it alleviates a lot of the issues with the “append to a text file” approach, such as concurrency. It’s great for this sort of thing.
- mjevans 7y ago
  Loosely running with your example: A file already exists with this content: [{"cmd": "git checkout", "when": 1234 }] Another tool wants to add a setting/element/etc, and simply creates a new config object with only the change in question included and APPENDS the existing config. [{"cmd": "git checkout", "when": 1234 }][{"cmd": "vagrant up", "when": 4567},{"comment": "Comments, notes, etc are kept even if they don't validate to the recognized configuration options."}] A configuration validator / etc loads the config and merges these in state-machine order, over-writing existing values with the latest ones from the end of the stream of objects and then determining if the result is a valid configuration. (Maybe file references fail to resolve / don't open / there's some combination of settings that's not supported...) [ {"cmd": "git checkout", "when": 1234 }, {"cmd": "vagrant up", "when": 4567}, {"comment": "... actually kept as above"} ]

pjc50 7y ago

I'm tempted to suggest CSV.

orf 7y ago

Obligatory "thanks for fish shell". Try just using line-delimited JSON objects (http://jsonlines.org/ http://jsonlines.org/). It ticks all of your boxes, especially 3: "jq -s '.cmd' fish_history | histogram". Neither YAML or Protobufs are quite as easy as that. All in all it's ridiculously simple, easy to parse in a variety of languages and each row is a single line that's simple to iteratively parse without loading the whole thing into memory.

keithnz 7y ago
this seems like it makes json useful for logging, but not too useful as configuration. For instance, it doesn't support commenting, and it seems like every line needs to have all its children compressed onto one line?
- orf 7y ago
  Indeed, it's not suitable for configuration. But we are talking about logging shell history, not configuring it.

breck 7y ago

Disclaimer: I work on Tree Notation. (https://github.com/treenotation/jtree https://github.com/treenotation/jtree) Here's a proposal: use a Tree Language. I created a demo for you called "Fished": https://github.com/breck7/fished https://github.com/breck7/fished. Took me just a few minutes but already get type check, autocomplete, syntax highlighting, and more. Tree Notation is early, and there will be kinks until the community is bigger, but I think it may be useful for you. http://treenotation.org/designer/#grammar%0A%20fishedNode%0A%20%20description%20A%20text%20serialization%20format%20for%20Fish%20Shell's%20history%20file.%0A%20%20root%0A%20%20inScope%20commandNode%0A%20anyCell%0A%20%20highlightScope%20string%0A%20intCell%0A%20%20highlightScope%20constant.numeric.integer%0A%20keywordCell%0A%20%20highlightScope%20keyword%0A%20abstractKeywordNode%0A%20%20firstCellType%20keywordCell%0A%20%20abstract%0A%20commandNode%0A%20%20extends%20abstractKeywordNode%0A%20%20inScope%20whenNode%20pwdNode%20pathsNode%0A%20%20cells%20anyCell%20anyCell%20anyCell%0A%20whenNode%0A%20%20extends%20abstractKeywordNode%0A%20%20cells%20intCell%0A%20pwdNode%0A%20%20extends%20abstractKeywordNode%0A%20%20cells%20anyCell%0A%20pathsNode%0A%20%20extends%20abstractKeywordNode%0A%20%20cells%20anyCell%0Asample%0A%20command%20git%20checkout%20file.txt%0A%20%20when%201565133286%0A%20%20pwd%20%2Fhome%2Fme%2Fdir%2F%0A%20%20paths%20file.txt http://treenotation.org/designer/#grammar%0A%20fishedNode%0A...

dfee 7y ago
Wow. This looks really cool. Is there a sort of design defense on how this was designed (tree notation)?
- breck 7y ago
  Not sure if I’m familiar with the term “design defense”. Can you explain? Stumbled into the idea. Basically just brute forced it. Tried thousands of things, built a huge database of languages, and tried to keep it simple.
  - dfee 7y ago
    I take the idea of “design defense” from Pyramid (Python Web Framework) [0], and have incorporated it into documentation on my projects. Basically, it’s a narrative discussion of how this solution came to be, the trade offs involved, and perhaps its relationships with prior art. [0] https://docs.pylonsproject.org/projects/pyramid/en/1.10-branch/designdefense.html https://docs.pylonsproject.org/projects/pyramid/en/1.10-bran...

kevin_thibedeau 7y ago

Tcl with control structure commands disabled and infix assignment for convenience. Jim Tcl is a lightweight implementation if the main line isn't workable.

theli0nheart 7y ago

TOML?

pitaj 7y ago
Can't recommend TOML enough. I use it for everything. Super simple and easy to edit. It fulfills all of the requirements. There are several available C++ TOML parsers, including one from Boost.
- jakear 7y ago
  Do you ever have the limitation that arrays must be of a single type come up as an issue?
  - pitaj 7y ago
    I haven't run into that. I think in the contexts for which TOML was meant, it isn't that big of an issue though. Also, I'm not entirely sure how many parsers enforce that restriction.
nine_k 7y ago
TOML is great when your data are mostly flat.
- meowface 7y ago
  It's great at representing nested data in a flat way, as well.
ziotom78 7y ago
TOML would be great, if not for an annoying obscure detail in the specification that makes it hard to use for my typical use cases (scientific computation) [1]. Moreover, I find quite unintuitive how you are supposed to specify array of tables [2]: this kind of is much easier in JSON (which is the format I am currently using, although it is far from perfect). [1] https://github.com/toml-lang/toml/issues/356 https://github.com/toml-lang/toml/issues/356 [2] https://github.com/toml-lang/toml#user-content-array-of-tables https://github.com/toml-lang/toml#user-content-array-of-tabl...

Mathnerd314 7y ago

S-Expressions are quite simple, there are some parsers floating around in well-known projects, although I'm not sure they're SAX-style: https://leon.bottou.org/projects/minilisp https://leon.bottou.org/projects/minilisp I also wonder if you need a text format, or if SQLite or systemd's journal API would work.

bhawks 7y ago

I love proto, but the textformat was an after thought. The binary format is rigorously defined, portable, extensible and optimized. The text format was reverse engineered from the c++ implementation after the fact when folks found textproto useful. Unfortunately there are discrepancies between languages around the corner cases of the textformat and that's the sad world we live in. Avoid letting textproto be part of your user exposed interface.