3 ms·

I'm interested, can you explain more about the merging idea? To clarify the requirement, history could be a JSON array of objects: [ {"cmd": "git

I'm interested, can you explain more about the merging idea? To clarify the requirement, history could be a JSON array of objects: [ {"cmd": "git checkout", "when": 1234 }, {"cmd": "vagrant up", "when": 4567 } ] To append an entry to this file and keep it valid, one must locate the closing square bracket and overwrite it. That work is what I hope to avoid.

ainar-g 7y ago
Why not use a “stream” of objects? {"cmd": "git checkout", "when": 1234} {"cmd": "vagrant up", "when": 4567} Not sure about other languages and libraries, but Go supports this out of the box[1]. And while we're at it, why not CSV? That can be processed with awk. #cmd,when "git checkout","1234" "vagrant up","4567" [1]: https://play.golang.org/p/sTN9z4Kv3DB https://play.golang.org/p/sTN9z4Kv3DB
- ridiculous_fish 7y ago
  CSV is fine for simple cases but has issues with versioning (adding new/optional fields) and nested data like arrays. The object stream idea is apparently supported widely and seems pretty strong. Thanks for the suggestion!
  - zenexer 7y ago
    I use JSON streams a lot with command line tools. Keep in mind that it’s limited in that each object must consume a single line. This allows you to recover from syntax errors in a single entry; each line is a fresh start. Have you considered SQLite? I know it’s not a friendly text format, but it alleviates a lot of the issues with the “append to a text file” approach, such as concurrency. It’s great for this sort of thing.
mjevans 7y ago
Loosely running with your example: A file already exists with this content: [{"cmd": "git checkout", "when": 1234 }] Another tool wants to add a setting/element/etc, and simply creates a new config object with only the change in question included and APPENDS the existing config. [{"cmd": "git checkout", "when": 1234 }][{"cmd": "vagrant up", "when": 4567},{"comment": "Comments, notes, etc are kept even if they don't validate to the recognized configuration options."}] A configuration validator / etc loads the config and merges these in state-machine order, over-writing existing values with the latest ones from the end of the stream of objects and then determining if the result is a valid configuration. (Maybe file references fail to resolve / don't open / there's some combination of settings that's not supported...) [ {"cmd": "git checkout", "when": 1234 }, {"cmd": "vagrant up", "when": 4567}, {"comment": "... actually kept as above"} ]

ainar-g 7y ago

Why not use a “stream” of objects? {"cmd": "git checkout", "when": 1234} {"cmd": "vagrant up", "when": 4567} Not sure about other languages and libraries, but Go supports this out of the box[1]. And while we're at it, why not CSV? That can be processed with awk. #cmd,when "git checkout","1234" "vagrant up","4567" [1]: https://play.golang.org/p/sTN9z4Kv3DB https://play.golang.org/p/sTN9z4Kv3DB

ridiculous_fish 7y ago
CSV is fine for simple cases but has issues with versioning (adding new/optional fields) and nested data like arrays. The object stream idea is apparently supported widely and seems pretty strong. Thanks for the suggestion!
- zenexer 7y ago
  I use JSON streams a lot with command line tools. Keep in mind that it’s limited in that each object must consume a single line. This allows you to recover from syntax errors in a single entry; each line is a fresh start. Have you considered SQLite? I know it’s not a friendly text format, but it alleviates a lot of the issues with the “append to a text file” approach, such as concurrency. It’s great for this sort of thing.

mjevans 7y ago

Loosely running with your example: A file already exists with this content: [{"cmd": "git checkout", "when": 1234 }] Another tool wants to add a setting/element/etc, and simply creates a new config object with only the change in question included and APPENDS the existing config. [{"cmd": "git checkout", "when": 1234 }][{"cmd": "vagrant up", "when": 4567},{"comment": "Comments, notes, etc are kept even if they don't validate to the recognized configuration options."}] A configuration validator / etc loads the config and merges these in state-machine order, over-writing existing values with the latest ones from the end of the stream of objects and then determining if the result is a valid configuration. (Maybe file references fail to resolve / don't open / there's some combination of settings that's not supported...) [ {"cmd": "git checkout", "when": 1234 }, {"cmd": "vagrant up", "when": 4567}, {"comment": "... actually kept as above"} ]