I like JSON a lot, it's my go to for ad-hoc data storage. I've found it extremely convenient because nearly all modern APIs can return json. Further, jq makes w

I like JSON a lot, it's my go to for ad-hoc data storage. I've found it extremely convenient because nearly all modern APIs can return json. Further, jq makes working with json extremely ergonomic. For certain tasks (personal use) I hand roll my own json document storage database. I make a bunch of rest API calls to collect data and store it in flat files. While doing this, I use JQ to store subsets of that information (the stuff I really care about) in smaller json blobs. I then write a script to aggregate all of the smaller blobs into a larger json array. That's almost no work and you already get a nice schema-less database. Write some commands (stored-procedures) to do any kind of filtering/modifications and you can immediately get whatever view you want of your data with one command. Write a wrapper script to identify documents by a field (primary key) and you can make data modifications in an ergonomic way. I run "modify-document <primary-key>" and it runs a tiny readline bash script prompting me for info which immediately modifies the corresponding database row. A work flow could be... 1. Make API requests (with gnu parallel) storing the response json and a smaller json blob. 2. Aggregate the small blobs into an array. 3. Filter blobs for any that are missing information. 4. Manually update blobs missing information with readline script. 5. Filter for blobs that need processing. 6. Process blobs, use readline script to mark them as processed. 7. Continue until all blobs are processed. This is the kind of thing I would use excel to do in the past. Hopefully I never make that mistake again.