3 ms·

Track changes to CLI tools by recording their help output

10 points by appwiz 5y ago

simonw 5y ago
My AWS tracker has caught a ton of a activity - they really do ship new feature in that tool every day of the week: https://github.com/simonw/help-scraper/commits/main/aws https://github.com/simonw/help-scraper/commits/main/aws My scraper against the GitHub GraphQL schema catches some interesting details too: https://github.com/simonw/help-scraper/commits/main/github https://github.com/simonw/help-scraper/commits/main/github
- wizwit999 5y ago
  For AWS, you don't need to hit the cli in this circuitous manner. The clis are all generated from their API models, which are published publicly in json format.
  - simonw 5y ago
    Right - but actually comprehending the differences in different versions of that JSON isn't at all easy. The diff of a CLI help output is a lot more readable than the diff of a JSON file.
bhupesh 5y ago
that's cool, but shouldn't the best way to solve this would be to ship the changelog with the CLI itself? Something like `cli --changelog`? I mean I don't understand why its not a general practice with CLI authors
- jitl 5y ago
  When I built developer tools at Airbnb, we had the tool show the new things in the changelog the first time you ran it after updating, and we had a subcommand to spit out the whole changelog as you suggest. But I think the general issue is that writing good, useful changelogs is manual work — you shouldn’t show end user something like `git log —-oneline`, which usually ends up pretty noisy and useless no matter what kind of grooming you apply to it. We only put things in the changelog if they affected the workflow of ~all our users, and spamming markdown in the terminal would materially reduce our support burden. For something like AWS, I guess you could do a changelog per service command? A changelog for the whole thing would be too detailed and contain too much stuff I don’t care about, I’d think.
jitl 5y ago
Interesting idea, it’s like running an integration test for someone else’s tool. I did something a bit like this for a few internal tools I built at Airbnb. I built a docs system that indexed all the markdown files in every repo or project’s .infra/docs folder into a central documentation browser website. Then in each tool project, we had a CI job that would loop over all the commands and spit the —-help output in Markdown format into that folder. That way when someone added commands or changed a command’s args or help text, the searchable docs were automatically updated. We didn’t use this to generate a change log to show to users per se, but it did mean that you could run `git log .infra/docs` to see a log of every change to the tool’s interface.

10 points by appwiz 5y ago

My AWS tracker has caught a ton of a activity - they really do ship new feature in that tool every day of the week: https://github.com/simonw/help-scraper/commits/main/aws https://github.com/simonw/help-scraper/commits/main/aws My scraper against the GitHub GraphQL schema catches some interesting details too: https://github.com/simonw/help-scraper/commits/main/github https://github.com/simonw/help-scraper/commits/main/github

wizwit999 5y ago
For AWS, you don't need to hit the cli in this circuitous manner. The clis are all generated from their API models, which are published publicly in json format.
- simonw 5y ago
  Right - but actually comprehending the differences in different versions of that JSON isn't at all easy. The diff of a CLI help output is a lot more readable than the diff of a JSON file.

bhupesh 5y ago

that's cool, but shouldn't the best way to solve this would be to ship the changelog with the CLI itself? Something like `cli --changelog`? I mean I don't understand why its not a general practice with CLI authors

jitl 5y ago
When I built developer tools at Airbnb, we had the tool show the new things in the changelog the first time you ran it after updating, and we had a subcommand to spit out the whole changelog as you suggest. But I think the general issue is that writing good, useful changelogs is manual work — you shouldn’t show end user something like `git log —-oneline`, which usually ends up pretty noisy and useless no matter what kind of grooming you apply to it. We only put things in the changelog if they affected the workflow of ~all our users, and spamming markdown in the terminal would materially reduce our support burden. For something like AWS, I guess you could do a changelog per service command? A changelog for the whole thing would be too detailed and contain too much stuff I don’t care about, I’d think.

jitl 5y ago

Interesting idea, it’s like running an integration test for someone else’s tool. I did something a bit like this for a few internal tools I built at Airbnb. I built a docs system that indexed all the markdown files in every repo or project’s .infra/docs folder into a central documentation browser website. Then in each tool project, we had a CI job that would loop over all the commands and spit the —-help output in Markdown format into that folder. That way when someone added commands or changed a command’s args or help text, the searchable docs were automatically updated. We didn’t use this to generate a change log to show to users per se, but it did mean that you could run `git log .infra/docs` to see a log of every change to the tool’s interface.