6 ms·

Show HN: Sol – A de-minifier for shell programs

147 points by noperator 2y ago

I've built a tool called sol (like "soul") that helps you inspect and format complex shell one-liners. Features: - Choose which transformations you want (break on pipe, args, redirect, whatever) - "Peeks" into stringified commands (think xargs, parallel) and formats those, too - Auto-breaks at a given width (e.g., 80 characters) - Shows you non-standard aliases, functions, files, etc. that you might not have in your shell environment - Breaks up long jq lines with jqfmt because—let's be honest—they're getting out of hand As a security researcher and tool developer, I often encounter (or create) long pipelined Bash commands. While quick and powerful, they can be a nightmare to read or debug. I created sol to make it easier to understand and share these commands with others.

kmarc 2y ago
I usually do this by hand. Good to see a tool for it :-) Feature request, which I would love to have in all my automation scripts: Replace short flags with the long switches. Short flags are great when typing in a terminal but I don't want to figure out 2 years from now what the obscurecommand -v -f -n does, and I have to assume that it's NOT --version --file --dry-run, but --verbose, --force, and --dont-ask-before-deleting-everything I try to use long options in my script, therefore (especially in a team, where not everyone is familiar with every single command)
- notpushkin 2y ago
  It would be a great rule for shellcheck, by the way. Line 6: curl -fsSL "${url}" ^-- SC8537 (warning): use long options instead (`--fail`, `--silent`, `--show-error`, `--location`).
  - yjftsjthsd-h 2y ago
    I would want it opt-in, because I use shellcheck on scripts that will be run on busybox or *BSD where there aren't long options
    - notpushkin 2y ago
      Of course.
      - yjftsjthsd-h 2y ago
        Oh, I didn't realize shellcheck already had optional checks (see `shellcheck --list-optional` for a list), so that was not obvious to me initially. Then yes, that'd be a good thing to have available.
- jakub_g 2y ago
  When I saw "deminifier for shell commands" in title I had exactly the same in mind.
snatchpiesinger 2y ago
Cool! My personal preference is Knuth-style line-breaks on binary operators and pipes, which means breaking before the operator/pipe symbol. foo -a -b \ | bar -c -d -e \ | baz -e -f instead of foo -a -b | \ bar -c -d -e | \ baz -e -f This doesn't seem to be an option, but could be easy to implement.
- basemi 2y ago
  `shfmt` formats multi-line pipes into: foo -a -b | bar -c -d -e | baz -e -f which it's not that bad
- ramses0 2y ago
  Next level: foo -a -b \ | bar -c -d -e \ | baz -e -f \ && echo "DONE." # && /bin/true ...means you can safely (arbitrarily) add more intermediate commands to the pipeline w/o having to worry about modifying the trailing slash (eg: `yyp` won't cause a syntax error).
  - pxc 2y ago
    I do it this way but I indent the rest of the pipeline (like one typically would in a functional language with a pipeline operator or thread macro, or in an OOP language with method chaining via `.`): foo --long-something \ | bar --long-other and if the lines in subsequent commands in the pipeline start to get long, I also indent their arguments: foo --long-flag \ | bar \ --long-other-flag \ --long-option a \ --long-option b \ | baz \ --long-another-flag \ --long-flag-again \ > /path/to/some/file I really like to use && on its own line like that. One of my favorite things about Fish is how it turns && and || from special syntax into commands, which it calls combiners, so you could write: foo \ | bar \ | baz and echo done I use this often for conditions whose bodies would only be one line, avoiding the nesting and indentation: test -n "$SOMETHING" or set -x SOMETHING some-default command $SOMETHING In Bash, I usually use parameter substitution for this, but in other situations (other than setting default values for vars) I throw a backslash at the end of a line, indent and use && or ||, imitating the Fish style. One of my favorite patterns for readability is to use indented, long-form pipelines like this to set variables. They work fine inside subshells, but for uniformity and clarity I prefer to do shopt -s lastpipe foo \ | bar \ | baz \ | read SOMEVAR I really enjoy 'maximizing' pipelines like this because it makes it possible to use long pipelines everywhere without making your program terse and mysterious, or creating unwieldy long lines. If you do this, you end up with a script is mostly 'flat' (having very little nested control flow, largely avoiding loops), has very few variable assignments, and predictably locates the variable assignments it does have at the ends of pipelines. Each pipeline is a singular train of thought requiring you to consider context and state only at the very beginning and very end, and you can typically likewise think of all the intermediate steps/commands in functional terms. I tend to write all of my shell scripts this way, including the ones I write interactively at my prompt. One really cool thing about shell languages is that unlike in 'real' programming languages, loops are actually composable! So you can freely mix ad-hoc/imperative and pipeline-centric styles like this (example is Fish): find -name whatever -exec basename '{}' \; | while read -l data set -l temp (some-series $data) set -l another (some-command $temp) blahdiblah --something $temp --other $another end \ | bar \ | baz \ > some-list.txt (I typically use 2 spaces to indent when writing Bash scripts, but Fish defaults to 4 in the prompt, which it also automatically indents for you. I'm not sure if that's configurable but I haven't attempted to change it.) I tend to follow my guideline suggested earlier and do this only close to the very beginning or very end of a pipeline if that loop actually modifies the filesystem or non-local variables, but it's really nice to have that flexibility imo. (It's especially handy if you want to embed some testing or conditional logic into the pipeline to filter results in a complex way.)
ComputerGuru 2y ago
This is really cool; for a second I thought I could use it to stop manually maintaining both the minified and full-text versions of my “shell prefix” that makes it possible to run rust source code directly as if it were a shell script [0] where I’ve accidentally diverged between the two in the past, but then I revisited it and saw that the real value was in the comments and explanations more than just placing output in variables and breaking up command pipelines across shell lines. But the opposite might work, does anyone have a good minifier they could recommend (preferably one that does more than just whitespace mangling, eg also renames variables, chains executions, etc) that doesn’t introduce bash-isms into the resulting script? [0]: https://neosmart.net/blog/self-compiling-rust-code/ https://neosmart.net/blog/self-compiling-rust-code/
pxc 2y ago
This looks really handy! I should add this to the environment for some of my shell-centric projects at work.

147 points by noperator 2y ago

kmarc 2y ago

I usually do this by hand. Good to see a tool for it :-) Feature request, which I would love to have in all my automation scripts: Replace short flags with the long switches. Short flags are great when typing in a terminal but I don't want to figure out 2 years from now what the obscurecommand -v -f -n does, and I have to assume that it's NOT --version --file --dry-run, but --verbose, --force, and --dont-ask-before-deleting-everything I try to use long options in my script, therefore (especially in a team, where not everyone is familiar with every single command)

notpushkin 2y ago
It would be a great rule for shellcheck, by the way. Line 6: curl -fsSL "${url}" ^-- SC8537 (warning): use long options instead (`--fail`, `--silent`, `--show-error`, `--location`).
- yjftsjthsd-h 2y ago
  I would want it opt-in, because I use shellcheck on scripts that will be run on busybox or *BSD where there aren't long options
  - notpushkin 2y ago
    Of course.
    - yjftsjthsd-h 2y ago
      Oh, I didn't realize shellcheck already had optional checks (see `shellcheck --list-optional` for a list), so that was not obvious to me initially. Then yes, that'd be a good thing to have available.
jakub_g 2y ago
When I saw "deminifier for shell commands" in title I had exactly the same in mind.

snatchpiesinger 2y ago

Cool! My personal preference is Knuth-style line-breaks on binary operators and pipes, which means breaking before the operator/pipe symbol. foo -a -b \ | bar -c -d -e \ | baz -e -f instead of foo -a -b | \ bar -c -d -e | \ baz -e -f This doesn't seem to be an option, but could be easy to implement.

basemi 2y ago
`shfmt` formats multi-line pipes into: foo -a -b | bar -c -d -e | baz -e -f which it's not that bad
ramses0 2y ago
Next level: foo -a -b \ | bar -c -d -e \ | baz -e -f \ && echo "DONE." # && /bin/true ...means you can safely (arbitrarily) add more intermediate commands to the pipeline w/o having to worry about modifying the trailing slash (eg: `yyp` won't cause a syntax error).
- pxc 2y ago
  I do it this way but I indent the rest of the pipeline (like one typically would in a functional language with a pipeline operator or thread macro, or in an OOP language with method chaining via `.`): foo --long-something \ | bar --long-other and if the lines in subsequent commands in the pipeline start to get long, I also indent their arguments: foo --long-flag \ | bar \ --long-other-flag \ --long-option a \ --long-option b \ | baz \ --long-another-flag \ --long-flag-again \ > /path/to/some/file I really like to use && on its own line like that. One of my favorite things about Fish is how it turns && and || from special syntax into commands, which it calls combiners, so you could write: foo \ | bar \ | baz and echo done I use this often for conditions whose bodies would only be one line, avoiding the nesting and indentation: test -n "$SOMETHING" or set -x SOMETHING some-default command $SOMETHING In Bash, I usually use parameter substitution for this, but in other situations (other than setting default values for vars) I throw a backslash at the end of a line, indent and use && or ||, imitating the Fish style. One of my favorite patterns for readability is to use indented, long-form pipelines like this to set variables. They work fine inside subshells, but for uniformity and clarity I prefer to do shopt -s lastpipe foo \ | bar \ | baz \ | read SOMEVAR I really enjoy 'maximizing' pipelines like this because it makes it possible to use long pipelines everywhere without making your program terse and mysterious, or creating unwieldy long lines. If you do this, you end up with a script is mostly 'flat' (having very little nested control flow, largely avoiding loops), has very few variable assignments, and predictably locates the variable assignments it does have at the ends of pipelines. Each pipeline is a singular train of thought requiring you to consider context and state only at the very beginning and very end, and you can typically likewise think of all the intermediate steps/commands in functional terms. I tend to write all of my shell scripts this way, including the ones I write interactively at my prompt. One really cool thing about shell languages is that unlike in 'real' programming languages, loops are actually composable! So you can freely mix ad-hoc/imperative and pipeline-centric styles like this (example is Fish): find -name whatever -exec basename '{}' \; | while read -l data set -l temp (some-series $data) set -l another (some-command $temp) blahdiblah --something $temp --other $another end \ | bar \ | baz \ > some-list.txt (I typically use 2 spaces to indent when writing Bash scripts, but Fish defaults to 4 in the prompt, which it also automatically indents for you. I'm not sure if that's configurable but I haven't attempted to change it.) I tend to follow my guideline suggested earlier and do this only close to the very beginning or very end of a pipeline if that loop actually modifies the filesystem or non-local variables, but it's really nice to have that flexibility imo. (It's especially handy if you want to embed some testing or conditional logic into the pipeline to filter results in a complex way.)

ComputerGuru 2y ago

This is really cool; for a second I thought I could use it to stop manually maintaining both the minified and full-text versions of my “shell prefix” that makes it possible to run rust source code directly as if it were a shell script [0] where I’ve accidentally diverged between the two in the past, but then I revisited it and saw that the real value was in the comments and explanations more than just placing output in variables and breaking up command pipelines across shell lines. But the opposite might work, does anyone have a good minifier they could recommend (preferably one that does more than just whitespace mangling, eg also renames variables, chains executions, etc) that doesn’t introduce bash-isms into the resulting script? [0]: https://neosmart.net/blog/self-compiling-rust-code/ https://neosmart.net/blog/self-compiling-rust-code/

pxc 2y ago

This looks really handy! I should add this to the environment for some of my shell-centric projects at work.