5 ms·

>However, if you instead organize all your data in a format that's sympathetic to line-oriented processing on stdin-stdout, then shell will work with you instea

>However, if you instead organize all your data in a format that's sympathetic to line-oriented processing on stdin-stdout, then shell will work with you instead of against. Not even that is necessary. Just use structured data formats like json. If you are consuming some API that is not json but still structured, use `rq` to convert it to json. Then use `jq` to slice and dice through the data. dmenu + fzf + jq + curl is my bread and butter in shell scripts. However, I still haven't managed to find a way to do a bunch of tasks concurrently. No, xargs and parallel don't cut it. Just give me an opinionated way to do this that is easily inspectable, loggable and debuggable. Currenly I hack together functions in a `((job_i++ < max_jobs)) || wait -n` spaghetti.

kqr 4y ago
I think this comment points to an even deeper insight: shell is a crappy programming language but with amazing extensibility. I would argue that once you pull in jq, you're no longer writing in "shell", you're writing in jq, which is a separate and different language. But that's precisely the point! Look at how effortless it is to (literally) shell out to a slew of other languages from shell. The power of shell isn't in the scripting language itself, it's in how fluidly it lets you embed snippets of tr, sed, awk, jq, and whatever else you need. And, critically, these languages callable from shell were not all there when shell was designed. The extension interface of spawning processes and communicating with arguments and pipes is just that powerful. That's where shell shines.
- chasil 4y ago
  The shell is an ambiguous language that cannot be directly implemented with an LR parser. Perhaps some of the power emerges from that ambiguity, but it is quite difficult to implement. This presentation sums up the woes of an implementor: https://archive.fosdem.org/2018/schedule/event/code_parsing_posix_s_hell/ https://archive.fosdem.org/2018/schedule/event/code_parsing_...
synergy20 4y ago
new to 'rq', it's not in active development, any other alternatives? it seems doing a lot other than convert structured data to json.
- ducktective 4y ago
  Not sure what it is doing more...I'm referring to this rq: https://github.com/dflemstr/rq#format-support-status https://github.com/dflemstr/rq#format-support-status It converts to/from the listed formats. There is also `jc` (written in Python) with the added benefit that it converts output of many common unix utilities to json. So you would not need to parse `ip` for example. https://github.com/kellyjonbrazil/jc#parsers https://github.com/kellyjonbrazil/jc#parsers
  - charsi 4y ago
    Also look at `yq` - https://github.com/mikefarah/yq https://github.com/mikefarah/yq This is a wrapper to jq that also supports yaml and other file formats.
maximus-decimus 4y ago
How do you use dmenu for your shell script? to launch it? to prompt the user for input while it's running? Do you have an example of a script you wrote?
- ducktective 4y ago
  Yes, for creating ad-hoc mini-UIs so the user can select an option. Same with fzf, but it's terminal-bound (rather than X-bound). The scripts are similar to this one: https://github.com/debxp/dmenu-scripts/blob/master/dmenu-kill-process https://github.com/debxp/dmenu-scripts/blob/master/dmenu-kil...
  - maximus-decimus 4y ago
    Thanks, I will definitively use that kill one.
- sargstuff 4y ago
  WASM gawk with html as user input/output more flexible.
rustyminnow 4y ago
Do you have examples of concurrent use-cases that xargs and parallel don't satisfy? I discovered parallel recently and was blown away by how much it improves things. I've only really used it in basic scenarios so far, just wondering where its limitations are.
- ducktective 4y ago
  running a bash function with its own private variables in parallel. without having to export it.
pcthrowaway 4y ago
Can you give an example of how you'd use rq in this pipeline? I'm not finding any good examples
- ducktective 4y ago
  curl -s "give.me.some/yaml" | rq --input-yaml --output-json | jq '.my.selected[1].field'

kqr 4y ago

I think this comment points to an even deeper insight: shell is a crappy programming language but with amazing extensibility. I would argue that once you pull in jq, you're no longer writing in "shell", you're writing in jq, which is a separate and different language. But that's precisely the point! Look at how effortless it is to (literally) shell out to a slew of other languages from shell. The power of shell isn't in the scripting language itself, it's in how fluidly it lets you embed snippets of tr, sed, awk, jq, and whatever else you need. And, critically, these languages callable from shell were not all there when shell was designed. The extension interface of spawning processes and communicating with arguments and pipes is just that powerful. That's where shell shines.

chasil 4y ago
The shell is an ambiguous language that cannot be directly implemented with an LR parser. Perhaps some of the power emerges from that ambiguity, but it is quite difficult to implement. This presentation sums up the woes of an implementor: https://archive.fosdem.org/2018/schedule/event/code_parsing_posix_s_hell/ https://archive.fosdem.org/2018/schedule/event/code_parsing_...

synergy20 4y ago

new to 'rq', it's not in active development, any other alternatives? it seems doing a lot other than convert structured data to json.

ducktective 4y ago
Not sure what it is doing more...I'm referring to this rq: https://github.com/dflemstr/rq#format-support-status https://github.com/dflemstr/rq#format-support-status It converts to/from the listed formats. There is also `jc` (written in Python) with the added benefit that it converts output of many common unix utilities to json. So you would not need to parse `ip` for example. https://github.com/kellyjonbrazil/jc#parsers https://github.com/kellyjonbrazil/jc#parsers
- charsi 4y ago
  Also look at `yq` - https://github.com/mikefarah/yq https://github.com/mikefarah/yq This is a wrapper to jq that also supports yaml and other file formats.

maximus-decimus 4y ago

How do you use dmenu for your shell script? to launch it? to prompt the user for input while it's running? Do you have an example of a script you wrote?

ducktective 4y ago
Yes, for creating ad-hoc mini-UIs so the user can select an option. Same with fzf, but it's terminal-bound (rather than X-bound). The scripts are similar to this one: https://github.com/debxp/dmenu-scripts/blob/master/dmenu-kill-process https://github.com/debxp/dmenu-scripts/blob/master/dmenu-kil...
- maximus-decimus 4y ago
  Thanks, I will definitively use that kill one.
sargstuff 4y ago
WASM gawk with html as user input/output more flexible.

rustyminnow 4y ago

Do you have examples of concurrent use-cases that xargs and parallel don't satisfy? I discovered parallel recently and was blown away by how much it improves things. I've only really used it in basic scenarios so far, just wondering where its limitations are.

ducktective 4y ago
running a bash function with its own private variables in parallel. without having to export it.

pcthrowaway 4y ago

Can you give an example of how you'd use rq in this pipeline? I'm not finding any good examples

ducktective 4y ago
curl -s "give.me.some/yaml" | rq --input-yaml --output-json | jq '.my.selected[1].field'