4 ms·

This is the only statement I disagree with: > PowerShell gets a lot right with structured data. CLI programs should operate on text. If you want to parse and

This is the only statement I disagree with: > PowerShell gets a lot right with structured data. CLI programs should operate on text. If you want to parse and format it, do so, but the default output mode should be plain text, so that I can pipe it into grep or awk without a second thought. I am continuously irritated that the AWS CLI defaults to outputting in JSON. No one (I hope…) is using that tool in programs; that’s what boto3 and its ilk are for. But if humans are reading it, why default to something that they’re almost certainly going to be piping into jq if only for the formatting help?

tredre3 3mo ago
Powershell commands automatically format the data for you, you can pipe it to grep just fine (I do it all the time). It's just that you can also access it in a structured way if you need to. It's the best of both worlds. Linux tools that are starting to output raw JSON by default are indeed a nuisance, but how else can you achieve structured output if no standard shell supports it? It's a chicken and egg problem.
- em-bee 3mo ago
  and new shells are developing features to handle structured data like json: here is an elvish shell command that converts a freetube playlist from json into a list of urls grouped by author: for i (cat 'freetube-playlist-favorites.db' | from-json)["videos"] { mkdir -p $i['author'] print http://youtu.be/$i['videoId'] >> $i['author']/get } here is one to get a list of devices connected to my zerotier network curl -s -H "Authorization: token <redacted>" "https://api.zerotier.com/api/v1/network/<redacted>/member" | all (from-json) | order &key={|d| put $d[name]} | each { |device| var t = (printf "%.0f" (/ $device[lastSeen] 1000)) if (> 20000000 (- (date '+%s') $t)) { print (date -R --date='@'$t) $device[config][ipAssignments] $device[name] "\n" } } those are not scripts saved in a file. i run these directly on the commandline. ignore the elvish syntax, focus on the ease of accessing values from the json data. those are just two examples, though i recently discovered an ls replacement that optionally outputs json, that will be interesting to use.
  - fragmede 3mo ago
    Interesting. That's a lot to type on the fly, but I can posit some benefits.
  - jcgl 3mo ago
    Great example! How do you like using elvish? Even though I am a proponent of structured data and like PowerShell a lot (mentioned in a nearby comment of mine), I use fish as my regular shell. Big fan of fish's careful focus on user experience, but would be open to trying something structured.
    - blooalien 3mo ago
      I'ma huge fan of `fish` myself, but I've been watching `elvish` for a while now, and some of their scripting stuff really is awfully nice.
nixon_why69 3mo ago
I always hated powershell for the same reason, and then I read a really long retro by the guy who ran the project and it makes sense now. Basically Unix has a long tradition of "everything is a file" and a big ecosystem of coreutils that are based around text and windows.. didn't. You can't look at /dev or /etc and learn anything about the machine. They had a few generations of APIs and wanted to give admins and power users any shell at all instead of a GUI. So the shell is centered around making those APIs accessible, rather than piping grep and sed or whatever.
- digdugdirk 3mo ago
  Ooo... Any chance you remember where you read that? I'd love a reference to check it out. Sounds interesting!
  - rellem 3mo ago
    I'm pretty sure they're referring to this one: https://corecursive.com/building-powershell-with-jeffrey-snover/ https://corecursive.com/building-powershell-with-jeffrey-sno...
    - nixon_why69 3mo ago
      That's the one!
    - justinhj 3mo ago
      Thanks for sharing that. It's relevant but also a great story about how to struggle against giant corporation bureaucracy and ship against all odds
hombre_fatal 3mo ago
I'm not sure I agree. In the AWS case the tools talk to an API server, so sure, you can call the API server directly or use a wrapper that does, but what about all the other CLI apps that don't? The CLI program is the API. I built a CLI program that wraps luks + btrfs, and they only offer a `--json` output option for a few commands. I have to write an ad hoc parser for each command since raw text includes arbitrary formatting and presentation lipstick that the creator came up with. And I have to do extra work to avoid breaking changes at the parser level instead of the higher data level. If I had to pick between the two, json would at least solve the data representation part so that I can build on top of it. And it's trivial to go data (json) -> pretty print rather than pretty print -> data. I can see it being annoying if all you care about is using CLI programs by hand, but it seems like a mild upside compared to the downsides the second you want to consume it programmatically, even if it's with a chain of awk, cut, and tr.
zbentley 3mo ago
Slightly off topic, but it is worth being familiar with AWSCLI’s built in jmespath JSON transformation system. That saves on the need for jq in a lot of cases. Given that awscli bundles that, I generally forgive it for defaulting to JSON (especially given how deeply nested and interlinked many AWS API results are), but reasonable minds can differ. AWSCLI’s schizophrenic use (or not) of pagers for tiny results, on the other hand, is less defensible.
jcgl 3mo ago
I think you're mistaking text-with-structured data for structured data itself. Because unix shell is irrevocably text-oriented, kludging in something like JSON is basically the best that can be done when you start to want to do structured operations on structured data. (I'm sympathetic to your point about the AWS CLI tools doing JSON by default though--that just sounds like bad design.) Being text-oriented imposes drastic limits on composability. Because there is no structure, every element of a pipeline needs to do its own parsing of the input data. This leads to brittle pipelines where every element is tightly coupled to its input's textual representation. As an exercise, try to write a pipeline that sorts podman images by size without removing the column headers[0]: $ podman image ls --all REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/prom/prometheus latest 937690d77350 2 months ago 367 MB quay.io/keycloak/keycloak latest da9433c9fac3 2 months ago 466 MB registry.fedoraproject.org/fedora-toolbox 43 a32da54355ca 4 months ago 2.19 GB docker.io/powerdns/pdns-auth-49 latest 8c1385c9deed 4 months ago 208 MB docker.io/testcontainers/ryuk 0.13.0 b75bc7ce94c3 6 months ago 7.21 MB As far as I can tell, there is no way to do this in a manner that's even remotely composable. Your best bet is to basically do everything from within awk. Whatever the result would be, it certainly won't be pretty! Contrast that with what you can do in PowerShell. You can write a couple of standalone functions[0] that are readable and composable, resulting in this pipeline: podman image ls --all | Replace-SpacesWithTabs | ConvertFrom-Csv -Delimiter "`t" | Sort-Object -Property {Convert-HumanSizeToBytes -Size $_.size} -Descending [0] Repurposing this from a blog post I wrote: https://www.cgl.sh/blog/posts/sh.html#this-should-be-basic https://www.cgl.sh/blog/posts/sh.html#this-should-be-basic
- alloyed 3mo ago
  the point is well-taken, but i do want to show the bash version just for fun: podman image ls --all | sed 's/\s\s\+/\t/g' | tee >(head -n 1) >(tail -n +2 | sort -hrk 5) >/dev/null this is _still_ all text, and we're relying heavily on sort to do a bunch of internal parsing and be in agreement with podman about how sizes should be formatted. also, for "real world" work, i dunno if the tee trick here has any kind of order guarantees, just that it works fine in this case. I'd probably just end up dropping the header and living with worse output in reality
tialaramex 3mo ago
I thought I would like Powershell but it was another big disappointment for me. It's got a lot of the unpleasant clunkiness that something like the Bourne shell owes to decades of compatibility, but Microsoft doesn't have that excuse. Despite this, it's gratuitously incompatible with itself, I had code which worked fine, we upgraded Powershell oh, now that won't work, just fix all the scripts. Crazy. It clearly wants an NPM-style experience where you seamlessly incorporate other people's work, but then it doesn't really deliver this well and so you often end up manually copy-pasting. If Powershell was a one man project and this was their Beta I'd say it is promising. But it's a project from the Microsoft corporation for 20 years. Do better.
- torginus 3mo ago
  This is my opinion as well. Powershell and object shells are much better and the way to go imo. Also Powershell code is fast owing to running on the .NET VM. But the actual UX is just convoluted and horrible. For example, the person who decided that commands without a ; at the end of the line should just vomit their outputs to the enclosing function's clearly hasn't heard of the principle of least surprise. PS just does so many plain weird and unintuitive things, which is worsened by the fact that it looks like a boring old programming language, while clearly not being one.

tredre3 3mo ago

Powershell commands automatically format the data for you, you can pipe it to grep just fine (I do it all the time). It's just that you can also access it in a structured way if you need to. It's the best of both worlds. Linux tools that are starting to output raw JSON by default are indeed a nuisance, but how else can you achieve structured output if no standard shell supports it? It's a chicken and egg problem.

em-bee 3mo ago
and new shells are developing features to handle structured data like json: here is an elvish shell command that converts a freetube playlist from json into a list of urls grouped by author: for i (cat 'freetube-playlist-favorites.db' | from-json)["videos"] { mkdir -p $i['author'] print http://youtu.be/$i['videoId'] >> $i['author']/get } here is one to get a list of devices connected to my zerotier network curl -s -H "Authorization: token <redacted>" "https://api.zerotier.com/api/v1/network/<redacted>/member" | all (from-json) | order &key={|d| put $d[name]} | each { |device| var t = (printf "%.0f" (/ $device[lastSeen] 1000)) if (> 20000000 (- (date '+%s') $t)) { print (date -R --date='@'$t) $device[config][ipAssignments] $device[name] "\n" } } those are not scripts saved in a file. i run these directly on the commandline. ignore the elvish syntax, focus on the ease of accessing values from the json data. those are just two examples, though i recently discovered an ls replacement that optionally outputs json, that will be interesting to use.
- fragmede 3mo ago
  Interesting. That's a lot to type on the fly, but I can posit some benefits.
- jcgl 3mo ago
  Great example! How do you like using elvish? Even though I am a proponent of structured data and like PowerShell a lot (mentioned in a nearby comment of mine), I use fish as my regular shell. Big fan of fish's careful focus on user experience, but would be open to trying something structured.
  - blooalien 3mo ago
    I'ma huge fan of `fish` myself, but I've been watching `elvish` for a while now, and some of their scripting stuff really is awfully nice.

nixon_why69 3mo ago

I always hated powershell for the same reason, and then I read a really long retro by the guy who ran the project and it makes sense now. Basically Unix has a long tradition of "everything is a file" and a big ecosystem of coreutils that are based around text and windows.. didn't. You can't look at /dev or /etc and learn anything about the machine. They had a few generations of APIs and wanted to give admins and power users any shell at all instead of a GUI. So the shell is centered around making those APIs accessible, rather than piping grep and sed or whatever.

digdugdirk 3mo ago
Ooo... Any chance you remember where you read that? I'd love a reference to check it out. Sounds interesting!
- rellem 3mo ago
  I'm pretty sure they're referring to this one: https://corecursive.com/building-powershell-with-jeffrey-snover/ https://corecursive.com/building-powershell-with-jeffrey-sno...
  - nixon_why69 3mo ago
    That's the one!
  - justinhj 3mo ago
    Thanks for sharing that. It's relevant but also a great story about how to struggle against giant corporation bureaucracy and ship against all odds

hombre_fatal 3mo ago

I'm not sure I agree. In the AWS case the tools talk to an API server, so sure, you can call the API server directly or use a wrapper that does, but what about all the other CLI apps that don't? The CLI program is the API. I built a CLI program that wraps luks + btrfs, and they only offer a `--json` output option for a few commands. I have to write an ad hoc parser for each command since raw text includes arbitrary formatting and presentation lipstick that the creator came up with. And I have to do extra work to avoid breaking changes at the parser level instead of the higher data level. If I had to pick between the two, json would at least solve the data representation part so that I can build on top of it. And it's trivial to go data (json) -> pretty print rather than pretty print -> data. I can see it being annoying if all you care about is using CLI programs by hand, but it seems like a mild upside compared to the downsides the second you want to consume it programmatically, even if it's with a chain of awk, cut, and tr.

zbentley 3mo ago

Slightly off topic, but it is worth being familiar with AWSCLI’s built in jmespath JSON transformation system. That saves on the need for jq in a lot of cases. Given that awscli bundles that, I generally forgive it for defaulting to JSON (especially given how deeply nested and interlinked many AWS API results are), but reasonable minds can differ. AWSCLI’s schizophrenic use (or not) of pagers for tiny results, on the other hand, is less defensible.

jcgl 3mo ago

I think you're mistaking text-with-structured data for structured data itself. Because unix shell is irrevocably text-oriented, kludging in something like JSON is basically the best that can be done when you start to want to do structured operations on structured data. (I'm sympathetic to your point about the AWS CLI tools doing JSON by default though--that just sounds like bad design.) Being text-oriented imposes drastic limits on composability. Because there is no structure, every element of a pipeline needs to do its own parsing of the input data. This leads to brittle pipelines where every element is tightly coupled to its input's textual representation. As an exercise, try to write a pipeline that sorts podman images by size without removing the column headers[0]: $ podman image ls --all REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/prom/prometheus latest 937690d77350 2 months ago 367 MB quay.io/keycloak/keycloak latest da9433c9fac3 2 months ago 466 MB registry.fedoraproject.org/fedora-toolbox 43 a32da54355ca 4 months ago 2.19 GB docker.io/powerdns/pdns-auth-49 latest 8c1385c9deed 4 months ago 208 MB docker.io/testcontainers/ryuk 0.13.0 b75bc7ce94c3 6 months ago 7.21 MB As far as I can tell, there is no way to do this in a manner that's even remotely composable. Your best bet is to basically do everything from within awk. Whatever the result would be, it certainly won't be pretty! Contrast that with what you can do in PowerShell. You can write a couple of standalone functions[0] that are readable and composable, resulting in this pipeline: podman image ls --all | Replace-SpacesWithTabs | ConvertFrom-Csv -Delimiter "`t" | Sort-Object -Property {Convert-HumanSizeToBytes -Size $_.size} -Descending [0] Repurposing this from a blog post I wrote: https://www.cgl.sh/blog/posts/sh.html#this-should-be-basic https://www.cgl.sh/blog/posts/sh.html#this-should-be-basic

alloyed 3mo ago
the point is well-taken, but i do want to show the bash version just for fun: podman image ls --all | sed 's/\s\s\+/\t/g' | tee >(head -n 1) >(tail -n +2 | sort -hrk 5) >/dev/null this is _still_ all text, and we're relying heavily on sort to do a bunch of internal parsing and be in agreement with podman about how sizes should be formatted. also, for "real world" work, i dunno if the tee trick here has any kind of order guarantees, just that it works fine in this case. I'd probably just end up dropping the header and living with worse output in reality

tialaramex 3mo ago

I thought I would like Powershell but it was another big disappointment for me. It's got a lot of the unpleasant clunkiness that something like the Bourne shell owes to decades of compatibility, but Microsoft doesn't have that excuse. Despite this, it's gratuitously incompatible with itself, I had code which worked fine, we upgraded Powershell oh, now that won't work, just fix all the scripts. Crazy. It clearly wants an NPM-style experience where you seamlessly incorporate other people's work, but then it doesn't really deliver this well and so you often end up manually copy-pasting. If Powershell was a one man project and this was their Beta I'd say it is promising. But it's a project from the Microsoft corporation for 20 years. Do better.

torginus 3mo ago
This is my opinion as well. Powershell and object shells are much better and the way to go imo. Also Powershell code is fast owing to running on the .NET VM. But the actual UX is just convoluted and horrible. For example, the person who decided that commands without a ; at the end of the line should just vomit their outputs to the enclosing function's clearly hasn't heard of the principle of least surprise. PS just does so many plain weird and unintuitive things, which is worsened by the fact that it looks like a boring old programming language, while clearly not being one.