4 ms·

I’m a big fan of tools producing json output, but I couldn’t help but chuckle a bit at: > This allows easier parsing in scripts by using JSON parsing tools lik

I’m a big fan of tools producing json output, but I couldn’t help but chuckle a bit at: > This allows easier parsing in scripts by using JSON parsing tools like jq, jello, jp, etc. without arcane awk, sed, cut, tr, reverse, etc. incantations. While I love jq, I find its syntax to be utterly baroque. Far more than awk and sed (let alone simple tools like cut and tr). To the point where I end up looking at the jq man page to remind myself how to do even seemingly simple things at times. And often jq is just the beginning of a pipeline to get hierarchical data into columnar format so I can hit it with awk and sed. But maybe this is all about familiarity; I have years of experience cobbling together shell tools and reach for jq comparatively less. I suspect the author may be the opposite.

Mordisquitos 3y ago
I would argue it's not just about familiarity. I had the same reaction to that quote and feel the same as you do, and yet I have been using jq before I first learned and started using awk. For a long time I had been familiar with jq, but every time I had to use it for any non-trivial task after not touching it for a while I had to go to the docs and really scratch my head. Meanwhile, I had no idea about awk, and for ages I would just see those awk one-liners on Stack Overflow as weird arcane alternatives to "simply" piping sed, grep and tr over bash. However, one day I finally had the time and the reason to have a look at AWK Programming for a specific use that I needed, and it immediately clicked. Awk makes sense and now I can easily fall back to it every time, even if I haven't used it for ages. At most, all I need to look up is the order of parameters in the gawk regex functions. And yet, time and time again I am baffled by jq, however much I have used it in the past. Whenever I need to parse a JSON I'm happy enough if I can use jq to get it to a good enough halfway point so that I can pipe it to awk to do the actual work.
DangitBobby 3y ago
ChatGPT flawlessly constructs jq commands for whatever banal task you have. Game changer.
- simonw 3y ago
  Completely agree. I use jq multiple times a day now, purely because GPT-4 or Claude Opus mean I never have to think about how to express anything in it.
- invalidator 3y ago
  Can you post an example of how you'd phrase the question to ChatGPT?
  - InvaderFizz 3y ago
    Here is a sample of my input json, here is what I expect the result to be, write a jq command to accomplish that.
  - DangitBobby 3y ago
    Write a command using jq which will take the following input and do x, y, and z to it: $INPUT
    - jasomill 3y ago
      Speaking as an inexperienced, infrequent jq user, using ChatGPT to create a jq version of my above PowerShell example took more time than writing a jq version by hand with the help of jq documentation (and to have any confidence that the result is actually correct in the general case, rather than merely producing expected results for my example, I'd need to inspect OpenGPT's final command and RTFM anyway): https://chat.openai.com/share/9c73bc02-da4b-494d-8cff-3d1202a82cc1 https://chat.openai.com/share/9c73bc02-da4b-494d-8cff-3d1202...
      - DangitBobby 3y ago
        Oof, that "don't" in the code comment within the single quotes was causing your single quoted command to end, causing the command to be parsed incorrectly in your shell. I wonder if that was the only issue originally because it actually looks pretty good to me. Regardless, I can tell you with confidence it would take much less time for me to construct a working version based on the first input than starting from RTFM, despite already having some jq experience myself. Was this 3.5 or 4?
candiddevmike 3y ago
I output JSON and embed a jq-like arg in in all of my projects (https://rotx.dev/docs/references/cli#jq https://rotx.dev/docs/references/cli#jq). Makes it really easy to slice and dice output.
arp242 3y ago
It's not you; parsing line-based text is fundamentally simpler than parsing structured data like JSON, especially nested JSON. It's just a lot easier to reason about, and a lot easier to split up in smaller chunks, too. Who hasn't ended up with something like: cmd | sed | grep | sed | grep Each bit gets you closer what you want in small easy-to-understand steps, making it fairly easy to develop. And for things you only do once or twice, who cares as long as it works, right? I find that's a lot harder to do with jq. I feel this is a problem waiting for a better solution (and gron isn't it).
- fuzztester 3y ago
  >Who hasn't ended up with something like: > cmd | sed | grep | sed | grep And the decorate-sort-undecorate pattern: https://en.m.wikipedia.org/wiki/Decorate-sort-undecorate https://en.m.wikipedia.org/wiki/Decorate-sort-undecorate and the related Schwartzian transform: https://en.m.wikipedia.org/wiki/Schwartzian_transform https://en.m.wikipedia.org/wiki/Schwartzian_transform (The former redirects to the latter.)
- lttlrck 3y ago
  I end up with that kind of thing frequently, I'd like an optimizer though because I'm sure I'm certainly missing some deeper flag/regex/... knowledge.
  - arp242 3y ago
    I probably have deeper knowledge of this kind of thing than most, and in most cases I found that doing a "crude" low-cognitive approach is better than spending brain cycles and effort on a "clever" high-cognitive approach.
jasomill 3y ago
I've been using sed, awk, and friends for I find myself reaching for 30+ years, refer to jq documentation 90% of the time I use it (vs. <<10% when using traditional UNIX tools), and still appreciate JSON output as an option. Incidentally, PowerShell has nice built-in facilities for working with JSON: /Users/jtm> ( >> diskutil list -plist ` >> | plutil -convert json -o - - ` >> | ConvertFrom-Json >> ).AllDisksAndPartitions.APFSVolumes ` >> | where MountPoint ` >> | select ` >> VolumeName,` >> @{L='Size';E={$_.Size/1GB}},` >> @{L='Free';E={($_.Size - $_.CapacityInUse)/1GB}} ` >> | where { >> $_.Free / $_.Size -lt 0.10 >> } ` >> | sort VolumeName ` >> | ConvertTo-Json [ { "VolumeName": "Gríðarvölr", "Size": 1862.8213348388672, "Free": 150.1973648071289 }, { "VolumeName": "Mjölnir", "Size": 1862.6963348388672, "Free": 92.34595108032227 } ]
jauntywundrkind 3y ago
I've definitely waxed & waned a number of times through jq familiarity. And sometimes it is a bit of a stress. But this feels like a vastly unfair basis for comparison. None of the old tools do 1/1000th what jq does. Arbitrary structure data in, a whole pipe of simple or complex commands, structured data out. Jq is complex maybe sure yeah, but it replaces some really gnarly inefficient awful zsh scripts I'd have to cobble together with something that, in the end, is sensible & clear & direct. That other people would probably also write the same way I did. Living in the structures data world is so much better than arbitrary text-line oriented systems. They may individually be better, but the utter lack of consistency in the old world meant reinventing too much, each time you want to do a task. Jq effectively is a mini shell, full of shell like concepts (operators, variables, pipes,...), and on those grounds, it's not like zsh has any shortage of capabilities most people don't know about & use well. Most of our shell scripting is beating on the same 10% and having to go hunt around through the other 90% on demand too.

Mordisquitos 3y ago

I would argue it's not just about familiarity. I had the same reaction to that quote and feel the same as you do, and yet I have been using jq before I first learned and started using awk. For a long time I had been familiar with jq, but every time I had to use it for any non-trivial task after not touching it for a while I had to go to the docs and really scratch my head. Meanwhile, I had no idea about awk, and for ages I would just see those awk one-liners on Stack Overflow as weird arcane alternatives to "simply" piping sed, grep and tr over bash. However, one day I finally had the time and the reason to have a look at AWK Programming for a specific use that I needed, and it immediately clicked. Awk makes sense and now I can easily fall back to it every time, even if I haven't used it for ages. At most, all I need to look up is the order of parameters in the gawk regex functions. And yet, time and time again I am baffled by jq, however much I have used it in the past. Whenever I need to parse a JSON I'm happy enough if I can use jq to get it to a good enough halfway point so that I can pipe it to awk to do the actual work.

DangitBobby 3y ago

ChatGPT flawlessly constructs jq commands for whatever banal task you have. Game changer.

simonw 3y ago
Completely agree. I use jq multiple times a day now, purely because GPT-4 or Claude Opus mean I never have to think about how to express anything in it.
invalidator 3y ago
Can you post an example of how you'd phrase the question to ChatGPT?
- InvaderFizz 3y ago
  Here is a sample of my input json, here is what I expect the result to be, write a jq command to accomplish that.
- DangitBobby 3y ago
  Write a command using jq which will take the following input and do x, y, and z to it: $INPUT
  - jasomill 3y ago
    Speaking as an inexperienced, infrequent jq user, using ChatGPT to create a jq version of my above PowerShell example took more time than writing a jq version by hand with the help of jq documentation (and to have any confidence that the result is actually correct in the general case, rather than merely producing expected results for my example, I'd need to inspect OpenGPT's final command and RTFM anyway): https://chat.openai.com/share/9c73bc02-da4b-494d-8cff-3d1202a82cc1 https://chat.openai.com/share/9c73bc02-da4b-494d-8cff-3d1202...
    - DangitBobby 3y ago
      Oof, that "don't" in the code comment within the single quotes was causing your single quoted command to end, causing the command to be parsed incorrectly in your shell. I wonder if that was the only issue originally because it actually looks pretty good to me. Regardless, I can tell you with confidence it would take much less time for me to construct a working version based on the first input than starting from RTFM, despite already having some jq experience myself. Was this 3.5 or 4?

candiddevmike 3y ago

I output JSON and embed a jq-like arg in in all of my projects (https://rotx.dev/docs/references/cli#jq https://rotx.dev/docs/references/cli#jq). Makes it really easy to slice and dice output.

arp242 3y ago

It's not you; parsing line-based text is fundamentally simpler than parsing structured data like JSON, especially nested JSON. It's just a lot easier to reason about, and a lot easier to split up in smaller chunks, too. Who hasn't ended up with something like: cmd | sed | grep | sed | grep Each bit gets you closer what you want in small easy-to-understand steps, making it fairly easy to develop. And for things you only do once or twice, who cares as long as it works, right? I find that's a lot harder to do with jq. I feel this is a problem waiting for a better solution (and gron isn't it).

fuzztester 3y ago
>Who hasn't ended up with something like: > cmd | sed | grep | sed | grep And the decorate-sort-undecorate pattern: https://en.m.wikipedia.org/wiki/Decorate-sort-undecorate https://en.m.wikipedia.org/wiki/Decorate-sort-undecorate and the related Schwartzian transform: https://en.m.wikipedia.org/wiki/Schwartzian_transform https://en.m.wikipedia.org/wiki/Schwartzian_transform (The former redirects to the latter.)
lttlrck 3y ago
I end up with that kind of thing frequently, I'd like an optimizer though because I'm sure I'm certainly missing some deeper flag/regex/... knowledge.
- arp242 3y ago
  I probably have deeper knowledge of this kind of thing than most, and in most cases I found that doing a "crude" low-cognitive approach is better than spending brain cycles and effort on a "clever" high-cognitive approach.

jasomill 3y ago

I've been using sed, awk, and friends for I find myself reaching for 30+ years, refer to jq documentation 90% of the time I use it (vs. <<10% when using traditional UNIX tools), and still appreciate JSON output as an option. Incidentally, PowerShell has nice built-in facilities for working with JSON: /Users/jtm> ( >> diskutil list -plist ` >> | plutil -convert json -o - - ` >> | ConvertFrom-Json >> ).AllDisksAndPartitions.APFSVolumes ` >> | where MountPoint ` >> | select ` >> VolumeName,` >> @{L='Size';E={$_.Size/1GB}},` >> @{L='Free';E={($_.Size - $_.CapacityInUse)/1GB}} ` >> | where { >> $_.Free / $_.Size -lt 0.10 >> } ` >> | sort VolumeName ` >> | ConvertTo-Json [ { "VolumeName": "Gríðarvölr", "Size": 1862.8213348388672, "Free": 150.1973648071289 }, { "VolumeName": "Mjölnir", "Size": 1862.6963348388672, "Free": 92.34595108032227 } ]

jauntywundrkind 3y ago

I've definitely waxed & waned a number of times through jq familiarity. And sometimes it is a bit of a stress. But this feels like a vastly unfair basis for comparison. None of the old tools do 1/1000th what jq does. Arbitrary structure data in, a whole pipe of simple or complex commands, structured data out. Jq is complex maybe sure yeah, but it replaces some really gnarly inefficient awful zsh scripts I'd have to cobble together with something that, in the end, is sensible & clear & direct. That other people would probably also write the same way I did. Living in the structures data world is so much better than arbitrary text-line oriented systems. They may individually be better, but the utter lack of consistency in the old world meant reinventing too much, each time you want to do a task. Jq effectively is a mini shell, full of shell like concepts (operators, variables, pipes,...), and on those grounds, it's not like zsh has any shortage of capabilities most people don't know about & use well. Most of our shell scripting is beating on the same 10% and having to go hunt around through the other 90% on demand too.