5 ms·

I'm a bit confused as to the use case. Is it just a way to interact with json/yaml style documents as if they were a structured database, but from the command l

I'm a bit confused as to the use case. Is it just a way to interact with json/yaml style documents as if they were a structured database, but from the command line? Kind of an in-between for those moments you don't want to write a quick script to batch modify files? It looks really well done, I think I'm just failing to see how this is more beneficial than just opening a single file in the editor and making changes, or writing a quick functional script so you have the history of the changes that were made to a batch of files. If someone could explain how I could (and why I should) add a new tool to my digital toolbelt, I'd greatly appreciate it.

paulddraper 2y ago
> or writing a quick functional script It's exactly a quick functional script.
0thgen 2y ago
one benefit (idk if it applies here) is if the select/put/delete statements didn’t require loading the data in memory; so you could query massive data files with limited RAM and not have to solve that problem yourself for each data storage format you’re working with
supriyo-biswas 2y ago
For things that are mostly shell scripts and things in a similar family (Ansible playbooks, deployment pipelines etc.) and where you need to modify a structured file quickly, it's usually much faster to use the DSL provided by the tool than calling out to various scripts to extract or modify a single JSON key. People often say that they'd prefer to write their shell scripts in Python or even Go these days, but the problem there is that the elements of structured programming makes the overall steps difficult to follow. Typically, the paradigm with use cases adjacent with shell scripts is to be able to view what it is doing without any sort of abstractions.
Lord_Zero 2y ago
This could be useful for CICD where you need to bump a version number in a file based on the build number.
- kate_bits 2y ago
  For this specific use case? sed would work just as well and probably already exists in your environment.
  - rout39574 2y ago
    Having done similar things to this in the past, I rebut: no it wouldn't. :) It's not trivial to change a particular version: 2.33 line in a JSON file, when it's possible that string is present in many places with different meanings. You can just wing it and be right 90% of the time; but that thoughtlessness will bite you in time. Multiply that by the number of different context you might want to make "the same" change in a project, and sed/awk get to be a poor fit. YAML and JSON are just plain not line oriented. You really need something with a featureset like xpath to set the semantically correct node to the right value, and every few years the kids decide they need YET ANOTHER thing that's not XML, or Yaml, or JSON, or TOML, or..
    - kate_bits 2y ago
      Maybe I am too prescriptive of your setup, but I don't really see "version: 2.33" making sense. Why not something akin to "version: __BUILD_NUMBER__" being actually checked into your tree, and then your build pipeline filling it in at build time? Keeping a copy of the build number checked into your source tree seems like asking for trouble when the build number is so intrinsically tied to a single build pipeline run.
macNchz 2y ago
I see the appeal of having a declarative syntax rather than writing a bunch of code to make the change reliably and safely.
fsckboy 2y ago
the in-between mode that you mention but seem to dismiss it is the way most traditional unixheads work with data most of the time: from the command line editor? when i pull up emacs, 50% of the time it's write emacs macros, and I do that because shell scripts don't easily go backward in the stream. (something rarely mentioned about teco was that it was a stream editor that would chew its way forward through files; you didn't need the memory to keep it all in core, and it could go backward within understandable limits) writing an actual shellscript is only for when it's really hairy, you are going to be repeating it and/or you need the types of error handling that cloud up the clarity of the commandline the commandline does provide rudimentary "records" in the saved history
simonw 2y ago
I use jq for this kind of thing several times a week. It’s great for piped data - things like running curl to fetch JSON, then piping it though to reformat it in different ways. Here’s a jq expression I used recently to turn a complete GitHub Issues thread into a single Markdown document: curl -s "https://api.github.com/repos/simonw/shot-scraper/issues/1/comments" \ | jq -r '.[] | "## Comment by \(.user.login) on \(.created_at)\n\n\(.body)\n"' I use this pattern a lot. Data often comes in slightly the wrong shape - being able to fix that with a one-liner terminal command is really useful.
- pmarreck 2y ago
  FYI if you pipe that to glow via adding `| glow -` at the end, it will pretty-print it in the Terminal, if that's something you like https://github.com/charmbracelet/glow https://github.com/charmbracelet/glow
tofflos 2y ago
I used yq last week to scan through all the Java projects (i.e. Maven pom.xml-files) within our org to check which ones inherit from the corporate pom. yq eval --input-format xml --output-format csv '[file_index, file_name, .project.parent.groupId, .project.parent.artifactId, .project.parent.version]' **/pom.xml
- hnlmorg 2y ago
  Which yq? Last time I checked, there seemed to be a few tools with the same name.
  - srott 2y ago
    The “go” yq https://github.com/mikefarah/yq https://github.com/mikefarah/yq | yq is a portable command-line YAML, JSON, XML, CSV, TOML and properties processor
Natfan 2y ago
PowerShell[0]'s built-in Microsoft.PowerShell.Utility[1] module has commands to `Convert-From...` or `Convert-To...` JSON, CSV, a version of XML (CliXML), or custom key/value pairs (StringData) into objects, which can then be manipulated. Combined with IO cmdlets from the built-in module Microsoft.PowerShell.Management[2] such as `Get-Content` and `Set-Content`, a fair chunk of flat-file storage should be able to be made mutable with PowerShell. --- [0]: https://microsoft.com/powershell https://microsoft.com/powershell [1]: https://learn.microsoft.com/powershell/module/microsoft.powershell.utility https://learn.microsoft.com/powershell/module/microsoft.powe... [2]: https://learn.microsoft.com/powershell/module/microsoft.powershell.management https://learn.microsoft.com/powershell/module/microsoft.powe...
- partdavid 2y ago
  Weirdly enough, INI is not that well-supported; but yeah, it's loads better than stuff like curl/jq for the very common JSON, YAML and other structured data cases (which includes near-100% native support for what you are commonly running shell utilities for: ls, ps, and so forth). The best part is that it's integrated into the shell language at the top-level so you can leverage whatever you learn to sort, filter and process all kinds of structured data, and explore and manipulate in a straightforward way. It's very powerful to freely intermix gci, gi, irm output and script together these various sources. And it's full of delightful tricks, like when you discover properly-throttled parallelism is easy (% -parallel {}) and you don't need to dive into yet another tool-specific abstruse sublanguage.

paulddraper 2y ago

> or writing a quick functional script It's exactly a quick functional script.

0thgen 2y ago

one benefit (idk if it applies here) is if the select/put/delete statements didn’t require loading the data in memory; so you could query massive data files with limited RAM and not have to solve that problem yourself for each data storage format you’re working with

supriyo-biswas 2y ago

For things that are mostly shell scripts and things in a similar family (Ansible playbooks, deployment pipelines etc.) and where you need to modify a structured file quickly, it's usually much faster to use the DSL provided by the tool than calling out to various scripts to extract or modify a single JSON key. People often say that they'd prefer to write their shell scripts in Python or even Go these days, but the problem there is that the elements of structured programming makes the overall steps difficult to follow. Typically, the paradigm with use cases adjacent with shell scripts is to be able to view what it is doing without any sort of abstractions.

Lord_Zero 2y ago

This could be useful for CICD where you need to bump a version number in a file based on the build number.

kate_bits 2y ago
For this specific use case? sed would work just as well and probably already exists in your environment.
- rout39574 2y ago
  Having done similar things to this in the past, I rebut: no it wouldn't. :) It's not trivial to change a particular version: 2.33 line in a JSON file, when it's possible that string is present in many places with different meanings. You can just wing it and be right 90% of the time; but that thoughtlessness will bite you in time. Multiply that by the number of different context you might want to make "the same" change in a project, and sed/awk get to be a poor fit. YAML and JSON are just plain not line oriented. You really need something with a featureset like xpath to set the semantically correct node to the right value, and every few years the kids decide they need YET ANOTHER thing that's not XML, or Yaml, or JSON, or TOML, or..
  - kate_bits 2y ago
    Maybe I am too prescriptive of your setup, but I don't really see "version: 2.33" making sense. Why not something akin to "version: __BUILD_NUMBER__" being actually checked into your tree, and then your build pipeline filling it in at build time? Keeping a copy of the build number checked into your source tree seems like asking for trouble when the build number is so intrinsically tied to a single build pipeline run.

macNchz 2y ago

I see the appeal of having a declarative syntax rather than writing a bunch of code to make the change reliably and safely.

fsckboy 2y ago

the in-between mode that you mention but seem to dismiss it is the way most traditional unixheads work with data most of the time: from the command line editor? when i pull up emacs, 50% of the time it's write emacs macros, and I do that because shell scripts don't easily go backward in the stream. (something rarely mentioned about teco was that it was a stream editor that would chew its way forward through files; you didn't need the memory to keep it all in core, and it could go backward within understandable limits) writing an actual shellscript is only for when it's really hairy, you are going to be repeating it and/or you need the types of error handling that cloud up the clarity of the commandline the commandline does provide rudimentary "records" in the saved history

simonw 2y ago

I use jq for this kind of thing several times a week. It’s great for piped data - things like running curl to fetch JSON, then piping it though to reformat it in different ways. Here’s a jq expression I used recently to turn a complete GitHub Issues thread into a single Markdown document: curl -s "https://api.github.com/repos/simonw/shot-scraper/issues/1/comments" \ | jq -r '.[] | "## Comment by \(.user.login) on \(.created_at)\n\n\(.body)\n"' I use this pattern a lot. Data often comes in slightly the wrong shape - being able to fix that with a one-liner terminal command is really useful.

pmarreck 2y ago
FYI if you pipe that to glow via adding `| glow -` at the end, it will pretty-print it in the Terminal, if that's something you like https://github.com/charmbracelet/glow https://github.com/charmbracelet/glow

tofflos 2y ago

I used yq last week to scan through all the Java projects (i.e. Maven pom.xml-files) within our org to check which ones inherit from the corporate pom. yq eval --input-format xml --output-format csv '[file_index, file_name, .project.parent.groupId, .project.parent.artifactId, .project.parent.version]' **/pom.xml

hnlmorg 2y ago
Which yq? Last time I checked, there seemed to be a few tools with the same name.
- srott 2y ago
  The “go” yq https://github.com/mikefarah/yq https://github.com/mikefarah/yq | yq is a portable command-line YAML, JSON, XML, CSV, TOML and properties processor

Natfan 2y ago

PowerShell[0]'s built-in Microsoft.PowerShell.Utility[1] module has commands to `Convert-From...` or `Convert-To...` JSON, CSV, a version of XML (CliXML), or custom key/value pairs (StringData) into objects, which can then be manipulated. Combined with IO cmdlets from the built-in module Microsoft.PowerShell.Management[2] such as `Get-Content` and `Set-Content`, a fair chunk of flat-file storage should be able to be made mutable with PowerShell. --- [0]: https://microsoft.com/powershell https://microsoft.com/powershell [1]: https://learn.microsoft.com/powershell/module/microsoft.powershell.utility https://learn.microsoft.com/powershell/module/microsoft.powe... [2]: https://learn.microsoft.com/powershell/module/microsoft.powershell.management https://learn.microsoft.com/powershell/module/microsoft.powe...

partdavid 2y ago
Weirdly enough, INI is not that well-supported; but yeah, it's loads better than stuff like curl/jq for the very common JSON, YAML and other structured data cases (which includes near-100% native support for what you are commonly running shell utilities for: ls, ps, and so forth). The best part is that it's integrated into the shell language at the top-level so you can leverage whatever you learn to sort, filter and process all kinds of structured data, and explore and manipulate in a straightforward way. It's very powerful to freely intermix gci, gi, irm output and script together these various sources. And it's full of delightful tricks, like when you discover properly-throttled parallelism is easy (% -parallel {}) and you don't need to dive into yet another tool-specific abstruse sublanguage.