7 ms·

Discussion of a similar tool last month: yq: command-line YAML, JSON, XML, CSV and properties processor https://news.ycombinator.com/item?id=34656022 https://

Discussion of a similar tool last month: yq: command-line YAML, JSON, XML, CSV and properties processor https://news.ycombinator.com/item?id=34656022 https://news.ycombinator.com/item?id=34656022 Also mentions gojq, Benthos, xsv, Damsel, a 2nd yq, htmlq, cfn-flip, csvq, zq, and zsv.

networked 4y ago
I have made a repository cataloguing tools like this: https://github.com/dbohdan/structured-text-tools https://github.com/dbohdan/structured-text-tools.
- account-5 4y ago
  Thanks for this, good job. I always mean to do something similar but just end up bookmarking HN threads I then never look at.
  - samstave 4y ago
    Guilty as charged... I have so many bookmarks I forget to look at... APPlet Idea: an app that picks a random bookmark from your saves and sends you a reminder to click it and read it... and schedule when you want to see it - like every 7AM send me a link from my bookmarks. Set an alarm for 15 minutes to get me back on schedule.
  - damascus 4y ago
    So chatgpt and I made this: https://github.com/erasei/randombookmark https://github.com/erasei/randombookmark Load in developer mode and when you click the icon it will open a new tab with a random bookmark.
- jefftk 4y ago
  There's also GoAWK, which supports CSV natively ("-i csv"): https://github.com/benhoyt/goawk https://github.com/benhoyt/goawk
- glomgril 4y ago
  Big shouts to visidata -- very underrated tool imo.
- horseRad 4y ago
  You should include qsv (https://github.com/jqnatividad/qsv https://github.com/jqnatividad/qsv) and goawk (https://github.com/benhoyt/goawk https://github.com/benhoyt/goawk) into the CSV section! Goawk got a csv/tsv mode and qsv is loaded with features being developed.
- elesiuta 4y ago
  Thank you! That list is perfect since I don't use these tools often enough to remember them and some are hard to find again without knowing exactly what to search for. I also discovered pawk on it which looks interesting, I made a somewhat related tool [1] which is probably out of scope for your list, but can be used in some of the same ways [2] and may of of interest nonetheless. [1] https://github.com/elesiuta/pyxargs https://github.com/elesiuta/pyxargs [2] cat /etc/hosts | pyxargs -d \n --im json --pre "d={}" --post "print(dumps(d))" --py "d['{}'.split()[0]] = '{}'.split()[1]"
asicsp 4y ago
Seems like you meant https://github.com/TomWright/dasel https://github.com/TomWright/dasel (instead of damsel)
- password4321 4y ago
  Correct; thanks!
dima55 4y ago
There's also https://github.com/dkogan/vnlog/ https://github.com/dkogan/vnlog/ which is a wrapper around the existing coreutils, so all the options work, and there's nothing to learn
nextaccountic 4y ago
the obvious missing reference is jq https://github.com/stedolan/jq https://github.com/stedolan/jq
- mdaniel 4y ago
  I've been getting a lot of mileage out of https://github.com/itchyny/gojq#readme https://github.com/itchyny/gojq#readme recently due to two things: its vastly superior error messages and the (regrettably verbose) `--yaml-input` option I also have https://github.com/01mf02/jaq#readme https://github.com/01mf02/jaq#readme installed but just haven't needed it
  - donio 4y ago
    -y and -Y unused so I patch those in as short flags for --yaml-input/output in my copy.
- password4321 4y ago
  True, for me that one's a given!
elesiuta 4y ago
Another similar discussion last year where Miller was mentioned: New(ish) command line tools https://news.ycombinator.com/item?id=31009313 https://news.ycombinator.com/item?id=31009313
donio 4y ago
Benthos is a really cool tool when you want to take this idea to the next level. It can do the usual text/csv/json/yaml mangling you'd do with awk or jq (since it includes a version of those in addition to its own processing langauge) but it also has decoders and encoders for a bunch of different binary formats like protobuf or Avro. And in addition to stdin/out and files it can talk to Kafka, MQTT, HTTP, ES and a bunch of other stuff. I was able to put together a log processor that consumes from Kafka, does a bunch of field mangling and then ingests into Elasticsearch in a couple dozen lines of yaml.
dmix 4y ago
Anyone else find jq really hard to use? I found the syntax/interface bewildering at times. Although not as bad as awk.
- password4321 4y ago
  I'm pretty sure that's one big reason for all these other tools!

networked 4y ago

I have made a repository cataloguing tools like this: https://github.com/dbohdan/structured-text-tools https://github.com/dbohdan/structured-text-tools.

account-5 4y ago
Thanks for this, good job. I always mean to do something similar but just end up bookmarking HN threads I then never look at.
- samstave 4y ago
  Guilty as charged... I have so many bookmarks I forget to look at... APPlet Idea: an app that picks a random bookmark from your saves and sends you a reminder to click it and read it... and schedule when you want to see it - like every 7AM send me a link from my bookmarks. Set an alarm for 15 minutes to get me back on schedule.
- damascus 4y ago
  So chatgpt and I made this: https://github.com/erasei/randombookmark https://github.com/erasei/randombookmark Load in developer mode and when you click the icon it will open a new tab with a random bookmark.
jefftk 4y ago
There's also GoAWK, which supports CSV natively ("-i csv"): https://github.com/benhoyt/goawk https://github.com/benhoyt/goawk
glomgril 4y ago
Big shouts to visidata -- very underrated tool imo.
horseRad 4y ago
You should include qsv (https://github.com/jqnatividad/qsv https://github.com/jqnatividad/qsv) and goawk (https://github.com/benhoyt/goawk https://github.com/benhoyt/goawk) into the CSV section! Goawk got a csv/tsv mode and qsv is loaded with features being developed.
elesiuta 4y ago
Thank you! That list is perfect since I don't use these tools often enough to remember them and some are hard to find again without knowing exactly what to search for. I also discovered pawk on it which looks interesting, I made a somewhat related tool [1] which is probably out of scope for your list, but can be used in some of the same ways [2] and may of of interest nonetheless. [1] https://github.com/elesiuta/pyxargs https://github.com/elesiuta/pyxargs [2] cat /etc/hosts | pyxargs -d \n --im json --pre "d={}" --post "print(dumps(d))" --py "d['{}'.split()[0]] = '{}'.split()[1]"

asicsp 4y ago

Seems like you meant https://github.com/TomWright/dasel https://github.com/TomWright/dasel (instead of damsel)

password4321 4y ago
Correct; thanks!

dima55 4y ago

There's also https://github.com/dkogan/vnlog/ https://github.com/dkogan/vnlog/ which is a wrapper around the existing coreutils, so all the options work, and there's nothing to learn

nextaccountic 4y ago

the obvious missing reference is jq https://github.com/stedolan/jq https://github.com/stedolan/jq

mdaniel 4y ago
I've been getting a lot of mileage out of https://github.com/itchyny/gojq#readme https://github.com/itchyny/gojq#readme recently due to two things: its vastly superior error messages and the (regrettably verbose) `--yaml-input` option I also have https://github.com/01mf02/jaq#readme https://github.com/01mf02/jaq#readme installed but just haven't needed it
- donio 4y ago
  -y and -Y unused so I patch those in as short flags for --yaml-input/output in my copy.
password4321 4y ago
True, for me that one's a given!

elesiuta 4y ago

Another similar discussion last year where Miller was mentioned: New(ish) command line tools https://news.ycombinator.com/item?id=31009313 https://news.ycombinator.com/item?id=31009313

donio 4y ago

Benthos is a really cool tool when you want to take this idea to the next level. It can do the usual text/csv/json/yaml mangling you'd do with awk or jq (since it includes a version of those in addition to its own processing langauge) but it also has decoders and encoders for a bunch of different binary formats like protobuf or Avro. And in addition to stdin/out and files it can talk to Kafka, MQTT, HTTP, ES and a bunch of other stuff. I was able to put together a log processor that consumes from Kafka, does a bunch of field mangling and then ingests into Elasticsearch in a couple dozen lines of yaml.

dmix 4y ago

Anyone else find jq really hard to use? I found the syntax/interface bewildering at times. Although not as bad as awk.

password4321 4y ago
I'm pretty sure that's one big reason for all these other tools!