4 ms·

The most effective combination I've found so far is jq + basic shell tools. I still think jq's syntax and data model is unbelievably elegant and powerful once

by xg15 3y ago

The most effective combination I've found so far is jq + basic shell tools. I still think jq's syntax and data model is unbelievably elegant and powerful once you get the hang of it - but its "standard library" is unfortunately sorely lacking in many places and has some awkward design choices in others, which means that a lot of practical everyday tasks - such as aggregations or even just set membership - are a lot more complicated than they ought to be. Luckily, what jq can do really well is bringing data of interest into a line-based text representation, which is ideal for all kinds of standard unix shell tools - so you can just use those to take over the parts of your pipeline that would be hard to do in "pure" jq. So I think my solution to the OP's task - get all distinct OSS licenses from the project list and count usages for each one - would be: curl ... | jq '.[].license.key' | sort | uniq -c That's it.

pcthrowaway 3y ago
> I still think jq's syntax and data model is unbelievably elegant and powerful once you get the hang of it - but its "standard library" is unfortunately sorely lacking in many places After a few years of stalled development, jq has been taken over recently by a new team of maintainers and is rapidly working through a lot of longstanding issues (https://github.com/jqlang/jq https://github.com/jqlang/jq), so I'm not sure if this is still the case
- xg15 3y ago
  Wasn't aware of that, that's great to hear! I think if there is one utility that deserves a great maintainer team then this one. But if we saw some actual improvements in the future, that would be awesome! I have a list of pet peeves that I'd really like to see fixed, so I'm gonna risk a bit of hope.
jimbokun 3y ago
The Unix philosophy continues to pass the test of time.
- pphysch 3y ago
  Yes and no. Many UNIX philosophy proponents are abhorred by powerful binaries like jq and awk.
  - michaelcampbell 3y ago
    Like any religion, there are zealots and "no true Scotsman" arguments a'plenty.
eru 3y ago
> I still think jq's syntax and data model is unbelievably elegant and powerful once you get the hang of it [...] It's basically just functional programming. (Or what you would get from a functional programmer given the task of writing such a tool as jq.) That's not to diminish jq, it's a great tool. I love it!
salmo 3y ago
As an old Unix guy this is exactly how I see jq: a gateway to a fantastic library of text processing tools. I see a lot of complicated things done inside the language, which is a valid approach. But I don’t need it to be a programming language itself, just a transform to meet my next command after the pipe. If I want logic beyond that, then I skip the shell and write “real” software. I personally find those both to be more readable and easier to fit in my head than long complex jq expressions. But that’s completely subjective and others may find the jq expression language easier to read than shell or (choose your programming language).
- digdugdirk 3y ago
  Your comment made me go look up jq (even more than the article did) and the first paragraph of the repo [0] feels like a secret club's secret language. I'm very interested, but not a Linux person, do you know of any good resources for learning the Linux shell as a programming language? [0] https://jqlang.github.io/jq/ https://jqlang.github.io/jq/
  - salmo 3y ago
    I’ll say, I did shell scripting for years from copy/paste, cribbing smarter people, and reading online guides. But I didn’t really understand until I read The Unix Programming Environment by Brian Kernighan and Rob Pike. It’s a very old book and the audience was using dumb terminals. But it made me understand why and how. I think I’ve read every Kernighan book at this point and most he was involved in because he is just so amazing and not just conveying facts, but teaching how to think idiomatically in the topic. I also used awk for 2 decades, kind of like how I use jq now. But when I read his memoir I suddenly “got it.” What I make with it now is intentional and not just me banging on the keyboard until it works. A great middle ground for something a little sophisticated, but not worth writing a full program for. Something else that helped me was to install a minimal distro… actually a base FreeBSD install would be great… and read the man pages for all the commands. I don’t remember the details, but I learned that things existed. I have many man pages that I look at the same options on every few months because I’m not positive I remember right. Heck, I ‘man test’ all the time still. (‘test’ and ‘[‘ are the same thing) I also had an advantage of 2 great coworkers. They’d been working on Unix since the 80s and their feedback helped me be more efficient, clean, and avoid “useless use of cat” problems. I also highly recommend using shellcheck. I sometimes disagree with it when I’m intentionally abusing shell behavior, but it’s a great way to train good habits and prevent bugs that only crop up with bad input, scale, etc. I get new devs to use it and it’s helped them “ramp up” quickly, with me explaining the “why” from time to time. But yeah. The biggest problem I see is that people think there is more syntax than there really is (like my test and [ comment). And remember it’s all text, processes, and files. Except when we pretend it’s not ;).
mightybyte 3y ago
Your command line solution doesn't give quite the same result as OP. The final output in OP is sorted by the count field, but your command line incantation doesn't do that. One might respond that all you need to do is add a second "| sort" at the end, but that doesn't quite do it either. That will use string sorting instead of proper numeric sorting. In this example with only three output rows it's not an issue. But with larger amounts of data it will become a problem. Your fundamental point about the power of basic shell tools is still completely valid. But if I could attempt to summarize OP's point, I think it would be that SQL is more powerful than ad-hoc jq incantations. And in this case, I tend to agree with OP. I've made substantial use of jq and yq over the course of years, as well as other tools for CSVs and other data formats. But every time I reach for them I have to spend a lot of time hunting the docs for just the right syntax to attack my specific problem. I know jq's paradigm draws from functional programming concepts and I have plenty of personal experience with functional programming, but the syntax and still feel very ad hoc and clunky. Modern OLAP DB tools like duckdb, clickhouse, etc that provide really nice ways to get all kinds of data formats into and out of a SQL environment seem dramatically more powerful to me. Then when you add the power of all the basic shell tools on top of that, I think you get a much more powerful combination. I like this example from the clickhouse-local documentation: $ ps aux | tail -n +2 | awk '{ printf("%s\t%s\n", $1, $4) }' \ | clickhouse-local --structure "user String, mem Float64" \ --query "SELECT user, round(sum(mem), 2) as memTotal FROM table GROUP BY user ORDER BY memTotal DESC FORMAT Pretty"
- tleb_ 3y ago
  For reference, sort(1) has -n, --numeric-sort: compare according to string numerical value.
- xg15 3y ago
  You can archive that by appending sort -n, so the whole thing becomes: curl ... | jq '.[].license.key' | sort | uniq -c | sort -n You can even turn it back into json by exploiting the fact that when uniq -c gets lines of json as input, it's output will be "accidentally" parseable as a sequence of json literals by jq, where every second literal is a count. You can use jq's (very weird) input function to transform each pair of literals into a "proper" json object: curl ... | jq '.[].license.key' | sort | uniq -c | sort -n | jq '{"count":., "value":input}'
DanielHB 3y ago
found out recently that jq can url-encode values, is there anything it _can't_ do?
- xg15 3y ago
  Finding out whether . is contained in a given array or not, evidently. (That's not strictly true - you can do it, you just have to bend over backwards for what is essentially the "in" keyword in python, sql, etc. jq has no less than four functions that look like they should do that - in(), has(), contains() and inside(), yet they all do something slightly different)
jb3689 3y ago
Driving that further - I don’t want to have to edit my query for minutes when I am in the shell. I don’t believe that the shell is conducive to complex SQL queries. You could write simpler SQL queries, but then you’re in a space where there are less verbose tools for those simpler tasks. DuckDB seems stronger for someone who needs to create a scripting library - which also has lots of options and competition - or someone who has a very specific workflow of working with JSON dumps for a huge percentage of their time.
hipjiveguy 3y ago
I do the same thing all the time, lol but have in the last year or so, tried to start writing things that will be read/used by other people in python or java why? because most people don't have a clue how bash scripts work, but can read/debug python and java with ease, and both can be run as shell scripts too the mac has sqlite installed by default as well, so there's a powerful combo available on every mac but I totally do what you're doing all the time, and it is my default go to if I have to get something done fast :D

by xg15 3y ago

pcthrowaway 3y ago

> I still think jq's syntax and data model is unbelievably elegant and powerful once you get the hang of it - but its "standard library" is unfortunately sorely lacking in many places After a few years of stalled development, jq has been taken over recently by a new team of maintainers and is rapidly working through a lot of longstanding issues (https://github.com/jqlang/jq https://github.com/jqlang/jq), so I'm not sure if this is still the case

xg15 3y ago
Wasn't aware of that, that's great to hear! I think if there is one utility that deserves a great maintainer team then this one. But if we saw some actual improvements in the future, that would be awesome! I have a list of pet peeves that I'd really like to see fixed, so I'm gonna risk a bit of hope.

jimbokun 3y ago

The Unix philosophy continues to pass the test of time.

pphysch 3y ago
Yes and no. Many UNIX philosophy proponents are abhorred by powerful binaries like jq and awk.
- michaelcampbell 3y ago
  Like any religion, there are zealots and "no true Scotsman" arguments a'plenty.

eru 3y ago

> I still think jq's syntax and data model is unbelievably elegant and powerful once you get the hang of it [...] It's basically just functional programming. (Or what you would get from a functional programmer given the task of writing such a tool as jq.) That's not to diminish jq, it's a great tool. I love it!

salmo 3y ago

As an old Unix guy this is exactly how I see jq: a gateway to a fantastic library of text processing tools. I see a lot of complicated things done inside the language, which is a valid approach. But I don’t need it to be a programming language itself, just a transform to meet my next command after the pipe. If I want logic beyond that, then I skip the shell and write “real” software. I personally find those both to be more readable and easier to fit in my head than long complex jq expressions. But that’s completely subjective and others may find the jq expression language easier to read than shell or (choose your programming language).

digdugdirk 3y ago
Your comment made me go look up jq (even more than the article did) and the first paragraph of the repo [0] feels like a secret club's secret language. I'm very interested, but not a Linux person, do you know of any good resources for learning the Linux shell as a programming language? [0] https://jqlang.github.io/jq/ https://jqlang.github.io/jq/
- salmo 3y ago
  I’ll say, I did shell scripting for years from copy/paste, cribbing smarter people, and reading online guides. But I didn’t really understand until I read The Unix Programming Environment by Brian Kernighan and Rob Pike. It’s a very old book and the audience was using dumb terminals. But it made me understand why and how. I think I’ve read every Kernighan book at this point and most he was involved in because he is just so amazing and not just conveying facts, but teaching how to think idiomatically in the topic. I also used awk for 2 decades, kind of like how I use jq now. But when I read his memoir I suddenly “got it.” What I make with it now is intentional and not just me banging on the keyboard until it works. A great middle ground for something a little sophisticated, but not worth writing a full program for. Something else that helped me was to install a minimal distro… actually a base FreeBSD install would be great… and read the man pages for all the commands. I don’t remember the details, but I learned that things existed. I have many man pages that I look at the same options on every few months because I’m not positive I remember right. Heck, I ‘man test’ all the time still. (‘test’ and ‘[‘ are the same thing) I also had an advantage of 2 great coworkers. They’d been working on Unix since the 80s and their feedback helped me be more efficient, clean, and avoid “useless use of cat” problems. I also highly recommend using shellcheck. I sometimes disagree with it when I’m intentionally abusing shell behavior, but it’s a great way to train good habits and prevent bugs that only crop up with bad input, scale, etc. I get new devs to use it and it’s helped them “ramp up” quickly, with me explaining the “why” from time to time. But yeah. The biggest problem I see is that people think there is more syntax than there really is (like my test and [ comment). And remember it’s all text, processes, and files. Except when we pretend it’s not ;).

mightybyte 3y ago

Your command line solution doesn't give quite the same result as OP. The final output in OP is sorted by the count field, but your command line incantation doesn't do that. One might respond that all you need to do is add a second "| sort" at the end, but that doesn't quite do it either. That will use string sorting instead of proper numeric sorting. In this example with only three output rows it's not an issue. But with larger amounts of data it will become a problem. Your fundamental point about the power of basic shell tools is still completely valid. But if I could attempt to summarize OP's point, I think it would be that SQL is more powerful than ad-hoc jq incantations. And in this case, I tend to agree with OP. I've made substantial use of jq and yq over the course of years, as well as other tools for CSVs and other data formats. But every time I reach for them I have to spend a lot of time hunting the docs for just the right syntax to attack my specific problem. I know jq's paradigm draws from functional programming concepts and I have plenty of personal experience with functional programming, but the syntax and still feel very ad hoc and clunky. Modern OLAP DB tools like duckdb, clickhouse, etc that provide really nice ways to get all kinds of data formats into and out of a SQL environment seem dramatically more powerful to me. Then when you add the power of all the basic shell tools on top of that, I think you get a much more powerful combination. I like this example from the clickhouse-local documentation: $ ps aux | tail -n +2 | awk '{ printf("%s\t%s\n", $1, $4) }' \ | clickhouse-local --structure "user String, mem Float64" \ --query "SELECT user, round(sum(mem), 2) as memTotal FROM table GROUP BY user ORDER BY memTotal DESC FORMAT Pretty"

tleb_ 3y ago
For reference, sort(1) has -n, --numeric-sort: compare according to string numerical value.
xg15 3y ago
You can archive that by appending sort -n, so the whole thing becomes: curl ... | jq '.[].license.key' | sort | uniq -c | sort -n You can even turn it back into json by exploiting the fact that when uniq -c gets lines of json as input, it's output will be "accidentally" parseable as a sequence of json literals by jq, where every second literal is a count. You can use jq's (very weird) input function to transform each pair of literals into a "proper" json object: curl ... | jq '.[].license.key' | sort | uniq -c | sort -n | jq '{"count":., "value":input}'

DanielHB 3y ago

found out recently that jq can url-encode values, is there anything it _can't_ do?

xg15 3y ago
Finding out whether . is contained in a given array or not, evidently. (That's not strictly true - you can do it, you just have to bend over backwards for what is essentially the "in" keyword in python, sql, etc. jq has no less than four functions that look like they should do that - in(), has(), contains() and inside(), yet they all do something slightly different)

jb3689 3y ago

Driving that further - I don’t want to have to edit my query for minutes when I am in the shell. I don’t believe that the shell is conducive to complex SQL queries. You could write simpler SQL queries, but then you’re in a space where there are less verbose tools for those simpler tasks. DuckDB seems stronger for someone who needs to create a scripting library - which also has lots of options and competition - or someone who has a very specific workflow of working with JSON dumps for a huge percentage of their time.

hipjiveguy 3y ago

I do the same thing all the time, lol but have in the last year or so, tried to start writing things that will be read/used by other people in python or java why? because most people don't have a clue how bash scripts work, but can read/debug python and java with ease, and both can be run as shell scripts too the mac has sqlite installed by default as well, so there's a powerful combo available on every mac but I totally do what you're doing all the time, and it is my default go to if I have to get something done fast :D