7 ms·

Since no one seems to know about it, jq is described in great detail on the github wiki page [0]. That flattens the learning curve a lot. It's not as arcane as

Since no one seems to know about it, jq is described in great detail on the github wiki page [0]. That flattens the learning curve a lot. It's not as arcane as it seems. The touted claim that is fundamentally stateless is not true. jq is also stateful in the sense that it has variables. If you want, you can write regular procedural code this way. Some examples [1] The real problem of jq is that it is currently lacking a maintainer to assess a number of PRs that have accumulated since 2018. [0] https://github.com/stedolan/jq/wiki/jq-Language-Description https://github.com/stedolan/jq/wiki/jq-Language-Description [1] https://github.com/fadado/JBOL/blob/master/fadado.github.io/array/array.jq https://github.com/fadado/JBOL/blob/master/fadado.github.io/...

klysm 4y ago
I didn't realize jq was missing a maintainer, it's one of my most used CLI tools.
- ethanwillis 4y ago
  It really is a fundamental problem where lots of these important projects aren't maintained simply because the reality is the maintainers can't beat the economics of a lot of rich freeloaders having no real short term incentive to compensate these maintainers..
  - lenkite 4y ago
    Pity he quit before Github opened up sponsorships.
    - da39a3ee 4y ago
      How would Github sponsorship help pay for the time of someone at a finance company earning who knows where between 200-500k?
      - lenkite 4y ago
        My bad. I guess earning ~$100k is in-sufficient. https://changelog.com/posts/i-just-hit-100000-per-year-on-github-sponsors https://changelog.com/posts/i-just-hit-100000-per-year-on-gi...
        da39a3ee 4y ago
        Cool, I think I read that also! But, yeah it is on the low-end for a skilled engineer working for US companies, and also I'm thinking it's on the high-end / an outlier for github sponsors.
  - jahewson 4y ago
    It’s not though, because in this case the (ex-)maintainer works at a Wall St firm.
    - ethanwillis 4y ago
      This is exactly my point. Will he quit the paying job to work for free? How long could he maintain this for free with no other job, or with a job and additional free hours without running out of money or burning out?
      - lillecarl 4y ago
        But everyone can't realistically live a wealthy life off FLOSS tools either. The people who write these things are usually very talented and will make killer pay anywhere they work. Usually a cool thing is when companies sponsor them to work X% for them and Y% on the FLOSS tool.
  - avgcorrection 4y ago
    > can't beat the economics This makes it sound like this is some antagonistic relationship where the OSS maintainer loses. But the idealistic scenario that you are alluding to[1] is about a developer who develops free OSS in their free time. And then, yes, very few end up paying or donating anything. But how is a predictable chain of events a loss? What is the “economics” of it? [1] Some OSS developers do it as their day job.
    - ethanwillis 4y ago
      This is unrelated to the argument but using references that aren't references made that really confusing to read. In any case, what I meant by the "economics" of it is that in general a person can only afford to work for free for so long before they need to pay bills, eat, have and/or acquire a standard of living that isn't poverty. If they have a day job where they are writing this software in their free time, how long can they do this before burning out?
      - avgcorrection 4y ago
        You say that this is unrelated yet your follow-up reinforces my initial impression. How does one afford to work for free? One has a day job. How does someone who volunteers for search-and-rescue afford it? That’s obviously a ridiculous question—they are volunteers so they necessarily must do something from nine to five. Or be independently wealthy. But how does one avoid burnout as a double-worked programmer? I think we have ourselves to blame on that point since we have put the double-worked programmer on a pedestal. So we can either: 1. Not work on things both professionally and in our free time; or 2. Force ourselves to do just that because we gain something extrinsic from it that we might need, like simply keeping up with the Joneses (having an answer for “where’s your private GitHub” in interviews…)
        ethanwillis 4y ago
        When I said "this is unrelated" I was commenting SOLELY on your writing style. > How does one afford to work for free? This is exactly my point. The work isn't done for free, the person is spending their own money and time which takes away from a limited pool of resources they own. If they're insanely rich, they could probably "afford" to do this work until they die. But you're making a mistake in your reasoning relating "volunteers", "free work", and "day jobs." Here is what I think you are missing in this assessment: A worker for a company/day job works with an obligation via contract for compensation for their time from their employer. A volunteer works without a contractual obligation of compensation for their time from the community that benefits from their work. In this latter case while there is no contractual obligation for a society/community to compensate the volunteer, it does not forbid it. Does someone who works as a volunteer search-and-rescue deserve to be compensated? I'd say yes, in fact, they do. They are providing a service. Now I'll get ahead of the next possible argument. "But there's not enough work or compensation for them to make a living!" This is two parts: 1. For not enough work - This is only true because of the example chosen and our human tendency to draw broad analogies. There can definitely be enough work in multiple domains (and especially in software) but also what about volunteer firefighters? 2. Not enough compensation - This is because people with the means to compensate the work, simply are not doing that. And it's not a good faith argument to tell me that in the original case enough people with enough money aren't using the project to compensate it's continued development and maintenance. To sum up all of the above: Yes, work like this is volunteer work, and it says a lot that societies and communities do not compensate this work. Simply because they don't compensate that work doesn't mean it's not able to be compensated. And there are key differences between this relationship of work and compensation that make it different from the colloquial "work" as in a day job. As this is an entity reserving your time under contract. Now for burnout as a double-worked programmer. I think you're right on these two points. Obviously the second situation is not ideal. If someone wants to do it, let them. There are plenty of open source projects still maintained by stretched thin developers. Is this a tenable solution long term? No in the vast majority of cases and that's my point!
- skybrian 4y ago
  In this case it doesn't seem too critical? It means jq remains stable, which is probably what should happen once a tool like this gets a lot of users.
jdnier 4y ago
Here's podcast interview with the creator of jq about what he's been working on at Jane Street: https://signalsandthreads.com/memory-management/ https://signalsandthreads.com/memory-management/
Beltalowda 4y ago
> It's not as arcane as it seems. The issue with jq is that I use it maybe once a month, or even less. The syntax is "arcane enough" that I keep forgetting how to use it because I use it so sporadically. In comparison awk – which I also don't use that often – has a much easier syntax that I can mostly remember. Not entirely convinced by the zq syntax either though; it also seems "arcane enough" that I would keep forgetting it.
- taude 4y ago
  Same issue. However, I do successfully rely on using ctrl-r a lot to search prior invoked commands. And have a few core aliases that I've cobbled together....
  - rgoodwintx 4y ago
    Here because.... I didn't know of ctrl-R. What a life changer (although I had an alias for "hg" to "history | grep" :) )
    - kalev 4y ago
      Please check FZF [1] and it’s integration with ctrl-r. It’s a huge productivity boost and I cannot live without it. [1] https://github.com/junegunn/fzf https://github.com/junegunn/fzf
      - lillecarl 4y ago
        There's also McFly[1] that does interactive history search. [1]: https://github.com/cantino/mcfly https://github.com/cantino/mcfly
        mathfailure 4y ago
        This sounds quite awesome, it's a pity that the realization is quite 'alien' to unix way of things. IMO shell commands history should be a database by default, not just a single file that gets auto-appended with both awesome commands and crap you don't really want to get stored for later. Thanks for linking this project, will try it, it may be a game changer.
j1elo 4y ago
Sadly very few authors seem to acknowledge or even know that github wiki pages are not indexed by search engines so if it wasn't for third-party sites like github-wiki-see.page (which could stop working at any time) their contents would be undiscoverable by the very same people they are usually intended...
- oblio 4y ago
  What? That's crazy! Does Github block indexing?
  - beembeem 4y ago
    https://github.com/robots.txt https://github.com/robots.txt I don't see anything here about wiki specifically but maybe one of the rules hits wiki pages?
    - crazysim 4y ago
      They've moved from robots.txt to blocking by headers.
  - bckygldstn 4y ago
    There's more details on https://github-wiki-see.page/ https://github-wiki-see.page/ and https://github.com/github/feedback/discussions/4992#discussioncomment-1448177 https://github.com/github/feedback/discussions/4992#discussi... > we have also introduced an x-robots-tag: none in the http response header of Wiki pages > Abusive behavior in Wikis had a negative impact on our search engine ranking > GitHub is currently permitting a select criteria of GitHub Wikis to be indexed
adamgordonbell 4y ago
I found it hard to approach at first, but I think it was just the lack of material that worked through simple examples step by step. I ended up writing my own guide to it, that in my unbiased opinion makes it easier to get the point where in-depth examples and language descriptions are easier to understand. Edit: Oh, wow, it's even mentioned in this article. Maybe I should read before commenting. https://earthly.dev/blog/jq-select/ https://earthly.dev/blog/jq-select/
- sfink 4y ago
  I discovered jq after I wrote my own (extremely limited) version of it. I need it quite often, and yet I've never managed to get up the activation energy to learn enough for it to be useful. I need to have some notion of the computation model before anything is going to make sense to me. I hate learning things in completely disparate pieces that I need to memorize in hopes that someday it will just click together and I'll derive the underlying principles. Your guide was great for this. It stepped me through enough of the bare basics in a way that the underlying model was obvious. It didn't get me nearly far enough for many of the tasks that I need jq for, but it got me started and that's all I really needed. Everything additional that I need to learn becomes obvious in retrospect—"of course there's an operator for this, there kind of has to be!". Thank you!
deleted 4y ago
[deleted]
dilap 4y ago
From that page: > The jq documentation is written in a style that hides a lot of important detail because the hope is that the language feels intuitive. Yeah, not so much boys! Also, that disclaimer should really be at the top of the manual, with a link to the wiki, rather than vice-versa, as it is now. The wiki is like secret information -- "oh, hey, here's the page that actually tells you how it works!"

klysm 4y ago

I didn't realize jq was missing a maintainer, it's one of my most used CLI tools.

ethanwillis 4y ago
It really is a fundamental problem where lots of these important projects aren't maintained simply because the reality is the maintainers can't beat the economics of a lot of rich freeloaders having no real short term incentive to compensate these maintainers..
- lenkite 4y ago
  Pity he quit before Github opened up sponsorships.
  - da39a3ee 4y ago
    How would Github sponsorship help pay for the time of someone at a finance company earning who knows where between 200-500k?
    - lenkite 4y ago
      My bad. I guess earning ~$100k is in-sufficient. https://changelog.com/posts/i-just-hit-100000-per-year-on-github-sponsors https://changelog.com/posts/i-just-hit-100000-per-year-on-gi...
      - da39a3ee 4y ago
        Cool, I think I read that also! But, yeah it is on the low-end for a skilled engineer working for US companies, and also I'm thinking it's on the high-end / an outlier for github sponsors.
- jahewson 4y ago
  It’s not though, because in this case the (ex-)maintainer works at a Wall St firm.
  - ethanwillis 4y ago
    This is exactly my point. Will he quit the paying job to work for free? How long could he maintain this for free with no other job, or with a job and additional free hours without running out of money or burning out?
    - lillecarl 4y ago
      But everyone can't realistically live a wealthy life off FLOSS tools either. The people who write these things are usually very talented and will make killer pay anywhere they work. Usually a cool thing is when companies sponsor them to work X% for them and Y% on the FLOSS tool.
- avgcorrection 4y ago
  > can't beat the economics This makes it sound like this is some antagonistic relationship where the OSS maintainer loses. But the idealistic scenario that you are alluding to[1] is about a developer who develops free OSS in their free time. And then, yes, very few end up paying or donating anything. But how is a predictable chain of events a loss? What is the “economics” of it? [1] Some OSS developers do it as their day job.
  - ethanwillis 4y ago
    This is unrelated to the argument but using references that aren't references made that really confusing to read. In any case, what I meant by the "economics" of it is that in general a person can only afford to work for free for so long before they need to pay bills, eat, have and/or acquire a standard of living that isn't poverty. If they have a day job where they are writing this software in their free time, how long can they do this before burning out?
    - avgcorrection 4y ago
      You say that this is unrelated yet your follow-up reinforces my initial impression. How does one afford to work for free? One has a day job. How does someone who volunteers for search-and-rescue afford it? That’s obviously a ridiculous question—they are volunteers so they necessarily must do something from nine to five. Or be independently wealthy. But how does one avoid burnout as a double-worked programmer? I think we have ourselves to blame on that point since we have put the double-worked programmer on a pedestal. So we can either: 1. Not work on things both professionally and in our free time; or 2. Force ourselves to do just that because we gain something extrinsic from it that we might need, like simply keeping up with the Joneses (having an answer for “where’s your private GitHub” in interviews…)
      - ethanwillis 4y ago
        When I said "this is unrelated" I was commenting SOLELY on your writing style. > How does one afford to work for free? This is exactly my point. The work isn't done for free, the person is spending their own money and time which takes away from a limited pool of resources they own. If they're insanely rich, they could probably "afford" to do this work until they die. But you're making a mistake in your reasoning relating "volunteers", "free work", and "day jobs." Here is what I think you are missing in this assessment: A worker for a company/day job works with an obligation via contract for compensation for their time from their employer. A volunteer works without a contractual obligation of compensation for their time from the community that benefits from their work. In this latter case while there is no contractual obligation for a society/community to compensate the volunteer, it does not forbid it. Does someone who works as a volunteer search-and-rescue deserve to be compensated? I'd say yes, in fact, they do. They are providing a service. Now I'll get ahead of the next possible argument. "But there's not enough work or compensation for them to make a living!" This is two parts: 1. For not enough work - This is only true because of the example chosen and our human tendency to draw broad analogies. There can definitely be enough work in multiple domains (and especially in software) but also what about volunteer firefighters? 2. Not enough compensation - This is because people with the means to compensate the work, simply are not doing that. And it's not a good faith argument to tell me that in the original case enough people with enough money aren't using the project to compensate it's continued development and maintenance. To sum up all of the above: Yes, work like this is volunteer work, and it says a lot that societies and communities do not compensate this work. Simply because they don't compensate that work doesn't mean it's not able to be compensated. And there are key differences between this relationship of work and compensation that make it different from the colloquial "work" as in a day job. As this is an entity reserving your time under contract. Now for burnout as a double-worked programmer. I think you're right on these two points. Obviously the second situation is not ideal. If someone wants to do it, let them. There are plenty of open source projects still maintained by stretched thin developers. Is this a tenable solution long term? No in the vast majority of cases and that's my point!
skybrian 4y ago
In this case it doesn't seem too critical? It means jq remains stable, which is probably what should happen once a tool like this gets a lot of users.

jdnier 4y ago

Here's podcast interview with the creator of jq about what he's been working on at Jane Street: https://signalsandthreads.com/memory-management/ https://signalsandthreads.com/memory-management/

Beltalowda 4y ago

> It's not as arcane as it seems. The issue with jq is that I use it maybe once a month, or even less. The syntax is "arcane enough" that I keep forgetting how to use it because I use it so sporadically. In comparison awk – which I also don't use that often – has a much easier syntax that I can mostly remember. Not entirely convinced by the zq syntax either though; it also seems "arcane enough" that I would keep forgetting it.

taude 4y ago
Same issue. However, I do successfully rely on using ctrl-r a lot to search prior invoked commands. And have a few core aliases that I've cobbled together....
- rgoodwintx 4y ago
  Here because.... I didn't know of ctrl-R. What a life changer (although I had an alias for "hg" to "history | grep" :) )
  - kalev 4y ago
    Please check FZF [1] and it’s integration with ctrl-r. It’s a huge productivity boost and I cannot live without it. [1] https://github.com/junegunn/fzf https://github.com/junegunn/fzf
    - lillecarl 4y ago
      There's also McFly[1] that does interactive history search. [1]: https://github.com/cantino/mcfly https://github.com/cantino/mcfly
      - mathfailure 4y ago
        This sounds quite awesome, it's a pity that the realization is quite 'alien' to unix way of things. IMO shell commands history should be a database by default, not just a single file that gets auto-appended with both awesome commands and crap you don't really want to get stored for later. Thanks for linking this project, will try it, it may be a game changer.

j1elo 4y ago

Sadly very few authors seem to acknowledge or even know that github wiki pages are not indexed by search engines so if it wasn't for third-party sites like github-wiki-see.page (which could stop working at any time) their contents would be undiscoverable by the very same people they are usually intended...

oblio 4y ago
What? That's crazy! Does Github block indexing?
- beembeem 4y ago
  https://github.com/robots.txt https://github.com/robots.txt I don't see anything here about wiki specifically but maybe one of the rules hits wiki pages?
  - crazysim 4y ago
    They've moved from robots.txt to blocking by headers.
- bckygldstn 4y ago
  There's more details on https://github-wiki-see.page/ https://github-wiki-see.page/ and https://github.com/github/feedback/discussions/4992#discussioncomment-1448177 https://github.com/github/feedback/discussions/4992#discussi... > we have also introduced an x-robots-tag: none in the http response header of Wiki pages > Abusive behavior in Wikis had a negative impact on our search engine ranking > GitHub is currently permitting a select criteria of GitHub Wikis to be indexed

adamgordonbell 4y ago

I found it hard to approach at first, but I think it was just the lack of material that worked through simple examples step by step. I ended up writing my own guide to it, that in my unbiased opinion makes it easier to get the point where in-depth examples and language descriptions are easier to understand. Edit: Oh, wow, it's even mentioned in this article. Maybe I should read before commenting. https://earthly.dev/blog/jq-select/ https://earthly.dev/blog/jq-select/

sfink 4y ago
I discovered jq after I wrote my own (extremely limited) version of it. I need it quite often, and yet I've never managed to get up the activation energy to learn enough for it to be useful. I need to have some notion of the computation model before anything is going to make sense to me. I hate learning things in completely disparate pieces that I need to memorize in hopes that someday it will just click together and I'll derive the underlying principles. Your guide was great for this. It stepped me through enough of the bare basics in a way that the underlying model was obvious. It didn't get me nearly far enough for many of the tasks that I need jq for, but it got me started and that's all I really needed. Everything additional that I need to learn becomes obvious in retrospect—"of course there's an operator for this, there kind of has to be!". Thank you!

deleted 4y ago

[deleted]

dilap 4y ago

From that page: > The jq documentation is written in a style that hides a lot of important detail because the hope is that the language feels intuitive. Yeah, not so much boys! Also, that disclaimer should really be at the top of the manual, with a link to the wiki, rather than vice-versa, as it is now. The wiki is like secret information -- "oh, hey, here's the page that actually tells you how it works!"