13 ms·

That's super weird, I think most people use jq for bash pipelines.

by f311a 6y ago

jeffbee 6y ago
Yes, I don't understand how people end up with assertions that the filename is a require argument. At least we've got /dev/stdin or /proc/self/fd/0 as workarounds.
- benibela 6y ago
  files might be faster, because you can mmap them?
  - f311a 6y ago
    Most people work on compressed JSON lines files. Sometimes they are stored on s3. Files do not give flexibility. When using jq, I can do a lot of things: aws s3 cp s3://bucket/file.json.gz - | zcat | head | jq .field | sort
    - jerf 6y ago
      Another common thing you can do is accept a generic stream as input, but have some code that penetrates the abstraction a bit to see what kind of stream it is, and if it is a file or something, do something special with it to go even faster. This way, you start with something maximally useful up front, and easy to use, but you can optimize things based on details as you go. That's how Go's static file web server works. It serves streams, but if you happen to io.Copy to that stream with something that is also an ∗os.File on Linux, it can use the sendfile call in the kernel instead. (A downside of making it so transparent is that if you wrap that stream with something you may not realize that you've wrecked the optimization because it no longer unwraps to an ∗os.File but whatever your wrapper is, but, well, nothing's perfect.)
    - Spivak 6y ago
      Which is sorta-kinda the same thing as jq .field <(aws s3 cp s3://bucket/file.json.gz - | zcat | head) | sort which is more annoying to type but works.
      - dullgiulio 6y ago
        Redirection is not a parameter, meaning that would still not work with this Q tool.
        jraph 6y ago
        This is not a simple redirection. cmd <(subcmd) is a bashism that redirects the output of command subcmd to something that looks like a regular file to command cmd. Command cmd receive a path at the place the <(subcmd) syntax is used. Different from cmd < f, which redirects the contents of file f to cmd's input. So, this should work :-)
        jeffbee 6y ago
        Works in other shells, not just bash % cat /proc/self/cmdline <(echo $SHELL) | tr '\0' ' ' cat /proc/self/cmdline /proc/self/fd/11 /bin/zsh
        jraph 6y ago
        Yep! Zsh supports a lot of bashisms :-) It won't work in dash though, and you should not use this in a shell that targets POSIX.
- caymanjim 6y ago
  Much Unix software today is written by people who really don't appreciate or understand Unix. It leads to things like Homebrew (early on, at least) completely taking over and breaking /usr/local; to command-line utilities using a single dash (-) precending both short and long option names (granted, --long-opts is a GNUism, but it's a well-established standard); commands that output color by default even when the output isn't a tty; etc. It's not hard to fix things like this, but it exemplifies a lack of familiarity with the Unix command line. There are an enormous number of tools out there that only exist because people don't know how to chain together basic 1970s Unix text-processing tools in a pipeline.
  - jeffbee 6y ago
    The `go` program uses -longopts and I think it would be hard to argue that Rob Pike lacks an appreciation of Unix traditions.
    - bfrydl 6y ago
      I would argue that Go's design as a whole is characterized by an attitude of ignoring established ideas for no other reason than that they think they know better.
      - thatjamesdude 6y ago
        Something being established is not a grand argument for it's usage. The reasons it got established are relavent, and if you feel the end result of said establishment is obtuse or inane, why would you use it? That's not to say Go's decisions to toss some established practices are "wise" or "sagely", just that broad acceptance is not a criteria they seemed concerned with. Which is fine. >they think they know better. It's safe to say Rob Pike is not clueless or without experience in unix tooling. You should listen to some of his experiences and thoughts with designing Go [0]. I don't always agree with him, [but it's very baseless to suggest he makes decisions on the grounds that they were his, not they have merrit.] Edit to clarify: [He makes decisions on merrit over authority] [0]: https://changelog.com/gotime/100 https://changelog.com/gotime/100
ashtonkem 6y ago
My most common jq usage is to copy and paste some json into quotes to make it easier to read. My second most common action is to chain curl and jq together. A replacement for jq that doesn’t use stdin is literally useless to me.
- yawaramin 6y ago
  I think at least two people in the thread pointed out that you can use /dev/stdin as a file.
  - monadic2 6y ago
    It's more than a little strange that isn't default behavior out of the box without specifying input. How do the authors think jq is used?
    - yawaramin 6y ago
      I'm pretty sure the author has also replied elsewhere saying he will add it since it was brought up as a concern.
    - davesnx 6y ago
      I trend to use jq with a file all the time, I prefer to curl first and later operate on top. /slug
  - ashtonkem 6y ago
    That’s a cumbersome extra step for unclear benefit. Yes, q is supposedly faster than jq. But it is exceedingly rare for me to ever have any performance problems with jq, especially since it’s essentially a one off utility I use occasionally, not as part of the hot loop of any workflow where performance matters.
    - yawaramin 6y ago
      In that case I suppose you wouldn't use it.
kilroy123 6y ago
Probably true but I use it regularly in the automation app huginn. https://github.com/huginn/huginn https://github.com/huginn/huginn

That's super weird, I think most people use jq for bash pipelines.

by f311a 6y ago

That's super weird, I think most people use jq for bash pipelines.

Yes, I don't understand how people end up with assertions that the filename is a require argument. At least we've got /dev/stdin or /proc/self/fd/0 as workarounds.

benibela 6y ago
files might be faster, because you can mmap them?
- f311a 6y ago
  Most people work on compressed JSON lines files. Sometimes they are stored on s3. Files do not give flexibility. When using jq, I can do a lot of things: aws s3 cp s3://bucket/file.json.gz - | zcat | head | jq .field | sort
  - jerf 6y ago
    Another common thing you can do is accept a generic stream as input, but have some code that penetrates the abstraction a bit to see what kind of stream it is, and if it is a file or something, do something special with it to go even faster. This way, you start with something maximally useful up front, and easy to use, but you can optimize things based on details as you go. That's how Go's static file web server works. It serves streams, but if you happen to io.Copy to that stream with something that is also an ∗os.File on Linux, it can use the sendfile call in the kernel instead. (A downside of making it so transparent is that if you wrap that stream with something you may not realize that you've wrecked the optimization because it no longer unwraps to an ∗os.File but whatever your wrapper is, but, well, nothing's perfect.)
  - Spivak 6y ago
    Which is sorta-kinda the same thing as jq .field <(aws s3 cp s3://bucket/file.json.gz - | zcat | head) | sort which is more annoying to type but works.
    - dullgiulio 6y ago
      Redirection is not a parameter, meaning that would still not work with this Q tool.
      - jraph 6y ago
        This is not a simple redirection. cmd <(subcmd) is a bashism that redirects the output of command subcmd to something that looks like a regular file to command cmd. Command cmd receive a path at the place the <(subcmd) syntax is used. Different from cmd < f, which redirects the contents of file f to cmd's input. So, this should work :-)
        jeffbee 6y ago
        Works in other shells, not just bash % cat /proc/self/cmdline <(echo $SHELL) | tr '\0' ' ' cat /proc/self/cmdline /proc/self/fd/11 /bin/zsh
        jraph 6y ago
        Yep! Zsh supports a lot of bashisms :-) It won't work in dash though, and you should not use this in a shell that targets POSIX.
caymanjim 6y ago
Much Unix software today is written by people who really don't appreciate or understand Unix. It leads to things like Homebrew (early on, at least) completely taking over and breaking /usr/local; to command-line utilities using a single dash (-) precending both short and long option names (granted, --long-opts is a GNUism, but it's a well-established standard); commands that output color by default even when the output isn't a tty; etc. It's not hard to fix things like this, but it exemplifies a lack of familiarity with the Unix command line. There are an enormous number of tools out there that only exist because people don't know how to chain together basic 1970s Unix text-processing tools in a pipeline.
- jeffbee 6y ago
  The `go` program uses -longopts and I think it would be hard to argue that Rob Pike lacks an appreciation of Unix traditions.
  - bfrydl 6y ago
    I would argue that Go's design as a whole is characterized by an attitude of ignoring established ideas for no other reason than that they think they know better.
    - thatjamesdude 6y ago
      Something being established is not a grand argument for it's usage. The reasons it got established are relavent, and if you feel the end result of said establishment is obtuse or inane, why would you use it? That's not to say Go's decisions to toss some established practices are "wise" or "sagely", just that broad acceptance is not a criteria they seemed concerned with. Which is fine. >they think they know better. It's safe to say Rob Pike is not clueless or without experience in unix tooling. You should listen to some of his experiences and thoughts with designing Go [0]. I don't always agree with him, [but it's very baseless to suggest he makes decisions on the grounds that they were his, not they have merrit.] Edit to clarify: [He makes decisions on merrit over authority] [0]: https://changelog.com/gotime/100 https://changelog.com/gotime/100

ashtonkem 6y ago

My most common jq usage is to copy and paste some json into quotes to make it easier to read. My second most common action is to chain curl and jq together. A replacement for jq that doesn’t use stdin is literally useless to me.

yawaramin 6y ago
I think at least two people in the thread pointed out that you can use /dev/stdin as a file.
- monadic2 6y ago
  It's more than a little strange that isn't default behavior out of the box without specifying input. How do the authors think jq is used?
  - yawaramin 6y ago
    I'm pretty sure the author has also replied elsewhere saying he will add it since it was brought up as a concern.
  - davesnx 6y ago
    I trend to use jq with a file all the time, I prefer to curl first and later operate on top. /slug
- ashtonkem 6y ago
  That’s a cumbersome extra step for unclear benefit. Yes, q is supposedly faster than jq. But it is exceedingly rare for me to ever have any performance problems with jq, especially since it’s essentially a one off utility I use occasionally, not as part of the hot loop of any workflow where performance matters.
  - yawaramin 6y ago
    In that case I suppose you wouldn't use it.

kilroy123 6y ago

Probably true but I use it regularly in the automation app huginn. https://github.com/huginn/huginn https://github.com/huginn/huginn