3 ms·
Most people work on compressed JSON lines files. Sometimes they are stored on s3. Files do not give flexibility. When using jq, I can do a lot of things:
by f311a 6y ago
Most people work on compressed JSON lines files. Sometimes they are stored on s3. Files do not give flexibility.
When using jq, I can do a lot of things:
aws s3 cp s3://bucket/file.json.gz - | zcat | head | jq .field | sort
- jerf 6y agoAnother common thing you can do is accept a generic stream as input, but have some code that penetrates the abstraction a bit to see what kind of stream it is, and if it is a file or something, do something special with it to go even faster. This way, you start with something maximally useful up front, and easy to use, but you can optimize things based on details as you go. That's how Go's static file web server works. It serves streams, but if you happen to io.Copy to that stream with something that is also an ∗os.File on Linux, it can use the sendfile call in the kernel instead. (A downside of making it so transparent is that if you wrap that stream with something you may not realize that you've wrecked the optimization because it no longer unwraps to an ∗os.File but whatever your wrapper is, but, well, nothing's perfect.)
- Spivak 6y agoWhich is sorta-kinda the same thing as jq .field <(aws s3 cp s3://bucket/file.json.gz - | zcat | head) | sort which is more annoying to type but works.
- dullgiulio 6y agoRedirection is not a parameter, meaning that would still not work with this Q tool.
- jraph 6y agoThis is not a simple redirection. cmd <(subcmd) is a bashism that redirects the output of command subcmd to something that looks like a regular file to command cmd. Command cmd receive a path at the place the <(subcmd) syntax is used. Different from cmd < f, which redirects the contents of file f to cmd's input. So, this should work :-)