wwader

10 ms·

31.
by wwader 2y ago
You might want to look into using jq --stream together with inputs/truncate_stream/fromstream and friends if you want to use jq with large inputs. Not a speed daemon but will probably use a lot less memory.
32.
by wwader 2y ago
I see, thanks for replying and no worries! yeap some of the "self-describing" formats like msgpack, cbor etc will because of how fq works have to be decoded into something more of a meta-msgpack etc. About blobs, if you want to ch
33.
by wwader 2y ago
hope it can be of use!
34.
by wwader 2y ago
Hey, fq author here. Happy to hear it's useful! could you elaborate a bit more how it was less intuitive? fq's query language is jq with some small additions so i wonder if you might mean the decoded structure is more detailed&#x2
35.
by wwader 3y ago
Ah sorry that is probably zsh specific, it makes sure to not glob expand * etc
36.
by wwader 3y ago
I do quite a lot of adhoc/exploratory programming to query and transform data then jq is very convenient as it works very well with "deep" data structures and the language itself it very composable. To debug in jq you can use
37.
by wwader 3y ago
Just had to try: $ function _select_aux () { clickhouse local -q "SELECT $* FORMAT Pretty" } $ alias SELECT='noglob _select_aux' $ SELECT COUNT(*) as count FROM file('repos.json', JSON) ┏━━━━━━━┓ ┃
38.
by wwader 3y ago
Interesting post! when prototyping what would end up being fq i did quite a lot of tinkering with how it could work by splitting it up into multiple tools, and use it like: <decode tool> | <query tool> | <display tool>. Ne
39.
by wwader 3y ago
Hey! fq author here. I have a bunch of related tools in the readme https://github.com/wader/fq?tab=readme-ov-file#tools two suggestions: gnu poke and wireshark (can decode lots of more things then just network protocol
40.
by wwader 3y ago
Yeap i've talked to itchyny quite a lot about various changes https://github.com/itchyny/gojq/issues/153 and also upstreamed quite a lot https://github.com/itchyny/gojq/issues?q
41.
by wwader 3y ago
Did someone say let's represent structured data as json? a bit of shameless plug: https://github.com/wader/fq :) It's using a fork of gojq btw!
42.
by wwader 3y ago
I think https://pkg.go.dev/encoding/json#Decoder do support steaming at least. Here is gojq's stream mode https://github.com/itchyny/gojq/blob/main/cli/stream.go
43.
by wwader 3y ago
Feels fishy and generated somehow. Quite sure there is no @pretty format in jaq or jq and the section about map function does not use map function.
44.
by wwader 3y ago
What might be confusing is that iterating an object iterates its values. add is defined something like this: def add: reduce .[] as $n (0; . + $n)
45.
by wwader 3y ago
If you spend lots of time with certain binary formats then i can recommend adding a decoder, happy to help with it also!
46.
by wwader 3y ago
https://github.com/wader/fq has a REPL and can read JSON. Tip is to use "paste | from_json | repl" in a REPl to paste JSON into a sub-REPL, you can also use `<text here>` with fq which is a raw string l
47.
by wwader 3y ago
Quite a lot! i use it to explode both JSON and tex (parse using jq functions). I also use it for exploring ane debug binary formats ( https://github.com/wader/fq ). Now a days i also use it for some adhoc programming and
48.
by wwader 3y ago
jq 1.7 do preserve large integers but will truncate if any operation is done on them. Unfortunetly it currently truncates to a decimal64 which is a bit confusing, this will be fixed in next release where it follow the suggestion from the JS
49.
by wwader 3y ago
Yeap was fixed in 1.7 https://github.com/jqlang/jq/pull/2646
50.
by wwader 3y ago
Great to hear and that was one of my hopes! but honestly it initially came to be because i was fiddling with some jq AST-tree stuff for fq :) weirdly it was much easier to implement than i expected. Hardest part was how to handle infix oper
51.
by wwader 3y ago
Thanks for the jqjq shoutout! :) i'm quite sure jq is turing complete, jq (and jqjq!) can implement brainfuck https://github.com/01mf02/jaq/blob/main/examples/bf.jq
52.
by wwader 3y ago
I think it's quite nice and intuitive to describe a pipe of filters as: filter | filter | filter. Also note that a single filter in jq is 1:N, 1 input N output where N can be zero which is a very useful feature. To express this in pyth
53.
by wwader 3y ago
Can recommend the documentary It's Quieter in the Twilight https://www.imdb.com/title/tt17658964/ that follows the engineers working with voyager 1 and 2
54.
by wwader 3y ago
Some joke/useless software i've written: - https://github.com/wader/jqjq - https://github.com/wader/flac.tcl In most cases i learned way more then i expected. Even made some friends.
55.
by wwader 3y ago
yeap that could be an extension of the shorthand syntax, but note that pick/1 also works with any "path expression" ex: pick(if .a then .b.c else .x end)
56.
by wwader 3y ago
If the task was the produce an object with the top 10 commiters with username as key and value number of commits maybe something liket this: group_by(.commit.author.name) | map({key: first.commit.author.name, value: length}) | sort_by
57.
by wwader 3y ago
jq support regex so now you can mix!
58.
by wwader 3y ago
I do lots of exploratory work in various structure data, in my case often debugging media filea via https://github.com/wader/fq , which mean doing lots of use-once-queries on the command line or REPL. In those cases jq
59.
by wwader 3y ago
jq has regex support! :)
60.
by wwader 3y ago
Nothing planned but there has been some talks about it. https://github.com/jqlang/jq/issues/2855 is probably the lastest issue mentioning it

More ›