3 ms·
Not anywhere near as sophisticated as yours but I have something vaguely similar for simplifying JSON documents (while maintaining what the data also looks like
by petercooper 1y ago
Not anywhere near as sophisticated as yours but I have something vaguely similar for simplifying JSON documents (while maintaining what the data also looks like) for feeding to LLMs to help them code against:
jq 'walk(if type == "array" then (if length > 0 then [.[0]] else . end) else . end)'
So that 70,000+ line Amazon example of yours would boil down to:
{
"syncToken": "1753114994",
"createDate": "2025-07-21-16-23-14",
"prefixes": [
{
"ip_prefix": "3.4.12.4/32",
"region": "eu-west-1",
"service": "AMAZON",
"network_border_group": "eu-west-1"
}
],
"ipv6_prefixes": [
{
"ipv6_prefix": "2600:1f69:7400::/40",
"region": "mx-central-1",
"service": "AMAZON",
"network_border_group": "mx-central-1"
}
]
}
.. which is easier/cheaper to feed to an LLM for getting it to write code to process, etc. than the multi-megabyte original.
- rwiggins 1y agoOh wow, that's fantastic. I love that it includes real values while still summarizing the doc's structure. I'm going to steal that. I'll probably keep jq-structure around because it's so easy to copy/paste paths I'm looking for, but yours is definitely better for understanding what the JSON doc actually contains.
- naniwaduni 1y agoGot a bit nerd-sniped here, but first of all we can reduce if A then B else . end === if A then B end since jq 1.7: jq 'walk(if type == "array" then (if length > 0 then [.[0]] end) end)' Now we could contract those conditionals: jq 'walk(if type == "array" and length > 0 then [.[0]] end)' but it turns out we can even more usefully express if length > 0 then [.[0]] end === [limit(1; .[])] == .[:1]: jq 'walk(if type == "array" then .[:1] end)' From here, we can golf it a little further (this is kind of a generic type-matching pattern): jq 'walk(arrays[:1] // .)' although this does incur a bit more overhead than checking type directly. Speaking of overhead, though, it turns out that the implementation of walk/1 (https://github.com/jqlang/jq/blob/master/src/builtin.jq#L212 https://github.com/jqlang/jq/blob/master/src/builtin.jq#L212) will actually run the filter on every element of an array, even though we're about to throw most of them out, which we can eliminate by writing the recursion explicitly: jq 'def w: if type=="array" then [limit(1; .[]|w)] elif type=="object" then .[] |= w end; w' which gets the operation down from ~200 ms on my machine (not long enough to really get distracted, but enough to feel the wait) to a perceptually instant ~40 ms (which is mostly just the cost of reading the input). Now we can golf it down a little more: jq 'def w: if type=="array" then [limit(1; .[]|w)] else objects[] |= w end; w' jq 'def w: (arrays[:1]|map(w)) // (objects[] |= w); w' (the precedence here actually allows us to eliminate the parens here...) jq 'def w: arrays |= .[:1]|iterables[] |= w; w' And, inaccessibility of the syntax aside, I think this does an incredible job of expressing the essence of what we're trying to do: we trim any array down to its first element, and then recursively apply the same transformation throughout the structure. jq is a very expressive language, it just looks like line noise...
- Bluestein 1y agoHat off.- PS. Also, if I may l, thanks for the walkthrough - I'd be clapping with just the short form at the end, but the reasoning is appreciated.-