8 ms·

Jtc – CLI tool to extract, manipulate and transform source JSON

122 points by vyuh 7y ago

alanbernstein 7y ago
Can anyone compare to jq? https://stedolan.github.io/jq/ https://stedolan.github.io/jq/
- vyuh 7y ago
  https://github.com/ldn-softdev/jtc#jtc-vs-jq https://github.com/ldn-softdev/jtc#jtc-vs-jq
meddlepal 7y ago
> jtc is written in idiomatic C++ (the most powerful programming language to date) Citation needed. The `jq` vs `jtc` section is interesting, but author seems a little full of himself with some of the explanations.
- ur-whale 7y ago
  I do agree with the author that the learning investment required by jq (learning a full blown DSL) is total overkill for most simple JSON based tasks. For those, I mostly use "jq ." to get all leaves on a single line and then feed the output to standard unix tools like grep cut awk and friends. And for more complex tasks, python, perl, even C++ if speed is needed.
  - unspecified 7y ago
    > For those, I mostly use "jq ." to get all leaves on a single line and then feed the output to standard unix tools like grep cut awk and friends. There's a tool called `gron` designed for workflows like this, I've found it incredibly useful. https://news.ycombinator.com/item?id=16727665 https://news.ycombinator.com/item?id=16727665 https://github.com/tomnomnom/gron/ https://github.com/tomnomnom/gron/
  - jolmg 7y ago
    > the learning investment required by jq (learning a full blown DSL) is total overkill for most simple JSON based tasks. There's no reason why you would need to learn the whole DSL for simple tasks. Just learn what you need. For simple stuff, the DSL is also simple. jtc also seems to have its own DSL by the way, and it doesn't really seem more intuitive than jq's: jtc -x'[0][:][name]<person>v [-1][children]<kids:false>f[0]<kids:true>v' -T'{"{person} has children":{kids}}' -r That's apparently the equivalent of: jq '.Directory | map({"\(.name) has children": (.children | length == 0)}) | .[]' There are some things that seem simpler on jtc, though, like searches of string values without regard to the json structure.
    - zeroimpl 7y ago
      I’d really like a CLI tool which solves this problem using a syntax I already know. I struggle with jq because I use it quite infrequently. IMO the ideal solution is something using pure JavaScript syntax, possibly with a library resembling jQuery for tree traversal.
      - jolmg 7y ago
        Most of jq's syntax is just a mix of JavaScript and a typical shell's. Every expression has an input and output like a shell command's stdin and stdout. Functions work on arguments and their stdin to produce their stdout. Here's an explanation of the syntax in the command I posted: jq ' # output the value at "Directory" from input object .Directory # pipe to map (JavaScript also has map()). The argument of map works in # the context of each element in map's input array. | map( # Produce an object where the property name is a string that # interpolates the value of the "name" property of this element. # Instead of interpolating, we could have also used this more # JavaScript-ish (ES5) syntax: # # {(.name + " has children"): ... # # The property value is an expression that gets the value of the # "children" property and pipes it to the expression `length != 0`. # `length` (which JavaScript also has) outputs the length of the # piped input, and then we compare that with 0. {"\(.name) has children": (.children | length != 0)} ) # map's output is a single record which is an array. We pipe that to # .[] to make multiple records, each an element of the array. The # syntax here is comprised of 2 parts: `.`, which is the input object, # and `[]` which is the subscript syntax without an index. | .[] ' > IMO the ideal solution is something using pure JavaScript syntax The greatest advantage of the current syntax is the great balance it has between legibility and terseness. I don't think making it pure JavaScript would be better. > possibly with a library resembling jQuery for tree traversal Using jQuery in JavaScript to traverse JavaScript objects? I don't know what to say...
- deleted 7y ago
  [deleted]
- kbenson 7y ago
  Oh yeah, immediately above that line: > jq is written in C, which drags all intrinsic problems the language has dated its creation And then to follow on the C++ claims: > Main JSON engine/library does not have a single new operator, nor it has a single naked pointer acting as a resource holder/owner, thus jtc is guaranteed to be free of memory leaks (at least one class of the problems is off the table) - STL guaranty. That's a lot more faith than I would be willing to put into C++ or C. Sure, that claim might be correct, but there's enough edge cases and undefined behavior in the language that I take it as a fairly bold claim unless it's also been thoroughly reviewed for any places some undefined behavior may be induced. I mean, it's probably fine, but just the willingness to make such a bold claim in that way communicates the opposite of what the author likely intended for me.
  - hyperpallium 7y ago
    > jq is written in C, which drags all intrinsic problems the language has dated its creation I can't even parse this sentence. What does it mean? FWIW jq was originally written in Haskell, then ported to C. The C source for the core parts is strikingly clean. I think, it's partly from this Haskell heritage that jq gets its nice composability. However, TBF, I do find I must refresh jq syntax whenever I use it (pr only bc I don't process json often). It will be interesting to see what jtc does - though this foreshadowing does not bode well...
  - otabdeveloper4 7y ago
    > "undefined behaviour" Yeah, you don't know what you're talking about. "Undefined behavior" simply means "stuff not covered by the ISO standard". So Python and Rust are 100% UB, and people don't really care.
    - kbenson 7y ago
      Undefined behavior can cause perfectly functioning code in one compiler version to be completely elided in a different version, as we've seen numerous times in the past. Review the article and comments here[1] for a good example of exactly this and discussion about the how and why, but the TL;DR is that a specific optimization in a specific compiler release determined that the adding of two variables, one of which a pointer, would result in an overflow which is undefined for pointers, and thus elided the check to see if the result was less than should be possible (overflowed), and the entire error handling case it resulted in. This caused the value to be used while invalid (as the error check and handling was elided), causing a segfault. Perhaps the features used in this project protect against that, but my point is that undefined behavior and the way compilers deal with it is variable and problematic. 1: https://news.ycombinator.com/item?id=14163111 https://news.ycombinator.com/item?id=14163111
vyuh 7y ago
The author has also written related tools. One to convert XML to JSON and back (https://github.com/ldn-softdev/jtm https://github.com/ldn-softdev/jtm) and another to convert JSON to SQLite tables (https://github.com/ldn-softdev/jsl https://github.com/ldn-softdev/jsl). Combining these with the hxnormalize tool ( https://www.w3.org/Tools/HTML-XML-utils/man1/hxnormalize.html https://www.w3.org/Tools/HTML-XML-utils/man1/hxnormalize.htm...), one can do very sophisticated manipulation on HTML web pages. HTML -> XML (via hxnormalize) -> JSON (via jtm) -> process using jtc (or even jq)
- peterwwillis 7y ago
  > convert XML to JSON and back This is basically impossible to do in a way that is compatible with other tools. Things like duplicate attributes of an object can exist in XML, but not in JSON. You can still work-around these limitations if you just have a pipeline using the same toolset, but part of the point of these tools is to then convert them back to a format that some other tool can use, which is where this pattern breaks down. Here's a list of pitfalls: https://stackoverflow.com/questions/33072812/potential-problems-of-mapping-json-to-xml https://stackoverflow.com/questions/33072812/potential-probl...
- namibj 7y ago
  This suggest a very scalable, easy approach to extract data from somewhat regular HTML...
  - benibela 7y ago
    or just use xpath
  - Yetanfou 7y ago
    I generally use xidel [1] for that type of task. Feed it xpath, css selectors or its own pattern matching thing. [1] https://github.com/benibela/xidel https://github.com/benibela/xidel
pmoriarty 7y ago
At first I was a little skeptical of this tool's claims of being simpler than jq, as I struggled to understand why (for instance) they were using <> in some parts of their example and [] in others, but after going through their step-by-step explanation it all made sense, and it does seem very simple. A tool that simplifies things is always welcome.
- bradknowles 7y ago
  The "User Guide.md" file is over 3000 lines long, comprising some 133KB. I read through about a quarter of it, before my eyes started to glaze over. I'm not convinced that I would call this "simpler" than jq.
  - ak217 7y ago
    jq is one of the most intentionally designed pieces of software I've seen. In due time, I expect it to become as ubiquitous and indispensable as ls, grep, sed and awk. The jq language could use a formal spec, but other than that jq has a sublime elegance. By contrast, this seems like one of the least intentionally designed pieces of CLI software I've seen.
munk-a 7y ago
> jtc stand for: JSON test console, but it's a legacy name, don't get misled. It's time to stop admitting that (if the name is so terrible) and re-brand or, more likely, come up with a better backronym meaning for your name.
- saghm 7y ago
  "JSON tool CLI"? Idk, naming is hard
- gojomo 7y ago
  JSON Transform Command
  - munk-a 7y ago
    That is pretty perfect - the term usage is a bit unnatural but nobody will notice since developers use weird names for things all the time (thanks, General Regular Expression Parser!)
justinsaccount 7y ago
Looks neat, but seems slow? $ wc -l conn.log 8505 conn.log $ cat conn.log |time ~/src/jtc-standard.json/jtc -a -w '<uid>l' -qq |shasum 0.44 real 0.43 user 0.00 sys 167f6d638a4ccf9e0be1ff4ed74caa01a461bd9a - $ cat conn.log | time jq -r .uid|shasum 0.10 real 0.10 user 0.00 sys 167f6d638a4ccf9e0be1ff4ed74caa01a461bd9a - $ cat conn.log | time json-cut uid |shasum 0.04 real 0.01 user 0.01 sys 167f6d638a4ccf9e0be1ff4ed74caa01a461bd9a - json-cut is a crappy tool I wrote that use github.com/buger/jsonparser
techie128 7y ago
It probably should not invent its own language or if it does, it should take inspiration from XPath/JSONPath which is arguably simpler.
- Rotareti 7y ago
  I agree. Modern SQL supports JSONPath. Postgresql 12 has support for it. A lot of people are/will be familiar with it.
  - chrismorgan 7y ago
    Be careful with terminology: the thing defined in SQL:2016 is called the SQL/JSON path language, and although it’s similar to the older JSONPath, it’s not the same. PostgreSQL also calls the SQL/JSON path language “JSON Path”, and the data type is called `JSONPATH`.
    - Rotareti 7y ago
      Thanks for the clarification! I guess it would make a lot of sense for a tool such as jtc to use SQL/JSON path language then. I'm sure it'll be the most common JSON query language soon.
- hyperpallium 7y ago
  > JsonPath has some drawbacks, such as a lack of operators for reaching parent or sibling nodes, -- https://www.baeldung.com/guide-to-jayway-jsonpath https://www.baeldung.com/guide-to-jayway-jsonpath Unfortunately, ".." just doesn't fit nicely with the "." operator of C-style languages. I'm almost inclined to go full-XPath, and use "/", so I can use "..".
bachmeier 7y ago
That's a complicated query tool. I'll be sticking with Taskwarrior for my simple use cases.
- triangleman 7y ago
  Taskwarrior?
mastrsushi 7y ago
What makes this different then all the 300 json libraries provided by and for the programming languages we already use? If I really wanted CLI, I would use one of those languages to write a program that reads arguments while making use of one of those JSON libraries.
jtms 7y ago
Sorry, but the api seems quite obtuse on first glance. I’ll just use Ruby or JavaScript or literally any modern language with good Json support
fergie 7y ago
JSON functionality is a subset of Javascript. Surely therefore the most sensible approach to pipe to a JavaScript runtime such as node.js? I can see how a C/C++ lib might me useful, but why build a standalone CLI just for JSON manipulation? Surely this functionality exists already? Or am I just missing something?
- aargh_aargh 7y ago
  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/eval#Do_not_ever_use_eval https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...!

122 points by vyuh 7y ago

alanbernstein 7y ago

Can anyone compare to jq? https://stedolan.github.io/jq/ https://stedolan.github.io/jq/

vyuh 7y ago
https://github.com/ldn-softdev/jtc#jtc-vs-jq https://github.com/ldn-softdev/jtc#jtc-vs-jq

meddlepal 7y ago

> jtc is written in idiomatic C++ (the most powerful programming language to date) Citation needed. The `jq` vs `jtc` section is interesting, but author seems a little full of himself with some of the explanations.

ur-whale 7y ago
I do agree with the author that the learning investment required by jq (learning a full blown DSL) is total overkill for most simple JSON based tasks. For those, I mostly use "jq ." to get all leaves on a single line and then feed the output to standard unix tools like grep cut awk and friends. And for more complex tasks, python, perl, even C++ if speed is needed.
- unspecified 7y ago
  > For those, I mostly use "jq ." to get all leaves on a single line and then feed the output to standard unix tools like grep cut awk and friends. There's a tool called `gron` designed for workflows like this, I've found it incredibly useful. https://news.ycombinator.com/item?id=16727665 https://news.ycombinator.com/item?id=16727665 https://github.com/tomnomnom/gron/ https://github.com/tomnomnom/gron/
- jolmg 7y ago
  > the learning investment required by jq (learning a full blown DSL) is total overkill for most simple JSON based tasks. There's no reason why you would need to learn the whole DSL for simple tasks. Just learn what you need. For simple stuff, the DSL is also simple. jtc also seems to have its own DSL by the way, and it doesn't really seem more intuitive than jq's: jtc -x'[0][:][name]<person>v [-1][children]<kids:false>f[0]<kids:true>v' -T'{"{person} has children":{kids}}' -r That's apparently the equivalent of: jq '.Directory | map({"\(.name) has children": (.children | length == 0)}) | .[]' There are some things that seem simpler on jtc, though, like searches of string values without regard to the json structure.
  - zeroimpl 7y ago
    I’d really like a CLI tool which solves this problem using a syntax I already know. I struggle with jq because I use it quite infrequently. IMO the ideal solution is something using pure JavaScript syntax, possibly with a library resembling jQuery for tree traversal.
    - jolmg 7y ago
      Most of jq's syntax is just a mix of JavaScript and a typical shell's. Every expression has an input and output like a shell command's stdin and stdout. Functions work on arguments and their stdin to produce their stdout. Here's an explanation of the syntax in the command I posted: jq ' # output the value at "Directory" from input object .Directory # pipe to map (JavaScript also has map()). The argument of map works in # the context of each element in map's input array. | map( # Produce an object where the property name is a string that # interpolates the value of the "name" property of this element. # Instead of interpolating, we could have also used this more # JavaScript-ish (ES5) syntax: # # {(.name + " has children"): ... # # The property value is an expression that gets the value of the # "children" property and pipes it to the expression `length != 0`. # `length` (which JavaScript also has) outputs the length of the # piped input, and then we compare that with 0. {"\(.name) has children": (.children | length != 0)} ) # map's output is a single record which is an array. We pipe that to # .[] to make multiple records, each an element of the array. The # syntax here is comprised of 2 parts: `.`, which is the input object, # and `[]` which is the subscript syntax without an index. | .[] ' > IMO the ideal solution is something using pure JavaScript syntax The greatest advantage of the current syntax is the great balance it has between legibility and terseness. I don't think making it pure JavaScript would be better. > possibly with a library resembling jQuery for tree traversal Using jQuery in JavaScript to traverse JavaScript objects? I don't know what to say...
deleted 7y ago
[deleted]
kbenson 7y ago
Oh yeah, immediately above that line: > jq is written in C, which drags all intrinsic problems the language has dated its creation And then to follow on the C++ claims: > Main JSON engine/library does not have a single new operator, nor it has a single naked pointer acting as a resource holder/owner, thus jtc is guaranteed to be free of memory leaks (at least one class of the problems is off the table) - STL guaranty. That's a lot more faith than I would be willing to put into C++ or C. Sure, that claim might be correct, but there's enough edge cases and undefined behavior in the language that I take it as a fairly bold claim unless it's also been thoroughly reviewed for any places some undefined behavior may be induced. I mean, it's probably fine, but just the willingness to make such a bold claim in that way communicates the opposite of what the author likely intended for me.
- hyperpallium 7y ago
  > jq is written in C, which drags all intrinsic problems the language has dated its creation I can't even parse this sentence. What does it mean? FWIW jq was originally written in Haskell, then ported to C. The C source for the core parts is strikingly clean. I think, it's partly from this Haskell heritage that jq gets its nice composability. However, TBF, I do find I must refresh jq syntax whenever I use it (pr only bc I don't process json often). It will be interesting to see what jtc does - though this foreshadowing does not bode well...
- otabdeveloper4 7y ago
  > "undefined behaviour" Yeah, you don't know what you're talking about. "Undefined behavior" simply means "stuff not covered by the ISO standard". So Python and Rust are 100% UB, and people don't really care.
  - kbenson 7y ago
    Undefined behavior can cause perfectly functioning code in one compiler version to be completely elided in a different version, as we've seen numerous times in the past. Review the article and comments here[1] for a good example of exactly this and discussion about the how and why, but the TL;DR is that a specific optimization in a specific compiler release determined that the adding of two variables, one of which a pointer, would result in an overflow which is undefined for pointers, and thus elided the check to see if the result was less than should be possible (overflowed), and the entire error handling case it resulted in. This caused the value to be used while invalid (as the error check and handling was elided), causing a segfault. Perhaps the features used in this project protect against that, but my point is that undefined behavior and the way compilers deal with it is variable and problematic. 1: https://news.ycombinator.com/item?id=14163111 https://news.ycombinator.com/item?id=14163111

vyuh 7y ago

The author has also written related tools. One to convert XML to JSON and back (https://github.com/ldn-softdev/jtm https://github.com/ldn-softdev/jtm) and another to convert JSON to SQLite tables (https://github.com/ldn-softdev/jsl https://github.com/ldn-softdev/jsl). Combining these with the hxnormalize tool ( https://www.w3.org/Tools/HTML-XML-utils/man1/hxnormalize.html https://www.w3.org/Tools/HTML-XML-utils/man1/hxnormalize.htm...), one can do very sophisticated manipulation on HTML web pages. HTML -> XML (via hxnormalize) -> JSON (via jtm) -> process using jtc (or even jq)

peterwwillis 7y ago
> convert XML to JSON and back This is basically impossible to do in a way that is compatible with other tools. Things like duplicate attributes of an object can exist in XML, but not in JSON. You can still work-around these limitations if you just have a pipeline using the same toolset, but part of the point of these tools is to then convert them back to a format that some other tool can use, which is where this pattern breaks down. Here's a list of pitfalls: https://stackoverflow.com/questions/33072812/potential-problems-of-mapping-json-to-xml https://stackoverflow.com/questions/33072812/potential-probl...
namibj 7y ago
This suggest a very scalable, easy approach to extract data from somewhat regular HTML...
- benibela 7y ago
  or just use xpath
- Yetanfou 7y ago
  I generally use xidel [1] for that type of task. Feed it xpath, css selectors or its own pattern matching thing. [1] https://github.com/benibela/xidel https://github.com/benibela/xidel

pmoriarty 7y ago

At first I was a little skeptical of this tool's claims of being simpler than jq, as I struggled to understand why (for instance) they were using <> in some parts of their example and [] in others, but after going through their step-by-step explanation it all made sense, and it does seem very simple. A tool that simplifies things is always welcome.

bradknowles 7y ago
The "User Guide.md" file is over 3000 lines long, comprising some 133KB. I read through about a quarter of it, before my eyes started to glaze over. I'm not convinced that I would call this "simpler" than jq.
- ak217 7y ago
  jq is one of the most intentionally designed pieces of software I've seen. In due time, I expect it to become as ubiquitous and indispensable as ls, grep, sed and awk. The jq language could use a formal spec, but other than that jq has a sublime elegance. By contrast, this seems like one of the least intentionally designed pieces of CLI software I've seen.

munk-a 7y ago

> jtc stand for: JSON test console, but it's a legacy name, don't get misled. It's time to stop admitting that (if the name is so terrible) and re-brand or, more likely, come up with a better backronym meaning for your name.

saghm 7y ago
"JSON tool CLI"? Idk, naming is hard
gojomo 7y ago
JSON Transform Command
- munk-a 7y ago
  That is pretty perfect - the term usage is a bit unnatural but nobody will notice since developers use weird names for things all the time (thanks, General Regular Expression Parser!)

justinsaccount 7y ago

Looks neat, but seems slow? $ wc -l conn.log 8505 conn.log $ cat conn.log |time ~/src/jtc-standard.json/jtc -a -w '<uid>l' -qq |shasum 0.44 real 0.43 user 0.00 sys 167f6d638a4ccf9e0be1ff4ed74caa01a461bd9a - $ cat conn.log | time jq -r .uid|shasum 0.10 real 0.10 user 0.00 sys 167f6d638a4ccf9e0be1ff4ed74caa01a461bd9a - $ cat conn.log | time json-cut uid |shasum 0.04 real 0.01 user 0.01 sys 167f6d638a4ccf9e0be1ff4ed74caa01a461bd9a - json-cut is a crappy tool I wrote that use github.com/buger/jsonparser

techie128 7y ago

It probably should not invent its own language or if it does, it should take inspiration from XPath/JSONPath which is arguably simpler.

Rotareti 7y ago
I agree. Modern SQL supports JSONPath. Postgresql 12 has support for it. A lot of people are/will be familiar with it.
- chrismorgan 7y ago
  Be careful with terminology: the thing defined in SQL:2016 is called the SQL/JSON path language, and although it’s similar to the older JSONPath, it’s not the same. PostgreSQL also calls the SQL/JSON path language “JSON Path”, and the data type is called `JSONPATH`.
  - Rotareti 7y ago
    Thanks for the clarification! I guess it would make a lot of sense for a tool such as jtc to use SQL/JSON path language then. I'm sure it'll be the most common JSON query language soon.
hyperpallium 7y ago
> JsonPath has some drawbacks, such as a lack of operators for reaching parent or sibling nodes, -- https://www.baeldung.com/guide-to-jayway-jsonpath https://www.baeldung.com/guide-to-jayway-jsonpath Unfortunately, ".." just doesn't fit nicely with the "." operator of C-style languages. I'm almost inclined to go full-XPath, and use "/", so I can use "..".

bachmeier 7y ago

That's a complicated query tool. I'll be sticking with Taskwarrior for my simple use cases.

triangleman 7y ago
Taskwarrior?

mastrsushi 7y ago

What makes this different then all the 300 json libraries provided by and for the programming languages we already use? If I really wanted CLI, I would use one of those languages to write a program that reads arguments while making use of one of those JSON libraries.

jtms 7y ago

Sorry, but the api seems quite obtuse on first glance. I’ll just use Ruby or JavaScript or literally any modern language with good Json support

fergie 7y ago

JSON functionality is a subset of Javascript. Surely therefore the most sensible approach to pipe to a JavaScript runtime such as node.js? I can see how a C/C++ lib might me useful, but why build a standalone CLI just for JSON manipulation? Surely this functionality exists already? Or am I just missing something?

aargh_aargh 7y ago
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/eval#Do_not_ever_use_eval https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...!