13 ms·

Show HN: Catj – A new way to display JSON files

388 points by soheilpro 7y ago

BaconJuice 7y ago
Just curious, what would be the use case for something like this?
- ofxartem 7y ago
  I could see it being useful when using grep or writing awk scripts. For the latter you usually have to carry around some state if you want to deduce the path to a desired leaf node. I could also see it useful for just finding the index of a particular array entity.
- z3t4 7y ago
  Easier to parse maybe. You can parse it line by line. I also think this format is easier to edit.
- soheilpro 7y ago
  Author here. I use it all the time when working with jq [1] to find the path of the nodes that I want to select or filter. It also makes it much easier to understand the structure of deeply nested JSON files. [1] https://stedolan.github.io/jq https://stedolan.github.io/jq
avidal 7y ago
Have you seen gron[0]? It's similar: flattens JSON documents to make them easily greppable. But it also can revert (ie, ungron) so you can pipe json to gron, grep -v to remove some data, then ungron to get it back to json. [0] https://github.com/tomnomnom/gron https://github.com/tomnomnom/gron
- Fnoord 7y ago
  Actively developed and written in Go. The project linked to is from 2014 with last update in 2015.and it is on NPM... What is left to say? Thank you!
  - heyoni 7y ago
    Used to be written in php right?
    - inoyeh 7y ago
      This "joke" is so incredibly old.
  - edflsafoiewq 7y ago
    What's wrong with it not being updated in 5 years? It's a simple script. It's probably stable and doesn't need to be "actively developed".
    - sethammons 7y ago
      I see this sentiment a lot. I agree with you, simple things can be done and don't need active development.
  - Apaec 7y ago
    I much prefer jq[1]. Actively developed and written in C. [1]:https://github.com/stedolan/jq https://github.com/stedolan/jq
- inferiorhuman 7y ago
  What does grep + gron give you over jq?
  - webo 7y ago
    as far as i can tell, jq doesn't do flattening .
    - lucb1e 7y ago
      Maybe GP meant that jq can do selection as well, i.e. that grepping is redundant after jq. But jq is much more complicated to learn and grep works on all inputs (not just json), so it makes a lot more sense to learn and use grep properly.
    - pstuart 7y ago
      Not built in, but @jolmg posted a script here which does the needful.
  - pfranz 7y ago
    jq seems very powerful. I don't deal with json all that often and my most common use case (by far) is `jq '.' -C` and it took a few tries for me to remember that syntax. The idea of flattening, grepping, then reverting sounds very appealing and sounds like a better fit for me.
    - jolmg 7y ago
      > `jq '.' -C` and it took a few tries for me to remember that syntax. I don't think you really need neither `.` nor `-C`. Just `jq` seems to do the same colored output of the input by default.
      - pfranz 7y ago
        It does look like neither are needed if you pipe a file in jq, but `jq . file.json` requires the `.` and if you're pipeing into a pager, like less, you need both `.` and `-C` to get colored output (that was the case with the alias I had pulled up). I am using 1.5 and haven't looked to see if 1.6 changes this.
- wooby 7y ago
  I haven't seen that one, but it looks very similar to the one I use: https://github.com/micha/json-table https://github.com/micha/json-table
- krick 7y ago
  That's actually a nice lifehack. Much simplier than jq. Unfortunately, would be harder to make all kinds of logical conditions for which jq allows (even if not that intuitively). It still feels like there must be something in between, some way to make queries with json more naturally, than with jq, yet with enough power.
  - jolmg 7y ago
    jq is certainly a unique language, which makes it unfamiliar to work with. Intuitiveness and natural feeling comes when one has gotten familiar with it after a bit of practice and reading the manual, though. It's a very well thought-out language. A very nice design. It might help to recognize how it's influenced by shell languages and XPath, if you're familiar with those.
    - krick 7y ago
      Well, no arguing there, it is indeed. And I use it from time to time. However, it's not like I need a tool like that every day, and if I'm no using it for a week I usually need to "learn" it all over again.
- news_hacker 7y ago
  Love the greppability and reconstructability. This should be the submission.
enriquto 7y ago
this is not merely a display, it is really a sanitation of the brain-damaged json syntax.
- gridlockd 7y ago
  Your opinion is bad.
  - enriquto 7y ago
    json will be mocked in a few years by the same people that today mock xml. In the end, unix always survives.
    - krapp 7y ago
      "unix" isn't a file format, nor is it in competition with either XML or JSON, so claiming it will "survive" either is a meaningless boast.
      - brennebeck 7y ago
        While I don’t disagree with you, if you look at the comment above this you’ll see @enriquto mentions just dumping lists to a text file. Pretty sure that’s what was implied by ‘unix’: use a simple text file that is dumped to the fs that can be easily parsed by standard tools on any Unix-like.
    - imtringued 7y ago
      People mock xml because it's an over complicated mess with too many features.
    - gridlockd 7y ago
      JSON is usually mocked by people who don't get to use Javascript objects natively. The benefits aren't obvious or tangible to them. XML is native to nothing. It's annoying to use everywhere. UNIX tools just don't compare. Sure, they stick around, because sometimes they're the easiest tool for some job, until they aren't and you regret starting out with them. In a few years, Javascript and Javascript-compatible languages are likely to be bigger than ever before. A whole generation of developers has been trained mainly on them. Billions of dollars have been invested into the ecosystem. Whether we like it or not, that's the reality. For that reason alone, JSON will stick around.
nfoz 7y ago
That's nice, I like that. Related, if you want more of a csv-style, see JSONLines. aka "newline-delimited JSON" http://jsonlines.org/ http://jsonlines.org/
- emmelaich 7y ago
  I don't see what jsonlines has over yaml; in fact the jsonlines examples presented there are almost trivially converted to yaml. e.g. the first example is valid yaml if you add a '- ' to each line. And JSON is (almost) a perfect subset of yaml. I've been using csv lately. It's reputation is overstated. What I like is that it's far more compact than yaml or json and trivially pulled into sqlite for ad-hoc queries.
  - mirimir 7y ago
    > ... and trivially pulled into sqlite for ad-hoc queries. Or spreadsheets, to work out a plan, and then MySQL.
  - geofft 7y ago
    One thing I like about JSON Lines is robustness to bad data - if an individual line is corrupted, you can discard it / print a warning and move on, and the parser can start again at the next newline. This makes it useful for log messages / metrics, because if something crashes while emitting a log line, you can recover. If something crashes while emitting an item in a YAML list, you might corrupt the entire rest of the document. Another is that it makes streaming processing a little easier. Once you have a line, you know you can attempt to process it, and you can shard processing on newlines without a full YAML processor. Tools that work on newlines or tools that can just split on lines can handle the first level of JSON Lines output.
deleted 7y ago
[deleted]
dymk 7y ago
Might want to put [2015] in the title? Certainly not a new way
sdegutis 7y ago
This is really similar to the format that AWS uses to represent recursive structures (arrays, maps) as a single array of key-value pairs for their APIs. Your catj could potentially be used to create that if ever working with the raw API directly instead of a SDK.
krapp 7y ago
The output format like a slightly better INI (in that it would support deeper hierarchies), although arrays would be a pain to write an index at a time like that. And for purely aesthetic, nitpicky reasons I think the leading period in each line is redundant.
- ucarion 7y ago
  I don't think that leading period is redundant. It indicates that the top-level value is an object, as opposed to an array or some basic value.
jolmg 7y ago
I was curious if this could be doable with jq, and apparently it is: jq -j ' [ [ paths(scalars) | map( if type == "number" then "[" + tostring + "]" else "." + . end ) | join("") ], [ .. | select(scalars) | @json ] ] | transpose | map(join(" = ") + "\n") | join("") ' EDIT: Got the string quoting and escaping. EDIT 2: For those who want to save this script, you can put just the jq code in an executable file with the shebang: #!/usr/bin/jq -jf
- pstuart 7y ago
  I was expecting some mention of jq for this post, and you did not disappoint. Thank you for the script -- it works great and I'm adding it to my collection.
- phiresky 7y ago
  Would probably be shorter if you used `tostream`
  - jolmg 7y ago
    You're right. Good tip: jq -r ' tostream | select(length > 1) | ( .[0] | map( if type == "number" then "[" + tostring + "]" else "." + . end ) | join("") ) + " = " + (.[1] | @json) ' EDIT: For those who want to save this script, you can put just the jq code in an executable file with the shebang: #!/usr/bin/jq -rf
    - hyperpallium 7y ago
      Appending | to each line and a last line . is a jq program that reproduces the original json. jq -r ' ( tostream | select(length > 1) | ( .[0] | map( if type == "number" then "[" + tostring + "]" else "." + . end ) | join("") ) + " = " + (.[1] | @json) + " |" ), "." '
      - 7y ago
tedivm 7y ago
For exploring large files I released a program called "JSONSmash" that runs a shell which lets you browse the data as if it was a filesystem. https://blog.tedivm.com/open-source/2017/05/introducing-jsonsmash-work-with-large-json-files-easily/ https://blog.tedivm.com/open-source/2017/05/introducing-json...
- ficklepickle 7y ago
  What an interesting idea! On my last contract, I had to deal with 100mb+ JSON responses from a barely-documented API. This would have come in handy when I was figuring it out. I've used JSONExplorer for this purpose, but it is web based and doesn't handle files this large. Extending the filesystem metaphor to JSON data and re-using the same commands strikes me as a great idea. Did another project inspire you, or did you come up with the concept yourself? Have you done a Show HN yet?
  - tedivm 7y ago
    I came up with this idea on my own when dealing with the AWS Bulk API files. I did make a "Show HN" but it got a total of four upvotes.
    - mkesper 7y ago
      HN is very sensitive to timing, afaik the moderators allow resubmissions when that seems to be the case.
- rahimnathwani 7y ago
  This looks cool! I'm curious: did you consider using FUSE for this, instead of taking a shell? Using FUSE might make installation harder for less technical users, but the upside is that they could use their choice of file manager (shell commands, curses-based or GUI). As you've already implemented most relevant commands (cd, ls, cat), it would probably be easy to make a FUSE version using fs-fuse / fuse-bindings
konsumer 7y ago
I made this to do similar, it goes both ways: https://github.com/konsumer/jsflat https://github.com/konsumer/jsflat
karxxm 7y ago
All those redundancies make me sad
matmann2001 7y ago
I would recommend an optional flag to disable the output coloring, in the case where someone wants to pipe this to a file or other program.
twp 7y ago
I wrote a similar tool: https://github.com/twpayne/flatjson https://github.com/twpayne/flatjson The flat format is great for diffs: --- testdata/a.json +++ testdata/b.json @@ -1,5 +1,6 @@ root = {}; root.menu = {}; +root.menu.disabled = true; root.menu.id = "file"; root.menu.popup = {}; root.menu.popup.menuitem = []; @@ -9,8 +10,5 @@ root.menu.popup.menuitem[1] = {}; root.menu.popup.menuitem[1].onclick = "OpenDoc()"; root.menu.popup.menuitem[1].value = "Open"; -root.menu.popup.menuitem[2] = {}; -root.menu.popup.menuitem[2].onclick = "CloseDoc()"; -root.menu.popup.menuitem[2].value = "Close"; -root.menu.value = "File"; +root.menu.value = "File menu";
etaioinshrdlu 7y ago
It looks almost executable! If it were, that might be handy now and then.
- rolltiide 7y ago
  add in a few nil and undefined checks automatically and it really is executable
flying_sheep 7y ago
https://github.com/stedolan/jq/issues/243 https://github.com/stedolan/jq/issues/243 jq can also do the same thing, with more flexibility. And it is possible to combine with bash alias to make it indistinguishable from catj
- jolmg 7y ago
  > combine with bash alias ... or you know, you could put the jq script in an executable file and add a shebang like #!/usr/bin/jq -jf or #!/usr/bin/jq -rf In my opinion, aliases should mostly be used to add default options only. Not really to insert whole scripts into them.
  - Splognosticus 7y ago
    I've always been of the mind that aliases should be "whatever the user finds convenient." I don't think I've ever seen code that depends on them.
laurent123456 7y ago
New from (2014)
hasahmed 7y ago
cat foo.json | jq .
kara_jade 7y ago
You can do this easily with the json_tree function from SQLite's JSON1 extension. It's given as an example in the documentation: https://sqlite.org/json1.html#jtree https://sqlite.org/json1.html#jtree SELECT big.rowid, fullkey, value FROM big, json_tree(big.json) WHERE json_tree.type NOT IN ('object','array');
- ComputerGuru 7y ago
  And with the fileio sqlite extension, you can even directly query files (and their contents, and directories recursively too, no less) from SQL.
jay-anderson 7y ago
Reminds me of how spring configuration works between yaml and property files. It does a similar flattening to translate between them.
jijji 7y ago
not much different than php print var_export(json_decode($json,TRUE), TRUE);
emmelaich 7y ago
Augeas can do something similar too. But not only JSON but XML and 200+ other config file formats.[0] $ augtool -r . -L --transform 'JSON.lns incl /catj-eg.json' <<< 'print /files/catj-eg.json' /files/catj-eg.json /files/catj-eg.json/dict /files/catj-eg.json/dict/entry = "movie" /files/catj-eg.json/dict/entry/dict /files/catj-eg.json/dict/entry/dict/entry[1] = "name" /files/catj-eg.json/dict/entry/dict/entry[1]/string = "Interstellar" /files/catj-eg.json/dict/entry/dict/entry[2] = "year" /files/catj-eg.json/dict/entry/dict/entry[2]/number = "2014" /files/catj-eg.json/dict/entry/dict/entry[3] = "is_released" /files/catj-eg.json/dict/entry/dict/entry[3]/const = "true" /files/catj-eg.json/dict/entry/dict/entry[4] = "director" /files/catj-eg.json/dict/entry/dict/entry[4]/string = "Christopher Nolan" /files/catj-eg.json/dict/entry/dict/entry[5] = "cast" /files/catj-eg.json/dict/entry/dict/entry[5]/array /files/catj-eg.json/dict/entry/dict/entry[5]/array/string[1] = "Matthew McConaughey" /files/catj-eg.json/dict/entry/dict/entry[5]/array/string[2] = "Anne Hathaway" /files/catj-eg.json/dict/entry/dict/entry[5]/array/string[3] = "Jessica Chastain" /files/catj-eg.json/dict/entry/dict/entry[5]/array/string[4] = "Bill Irwin" /files/catj-eg.json/dict/entry/dict/entry[5]/array/string[5] = "Ellen Burstyn" /files/catj-eg.json/dict/entry/dict/entry[5]/array/string[6] = "Michael Caine" [0] $ ls .../share/augeas/lenses/dist/|wc 221 221 2867
stdcall83 7y ago
Looks like a good candidate for lesspipe plugin.
not_kurt_godel 7y ago
Hm, essentially JSON->Properties file. Cool. I wrote a little script the other day and decided Properties format was a pleasant way to define the config, maybe this could dovetail with similar future endeavors.
deleted 7y ago
[deleted]
danschumann 7y ago
Okay I made one with javascript ( go to https://underscorejs.org/ https://underscorejs.org/ and open console ) var json = JSON.parse('{"my": "json"}'); (function printRecursively(ob, _keys = []){ _.map(ob, (val, key) => { var k = _.isNumber(key) ? '[' + key + ']' : '.' + key; var keys = _keys.concat(k); if (_.isObject(val)) printRecursively(val, keys); else console.log(keys.join('') + ' = ' + typeof val + ' ' + val); }); })(json); EDIT: how do I markup code on HN?
- ComputerGuru 7y ago
  > how do I markup code Indent each line with four spaces. Please please keep line length very short (under forty?) as HN's pre tags are absolutely not mobile friendly and can trash the entire page. Edit: actually it seems to at least scroll within the comment div on overflow now, that's a huge improvement!
usamaejaz 7y ago
this looks nice
fareesh 7y ago
I use the "JSON viewer awesome" extension on Chrome. It displays JSON in a collapsible verticle tree and has a copy path button. This is a nice CLI based alternative
hokus 7y ago
this is something between a joke and a thought experiment. function cason(x){ switch(x[0]){ case "movie": switch(x[1]) { case "name" : return "Interstellar"; case "year" : return 2014; case "is_released": return true; case "director" : return "Christopher Nolan"; case "cast": switch(x[2]){ case 0: return "Matthew McConaughey"; case 1: return "Anne Hathaway"; case 2: return "Jessica Chastain"; case 3: return "Bill Irwin"; case 4: return "Ellen Burstyn"; case 5: return "Michael Caine"; } } } }
- geofft 7y ago
  Well, that's kind of the inverse of writing switch statements like def license(kernel): return {"Linux": "GPL", "FreeBSD": "BSD", "NT": "Proprietary"}[kernel]
yason 7y ago
When XML came out into popularity I think the first thing I wrote was a small Python program to flatten/unflatten XML into per-line entries quite similar to the example output in the article. The text streams that are processed line-by-line by dozens or hundreds of line-based tools are immensely powerful and universal. It's all Unix heritage and often overlooked by fancy modern designs that more often follow a fashion rather than root themselves in substance. Surely text streams have their share of limitations like everything else but in practise you can retrofit nearly anything into line-based text streams and get an immediate productivity multiplier by being able to apply a whole array of established tools to process that data. Proof of that power is that it has been worthwhile to write converters to and from text and other formats. Not only you can find translators to turn various hierarchical or object-oriented formats into text but you can even convert a PNG into text and back (with SNG). Text streams are like roads with lanes. They're ages old, they're pretty good at separating and guiding traffic, and they're somehow suboptimal in several senses yet rarely can anyone point out a single, clear practical improvement on laned roads, not to mention a system for containing traffic flows that is superior to them.
ravinizme 7y ago
Similar to python-jsonpipe (8 years ago). https://github.com/zacharyvoase/jsonpipe https://github.com/zacharyvoase/jsonpipe It includes 'jsonunpipe'. So you could grep part of the JSON and still get a JSON back. ``` echo '{"a": 1, "b": 2}' |grep b| jsonpipe | jsonunpipe #{""b": 2} ```
chrismorgan 7y ago
There is actually a standard around writing paths into JSON objects: JSON Pointer, https://tools.ietf.org/html/rfc6901 https://tools.ietf.org/html/rfc6901. It’s straightforward, and avoids ambiguity between separator and key name by simple replacement, e.g. `/foo/bar~0baz~1quux` looks up a key named "foo", then a key named "bar~baz/quux" inside it. It’s not particularly widely used, but I’ve come across it in a few places over the years (it’s not a common thing to need to do), and probably most recently JMAP uses it for backreferences. (I haven’t run it, but a skim of the code suggests that this tool will turn `{"foo.bar": "baz", "foo": {"bar": "baz"}}` into `["foo.bar"] = "baz"` and `.foo.bar = "baz"`, resolving the separator ambiguity in a pretty JavaScripty way.)
venthur 7y ago
Looks a lot like ye olde gron: https://github.com/tomnomnom/gron https://github.com/tomnomnom/gron Here's a Python implementation of gron https://github.com/venthur/python-gron https://github.com/venthur/python-gron
bborud 7y ago
Oh no. This looks like the horrible config format a colleague of mine invented at Yahoo when making an absolutely horrific config system. This brings back bad memories. This may look cute but it is horrific when dealing with large configs and you have to reconstruct all the structure in your head. Also, when you have a format that nests using brackets, braces and parenthesis, you can get help from the editor. This format does not give you that. I'm not a huge fan of JSON (and the above mentioned format was invented because none of us were fans of XML at the time), but it turns out that both XML and JSON are actually easier to work with in practice than this format. Not least because there is ample tooling for JSON (and XML). The lesson I learnt: I may hate XML (or in this case JSON), but finding an alternative that is better is not easy.
- geofft 7y ago
  I think the idea is not that this is for storage or editing, but just for querying - you should keep your data in JSON, but if you want to find where something is, do `catj foo.json | grep something` and it'll tell you all the paths in the document where you can find the string "something". The intent is not to open catj format in a text editor, or to use the output of catj for anything other than ad-hoc purposes.
htk 7y ago
What a simple and clever idea. I would love something like that for XML in C#.
blablabla123 7y ago
That's really smart, in fact this was - maybe until now - the only reason for me to resort to csv. (With pandas it's by the way really easy to flatten jsons into csv) JSON is such a nice format but the tools are really not there yet. I guess should should make it then possible to combine with line-based tools like head, tail, sort, uniq etc.
QuadrupleA 7y ago
Please don't write JSON like the example - it's like putting 4 files in a hierarchy of 20 folders. Way over-structured. That said, if you're stuck dealing with bad JSON like this with low signal to noise this is a decent way to redisplay it.
limsup 7y ago
Try it with deno: deno install catj https://deno.land/std/examples/catjson.ts https://deno.land/std/examples/catjson.ts --allow-read
nailer 7y ago
pwsh is an open source shell that runs on Unix and supports JSON natively, so you don't need to flatten or use jq. Here's an example from a couple of days ago: https://twitter.com/mikemaccana/status/1141706132823695362?s=21 https://twitter.com/mikemaccana/status/1141706132823695362?s...
deostroll 7y ago
Sweet! I find this tool extremely useful for parsing dmn files in javascript (via xml2js). But right now I am more interested in a similar feature for xml/html documents. Hoping if someone can point me to it. Thanks.

388 points by soheilpro 7y ago

BaconJuice 7y ago

Just curious, what would be the use case for something like this?

ofxartem 7y ago
I could see it being useful when using grep or writing awk scripts. For the latter you usually have to carry around some state if you want to deduce the path to a desired leaf node. I could also see it useful for just finding the index of a particular array entity.
z3t4 7y ago
Easier to parse maybe. You can parse it line by line. I also think this format is easier to edit.
soheilpro 7y ago
Author here. I use it all the time when working with jq [1] to find the path of the nodes that I want to select or filter. It also makes it much easier to understand the structure of deeply nested JSON files. [1] https://stedolan.github.io/jq https://stedolan.github.io/jq

avidal 7y ago

Have you seen gron[0]? It's similar: flattens JSON documents to make them easily greppable. But it also can revert (ie, ungron) so you can pipe json to gron, grep -v to remove some data, then ungron to get it back to json. [0] https://github.com/tomnomnom/gron https://github.com/tomnomnom/gron

Fnoord 7y ago
Actively developed and written in Go. The project linked to is from 2014 with last update in 2015.and it is on NPM... What is left to say? Thank you!
- heyoni 7y ago
  Used to be written in php right?
  - inoyeh 7y ago
    This "joke" is so incredibly old.
- edflsafoiewq 7y ago
  What's wrong with it not being updated in 5 years? It's a simple script. It's probably stable and doesn't need to be "actively developed".
  - sethammons 7y ago
    I see this sentiment a lot. I agree with you, simple things can be done and don't need active development.
- Apaec 7y ago
  I much prefer jq[1]. Actively developed and written in C. [1]:https://github.com/stedolan/jq https://github.com/stedolan/jq
inferiorhuman 7y ago
What does grep + gron give you over jq?
- webo 7y ago
  as far as i can tell, jq doesn't do flattening .
  - lucb1e 7y ago
    Maybe GP meant that jq can do selection as well, i.e. that grepping is redundant after jq. But jq is much more complicated to learn and grep works on all inputs (not just json), so it makes a lot more sense to learn and use grep properly.
  - pstuart 7y ago
    Not built in, but @jolmg posted a script here which does the needful.
- pfranz 7y ago
  jq seems very powerful. I don't deal with json all that often and my most common use case (by far) is `jq '.' -C` and it took a few tries for me to remember that syntax. The idea of flattening, grepping, then reverting sounds very appealing and sounds like a better fit for me.
  - jolmg 7y ago
    > `jq '.' -C` and it took a few tries for me to remember that syntax. I don't think you really need neither `.` nor `-C`. Just `jq` seems to do the same colored output of the input by default.
    - pfranz 7y ago
      It does look like neither are needed if you pipe a file in jq, but `jq . file.json` requires the `.` and if you're pipeing into a pager, like less, you need both `.` and `-C` to get colored output (that was the case with the alias I had pulled up). I am using 1.5 and haven't looked to see if 1.6 changes this.
wooby 7y ago
I haven't seen that one, but it looks very similar to the one I use: https://github.com/micha/json-table https://github.com/micha/json-table
krick 7y ago
That's actually a nice lifehack. Much simplier than jq. Unfortunately, would be harder to make all kinds of logical conditions for which jq allows (even if not that intuitively). It still feels like there must be something in between, some way to make queries with json more naturally, than with jq, yet with enough power.
- jolmg 7y ago
  jq is certainly a unique language, which makes it unfamiliar to work with. Intuitiveness and natural feeling comes when one has gotten familiar with it after a bit of practice and reading the manual, though. It's a very well thought-out language. A very nice design. It might help to recognize how it's influenced by shell languages and XPath, if you're familiar with those.
  - krick 7y ago
    Well, no arguing there, it is indeed. And I use it from time to time. However, it's not like I need a tool like that every day, and if I'm no using it for a week I usually need to "learn" it all over again.
news_hacker 7y ago
Love the greppability and reconstructability. This should be the submission.

enriquto 7y ago

this is not merely a display, it is really a sanitation of the brain-damaged json syntax.

gridlockd 7y ago
Your opinion is bad.
- enriquto 7y ago
  json will be mocked in a few years by the same people that today mock xml. In the end, unix always survives.
  - krapp 7y ago
    "unix" isn't a file format, nor is it in competition with either XML or JSON, so claiming it will "survive" either is a meaningless boast.
    - brennebeck 7y ago
      While I don’t disagree with you, if you look at the comment above this you’ll see @enriquto mentions just dumping lists to a text file. Pretty sure that’s what was implied by ‘unix’: use a simple text file that is dumped to the fs that can be easily parsed by standard tools on any Unix-like.
  - imtringued 7y ago
    People mock xml because it's an over complicated mess with too many features.
  - gridlockd 7y ago
    JSON is usually mocked by people who don't get to use Javascript objects natively. The benefits aren't obvious or tangible to them. XML is native to nothing. It's annoying to use everywhere. UNIX tools just don't compare. Sure, they stick around, because sometimes they're the easiest tool for some job, until they aren't and you regret starting out with them. In a few years, Javascript and Javascript-compatible languages are likely to be bigger than ever before. A whole generation of developers has been trained mainly on them. Billions of dollars have been invested into the ecosystem. Whether we like it or not, that's the reality. For that reason alone, JSON will stick around.

nfoz 7y ago

That's nice, I like that. Related, if you want more of a csv-style, see JSONLines. aka "newline-delimited JSON" http://jsonlines.org/ http://jsonlines.org/

emmelaich 7y ago
I don't see what jsonlines has over yaml; in fact the jsonlines examples presented there are almost trivially converted to yaml. e.g. the first example is valid yaml if you add a '- ' to each line. And JSON is (almost) a perfect subset of yaml. I've been using csv lately. It's reputation is overstated. What I like is that it's far more compact than yaml or json and trivially pulled into sqlite for ad-hoc queries.
- mirimir 7y ago
  > ... and trivially pulled into sqlite for ad-hoc queries. Or spreadsheets, to work out a plan, and then MySQL.
- geofft 7y ago
  One thing I like about JSON Lines is robustness to bad data - if an individual line is corrupted, you can discard it / print a warning and move on, and the parser can start again at the next newline. This makes it useful for log messages / metrics, because if something crashes while emitting a log line, you can recover. If something crashes while emitting an item in a YAML list, you might corrupt the entire rest of the document. Another is that it makes streaming processing a little easier. Once you have a line, you know you can attempt to process it, and you can shard processing on newlines without a full YAML processor. Tools that work on newlines or tools that can just split on lines can handle the first level of JSON Lines output.

deleted 7y ago

[deleted]

dymk 7y ago

Might want to put [2015] in the title? Certainly not a new way

sdegutis 7y ago

This is really similar to the format that AWS uses to represent recursive structures (arrays, maps) as a single array of key-value pairs for their APIs. Your catj could potentially be used to create that if ever working with the raw API directly instead of a SDK.

krapp 7y ago

The output format like a slightly better INI (in that it would support deeper hierarchies), although arrays would be a pain to write an index at a time like that. And for purely aesthetic, nitpicky reasons I think the leading period in each line is redundant.

ucarion 7y ago
I don't think that leading period is redundant. It indicates that the top-level value is an object, as opposed to an array or some basic value.

jolmg 7y ago

I was curious if this could be doable with jq, and apparently it is: jq -j ' [ [ paths(scalars) | map( if type == "number" then "[" + tostring + "]" else "." + . end ) | join("") ], [ .. | select(scalars) | @json ] ] | transpose | map(join(" = ") + "\n") | join("") ' EDIT: Got the string quoting and escaping. EDIT 2: For those who want to save this script, you can put just the jq code in an executable file with the shebang: #!/usr/bin/jq -jf

pstuart 7y ago
I was expecting some mention of jq for this post, and you did not disappoint. Thank you for the script -- it works great and I'm adding it to my collection.
phiresky 7y ago
Would probably be shorter if you used `tostream`
- jolmg 7y ago
  You're right. Good tip: jq -r ' tostream | select(length > 1) | ( .[0] | map( if type == "number" then "[" + tostring + "]" else "." + . end ) | join("") ) + " = " + (.[1] | @json) ' EDIT: For those who want to save this script, you can put just the jq code in an executable file with the shebang: #!/usr/bin/jq -rf
  - hyperpallium 7y ago
    Appending | to each line and a last line . is a jq program that reproduces the original json. jq -r ' ( tostream | select(length > 1) | ( .[0] | map( if type == "number" then "[" + tostring + "]" else "." + . end ) | join("") ) + " = " + (.[1] | @json) + " |" ), "." '
    - 7y ago

tedivm 7y ago

For exploring large files I released a program called "JSONSmash" that runs a shell which lets you browse the data as if it was a filesystem. https://blog.tedivm.com/open-source/2017/05/introducing-jsonsmash-work-with-large-json-files-easily/ https://blog.tedivm.com/open-source/2017/05/introducing-json...

ficklepickle 7y ago
What an interesting idea! On my last contract, I had to deal with 100mb+ JSON responses from a barely-documented API. This would have come in handy when I was figuring it out. I've used JSONExplorer for this purpose, but it is web based and doesn't handle files this large. Extending the filesystem metaphor to JSON data and re-using the same commands strikes me as a great idea. Did another project inspire you, or did you come up with the concept yourself? Have you done a Show HN yet?
- tedivm 7y ago
  I came up with this idea on my own when dealing with the AWS Bulk API files. I did make a "Show HN" but it got a total of four upvotes.
  - mkesper 7y ago
    HN is very sensitive to timing, afaik the moderators allow resubmissions when that seems to be the case.
rahimnathwani 7y ago
This looks cool! I'm curious: did you consider using FUSE for this, instead of taking a shell? Using FUSE might make installation harder for less technical users, but the upside is that they could use their choice of file manager (shell commands, curses-based or GUI). As you've already implemented most relevant commands (cd, ls, cat), it would probably be easy to make a FUSE version using fs-fuse / fuse-bindings

konsumer 7y ago

I made this to do similar, it goes both ways: https://github.com/konsumer/jsflat https://github.com/konsumer/jsflat

karxxm 7y ago

All those redundancies make me sad

matmann2001 7y ago

I would recommend an optional flag to disable the output coloring, in the case where someone wants to pipe this to a file or other program.

twp 7y ago

I wrote a similar tool: https://github.com/twpayne/flatjson https://github.com/twpayne/flatjson The flat format is great for diffs: --- testdata/a.json +++ testdata/b.json @@ -1,5 +1,6 @@ root = {}; root.menu = {}; +root.menu.disabled = true; root.menu.id = "file"; root.menu.popup = {}; root.menu.popup.menuitem = []; @@ -9,8 +10,5 @@ root.menu.popup.menuitem[1] = {}; root.menu.popup.menuitem[1].onclick = "OpenDoc()"; root.menu.popup.menuitem[1].value = "Open"; -root.menu.popup.menuitem[2] = {}; -root.menu.popup.menuitem[2].onclick = "CloseDoc()"; -root.menu.popup.menuitem[2].value = "Close"; -root.menu.value = "File"; +root.menu.value = "File menu";

etaioinshrdlu 7y ago

It looks almost executable! If it were, that might be handy now and then.

rolltiide 7y ago
add in a few nil and undefined checks automatically and it really is executable

flying_sheep 7y ago

https://github.com/stedolan/jq/issues/243 https://github.com/stedolan/jq/issues/243 jq can also do the same thing, with more flexibility. And it is possible to combine with bash alias to make it indistinguishable from catj

jolmg 7y ago
> combine with bash alias ... or you know, you could put the jq script in an executable file and add a shebang like #!/usr/bin/jq -jf or #!/usr/bin/jq -rf In my opinion, aliases should mostly be used to add default options only. Not really to insert whole scripts into them.
- Splognosticus 7y ago
  I've always been of the mind that aliases should be "whatever the user finds convenient." I don't think I've ever seen code that depends on them.

laurent123456 7y ago

New from (2014)

hasahmed 7y ago

cat foo.json | jq .

kara_jade 7y ago

You can do this easily with the json_tree function from SQLite's JSON1 extension. It's given as an example in the documentation: https://sqlite.org/json1.html#jtree https://sqlite.org/json1.html#jtree SELECT big.rowid, fullkey, value FROM big, json_tree(big.json) WHERE json_tree.type NOT IN ('object','array');

ComputerGuru 7y ago
And with the fileio sqlite extension, you can even directly query files (and their contents, and directories recursively too, no less) from SQL.

jay-anderson 7y ago

Reminds me of how spring configuration works between yaml and property files. It does a similar flattening to translate between them.

jijji 7y ago

not much different than php print var_export(json_decode($json,TRUE), TRUE);

emmelaich 7y ago

Augeas can do something similar too. But not only JSON but XML and 200+ other config file formats.[0] $ augtool -r . -L --transform 'JSON.lns incl /catj-eg.json' <<< 'print /files/catj-eg.json' /files/catj-eg.json /files/catj-eg.json/dict /files/catj-eg.json/dict/entry = "movie" /files/catj-eg.json/dict/entry/dict /files/catj-eg.json/dict/entry/dict/entry[1] = "name" /files/catj-eg.json/dict/entry/dict/entry[1]/string = "Interstellar" /files/catj-eg.json/dict/entry/dict/entry[2] = "year" /files/catj-eg.json/dict/entry/dict/entry[2]/number = "2014" /files/catj-eg.json/dict/entry/dict/entry[3] = "is_released" /files/catj-eg.json/dict/entry/dict/entry[3]/const = "true" /files/catj-eg.json/dict/entry/dict/entry[4] = "director" /files/catj-eg.json/dict/entry/dict/entry[4]/string = "Christopher Nolan" /files/catj-eg.json/dict/entry/dict/entry[5] = "cast" /files/catj-eg.json/dict/entry/dict/entry[5]/array /files/catj-eg.json/dict/entry/dict/entry[5]/array/string[1] = "Matthew McConaughey" /files/catj-eg.json/dict/entry/dict/entry[5]/array/string[2] = "Anne Hathaway" /files/catj-eg.json/dict/entry/dict/entry[5]/array/string[3] = "Jessica Chastain" /files/catj-eg.json/dict/entry/dict/entry[5]/array/string[4] = "Bill Irwin" /files/catj-eg.json/dict/entry/dict/entry[5]/array/string[5] = "Ellen Burstyn" /files/catj-eg.json/dict/entry/dict/entry[5]/array/string[6] = "Michael Caine" [0] $ ls .../share/augeas/lenses/dist/|wc 221 221 2867

stdcall83 7y ago

Looks like a good candidate for lesspipe plugin.

not_kurt_godel 7y ago

Hm, essentially JSON->Properties file. Cool. I wrote a little script the other day and decided Properties format was a pleasant way to define the config, maybe this could dovetail with similar future endeavors.

deleted 7y ago

[deleted]

danschumann 7y ago

Okay I made one with javascript ( go to https://underscorejs.org/ https://underscorejs.org/ and open console ) var json = JSON.parse('{"my": "json"}'); (function printRecursively(ob, _keys = []){ _.map(ob, (val, key) => { var k = _.isNumber(key) ? '[' + key + ']' : '.' + key; var keys = _keys.concat(k); if (_.isObject(val)) printRecursively(val, keys); else console.log(keys.join('') + ' = ' + typeof val + ' ' + val); }); })(json); EDIT: how do I markup code on HN?

ComputerGuru 7y ago
> how do I markup code Indent each line with four spaces. Please please keep line length very short (under forty?) as HN's pre tags are absolutely not mobile friendly and can trash the entire page. Edit: actually it seems to at least scroll within the comment div on overflow now, that's a huge improvement!

usamaejaz 7y ago

this looks nice

fareesh 7y ago

I use the "JSON viewer awesome" extension on Chrome. It displays JSON in a collapsible verticle tree and has a copy path button. This is a nice CLI based alternative

hokus 7y ago

this is something between a joke and a thought experiment. function cason(x){ switch(x[0]){ case "movie": switch(x[1]) { case "name" : return "Interstellar"; case "year" : return 2014; case "is_released": return true; case "director" : return "Christopher Nolan"; case "cast": switch(x[2]){ case 0: return "Matthew McConaughey"; case 1: return "Anne Hathaway"; case 2: return "Jessica Chastain"; case 3: return "Bill Irwin"; case 4: return "Ellen Burstyn"; case 5: return "Michael Caine"; } } } }

geofft 7y ago
Well, that's kind of the inverse of writing switch statements like def license(kernel): return {"Linux": "GPL", "FreeBSD": "BSD", "NT": "Proprietary"}[kernel]

yason 7y ago

When XML came out into popularity I think the first thing I wrote was a small Python program to flatten/unflatten XML into per-line entries quite similar to the example output in the article. The text streams that are processed line-by-line by dozens or hundreds of line-based tools are immensely powerful and universal. It's all Unix heritage and often overlooked by fancy modern designs that more often follow a fashion rather than root themselves in substance. Surely text streams have their share of limitations like everything else but in practise you can retrofit nearly anything into line-based text streams and get an immediate productivity multiplier by being able to apply a whole array of established tools to process that data. Proof of that power is that it has been worthwhile to write converters to and from text and other formats. Not only you can find translators to turn various hierarchical or object-oriented formats into text but you can even convert a PNG into text and back (with SNG). Text streams are like roads with lanes. They're ages old, they're pretty good at separating and guiding traffic, and they're somehow suboptimal in several senses yet rarely can anyone point out a single, clear practical improvement on laned roads, not to mention a system for containing traffic flows that is superior to them.

ravinizme 7y ago

Similar to python-jsonpipe (8 years ago). https://github.com/zacharyvoase/jsonpipe https://github.com/zacharyvoase/jsonpipe It includes 'jsonunpipe'. So you could grep part of the JSON and still get a JSON back. ``` echo '{"a": 1, "b": 2}' |grep b| jsonpipe | jsonunpipe #{""b": 2} ```

chrismorgan 7y ago

There is actually a standard around writing paths into JSON objects: JSON Pointer, https://tools.ietf.org/html/rfc6901 https://tools.ietf.org/html/rfc6901. It’s straightforward, and avoids ambiguity between separator and key name by simple replacement, e.g. `/foo/bar~0baz~1quux` looks up a key named "foo", then a key named "bar~baz/quux" inside it. It’s not particularly widely used, but I’ve come across it in a few places over the years (it’s not a common thing to need to do), and probably most recently JMAP uses it for backreferences. (I haven’t run it, but a skim of the code suggests that this tool will turn `{"foo.bar": "baz", "foo": {"bar": "baz"}}` into `["foo.bar"] = "baz"` and `.foo.bar = "baz"`, resolving the separator ambiguity in a pretty JavaScripty way.)

venthur 7y ago

Looks a lot like ye olde gron: https://github.com/tomnomnom/gron https://github.com/tomnomnom/gron Here's a Python implementation of gron https://github.com/venthur/python-gron https://github.com/venthur/python-gron

bborud 7y ago

Oh no. This looks like the horrible config format a colleague of mine invented at Yahoo when making an absolutely horrific config system. This brings back bad memories. This may look cute but it is horrific when dealing with large configs and you have to reconstruct all the structure in your head. Also, when you have a format that nests using brackets, braces and parenthesis, you can get help from the editor. This format does not give you that. I'm not a huge fan of JSON (and the above mentioned format was invented because none of us were fans of XML at the time), but it turns out that both XML and JSON are actually easier to work with in practice than this format. Not least because there is ample tooling for JSON (and XML). The lesson I learnt: I may hate XML (or in this case JSON), but finding an alternative that is better is not easy.

geofft 7y ago
I think the idea is not that this is for storage or editing, but just for querying - you should keep your data in JSON, but if you want to find where something is, do `catj foo.json | grep something` and it'll tell you all the paths in the document where you can find the string "something". The intent is not to open catj format in a text editor, or to use the output of catj for anything other than ad-hoc purposes.

htk 7y ago

What a simple and clever idea. I would love something like that for XML in C#.

blablabla123 7y ago

That's really smart, in fact this was - maybe until now - the only reason for me to resort to csv. (With pandas it's by the way really easy to flatten jsons into csv) JSON is such a nice format but the tools are really not there yet. I guess should should make it then possible to combine with line-based tools like head, tail, sort, uniq etc.

QuadrupleA 7y ago

Please don't write JSON like the example - it's like putting 4 files in a hierarchy of 20 folders. Way over-structured. That said, if you're stuck dealing with bad JSON like this with low signal to noise this is a decent way to redisplay it.

limsup 7y ago

Try it with deno: deno install catj https://deno.land/std/examples/catjson.ts https://deno.land/std/examples/catjson.ts --allow-read

nailer 7y ago

pwsh is an open source shell that runs on Unix and supports JSON natively, so you don't need to flatten or use jq. Here's an example from a couple of days ago: https://twitter.com/mikemaccana/status/1141706132823695362?s=21 https://twitter.com/mikemaccana/status/1141706132823695362?s...

deostroll 7y ago

Sweet! I find this tool extremely useful for parsing dmn files in javascript (via xml2js). But right now I am more interested in a similar feature for xml/html documents. Hoping if someone can point me to it. Thanks.