8 ms·
JC converts the output of popular command-line tools to JSON
- calvinmorrison 3y agoIn a certain sense, files everywhere is great, that's the promise of unix, or plan9 to a further extent. However, unstructured files, or files that all have their own formats, is also equally hampering. Trying to even parse an nginx log file can be annoying with just awk or some such. One of the big disadvantages is that large system rewrites and design changes cannot be executed in the linux userland. All to say, I'd love a smarter shell, I love files, I have my awk book sitting next to me, but I think it's high time to get some serious improvements on parsing data. In the same way programs are smart enough to know to render colored output or not, I'd love it if it could dump structured output (or not)
- mikepurvis 3y agoWhen it comes to parsing server logs, it's too bad the functionality can't be extracted out of something like logstash, since that's already basically doing the same thing. Though I guess the real endgame here is for upstream tools to eventually recognize the value and learn how to directly supply structured output.
- dale_glass 3y agoYou can get that out of journald. journalctl -o json And applications using journald directly can provide their own custom fields.
- hoherd 3y ago> In the same way programs are smart enough to know to render colored output or not, I'd love it if it could dump structured output (or not) The even lower hanging fruit is to implement json output as a command line argument in all cli tools. I would love to see this done for the gnu core utils.
- teddyh 3y agoAt least “du”, “env”, and “printenv” in Coreutils all support the “--null” option to separate output into being NUL-separated instead of being text lines.
- bryanlarsen 3y agoIt was a really pleasant surprise to find the "-j" option to do this for the "ip" command from the iproute2 project.
- mr_mitm 3y agoAFAIK the idea is that if you need that kind of interoperability in unix/gnu, you're supposed to write your tool in C and include some libraries. Clearly not realistic in many use cases.
- dale_glass 3y agoYup. It really grinds my gears that people came up with fairly decent ideas half a century ago, and a large amount of people decided to take that as gospel rather than as something to improve on. And it's like pulling teeth to get any improvement, because the moment somebody like Lennart tries to get rid of decades of old cruft, drama erupts. And even JSON is still not quite there. JSON is an okay-ish idea, but to do this properly what we need is a format that can expose things like datatypes. More like PowerShell. So that we can do amazing feats like treating a number like a number, and calculating differences between dates by doing $a - $b.
- CyberDildonics 3y agoIf you want something more complex and restrictive you could easily make it out of JSON. JSON works because it is simple and isn't being constantly distorted into something more complicated to cover niche use cases.
- zlg_codes 3y agoYeah dude I totally love it when journald eats my logging and power interruptions create unrecoverable log loss! You can forget just mounting a drive and checking its journald logs -- your machine has to be started by systemd in order to read journald logs! I totally love it when liars frontmanning the project say it's a project and an init system and a system layer that will replace everything. But it won't! Pay no attention to the inconsistent messaging, the "gentle pushes" to get other distros to use it, etc. Also, let's not mention he basically hoodwinked the entire community by switching teams to Microsoft. And people like you eat his work up! Are you sure you like free software?
- dale_glass 3y ago> Yeah dude I totally love it when journald eats my logging and power interruptions create unrecoverable log loss! So does everything else, because of consumer drives. Fun fact: filesystems by default only guarantee the integrity of the filesystem's structure itself. The promise is that after power loss, the basic structures of the filesystem won't be corrupt, but makes no big claims of reliability about the data written. Blocks of random junk in your data, blocks of NULLs, even parts of other files (maybe deleted data) are all things I've seen happen. Databases and the like take serious effort to ensure safety, but that greatly slows down performance. So it's kind of a hard sell for a log system that's not supposed to be a performance impact. For this contingency, you can do log shipping, but in general, system logs shouldn't be expected to be reliable in the face of a crash, since to my knowledge all normal daemons (including rsyslogd) buffer data before flushing to disk. > You can forget just mounting a drive and checking its journald logs -- your machine has to be started by systemd in order to read journald logs! No, it doesn't. There are file arguments to journalctl. Read the manpage, sheesh. > I totally love it when liars frontmanning the project say it's a project and an init system and a system layer that will replace everything. But it won't! Pay no attention to the inconsistent messaging, the "gentle pushes" to get other distros to use it, etc. Meh. Paranoia. > Also, let's not mention he basically hoodwinked the entire community by switching teams to Microsoft. And people like you eat his work up! Are you sure you like free software? Free Software is a licensing/distribution concept completely unrelated to whether one likes or not Microsoft's technical decisions. Some I really hate, and some are actually pretty cool. I'm for Free Software because I like the licensing philosophy, not because I believe Unix is the best thing since sliced bread. In fact I believe Unix started as a bunch of good ideas but that have not kept up and so needs a bit of work to remain a good system to use.
- da_chicken 3y agoThere is always Powershell. The trouble there is that it's so rooted in .Net and objects that it's very difficult to integrate with existing native commands on any platform.
- uxp8u61q 3y agoIf these "native commands" had a sensible output format, integrating them with powershell would be as simple as putting ConvertFrom-Json or ConvertFrom-Csv in the middle of the pipeline. And let's be real, it's as poorly "integrated" with "native commands" as bash or zsh is poorly integrated with native commands.
- da_chicken 3y agoYou're not wrong, but the fix is then, "hey, let's extend the functionality of literally every *nix command". Which is hard to achieve. Building sensible serialized object-oriented output is the same problem as building sensible object-oriented output. That's why jc exists, I suppose. There is always ConvertFrom-String. The problem is that it is one of the least intuitive and worst-performing commands I've used in Powershell. It's awful and I hate it. It's like writing sed and awk commands without the benefit of sed and awk's maturity and ubiquity. IMX, only Compare-Object has been worse.
- uxp8u61q 3y ago> "hey, let's extend the functionality of literally every *nix command". Which is hard to achieve. It's pretty much what powershell did, though.
- imtringued 3y agoIt sounds like there needs to be a monolithic solution built from the ground up for interoperation. Updating coreutils is a losing game at this point. Of course people will get angry when a well designed solution gradually takes over simply, because it is better.
- mistercow 3y agoPart of the problem is that the output of commands is both a UI and an API, and because any text UI can be used as an API, the human readable text gets priority. Shell scripting is therefore kind of like building third party browser extensions. You look and you guess, and then you hack some parser up based on your guess, and hope for the best. I actually wish there was just a third standard output for machine readable content, which your terminal doesn’t print by default. When you pipe, this output is what gets piped (unless you redirect), it’s expected to be jsonl, and the man page is expected to specify a contract. Then stdout can be for humans, and while you can parse it, you know what you’re doing is fragile. Of course, that’s totally backwards incompatible, and as long as we’re unrealistically reinventing CLIs from the foundations to modernize them, I have a long list of changes I’d make.
- theblazehen 3y agoHave you seen some of the existing projects currently working on it? Most well known is https://www.nushell.sh/ https://www.nushell.sh/, amongst some others
- chongli 3y agoI really want to agree because it seems to make so much sense in theory. It gets rid of the need to parse by standardizing on syntax. But it doesn't solve everything, so we still don't get to live the dream. Namely, it does not solve the issue of schemas, versioning, and migration. And this is a really big issue that threatens to derail the whole project. If my script runs a pipeline `a | b | c` and utility b gets updated with a breaking change to the schema, it breaks the entire script. Now I've got to go deep into the weeds to figure out what happened, and the breakage might not be visible in the human-readable output. So to debug I'll have to pass the flag to each tool to get it to print all the json to stdout, and then sit there eyeballing the json to figure out how the schema changed and what I need to do to fix my script. Seems like a big mess to me. Unless there's something I'm missing?
- kbrazil 3y agoThere could be a major schema change that breaks the contract, but one of the nice things about JSON output is that it allows the creation of new fields without affecting downstream consumers. That is, if I have a CLI program that spits out a list of IP addresses and one day I want to also output the corresponding dns names, I can simply add the "dns" field and existing pipelines will ignore the field and work just fine. This is better than grep/awking/etc. unstructured text to STDOUT because, depending on how the author decides to add the new field, it can easily break existing pipelines that rely on the shape of the data to stay the same.
- numbsafari 3y ago> Trying to even parse an nginx log file can be annoying with just awk or some such. You probably already know this, but for those who do not, you can configure nginx to generate JSON log output. Quite handy if you are aggregating structured logs across your stack.
- imtringued 3y agoThe problem is that nobody has built an actual ffi solution except maybe the GObject guys. C isn't an ffi, because you need a C compiler to make it work. By that I mean it is not an interface, but rather just C code whose calling part has been embedded into your application.
- pushedx 3y agoI salute whoever chooses to maintain this
- alex_suzuki 3y agoI wonder how they will address versions… `aws s3 ls | jc —-aws=1.2.3` What a nightmare.
- dj_mc_merlin 3y agoWhat about jc 'aws sts get-caller-identity' | jq [..] That way the aws process can be a subprocess of jc, which can read where the binary is and get its version automatically.
- jasonjayr 3y agoThat can get thorny, because it adds another level of shell-quoting/escapes, and that is a notorious vector for security problems.
- sesm 3y agojc already has this, see ‘jc dig example.com’ in examples. They call it ‘alternative magic syntax’, but IMO it should be the primary syntax, while piping and second-guessing the previous commands parameters and version should be used only in exceptional cases.
- dtech 3y agoAws cli isn't the best example because it supports outputting json natively. I'd expect this to not be a huge problem in practice because this is mostly for those well established unix cli tools of which the output has mostly ossified anyway. Many modern and frequently updated tools support native JSON output.
- hnlmorg 3y ago> Aws cli isn't the best example because it supports outputting json natively. The s3 sub command annoyingly doesn’t. Which I’m guessing is the reason the GP used that specifically.
- PreInternet01 3y agoOh, this is cool. I'm a huge proponent of CLI tools supporting sensible JSON output, and things like https://github.com/WireGuard/wireguard-tools/blob/master/contrib/json/wg-json https://github.com/WireGuard/wireguard-tools/blob/master/con... and PowerShell's |ConvertTo-Json are a huge part of my management/monitoring automation efforts. But, unfortunately, sensible is doing some heavy lifting here and reality is... well, reality. While the output of things like the LSI/Broadcom StorCLI 'suffix the command with J' approach and some of PowerShell's COM-hiding wrappers (which are depressingly common) is technically JSON, the end result is so mindbogglingly complex-slash-useless, that you're quickly forced to revert to 'OK, just run some regexes on the plain-text output' kludges anyway. Having said that, I'll definitely check this out. If the first example given, parsing dig output, is indeed representative of what this can reliably do, it should be interesting...
- zubairq 3y agoSimple idea, really great to see this!
- kbknapp 3y agoReally cool idea but this gives me anxiety just thinking about how it has to be maintained. Taking into account versions, command flags changing output, etc. all seems like a nightmare to maintain to the point where I'm assuming actual usage of this will work great for a few cases but quickly lose it's novelty beyond basic cases. Not to mention using `--<CMD>` for the tool seems like a poor choice as your help/manpage will end up being thousands of lines long because each new parser will require a new flag.
- verdverm 3y agoThis is one of the better use cases for LLMs, which have shown good capability at turning unstructured text into structured objects
- ninkendo 3y agoIf LLM’s were local and cheap, sure. They’re just too heavyweight of a tool to use for simple CLI output manipulation today. I don’t want to send everything to the cloud (and pay a fee), and even if it was a local LLM, I don’t want it to eat all my RAM and battery to do simple text manipulation. In 20 years, assuming some semblance of moore’s law still holds for storage/RAM/gpu, I’m right there with you.
- d3nj4l 3y agoOn my M1 Pro/16GB RAM mac I get decently fast, fully local LLMs which are good enough to do this sort of thing. I use them in scripts all the time. Granted, I haven’t checked the impact on the battery life I get, but I definitely haven’t noticed any differences in my regular use.
- nickster 3y agoAll output being an object is one of my favorite things about powershell. I miss it when I have to write a bash script.
- timetraveller26 3y agoDoes anybody know of a listof modern unix command-line tools accepting a --json option? It may even be useful to add that information to this repo.
- chungy 3y agoBasically everything on FreeBSD supports it via libxo.
- user3939382 3y agoProbably not what you had in mind but, AWS CLI.
- pirates 3y agokubectl with “-o json”
- geraldcombs 3y agoTShark (the CLI companion to Wireshark) does with the `-T json` flag.
- bravetraveler 3y agoI don't have a list, but the modern replacement for "ifconfig" does JSON: "ip" As does "lldpctl" Ansible provides details about systems in JSON called 'facts'. The intention is to use these to inform automation
- pkkm 3y agolsblk accepts a --json flag and can give you a lot of information (try lsblk --json --output-all). Very useful if your script needs to check what disks and partitions there are in the system.
- Mister_Snuggles 3y agoIn FreeBSD, this problem was solved with libxo[0]: $ ps --libxo=json | jq { "process-information": { "process": [ { "pid": "41389", "terminal-name": "0 ", "state": "Is", "cpu-time": "0:00.01", "command": "-bash (bash)" }, [...] It's not perfect though. ls had support, but it was removed for reasons[1]. It's not supported by all of the utilities, etc. This seems to be a great stop-gap with parsers for a LOT of different commands, but it relies on parsing text output that's not necessarily designed to be parsed. It would be nice if utilities coalesced around a common flag to emit structured output. In PowerShell, structured output is the default and it seems to work very well. This is probably too far for Unix/Linux, but a standard "--json" flag would go a long way to getting the same benefits. [0] https://wiki.freebsd.org/LibXo https://wiki.freebsd.org/LibXo [1] https://reviews.freebsd.org/D13959 https://reviews.freebsd.org/D13959
- imtringued 3y agoNow they only need to do the same thing for input and let the operating system or the shell handle the argument parsing so that it is consistent accross the entire operating system.
- supriyo-biswas 3y agoSimilarly, in SerentityOS, stuff under /proc return JSON data rather than unstructured text files. A better, structural way in which this could be fixed is to allow data structures to be exported in ELFs and have those data structures serialized into terminal output, which can then be outputted in the preferred format of the user, such as JSON, YAML, or processed accordingly.
- crotchfire 3y agoI mean if you have a filesystem you've already got a way to tree-structure your data...
- nijave 3y agoLibxo is neat, in theory, but it seems like applications are left to implement their own logic for a given output format rather than being able to pass a structure to libxo and let it do the formatting. I can't remember the exact utility--I think it was iostat--would use string interpolation to format output lines in JSON and combined with certains flags produced completely mangled output. Not sure if things have improved but I would have expected something like JSON lines when interval is provided. Powershell and kubectl are miles ahead of libxo in useability imo
- timetraveller26 3y agoThe modern kids want it all easy, when I was learning Linux we used null delimiters, xargs, cut, sed & awk and that was enough! \s
- rplnt 3y agoWish it would have automatic parser selection by default. Even if just for a (possible) selected subset. Typing `foo | jc | jq ...` would be more convenient than `foo | jc --foo | jq ...`.
- mikecarlton 3y agoIt supports `jc foo | jq` which is quite handy. E.g. `jc dig google.com txt | jq '.[]|.answer[]|.data'`
- kbrazil 3y agoAlso, `jc` automatically selects the correct /proc/file parser so you can just do `jc /proc/meminfo` or `cat /proc/meminfo | jc --proc` without specifying the actual proc parser (though you can do that if you want) Disclaimer: I'm the author of `jc`.
- js2 3y agoPrevious discussion (linked to from the project's readme): https://news.ycombinator.com/item?id=28266193 https://news.ycombinator.com/item?id=28266193
- crazysim 3y agoJesus Christ, it's JSON for Bourne! I wonder how well this could work/interact with Powershell.
- majkinetor 3y agoPerfectly: <json stringn input> | ConvertFrom-Json
- Pxtl 3y agoHonestly this is half the reason I use Powershell for everything. Bash-like experience but everything returns objects. It's a messy, hairy, awful language. Consistently inconsistent, dynamically-typed in the worst ways, "two googles per line" complexity, etc. But for the convenience of being able to combine shell-like access to various tools and platforms combined with the "everything is a stream of objects" model, it can't be beat in my experience. And you can still do all the bash-like things for tools that don't have good Powershell wrappers that will convert their text-streams into objects. Which, sadly, is just about everything.
- zlg_codes 3y agoI love this comment, measuring complexity via googles/line. That really sends home how difficult it is to grok PowerShell. What are you building with it that would be harder in bash or another shell? I'm not seeing the value of passing around opaque objects instead of text.
- Pxtl 3y agoThe objects aren't opaque, you can catch them in a variable and inspect it with get-member. I find it very nice to work interactively when developing a script -- call a getter command and either let it out to the console or store it in variable and then play with the variable. Jason/XML/csv files and API results all get turned into PS objects easily, as do things with config objects like file permissions or SQL servers or whatnot. There's a bunch of dumb ideas like non-file filesystems for things like SQL and the Windows Registry, but the basic concept of "call command-line tools and navigate the filesystem with the ease of Bash, but also work with objects like Python or JS" works well for me.
- abound 3y agoNushell [1] ends up at mostly the same place (structured data from shell commands) with a different approach, mostly just being a shell itself. [1] http://www.nushell.sh/ http://www.nushell.sh/
- saghm 3y agoI had glanced at nunshell every now and then since it was initially announced, but it wasn't until a month or two before that I finally really "got" the point of it. I was trying to write a script to look through all of the files in a directory matching a certain pattern and pruning them to get rid of ones with modified timestamps within 10 minutes of each other. I remembered that nushell was supposed to be good for things like this, and after playing around with it for a minute, it finally "clicked" and now I'm hooked. Even when dealing with unstructured data, there's a lot of power in being able to convert it even into something as a list of records (sort of like structs) and process it from there.
- deleted 3y ago[deleted]
- danyx23 3y agoNushell actually pairs really well with JC, given that nushell has a "from json" operation. I recorded a video some time ago that shows a few nice features of Nushell and I bring up combining it with jc at around minute 19: https://www.youtube.com/watch?v=KF5dtxVsn1E https://www.youtube.com/watch?v=KF5dtxVsn1E
- freedomben 3y agoReally glad to see this is already packaged for most linux distributions. So many utilities nowadays seem be written in Python, and python apps are such a PITA to install without package manager packages. There's so many different ways to do it and everything seems to be a little different. Some will require root and try to install on top of package manager owned locations, which is a nightmare. Fedora Toolbox has been wonderful for this exact use case (installing Python tools), but for utilities like this that will be part of a bash pipe chain for me, toolbox won't cut it.
- Spivak 3y agoInstalling self-contained programs written in Python not packaged for your distro: PIPX_HOME=/usr/local/pipx PIPX_BIN_DIR=/usr/local/bin pipx install app==1.2.3 It sets up an isolated install for each app with only its deps and makes it transparent. The distro installation tree of Python is for the exclusive use of your distro because core apps cloud-init, dnf, firewalld are built against those versions.
- freedomben 3y agothank you! That's amazingly helpful. I had no idea pipx was a thing For others: https://github.com/pypa/pipx https://github.com/pypa/pipx It's also in the Fedora repos: dnf install -y pipx
- bottled_poe 3y agoI’d bet money on people (here) using tools like this to process millions of records or more. It’s a sad truth those people won’t have jobs in a few years when AI, which will know better, takes hold :(
- deleted 3y ago[deleted]
- moss2 3y agoExcellent
- sesm 3y agoIMO ‘jc dig example.com’ should be the primary syntax, because ‘dig example.com | jc —dig’ has to retroactively guess the flags and parameters of previous command to parse the output.
- mejutoco 3y agoI wonder if a tool could parse any terminal output into json in a really dumb and deterministic way: { "lines": [ "line1 bla bla", "line1 bla bla", ], "words": [ "word1", "word2", ], } With enough representations (maybe multiple of the same thing) to make it easy to process, without knowing sed or similar. It seems hacky but it would not require any maintenance for each command, and would only change if the actual output changes.
- hk__2 3y agoWhat’s the point of JSON, then?
- mejutoco 3y ago> without knowing sed or similar Not having to know sed or a similar tool. Most unix tools are structured in lines, columns, etc. anyway.
- hk__2 3y ago> Not having to know sed or a similar tool. But you get a JSON with a list of lines that you still have to process in some way. Instead of having a program that reads the input line by line, you read a JSON that contains a list of lines.
- mejutoco 3y agoYou could have multiple representations of it. For example split by tabs to represent columns in a cronfile, so you could choose lines to get the second lines or columns to get the 3rd column etc. Anyway, just brainstorming here.
- codedokode 3y agoThey are doing all wrong, instead of using human-readable and machine-readable formats, all CLI tools should use human-and-machine-readable format.
- mathfailure 3y agoYou mean YAML?
- codedokode 3y agoYAML is over-complicated.
- Cyph0n 3y agoInteresting project! But I expected them to be using textfsm (or something similar) as a first step parser. textfsm is heavily used to parse CLI outputs in networking devices. https://github.com/google/textfsm https://github.com/google/textfsm
- nailer 3y agoNice. Too many "lets fix the command line" (nushell, pwsh) have noble goals, but also start with "first let's boil the ocean". We need to easily ingest old shitty text output for a little while to move to the new world of structured IO.
- kazinator 3y ago$ dig example.com | txr dig.txr [{"query_time":"1","rcvd":"56","answer_num":1,"status":"NOERROR", "when_epoch":1702030676,"opcode":"QUERY","udp":"65494","opt_pseudosection":{"edns":{"udp":65494,"flags":[],"version":"0"}}, "query_num":1,"question":{"name":"example.com.","type":"A","class":"IN"}, "server":"127.0.0.53#53(127.0.0.53)","id":"48295","authority_num":0, "answer":[{"name":"example.com.","type":"A","data":"93.184.216.34","ttl":"4441", "class":"IN"}], "additional_num":1,"when":"Fri Dec 08 10:17:56 PST 2023"}] $ cat dig.txr @(bind sep @#/[\s\t]+/) @(skip) ;; ->>HEADER<<- opcode: @opcode, status: @status, id: @id ;; flags: qr rd ra; QUERY: @query, ANSWER: @answer, AUTHORITY: @auth, ADDITIONAL: @additional @(skip) ;; OPT PSEUDOSECTION: ; EDNS: version: @edns_ver, flags:@flags; udp: @udp @(skip) ;; QUESTION SECTION: ;@qname@sep@qclass@sep@qtype @(skip) ;; ANSWER SECTION: @aname@sep@ttl@sep@aclass@sep@atype@sep@data ;; Query time: @qtime msec ;; SERVER: @server ;; WHEN: @when ;; MSG SIZE rcvd: @rcvd @(do (put-jsonl #J^[{ "id" : ~id, "opcode" : ~opcode, "status" : ~status, "udp" : ~udp, "query_num" : ~(tofloat query), "answer_num" : ~(tofloat answer), "authority_num" : ~(tofloat auth), "additional_num" : ~(tofloat additional), "opt_pseudosection" : { "edns" : { "version" : ~edns_ver, "flags" : [], "udp" : ~(tofloat udp) } }, "question" : { "name" : ~qname, "class" : ~qclass, "type" : ~qtype }, "answer" : [ { "name" : ~aname, "class" : ~aclass, "type" : ~atype, "ttl" : ~ttl, "data" : ~data } ], "query_time" : ~qtime, "server" : ~server, "rcvd" : ~rcvd, "when" : ~when, "when_epoch" : ~(time-parse "%a %b %d %T %Z %Y" when).(time-utc) }])) The latest TXR (292 as of time of writing) allows integers in JSON data, so (toint query) could be used.
- AtlasBarfed 3y agoMy God, doesn't properly handle ls?
- da39a3ee 3y agoAwesome, does it work for man pages? They're a huge elephant in the room -- people get really upset if you point out that man pages are an unsearchable abomination, locking away vast amounts of important information about unix systems in an unparseable mess. But, it's true.
- Animats 3y agoI always felt that was a design flaw of UNIX. Programs accept command line parameters and environment variables as input, but all they output for their calling program is an integer exit code. It's not like exit(II) has an argv and an argc. GUI programs that call command line programs thus tend to be nearly blind to what the called program did. You can't treat command line programs as subroutines. I know why it worked that way in Research Unix for the PDP-11. It's a property of the hokey trick used to make fork(II) work on tiny machines. It didn't have to stay that way for four decades.