9 ms·
So I've been writing shell scripts for about two decades, about 75% of that time professionally. Parsing the unstructured, text-based output of utilities is not
by c0l0 5y ago
So I've been writing shell scripts for about two decades, about 75% of that time professionally. Parsing the unstructured, text-based output of utilities is not the problem for anyone who's had maybe a few weeks of training. Most `... | grep ... | cut ... | sed ... | awk ...`-abominations the post laments can be replaced by a single informed call to `awk`, making everything a lot more elegant and concise.
Having JSON as an intermediate representation to work on instead is not going to save anyone - what we'd really need is for all tool versions/variants output on all platforms (all GNU/Linux distros, all the BSDs, all embedded Linux variants, all commercial UNICES, etc.) to be the same, all the time. That's not going to happen, so shell scripting is going to stay messy.
Also, for my INTERACTIVE shell, anyone can try to pry free-form, text-based and semi-structured output from my cold, dead hands. JSON or YAML output might be an acceptable compromise between being easy to parse and bearable for human consumption, but for my daily work, I would rather have my tools make it easy for me, the human part in the whole equation, and not some parsing logic that might not even (need to) exist. Shell scripting provides most of its value from the fact that since I'm in its repl-of-sorts all the time, I can translate that familiarity to scripts and executables effortlessly, and I would not want that going away. But I am rather certain it would, if we had JSON (or another form of more structured data interchange syntax) adopted as the "universal" interface between UNIX tools.
- kbrazil 5y agoI don’t think the issue is that it’s hard to manually parse. The problem is that it’s hard for someone else to read your ad-hoc parser years later and reason about what you did if they need to modify it. Disclaimer: I am the author of the article and JC.
- pdimitar 5y agoThis is even more true for the ungodly long `jq` incantations that people write. It's like I get it, the old way is ugly and not always easy to decipher but at least it's shorter and your chances of understanding it are better. I've had both -- the classic piped chain of UNIX commands and various JSON-producing tools piped to `jq`. The former were still easier to work with.
- kbrazil 5y agoYes, I have seen those too! That’s why I also wrote Jello, which is like jq but uses pure python without the boilerplate. Python is nearly universal now and typically easy to read, though more verbose. Jq is just as much a write-once tool as awk and perl for more complex queries. For simple attribute calls, though, it’s both terse and readable.
- NegativeLatency 5y agoI wrote something similar by description in ruby, I’d be curious to see your python implementation. I considered python but Ruby’s easier chaining with map/filter/etc made it easier for me to use when writing just one line with it to transform some json. https://github.com/nburns/utilities/blob/master/rson https://github.com/nburns/utilities/blob/master/rson
- kbrazil 5y agoSure thing! https://github.com/kellyjonbrazil/jello https://github.com/kellyjonbrazil/jello Other languages are superior in their handling of maps/arrays, but Python is just so damned popular now I thought it was a good choice to democratize JSON handling. https://blog.kellybrazil.com/2020/03/25/jello-the-jq-alternative-for-pythonistas/ https://blog.kellybrazil.com/2020/03/25/jello-the-jq-alterna...
- hnlmorg 5y agoThis is why I wrote murex shell (https://github.com/lmorg/murex https://github.com/lmorg/murex), it's an alternative $SHELL, so you'd use it in place of Bash or Zsh, but it's optimised for modern DevOps tools. Which means JSON and YAML are first class citizens. It's syntax isn't 100% POSIX compatible so there is some new stuff to learn but it works with all the existing POSIX tools and is more readable than AWK and Perl but while also being terse enough to write one liners.
- hackerbrother 5y ago(Sorry, this is my first time trying to do a formatted comment here.) What I like to do is comments like: /* * Collates * [ * { * id: 4 * dept: 'oncology', * name: 'Joe S.' * } * . * . * . * ] * * into * * { * 4: { // id * 'Joe. S': { // name * dept: oncology * } * . * . * . * } */ ( Then insert horrible one-liner that does the transformation. )
- rodgerd 5y agoBut it's very gratifying to the person who re-invents the wheel for the millionth time and feels good because of it. The Unix command line is the Candy Crush of interfaces, giving us a little dopamine hit every time we solve a problem that didn't need to be solved.
- jl6 5y ago> what we'd really need is for all tool versions/variants output on all platforms (all GNU/Linux distros, all the BSDs, all embedded Linux variants, all commercial UNICES, etc.) to be the same, all the time The industry has approximated this state of affairs by approximating a GNU+Linux monoculture. Doesn’t matter that BSD tar behaves differently if ~nobody uses it.
- deleted 5y ago[deleted]
- 1MachineElf 5y agoAbout this few weeks training you mentioned for Unix-style parsing of unstructured text - what resources do you recommend?
- tuatoru 5y agoThe GNU Awk manual has several practical examples in it - as manuals should. (Looking at you, nearly all man pages in linux distros.)
- yesenadam 5y agoThe original 1988 book The AWK Programming Language by A, W and K is, in my opinion, one of the finest pieces of documentation ever written, on any subject. A joy to read and be instructed by.
- seewhat 5y agoI have occasionally dipped into this book for instruction and consider it a freely accessible reference: Unix Text Processing, Dougherty and O'Reilly, 1987 https://www.oreilly.com/openbook/utp/ https://www.oreilly.com/openbook/utp/ Cherry pick chapters like those on the Shell and AWK, and avoid those on troff, macros etc unless specifically interested.
- athrowaway3z 5y agoI know enough awk to know it solves my problem, but use it to little to remember how, and the development experience is never very clean. What i would like is an cli awk ide. (Rolls of the tongue right? ) Ideally i could write ... | awk-ide "scripts/thing.awk" | ... If it exists, run it. Otherwise have it open an editor session that runs an awk program as i type it on some buffered data, shows the result, and have some hints on what syntax/variables are available.
- cle 5y agoI use this occasionally in Emacs with awk-mode to live edit awk programs against a data file: https://github.com/danlamanna/live-awk-mode https://github.com/danlamanna/live-awk-mode
- xelxebar 5y agoWhat about something like while inotifywait --event modify,move_self,delete_self script.awk; do awk -f script.awk input.txt; done in a tmux pane on the right with vim/emacs/whatever on the left? Note: You need to watch the move_self and delete_self events, since editors might use some file move/deletion dance instead of a direct modify to help prevent data loss in the event of a crash.
- kbrazil 5y agoI wrote something similar to this to query JSON and JSON lines with python instead of awk for text. It’s called Jellex (Jello Explorer) which is a TUI front-end to Jello. Jello is a python analog to JQ. https://github.com/kellyjonbrazil/jellex https://github.com/kellyjonbrazil/jellex
- JJMcJ 5y ago> awk Or Perl. That's what Perl was originally created for. Or sed. If awk is too mainstream for you. Or Python. Or just write the ugly pipeline in your script, and promptly forget how it actually works. I've done that a lot.
- u801e 5y ago> Or sed After learning how to use ed in scripts, I've found it's actually easier to use it compared to sed because it effectively has random access through the input rather than going from beginning to end. But both awk and Perl have the advantage of storing values in variables over sed or ed.
- Koshkin 5y agosed, being a stream editor, is more efficient than ed, in that it does not need to keep the entire file in memory.
- u801e 5y agoTrue, but if the file isn't that big, then the difference isn't noticable. Being able to do something like /some string/-2 d Which deletes a line 2 lines above where some string matched is something that's trivial to do in ed, but takes a bit of work in sed.
- kaliszad 5y agoI can see the potential in Babashka and other Clojure-based systems from the approach point of view. There needs to be a simple integrated editor or much better readline implementation though that needs to work out of the box. That should improve the interactivity in a major way. We also need to have a set of very simple tools to work with files, network connections (e.g. traffic dumps, netcat) and other typical shell tasks. This isn't really as efficient when used interactively in the Clojure ecosystem or anywhere else from what I have seen besides just using e.g. ls, du, df, fdisk, tcpdump, nc ... More long term, I have high hopes of having an environment above the kernel that basically does what /proc and /sys and others do currently. Programs equivalent to those in /bin /usr/bin etc. would be just dynamically loadable modules or built into the basic tool set. No shell scripts anymore, no random filesystems with custom formats for everything, just a JITed VM that has a strong set of tools but anybody can extend it with either stuff written in some Clojure-like language, or something compiling to the VM or something native that has some kind of interface (FFI?) to be usable from the programs running on the VM (e.g. for cryptographic stuff or stuff that needs to be as efficient as possible). We would also need something like SSH but for structured data that would support a SHELL/ REPL-like workflow as a byproduct but really be meant for more or less high performance, efficient communication (e.g. useable even for large file copy operations and such). In the end, parts of this system could connect to a in kernel VM (BPF?) and execute there but we would interact with them using the nice, structured REPL. This would be a huge undertaking but I can't really see, how we can radically improve the efficiency of work with the current systems. It seems, we are mostly just patching old approaches to do new tricks and to me, it seems to be falling apart. The complexity we impose upon us is crushing and I don't think all of it is necessary.
- coliveira 5y agoYou mentioned one pain point of shell scripting, variability among platforms. The other pain point is that different tools have different methods to treat errors and warnings. This makes debugging shell scripts a nightmare, compare to scripting languages like python. If you need to debug your work, you need to check several levels of script vs. commands, and use different ways to check for errors depending on what is causing the problem. If I had to decide on a tool to replace shell script I would vote for tcl, since it maintains many of the advantages of the shell but provides a better handling of the programming aspect. Unfortunately these days it seems that you either use shell or some full-scale language like python.
- pjmlp 5y agoTcl is also a full scale language, the startup I was on during the com wave was shipping products written in Tcl + C, just like Python.
- 1vuio0pswjnm7 5y agoCan JSON compare with line-based data, which many original UNIX utilities seem to target. JSON's design assumes the user can read the entire file into memory. It's really easy to exhaust resources with JSON. And fast, crash-proof JSON parsers become more challenging to write. Whereas it's not nearly as easy to exhaust memory with line-based data processing nor to crash utilities that read line-by-line, e.g., sed. If lines are too long, I can chop them down to a reasonable size on some sentinel. IMO, JSON, like Javascript, is web/browser centric. For someone who rarely uses a browser or Javascript and is comfortable with UNIX, e.g., yours truly, JSON is not particularly advantageous. For large data, line-based is more robust (and memory-efficient) than JSON, IME. Better than JSON is netstrings or bencode. https://en.wikipedia.org/wiki/Netstrings https://en.wikipedia.org/wiki/Netstrings https://en.wikipedia.org/wiki/Bencode https://en.wikipedia.org/wiki/Bencode
- kbrazil 5y agoJSON Lines is often used for this type of thing. https://jsonlines.org/ https://jsonlines.org/
- mananaysiempre 5y ago> JSON’s design assumes the user can read the entire file into memory No? The design of most JSON libraries assumes that, but there are perfectly good incremental JSON parsers out there[1–3]. It’s just that people don’t seem to have figured out a good API for not-completely-incremental parsing (please prove me wrong here!), but this applies equally to any structured data format as soon as you want to pull out pieces of data that are nested more than one level down. The lack of length prefixes in JSON does indeed make a solid parser somewhat more difficult, but you get the ability to author and validate it manually instead. All in all a draw and not because of the incremental parsing thing. (Tabular or otherwise homogeneous data is indeed reprsented wastefully, but unless the individual records are huge json+gzip is a perfectly serviceable “worse-is-better” solution, and its self-describing nature along with support for structured cell values can at times make for a better experience than TSV. And at other times not.) [1] https://github.com/ICRAR/ijson https://github.com/ICRAR/ijson [2] https://github.com/AMDmi3/jsonslicer https://github.com/AMDmi3/jsonslicer [3] https://github.com/danielyule/naya https://github.com/danielyule/naya
- emodendroket 5y agoPowerShell is just so much nicer to use than anything where text munging is the only way to do things, and it’s just as “pluggable” as Unix shell commands. And it can output text (or JSON or XML or YAML) or whatever you want easily by piping into relevant commands (or just not piping anywhere if you wanted text). I don’t imagine those are going anywhere anytime soon but I think it is wrong to say the system cannot be improved upon.
- diegocg 5y agoI had to write a powershell script and my impression was that objects are a much worse interface than the worst text manipulation tricks. Because you are passing objects around it quickly becomes really hard to understand what is going on on the code and which object interfaces the script is using, because the object lives behind the pipe so to speak. It was my impression that it is too easy to create unreadable powershell scripts It also was unbearably slow in a modern computer, to the point that it made debugging difficult.
- emodendroket 5y agoI'm not sure I understand exactly what you mean. I would rather read Select { $_.Property } than some awk or sed stuff to fetch the same thing any day. I'm not sure how this is different than most other modern languages used to write scripts.
- smaudet 5y agoWhile you can produce unreadable sphagetti in any language, powershell is just a poor mans Python without any real language features. I'd much rather use csharp(script) if I have a need to interface with some ms stuff (.net system libraries or whatever), powershell is just underwhelming and bloated without any real performance or productivity gains. Unix shells, for all their idiosynchronacies, tend to focus on small bits of functionality directly composed from common shell or system utilities. The complexities lie in outdated design, not in the language or form. We could definitely do with an upgrade to the underlying tooling, I don't find anything otherwise lacking in shell scripting otherwise. I don't agree with the OP that we need a web readable data format - perhaps the jq syntax could be useful but the best part of shell scripting is it is approachable by humans - if it is not you are doing too much and should be writing your missile control program in python or c, or similar. I dont think it should be easier to produce functioning junk nor should it be a requirement to master a data syntax in order to print a formated date. Maybe you need a different date program if you need to do something special.