13 ms·
Text Processing in the Shell
- seemslegit 7y agoJust Don't. Unix-style text stream processing was super cool in 80s-90s but is born tech-debt today.
- thekelvinliu 7y agowhat do you do instead?
- worthless-trash 7y agoNot OP, but I imagine he's bought into the Hadoop and other big data stories. Most data processing of large data sets can probably be done with standard UNIX tools. There is nothing new under the sun, it's all just rebranded.
- alkonaut 7y agoIf you do "object manipulation" e.g. columns/records/objects etc then at least Powershell offers some sanity.
- hnlmorg 7y agoShameless plug but this is what I wrote murex for. Murex is a "UNIX" shell that's keeps enough similarities with POSIX that you can use it like a Bash REPL with (hopefully) minimal disruption but it breaks from POSIX compatibility where it makes sense. And one significant area it does break compatibility is how it handles structured data formats like JSON, S-Expressions, CSV's, and other tabulated data (to name a few). It's designed from the ground up to support object manipulation while still retaining compatibility with the UNIX pipeline. I do this by building a suite of builtin tools that are aware of structured data files (primarily because that information is passed down the pipeline as a data-type) but it still breaks into normal pipeline when forking an external executable. https://github.com/lmorg/murex https://github.com/lmorg/murex
- oweiler 7y agoWe use Groovy or Kotlin with KScript.
- zmmmmm 7y agoGroovy is particularly nice I find. Completely cross platform and supports command line args similar to Perl to enable inline / pipe style processing easily, but if you are using Java in your back end you can throw any of your business logic in there too and use those.
- kuschku 7y agoOutput data as JSON, manipulate and show it with jq
- bryanrasmussen 7y agonot all data comes from sources that you control and have chosen how to output.
- BaltoRouberol 7y agohttps://github.com/kellyjonbrazil/jc https://github.com/kellyjonbrazil/jc can come pretty handy there
- bryanrasmussen 7y agothanks!
- enriquto 7y agooh my god, why?! just why!? What the world needs is the inverse program of "jc", where an unparseable json string is expanded into a flat list of lines all of the form "field.subfield=value"
- bryanrasmussen 7y agook well I can understand not wanting json when all you need is something simpler, but I'm not sure if I understand unparseable - I mean if it is JSON then it is parseable.
- enriquto 7y agoYou can only parse json easily by using json libraries. Plain text, or "field=value" pairs, you can easily cut(1) or grep(1), or sed(1) to your pleasure. This is what I mean by parseable. Parseable trivially by tools that do not understand the format. I can also sed and awk json files, and I do, but it is extremely painful; and more often than not these files are nothing more than simple lists of variables with values, for which the use of json is a ridiculous overkill.
- seemslegit 7y agoUse any modern programming language.
- spurdoman77 7y agoThe gnu tools are super fast and do the job. Ive seen people spinning their own solutions which end up being super slow and arguable take more time to develop. With any gnu tools it is great that they will stay there for your life and are usually by default installed on every system
- seemslegit 7y agoYeah, except anything that is not trivial ends up being write-only
- matvore 7y agoModern programming language features are overrated. I prefer to choose the tools that are the most portable and involve the least social friction (i.e. something that is widely understood by people I work with). For "fun" projects (stuff I'm not paid for) and workflow optimization, I just care about portability, which means C (with heavy use of the C stream library, a simple collections library of about 200 LOC, and occasionally POSIX syscalls) and shell scripting. After spending a lot of time learning languages as a hobby, I just don't believe the dark corners and warts of C and shell scripting are any worse than other languages.
- dejj 7y agoI love to cook 5-course meals for friends and family. But sometimes when I'm all alone by myself, I slice up a bun and slap meat on it. It's as fast and tastes just as good as in the 80ies.
- seemslegit 7y agoSandwhiches don't need to be maintained
- Koshkin 7y agoSure, but try to eat a juicy sandwich that was made in the 80s.
- jbkiv 7y agoI really like that post, this is very good. I always loved NLP and that gave me a different angle. Where are you going now? I would love to know.
- SomewhatLikely 7y agoAre you a bot? This has nothing to do with NLP.
- jbkiv 7y agoNope! but that reply was intended for a different post, sorry :-)
- MR4D 7y agoI wish man pages were this good!
- bori5 7y agohttps://tldr.sh/ https://tldr.sh/
- leadingthenet 7y agoEven better: https://github.com/chubin/cheat.sh https://github.com/chubin/cheat.sh This includes man pages from tldr, and more! The command line utility had been a great help for me over the past few months.
- bori5 7y agoThank you, did not know about this one!
- knolan 7y agoI like cheat.sh, works great with curl. http://cheat.sh/ http://cheat.sh/
- boshomi 7y agoPleased add the "join"
- RMPR 7y agoA little suggestion for the authors, they mentioned xargs, I think [GNU parallel](https://www.gnu.org/software/parallel/ https://www.gnu.org/software/parallel/) might work a mention too, since it is a kind of modern successor that can use many computers to run tasks.
- shp0ngle 7y agoSometimes I find it strange - in both good and bad way - that we are, in 2020, learning tools and languages designed and build in the 80s, with models and constraints of the time, with 40 years of layers of backwards compatibility, and actually going sometimes back to the 70s I am still learning tools designed around the constraints of teleprinters Sure, it’s the same on Windows side (and macOS side with their classic OS compatibility layers still present, like all the HFS stuff). Not bashing bash here. Surely our computers have very different models of operation than PDP-11, yet we are sometimes pretending it doesn’t
- throwaway_pdp09 7y agoI find these posts annoying that say we're anchored in past tech and we must be able to do better, but never suggest how. Please make a suggestion or two. (NB ultimately our character handling is based on 'writing' which goes back thousands of years, not 50, and it survives well).
- zokier 7y agoStructured data and tools that operate on that structure. Maybe steal a thing or two from LISP machines. I remember the one time I had to work on an AS/400 (iirc) how interesting it was that one of the fundamental building blocks were records instead of dumb files. PowerShell has interesting ideas too, but it'd need to be far more built-in and integrated to the OS to truly shine.
- JdeBP 7y agoI suggest learning from history. The doing better has already happened. The 1960s terminal I/O model evolved into the console I/O model during the 1980s; from firmware-mediated access to CGA, through the VIO+KBD+MOU subsystems of OS/2, to Windows NT console objects. * http://jdebp.uk./FGA/tui-console-and-terminal-paradigms.html http://jdebp.uk./FGA/tui-console-and-terminal-paradigms.html * https://news.ycombinator.com/item?id=17238350 https://news.ycombinator.com/item?id=17238350
- throwaway_pdp09 7y ago
- SPBS 7y agoI've always viewed `awk '!a[$0]++'` as superior to `sort | uniq` because it preserves order and does not have to sort the data first before deduplicating. But `sort | uniq` is much easier to remember.
- asicsp 7y agoOr just `sort -u` (if you are using GNU sort, not sure about other implementations) Another difference is that sort is optimized to handle large files [0] [0] https://unix.stackexchange.com/questions/279096/scalability-of-sort-u-for-gigantic-files https://unix.stackexchange.com/questions/279096/scalability-...
- dyu 7y agoIt appears sort has a --stable flag for order preservation. I actually thought sort was stable by default already, but now I know.
- fnord123 7y agoThis is cool for English text. But once you get Unicode with various ways to represent é, whew lad. This get shitty quickly in the shell.
- zokier 7y agoI would generalize that further and say that shell processing works well when your data is well-behaved wrt the thing you are trying to do. But once you start venturing into untrusted data, escaping, and yes, Unicode, things can become hairy. So basically you should always be doing some sanity checks when working in shell
- bmn__ 7y agoEnglish is Unicode. Pretending otherwise would be quite naïve. https://www.azabani.com/pages/gbu/#slide4 https://www.azabani.com/pages/gbu/#slide4
- fnord123 7y agoYou claim is not clear. "Unicode with various ways to represent é" is a shit show to parse using shell tools. e.g. Try scraping Spanish language Twitter feeds. When I have done this kind of work, I made a tool to canonicalize glyphs and had to put it between every step of a pipeline.
- dublin 7y agoJust drop the eighth bit and use 7-bit ASCII like God intended and it's not a problem... :-)
- deleted 7y ago[deleted]
- latenightcoding 7y agoOP you should mention perl one-liners in upcoming chapters https://catonmat.net/introduction-to-perl-one-liners https://catonmat.net/introduction-to-perl-one-liners
- thesuperbigfrog 7y agoAgreed. Perl makes it easy to do complex text processing and can replace many individual command line text processing tools. Recent versions of Perl also support UTF-8 so they can support text processing in different natural languages or internationalization needs. See https://en.wikibooks.org/wiki/Perl_Programming/Unicode_UTF-8 https://en.wikibooks.org/wiki/Perl_Programming/Unicode_UTF-8
- bmn__ 7y ago> Recent versions of Perl Make sure it's recent enough and released after 2002!
- thesuperbigfrog 7y agoI was thinking of Perl versions 5.14 (released in 2011) and later since that release fixed several Unicode-related bugs. There have been other improvements and fixes in the versions up to 5.30, so the Unicode support now is pretty transparent. Many of the classic command line text processing tools are not Unicode aware.
- superasn 7y agoTotally agree. I always use Perl instead of sed because sed doesn't support PCRE and it's far easier to write something like '\d+' instead of [:digit:] or whatever the sed equivalents are.
- nailer 7y agoText processing is scraping. Modern shells have structured output from their stdlib, allow you to pipe to 'where' and 'select', and can read JSON, yaml, etc natively.
- Gnouc 7y agoI invite you to read https://unix.stackexchange.com/q/169716/38906 https://unix.stackexchange.com/q/169716/38906
- clircle 7y agoI'll preface my question by saying that I'm not a dev. Why do this kind of work in the shell? Isn't it better to do this in a programming language that can run on all operating systems? What are Windows users supposed to do?
- asicsp 7y agoWell, there are ways to use such commands on windows too, like WSL, git-bash, cygwin, etc. And, there's powershell on windows (though I haven't used it and not sure about its capabilities) As to why use shell, that depends on your use case and working environment. Shell is something like an IDE [0] where you can solve multiple tasks from single environment. You don't have to use multiple programs (window manager, text editor, IDE, etc). Since it is all text, you can save and repeat a command, share it with others, edit a previously written command, etc. This is quite different from a GUI based workflow. Personally, I find using command line more productive, but as mentioned earlier, it'll depend on the task at hand. [0] https://sanctum.geek.nz/arabesque/series/unix-as-ide/ https://sanctum.geek.nz/arabesque/series/unix-as-ide/
- vlovich123 7y ago> Why do this kind of work in the shell It can be quicker & easier to write. Also individually these tools can outperform any code you write by hand. Shells are also common on a far wider range of operating systems than typical programming languages like Python et all. Additionally if you want to improve your productivity you may create your own shortcuts (e.g. "build_myproject" which understands what that entails & may involve some amount of text processing among other things). It's typically far more convenient (shorter, simpler & generally easier to understand) to invoke other programs from shell languages since that's what their programming interface is optimized around. Sometimes it's good to even wrap the entrypoint for common scripting languages like Python in shell so that, for example, you can setup a virtual environment to run out of or use the proper version of Python. > Isn't it better to do this in a programming language that can run on all operating systems? Bash & coreutils have been ported to every operating system under the sun (including Sun operating systems). They're even more common & available than any other programming language that doesn't require a compiler (e.g. `adb shell` will get you into an environment where you can grep & do these operations even though there's generally no python or other scripting language available). > What are Windows users supposed to do? * cygwin * WSL * Windows ports of Bash (http://win-bash.sourceforge.net/ http://win-bash.sourceforge.net/, https://gitforwindows.org/ https://gitforwindows.org/, etc) * msys Let me conclude this post that you shouldn't really write anything complex or maintained by multiple people in shell if you can avoid it and if you can make guarantees about (for example) the available of a Python interpreter. That doesn't mean that shell scripts aren't a valuable and important part of the development ecosystem.
- 411111111111111 7y agoi was just missing a small note akin to "which becomes all the more powerful by combining these commands" and then make a totally readable example such as this one some_file=example.sh; tail -n +$(( $(wc -l $some_file | grep -o "[0-9]\+") - 5 )) $some_file
- assafmo 7y agoLately I've started to use Perl instead of sed for replacing text. Its regex support is much better IMO: cat a.txt | perl -pe 's/banana-(\d)/papaya-$1/g' Or in-place: perl -i -pe 's/banana-(\d)/papaya-$1/g' a.txt