9 ms·

Unix Text Processing (1987)

108 points by rhollos 14y ago

kamaal 14y ago
A very nice book available for free. It makes me very happy that I can learn new ways to do my work smarter. Some of the best known advantages of Unix text processing tools is, as long as you can reduce your problem to 'Text'- There exist some very powerful, succinct and quick solutions to even some very difficult problems. Well it takes some time to get a grip on how to work with Unix text processing utilities and tools like Perl. Once you are upto speed, you see how much work you can do so quickly with so much little effort. In fact the more you get into it, you realize how much useless code you have writing over the years. While all you needed was a command with a few options.
clicks 14y ago
I see resources about Unix text processing utilities, about Bash, about readline shortcuts, etc. etc. submitted very often to HN. Exactly how useful are they? Would you say they're among the most important skills a programmer could have? Or do we just have a disproportionately large amount of sysadmins for HN readers? Isn't some general-purpose language like Python or Ruby almost always much better, much faster than these solutions? Isn't it worth it to rather invest your time learning Python well -- instead of getting familiar to the segregated and messy environment of modern Unix-land? Certainly, the learning curve is much steep for Unix utilities than, say, Python.
- srean 14y ago
  Depends, for one thing one how frequently you intend to use it. Consider variable names, it is often said that it is desirable to have nice, long explanatory names. But again it depends. For variables that have big scopes, it is indeed important to have descriptive names, but those that disappear after the line, its ok to use i,j,k. In fact short names in such cases increase comprehension rather than impede it. The reverse holds as well. > Certainly, the learn curve is much steep for Unix > utilities than, say, Python I am not certain about that at all. Yes they may have trucload of options, but I have never had to memorize them. Many disagree that these tools really dont strictly adhere to the "do one thing but do it well" philosophy they do that to a satisfying degree of approximation. Coreutils, textutils, find and xargs can go a really really long way.
- zainny 14y ago
  Honestly, I think it's a little of column A and a little of column B. While sometimes it can be useful and faster to bust out a one line bash shortcut, I think the HN community is also a bit guilty of over-romanticizing Unix/Bash/<Insert non-visual & somewhat geeky tool/language>.
  - hazov 14y ago
    About non-visual stuff I can simply answer that I am more productive with a shell, R, ggplot2 and python than with excel, just because I know this better. This is my stack along with matlab and mathematica in the bank where I work now, both basically at cli level. The majority of people in my division uses excel. I also think of things in an every unix way in my using of computers largely because of 15 years of using unix like systems, it's the same when I need to code more than a couple of lines and do not have vim, for me the brain became too much used to it, I believe this will be the case for many people in HN.
  - zmoney 14y ago
    Well, it's not romantic if it's actually practical. Bottom line, I think, is if you really get the shell, it is almost infinitely useful.
- dmlorenzetti 14y ago
  A number of things make me more inclined to look at writeups on using the command line, and command-line tools, than those on higher-level languages like Python. First, many of the things I want to automate are most naturally done at the command line. For these, I already know the commands I want to run, and just need to script the logic that glues them together. Second, I mostly want small scripts I can send to colleagues, and not have to worry about whether or not they have Python (or whatever) installed. I do write scripts in Python and Ruby, but they tend to be longer, since they reflect tasks where the data have to be pulled apart and put back together in multiple ways (for example, a dependency-generator for some custom makefiles I maintain). This sort of task favors building up structures in memory, over pipelines, and I use the appropriate tool accordingly. As for the question whether any of these tools are the most important skills a programmer could have, no, I don't think so. As a programmer, your most important skills are in the language you use all the time, the language your "deliverable" is written in. Most people probably don't use the unix tools as their primary programming platform. But those tools support and extend the environment in which we get our "real" programming done. To borrow the woodworking analogy from (I believe) the Pragmatic Programmers, all those articles about the Unix tools are probably the equivalent of articles on keeping your chisels and saws sharp. No, the file you use to keep your chisel sharp isn't your most important tool; your chisel or saw is your most important tool. But the craft of keeping your most important tool sharp can be fun and rewarding in itself. By the way, speed is never an issue with any of the scripts I'm likely to run. But if it was, I doubt Python would be faster, since those scripts involve a lot of calls on system resources.
- vidarh 14y ago
  Scripting languages still usually represents a substantial typing overhead compared to throwing together a pipeline on the command line with standard Unix tools for many simple tasks. The moment I see I'll need to do something many times, I'll consider writing a script, and then I'll often pick Ruby. But for one-off stuff, the command line is often faster once you get comfortable with a handful Unix tools. In fact I often find that even when I do things many times, the mental overhead of remembering "yet another script" is often high enough to make it faster to just re-compose the command line I want. For example, I very frequently do some variation over "grep [some term] | sort | uniq -c | sort -n" to get a sorted list by number of occurrences of [some term], but the key part is "some variation", and that makes adding and remembering an alias less useful. Another big consideration is that these tools are present on all or most machines many of us use. For larger applications, having to install other packages is often no big deal, but for example I don't want to find myself dealing with an emergency and suddenly having to pull down tons of packages to use Ruby because I'm not comfortable with the tools that are already on the machine.
- deleted 14y ago
  [deleted]
- dragonquest 14y ago
  While I can't speak for other texts, UTP is my and probably the go-to book for learning Troff. I owe the typesetting of my ebook to it. Since it occupies that niche area, it is still quite relevant and useful today.
  - Surio 14y ago
    somewhat curious... What was your book on?
    - ralph 14y ago
      Scanning his submissions, I think http://www.dreamincode.net/forums/blog/48/entry-4298-lessons-learnt-from-publishing-my-first-ebook/ http://www.dreamincode.net/forums/blog/48/entry-4298-lessons... is relevant.
      - Surio 14y ago
        Ralph, thank you very very much for going out of your way to be helpful in this entire thread. Very much appreciated. :-)
- aidenn0 14y ago
  Automating stuff at the command line is super important. If it's easier for you to do in Python, then that's fine. I use awk and bash for a lot of that stuff because it's what I happen to know. Other people I know use perl. The important thing is to be able to automate anything non-trivial that you will do more than a single-digit number of times. One time I spent 30 minutes writing a script for things that a dozen people were doing a dozen times a day. It maybe saved a minute each time, but that's over 2 hours in the first day we had it. It paid off in time for me in 3 days, and probably less than that it terms of "damnit this is boring" For me, at least, Python has a much higher friction to get from "here's the list of what I do" to "here's an automated version" I do use Python for some of the more involved things (manipulating timestamps from bash is a pain), and I've converted some bash scripts into "real language" scripts when things started to get hairy, but most of the time, you get 90%+ of the time savings from the first 1% of effort.
  - codewright 14y ago
    Python scripting is an investment, eventually it becomes as frictionless as bash scripting. Especially if you have a good shell scripting library. I usually use Fabric, SaltStack, or https://github.com/kennethreitz/envoy https://github.com/kennethreitz/envoy
- zmoney 14y ago
  >Certainly, the learning curve is much steep for Unix > utilities than, say, Python. Now what makes you think that? UNIX shell tools have a very consistent interface which is really pretty simple (pipes in, pipes out,) they are well documented, and they're interactive and easy to play around with no zero setup. Also, the UNIX shell has been around for 30 years now and isn't going anywhere. So what's the best investment? Python will change more in the next five years.
- edge17 14y ago
  I would say they are among the most important skills a programmer could have - the amount of time I have saved by being able to do fast manipulation with chains of things like sort, tr, sed, awk, head, tail, paste, strings, nm, hexdump, xargs, etc. and get diagnostic information for deeper issues it's like anything else; each tool is just a building block. the number of creative things you can do with very little effort by piping the output of one to the input of another is incredibly powerful. it's not always computationally efficient, but many times that doesn't matter. Python is another great tool, but sometimes a machete is quicker and easier. Different tools for different situations.
- bcbrown 14y ago
  It might be that at any given point of time, there are a lot of people who want or need to learn Unix. I already know Python, but I think there is value in learning the fundamentals of the Unix shell programs. I've spent the first five years of my career on Windows, am now on Linux, and this book looks like a useful resource.
- hiperlink 14y ago
  Yeah, it's a good idea to learn Python well. And perl. But... sitting here in this cubicle, in front of a PuTTy window working on a production server for a large financial institute to manipulate text files and I don't have access to either of those. Ruby is not installed. Most of the time cut, comm, paste, diff, tr and ed|sed does my job. When they aren't, then there is an old awk binary. So, it depends on the job and the tools you have. IMO every developer should learn the bare minimum about UTP utilities. It won't hurt.
- chubot 14y ago
  I used to have that attitude, and Python is still my main language. But I do more and more stuff in shell now. Every project I write has a whole bunch of shell snippets now, which saves a ton of code and lowers the barrier to automation considerably. If you're trying to write shell scripts in Python, they're going to be 3-5x longer. Bash is a higher level language than Python. Every tool has its place, and bash and Python are complementary. Also I realize there is some useless ceremony in Python standard practice. Do you want to know what my Python test runner looks like now? $ find . -name \*_test.py | sh -x -e That's it... no BS. I don't know what people are using in the Python world these days but some of it has drifted toward "framework land". Also, it's not very hard to make this parallel, whereas it is somewhat annoying in Python. And you don't have to worry about global variables polluting each other -- tests stay independent. I test a big part of my C/C++ code with shell scripts as well. So basically I find it very helpful to think of yourself as writing shell utilities in Python. Python's not your world. It's part of your world.
- twistedanimator 14y ago
  Just yesterday I was able to write a one liner on the command line to analyze my skype usage. I wanted to see if it was making financial sense to keep skype alongside my prepaid wireless plan. You can download your last six months of skype activity from skype.com and they are in the form: Date;Date;Item;Destination;Type;Rate;Duration;Amount;Currency "July 31, 2012 21:16";"2012-07-31T21:16:01+00:00";"+11234567890";"USA";"Call";0.000;00:00:10;0.000;USD "July 31, 2012 21:15";"2012-07-31T21:15:38+00:00";"+11234567890";"USA";"Call";0.000;01:17:02;0.000;USD After 15 minutes or so, I came up with the following one liner: cut -f7 -d";" call_history* | grep -v "Duration" | awk '{ FS=":"; s+=$1*60; s+=$2; if ($3 != 00) { s+=1 } } END {print s " minutes"}' I could have done the same thing in perl or python in 5 minutes, but it was interesting to "program" only by hooking programs together to achieve the same thing. After analyzing my skype logs, I found I used 1800 minutes. That would have cost me $180 with prepaid minutes, but only cost $30 with skype.
  - ralph 14y ago
    Another way would be all awk. awk -F\; ' $7 != "Duration" { split($7, t, ":") s += t[1] * 60 + t[2] + (t[3] != 0) } END {print s + 0} ' Note the handling of s == "" in END.
  - ralph 14y ago
    The sed and dc combo are perhaps not as readable, but there was a time before the dawn of the One True Awk. :-) sed '1s/.*/0/; s/;[^;]*$//; s///; s/.*;// s/00$//; s/:..$/+1++/; s/:$/++/; s/:/ 60*/; $s/$/p/' | dc
kephra 14y ago
I was even writing good old roff a few days ago. So this book is not even out of date.
- Surio 14y ago
  >> I was even writing good old roff a few days ago Just curious, what was it for? :) PS: I had a groff based CV (2006-ish). Now it is a LaTeX based CV.
  - ralph 14y ago
    I still regularly use groff; letters, invoices, ad hoc formatting of one-off tasks. tbl(1) is nice and the overall speed makes it handy for producing PDF in the back-end.
    - fusiongyro 14y ago
      I was going to ask why one would prefer groff to some TeX variant. Speed is a pretty good reason; I'm impressed at how slow TeX is even on modern hardware. I usually generate HTML unless I want something beautiful, and then I usually use ConTeXt. I'm pretty fond of classic Unix stuff so maybe I'll delve more into groff.
      - Surio 14y ago
        +1 on speed and memory.. Some table and figure heavy documents I've done have had internal memory overruns on latex. A lot of my forays into programming started with the Bell Labs books (with troff | pic | eqn ...) prominently on the copyright page. So, it was one of the first things I looked up when I got access to a UNIX terminal. Then I discovered that all the UNIX books by Stevens were done up with troff/groff. From there, it was all "steadily downhill" for me ;-) P.S: If you already know (La/Con)TeX then groff is literally a walk in the park. And it is always good to know more than one way to do things imho. Good luck with it.
        fusiongyro 14y ago
zmoney 14y ago
So much of this is still useful and relevant. Kind of amazing how much the web relies upon these ancient (by our standards) tools. UNIX is one of the most incredible technology stories ever.
contingencies 14y ago
roff: still the normal way to generate an IETF Internet Standards Draft text document. This mechanism is showing its age though, especially regarding UTF8 and width limitations.
- 4ad 14y ago
  I don't know about other systems, but on Plan 9, where UTF-8 was invented, troff handles it just fine: http://plan9.bell-labs.com/magic/man2html/1/troff http://plan9.bell-labs.com/magic/man2html/1/troff
  - ralph 14y ago
    groff does it with the help of its preconv(1), also invoked with groff's -k option. $ printf 'Hello ①②③\n' | preconv .lf 1 - Hello \[u2460]\[u2461]\[u2462] $ printf 'Hello ①②③\n' | groff -k -Tutf8 | grep . Hello ①②③ $
  - contingencies 14y ago
    Yeah, but UTF8's not acceptable to the IETF Secretariat.
ghshephard 14y ago
This book is an absolute treasure, even with it's age, is still one of the best descriptions of troff/nroff. One thing that I do find interesting, is that the book (on text processing, no less), that is 680 pages x 30 lines x 80 columns (plus some minimal line art) - which, in theory, is around 1.6 megabytes of data, weighs in at 28 MBytes in this PDF. Regardless of the Irony, great book which the authors clearly put blood, sweat and tears into.
- przemoc 14y ago
  26.7 MB is because it's scanned book with text layer on top. If it was retyped in, let's say, LaTeX, and outputted to PDF it wouldn't take more than one tenth of the current size. Remember that scanning the book has also archival purpose.
  - ralph 14y ago
    The book has been re-typed, as troff, just like the original. See my comment elsewhere on this post.
    - przemoc 14y ago
      No reason to downvote me (whoever it was). The PDF http://oreilly.com/openbook/utp/UnixTextProcessing.pdf http://oreilly.com/openbook/utp/UnixTextProcessing.pdf has _scanned_ pages with text layer on top of it. That's the reason why it's so big. I applaud retyping effort, as it's always better to preserve real content than images of it (even if OCRed), but I cannot say that I'm happy about troff being used for this purpose. It can be a matter of taste, but I don't like the way formatting is done in troff/nroff. That's why I never use it directly (e.g. using ronn to convert markdown text to man page, etc.). But I understand it's done that way to preserve "the creation process" too, which is also appreciated. And the book is about troff/nroff, so dogfooding is present. ;)
ralph 14y ago
I sought permission from Tim O'Reilly, he co-wrote this back when he was an author, to re-enter the text of the book as troff source, the original being lost. He kindly gave it and volunteers from the groff@gnu.org mailing list split the chapters amongst themselves; it was quite fun. http://home.windstream.net/kollar/utp/ http://home.windstream.net/kollar/utp/ is the result.
- super_mario 14y ago
  Thanks. This is the best version by far. Is it also possible to generate the PDF hyperlinks/outline so it is easier to navigate the document?
  - ralph 14y ago
    That's a good point. Back when it was created, doing that with the toolchain wasn't so straightforward as I understand it is now; I'll raise it on the groff@gnu.org list.
  - dredmorbius 14y ago
    You can generate PDF (without the hyperlinks) with standard psutils (e.g.: ps2pdf). A good PDF reader will offer text search, which is a bit of a help. Otherwise, I'd have to investigate groff and text conversion utilities to see if the metadata are readily accessible. Interesting question, though I suspect it might take a slight rewrite.
    - ralph 14y ago
      PDF is provided on the book's page I gave above. You're correct, small mark-up changes are needed to make use of the PDF-linking macros now available that weren't then.
      - deleted 14y ago
        [deleted]
- Surio 14y ago
new_test 14y ago
Is there any advantage (speed?) in using something like Awk over something like Python?
- ralph 14y ago
  It's awkward to embed Python in a pipeline and the awk can be quicker to write if the problem suits its domain. Execution speed may be slower than Python, depends on the work needed and the awk used; gawk, mawk, Kernighan's One True awk, ... awk 'NR > 3 && !/foo/ {s += $(NF - 2)} END {print NR, s + 0}'
- npsimons 14y ago
  Open up a terminal on Linux, OSX, or heck even Cygwin or MinGW on Windows. I'm willing to bet that awk is already pre-installed, even on a base or fresh install. Awk, like grep, sed and all the rest are pretty standard and very powerful. Not knocking Python, shell programming is just another paradigm, and the tools are pretty universally available.
  - dredmorbius 14y ago
    Awk is part of the POSIX specification, so yes, it's going to be present on pretty much any Unix-like environment. Even those which don't aim at POSIX compliance will almost certainly have an awk interpreter. Busybox includes one, which means that many minimal / embedded systems will include awk by default (as they use busybox to provide core utilities).
Surio 14y ago
Question for the groff experts out there: XeLaTeX has finally moved LaTeX into the realm of directly embedding virtually any true type font into the final document. Literally 5-6 lines of LaTeX commands and you are done! I was amazed when I first did it. It was so convenient, when compared to messing with pfbs, then afms, and then.... you get the idea, no? Is there a similar groff mechanism to pull TTF/OTF fonts into the final PS/PDF document with similar minimal effort? If, could someone be kind enough to point a resource to me. Thanks. P.S: I was searching, but could not home in on the right keywords to drive me to an answer.
- ralph 14y ago
  I think TrueType fonts with groff is fairly painless; the topic comes up on the groff@gnu.org list now and again but I don't pay much attention. The list is low volume and very friendly, with some real old Unix hands on there; feel free to ask, say I sent you if you like. https://lists.gnu.org/mailman/listinfo/groff https://lists.gnu.org/mailman/listinfo/groff Gunnar's Heirloom troff has TrueType support. "troff can access PostScript Type 1, OpenType, and TrueType fonts directly, that is, it can read font metrics from AFM, OpenType, or TrueType files, and can instruct its dpost post-processor to include glyph data from PFB, PFA, OpenType, and TrueType files into the output it generates". http://heirloom.sourceforge.net/doctools.html http://heirloom.sourceforge.net/doctools.html
  - Surio 14y ago
    Thanks. I will check both the list and heirloom. Hadn't heard of heirloom before. So, thanks again.

108 points by rhollos 14y ago

A very nice book available for free. It makes me very happy that I can learn new ways to do my work smarter. Some of the best known advantages of Unix text processing tools is, as long as you can reduce your problem to 'Text'- There exist some very powerful, succinct and quick solutions to even some very difficult problems. Well it takes some time to get a grip on how to work with Unix text processing utilities and tools like Perl. Once you are upto speed, you see how much work you can do so quickly with so much little effort. In fact the more you get into it, you realize how much useless code you have writing over the years. While all you needed was a command with a few options.

clicks 14y ago

I see resources about Unix text processing utilities, about Bash, about readline shortcuts, etc. etc. submitted very often to HN. Exactly how useful are they? Would you say they're among the most important skills a programmer could have? Or do we just have a disproportionately large amount of sysadmins for HN readers? Isn't some general-purpose language like Python or Ruby almost always much better, much faster than these solutions? Isn't it worth it to rather invest your time learning Python well -- instead of getting familiar to the segregated and messy environment of modern Unix-land? Certainly, the learning curve is much steep for Unix utilities than, say, Python.

srean 14y ago
Depends, for one thing one how frequently you intend to use it. Consider variable names, it is often said that it is desirable to have nice, long explanatory names. But again it depends. For variables that have big scopes, it is indeed important to have descriptive names, but those that disappear after the line, its ok to use i,j,k. In fact short names in such cases increase comprehension rather than impede it. The reverse holds as well. > Certainly, the learn curve is much steep for Unix > utilities than, say, Python I am not certain about that at all. Yes they may have trucload of options, but I have never had to memorize them. Many disagree that these tools really dont strictly adhere to the "do one thing but do it well" philosophy they do that to a satisfying degree of approximation. Coreutils, textutils, find and xargs can go a really really long way.
zainny 14y ago
Honestly, I think it's a little of column A and a little of column B. While sometimes it can be useful and faster to bust out a one line bash shortcut, I think the HN community is also a bit guilty of over-romanticizing Unix/Bash/<Insert non-visual & somewhat geeky tool/language>.
- hazov 14y ago
  About non-visual stuff I can simply answer that I am more productive with a shell, R, ggplot2 and python than with excel, just because I know this better. This is my stack along with matlab and mathematica in the bank where I work now, both basically at cli level. The majority of people in my division uses excel. I also think of things in an every unix way in my using of computers largely because of 15 years of using unix like systems, it's the same when I need to code more than a couple of lines and do not have vim, for me the brain became too much used to it, I believe this will be the case for many people in HN.
- zmoney 14y ago
  Well, it's not romantic if it's actually practical. Bottom line, I think, is if you really get the shell, it is almost infinitely useful.
dmlorenzetti 14y ago
A number of things make me more inclined to look at writeups on using the command line, and command-line tools, than those on higher-level languages like Python. First, many of the things I want to automate are most naturally done at the command line. For these, I already know the commands I want to run, and just need to script the logic that glues them together. Second, I mostly want small scripts I can send to colleagues, and not have to worry about whether or not they have Python (or whatever) installed. I do write scripts in Python and Ruby, but they tend to be longer, since they reflect tasks where the data have to be pulled apart and put back together in multiple ways (for example, a dependency-generator for some custom makefiles I maintain). This sort of task favors building up structures in memory, over pipelines, and I use the appropriate tool accordingly. As for the question whether any of these tools are the most important skills a programmer could have, no, I don't think so. As a programmer, your most important skills are in the language you use all the time, the language your "deliverable" is written in. Most people probably don't use the unix tools as their primary programming platform. But those tools support and extend the environment in which we get our "real" programming done. To borrow the woodworking analogy from (I believe) the Pragmatic Programmers, all those articles about the Unix tools are probably the equivalent of articles on keeping your chisels and saws sharp. No, the file you use to keep your chisel sharp isn't your most important tool; your chisel or saw is your most important tool. But the craft of keeping your most important tool sharp can be fun and rewarding in itself. By the way, speed is never an issue with any of the scripts I'm likely to run. But if it was, I doubt Python would be faster, since those scripts involve a lot of calls on system resources.
vidarh 14y ago
Scripting languages still usually represents a substantial typing overhead compared to throwing together a pipeline on the command line with standard Unix tools for many simple tasks. The moment I see I'll need to do something many times, I'll consider writing a script, and then I'll often pick Ruby. But for one-off stuff, the command line is often faster once you get comfortable with a handful Unix tools. In fact I often find that even when I do things many times, the mental overhead of remembering "yet another script" is often high enough to make it faster to just re-compose the command line I want. For example, I very frequently do some variation over "grep [some term] | sort | uniq -c | sort -n" to get a sorted list by number of occurrences of [some term], but the key part is "some variation", and that makes adding and remembering an alias less useful. Another big consideration is that these tools are present on all or most machines many of us use. For larger applications, having to install other packages is often no big deal, but for example I don't want to find myself dealing with an emergency and suddenly having to pull down tons of packages to use Ruby because I'm not comfortable with the tools that are already on the machine.
deleted 14y ago
[deleted]
dragonquest 14y ago
While I can't speak for other texts, UTP is my and probably the go-to book for learning Troff. I owe the typesetting of my ebook to it. Since it occupies that niche area, it is still quite relevant and useful today.
- Surio 14y ago
  somewhat curious... What was your book on?
  - ralph 14y ago
    Scanning his submissions, I think http://www.dreamincode.net/forums/blog/48/entry-4298-lessons-learnt-from-publishing-my-first-ebook/ http://www.dreamincode.net/forums/blog/48/entry-4298-lessons... is relevant.
    - Surio 14y ago
      Ralph, thank you very very much for going out of your way to be helpful in this entire thread. Very much appreciated. :-)
aidenn0 14y ago
Automating stuff at the command line is super important. If it's easier for you to do in Python, then that's fine. I use awk and bash for a lot of that stuff because it's what I happen to know. Other people I know use perl. The important thing is to be able to automate anything non-trivial that you will do more than a single-digit number of times. One time I spent 30 minutes writing a script for things that a dozen people were doing a dozen times a day. It maybe saved a minute each time, but that's over 2 hours in the first day we had it. It paid off in time for me in 3 days, and probably less than that it terms of "damnit this is boring" For me, at least, Python has a much higher friction to get from "here's the list of what I do" to "here's an automated version" I do use Python for some of the more involved things (manipulating timestamps from bash is a pain), and I've converted some bash scripts into "real language" scripts when things started to get hairy, but most of the time, you get 90%+ of the time savings from the first 1% of effort.
- codewright 14y ago
  Python scripting is an investment, eventually it becomes as frictionless as bash scripting. Especially if you have a good shell scripting library. I usually use Fabric, SaltStack, or https://github.com/kennethreitz/envoy https://github.com/kennethreitz/envoy
zmoney 14y ago
>Certainly, the learning curve is much steep for Unix > utilities than, say, Python. Now what makes you think that? UNIX shell tools have a very consistent interface which is really pretty simple (pipes in, pipes out,) they are well documented, and they're interactive and easy to play around with no zero setup. Also, the UNIX shell has been around for 30 years now and isn't going anywhere. So what's the best investment? Python will change more in the next five years.
edge17 14y ago
I would say they are among the most important skills a programmer could have - the amount of time I have saved by being able to do fast manipulation with chains of things like sort, tr, sed, awk, head, tail, paste, strings, nm, hexdump, xargs, etc. and get diagnostic information for deeper issues it's like anything else; each tool is just a building block. the number of creative things you can do with very little effort by piping the output of one to the input of another is incredibly powerful. it's not always computationally efficient, but many times that doesn't matter. Python is another great tool, but sometimes a machete is quicker and easier. Different tools for different situations.
bcbrown 14y ago
It might be that at any given point of time, there are a lot of people who want or need to learn Unix. I already know Python, but I think there is value in learning the fundamentals of the Unix shell programs. I've spent the first five years of my career on Windows, am now on Linux, and this book looks like a useful resource.
hiperlink 14y ago
Yeah, it's a good idea to learn Python well. And perl. But... sitting here in this cubicle, in front of a PuTTy window working on a production server for a large financial institute to manipulate text files and I don't have access to either of those. Ruby is not installed. Most of the time cut, comm, paste, diff, tr and ed|sed does my job. When they aren't, then there is an old awk binary. So, it depends on the job and the tools you have. IMO every developer should learn the bare minimum about UTP utilities. It won't hurt.
chubot 14y ago
I used to have that attitude, and Python is still my main language. But I do more and more stuff in shell now. Every project I write has a whole bunch of shell snippets now, which saves a ton of code and lowers the barrier to automation considerably. If you're trying to write shell scripts in Python, they're going to be 3-5x longer. Bash is a higher level language than Python. Every tool has its place, and bash and Python are complementary. Also I realize there is some useless ceremony in Python standard practice. Do you want to know what my Python test runner looks like now? $ find . -name \*_test.py | sh -x -e That's it... no BS. I don't know what people are using in the Python world these days but some of it has drifted toward "framework land". Also, it's not very hard to make this parallel, whereas it is somewhat annoying in Python. And you don't have to worry about global variables polluting each other -- tests stay independent. I test a big part of my C/C++ code with shell scripts as well. So basically I find it very helpful to think of yourself as writing shell utilities in Python. Python's not your world. It's part of your world.
twistedanimator 14y ago
Just yesterday I was able to write a one liner on the command line to analyze my skype usage. I wanted to see if it was making financial sense to keep skype alongside my prepaid wireless plan. You can download your last six months of skype activity from skype.com and they are in the form: Date;Date;Item;Destination;Type;Rate;Duration;Amount;Currency "July 31, 2012 21:16";"2012-07-31T21:16:01+00:00";"+11234567890";"USA";"Call";0.000;00:00:10;0.000;USD "July 31, 2012 21:15";"2012-07-31T21:15:38+00:00";"+11234567890";"USA";"Call";0.000;01:17:02;0.000;USD After 15 minutes or so, I came up with the following one liner: cut -f7 -d";" call_history* | grep -v "Duration" | awk '{ FS=":"; s+=$1*60; s+=$2; if ($3 != 00) { s+=1 } } END {print s " minutes"}' I could have done the same thing in perl or python in 5 minutes, but it was interesting to "program" only by hooking programs together to achieve the same thing. After analyzing my skype logs, I found I used 1800 minutes. That would have cost me $180 with prepaid minutes, but only cost $30 with skype.
- ralph 14y ago
  Another way would be all awk. awk -F\; ' $7 != "Duration" { split($7, t, ":") s += t[1] * 60 + t[2] + (t[3] != 0) } END {print s + 0} ' Note the handling of s == "" in END.
- ralph 14y ago
  The sed and dc combo are perhaps not as readable, but there was a time before the dawn of the One True Awk. :-) sed '1s/.*/0/; s/;[^;]*$//; s///; s/.*;// s/00$//; s/:..$/+1++/; s/:$/++/; s/:/ 60*/; $s/$/p/' | dc

kephra 14y ago

I was even writing good old roff a few days ago. So this book is not even out of date.

Surio 14y ago
>> I was even writing good old roff a few days ago Just curious, what was it for? :) PS: I had a groff based CV (2006-ish). Now it is a LaTeX based CV.
- ralph 14y ago
  I still regularly use groff; letters, invoices, ad hoc formatting of one-off tasks. tbl(1) is nice and the overall speed makes it handy for producing PDF in the back-end.
  - fusiongyro 14y ago
    I was going to ask why one would prefer groff to some TeX variant. Speed is a pretty good reason; I'm impressed at how slow TeX is even on modern hardware. I usually generate HTML unless I want something beautiful, and then I usually use ConTeXt. I'm pretty fond of classic Unix stuff so maybe I'll delve more into groff.
    - Surio 14y ago
      +1 on speed and memory.. Some table and figure heavy documents I've done have had internal memory overruns on latex. A lot of my forays into programming started with the Bell Labs books (with troff | pic | eqn ...) prominently on the copyright page. So, it was one of the first things I looked up when I got access to a UNIX terminal. Then I discovered that all the UNIX books by Stevens were done up with troff/groff. From there, it was all "steadily downhill" for me ;-) P.S: If you already know (La/Con)TeX then groff is literally a walk in the park. And it is always good to know more than one way to do things imho. Good luck with it.
      - fusiongyro 14y ago

zmoney 14y ago

So much of this is still useful and relevant. Kind of amazing how much the web relies upon these ancient (by our standards) tools. UNIX is one of the most incredible technology stories ever.

contingencies 14y ago

roff: still the normal way to generate an IETF Internet Standards Draft text document. This mechanism is showing its age though, especially regarding UTF8 and width limitations.

4ad 14y ago
I don't know about other systems, but on Plan 9, where UTF-8 was invented, troff handles it just fine: http://plan9.bell-labs.com/magic/man2html/1/troff http://plan9.bell-labs.com/magic/man2html/1/troff
- ralph 14y ago
  groff does it with the help of its preconv(1), also invoked with groff's -k option. $ printf 'Hello ①②③\n' | preconv .lf 1 - Hello \[u2460]\[u2461]\[u2462] $ printf 'Hello ①②③\n' | groff -k -Tutf8 | grep . Hello ①②③ $
- contingencies 14y ago
  Yeah, but UTF8's not acceptable to the IETF Secretariat.

ghshephard 14y ago

This book is an absolute treasure, even with it's age, is still one of the best descriptions of troff/nroff. One thing that I do find interesting, is that the book (on text processing, no less), that is 680 pages x 30 lines x 80 columns (plus some minimal line art) - which, in theory, is around 1.6 megabytes of data, weighs in at 28 MBytes in this PDF. Regardless of the Irony, great book which the authors clearly put blood, sweat and tears into.

przemoc 14y ago
26.7 MB is because it's scanned book with text layer on top. If it was retyped in, let's say, LaTeX, and outputted to PDF it wouldn't take more than one tenth of the current size. Remember that scanning the book has also archival purpose.
- ralph 14y ago
  The book has been re-typed, as troff, just like the original. See my comment elsewhere on this post.
  - przemoc 14y ago
    No reason to downvote me (whoever it was). The PDF http://oreilly.com/openbook/utp/UnixTextProcessing.pdf http://oreilly.com/openbook/utp/UnixTextProcessing.pdf has _scanned_ pages with text layer on top of it. That's the reason why it's so big. I applaud retyping effort, as it's always better to preserve real content than images of it (even if OCRed), but I cannot say that I'm happy about troff being used for this purpose. It can be a matter of taste, but I don't like the way formatting is done in troff/nroff. That's why I never use it directly (e.g. using ronn to convert markdown text to man page, etc.). But I understand it's done that way to preserve "the creation process" too, which is also appreciated. And the book is about troff/nroff, so dogfooding is present. ;)

ralph 14y ago

I sought permission from Tim O'Reilly, he co-wrote this back when he was an author, to re-enter the text of the book as troff source, the original being lost. He kindly gave it and volunteers from the groff@gnu.org mailing list split the chapters amongst themselves; it was quite fun. http://home.windstream.net/kollar/utp/ http://home.windstream.net/kollar/utp/ is the result.

super_mario 14y ago
Thanks. This is the best version by far. Is it also possible to generate the PDF hyperlinks/outline so it is easier to navigate the document?
- ralph 14y ago
  That's a good point. Back when it was created, doing that with the toolchain wasn't so straightforward as I understand it is now; I'll raise it on the groff@gnu.org list.
- dredmorbius 14y ago
  You can generate PDF (without the hyperlinks) with standard psutils (e.g.: ps2pdf). A good PDF reader will offer text search, which is a bit of a help. Otherwise, I'd have to investigate groff and text conversion utilities to see if the metadata are readily accessible. Interesting question, though I suspect it might take a slight rewrite.
  - ralph 14y ago
    PDF is provided on the book's page I gave above. You're correct, small mark-up changes are needed to make use of the PDF-linking macros now available that weren't then.
    - deleted 14y ago
      [deleted]
Surio 14y ago

new_test 14y ago

Is there any advantage (speed?) in using something like Awk over something like Python?

ralph 14y ago
It's awkward to embed Python in a pipeline and the awk can be quicker to write if the problem suits its domain. Execution speed may be slower than Python, depends on the work needed and the awk used; gawk, mawk, Kernighan's One True awk, ... awk 'NR > 3 && !/foo/ {s += $(NF - 2)} END {print NR, s + 0}'
npsimons 14y ago
Open up a terminal on Linux, OSX, or heck even Cygwin or MinGW on Windows. I'm willing to bet that awk is already pre-installed, even on a base or fresh install. Awk, like grep, sed and all the rest are pretty standard and very powerful. Not knocking Python, shell programming is just another paradigm, and the tools are pretty universally available.
- dredmorbius 14y ago
  Awk is part of the POSIX specification, so yes, it's going to be present on pretty much any Unix-like environment. Even those which don't aim at POSIX compliance will almost certainly have an awk interpreter. Busybox includes one, which means that many minimal / embedded systems will include awk by default (as they use busybox to provide core utilities).

Surio 14y ago

Question for the groff experts out there: XeLaTeX has finally moved LaTeX into the realm of directly embedding virtually any true type font into the final document. Literally 5-6 lines of LaTeX commands and you are done! I was amazed when I first did it. It was so convenient, when compared to messing with pfbs, then afms, and then.... you get the idea, no? Is there a similar groff mechanism to pull TTF/OTF fonts into the final PS/PDF document with similar minimal effort? If, could someone be kind enough to point a resource to me. Thanks. P.S: I was searching, but could not home in on the right keywords to drive me to an answer.

ralph 14y ago
I think TrueType fonts with groff is fairly painless; the topic comes up on the groff@gnu.org list now and again but I don't pay much attention. The list is low volume and very friendly, with some real old Unix hands on there; feel free to ask, say I sent you if you like. https://lists.gnu.org/mailman/listinfo/groff https://lists.gnu.org/mailman/listinfo/groff Gunnar's Heirloom troff has TrueType support. "troff can access PostScript Type 1, OpenType, and TrueType fonts directly, that is, it can read font metrics from AFM, OpenType, or TrueType files, and can instruct its dpost post-processor to include glyph data from PFB, PFA, OpenType, and TrueType files into the output it generates". http://heirloom.sourceforge.net/doctools.html http://heirloom.sourceforge.net/doctools.html
- Surio 14y ago
  Thanks. I will check both the list and heirloom. Hadn't heard of heirloom before. So, thanks again.