18 ms·

Was Knuth Really Framed?

130 points by DSpinellis 7y ago

dang 7y ago
The previous episode: https://news.ycombinator.com/item?id=22406070 https://news.ycombinator.com/item?id=22406070.
- giancarlostoro 7y ago
  According to a sibling comment this is the follow up to this post. Sibling comment here: https://news.ycombinator.com/item?id=22436816 https://news.ycombinator.com/item?id=22436816
  - dang 7y ago
    Sorry, I'm afraid I don't understand.
kencausey 7y ago
I feel like 'by Jon Bentley' should be restored to the end of the title to help differentiate it from the earlier posting to which this is a reply.
jagged-chisel 7y ago
“Through this demonstration I haven't proven that Bentley didn't frame Knuth; it seems that at some point McIlroy admitted that the criticism was unfair.“
benchaney 7y ago
Yes he was. The modification to the problem being discussed here is really besides the point.
samatman 7y ago
Thank you for adding to the considerable weight of literature completely missing the point of Dr. Knuth's literate software, which is: tr, sort, uniq, and sed, should all be literate programs. They would be easier to read, reason about, modify, and extend. At this point, tooling for literate programming lags considerably compared to illiterate programming, and that's entirely because of the determination to miss the point exhibited here. Too bad really.
- lonelappde 7y ago
  Literate programs are essays. They might be easy to read and reason about, but not modify or extend. A computer program is not best understood and managed as a linear artifact. Much of it's power is in graph nature. Documentation comments are great, but that's not the same as literate programming.
  - gmfawcett 7y ago
    I don't think a literate program is more or less linear than the source code that is extracted/tangled from it. Both artifacts have a sequence: for a C program, the tangled version would put the #includes before declarations before definitions, for example. In that sense, the LP program is an alternate linearization of the program, in that the authors can choose the order in which to introduce the program. But few LP programs are naively linear -- they typically impose a tree structure on the code, made up of labelled sections and subsections. Readers don't have to start at line 1 of the program/essay, they can navigate from the table of contents to the section of interest. A compelling argument for LP is that it's an additive technology. If you don't want to read the essay, that's fine -- just tangle the code, and read the source-code artifact instead. With the right tooling (which admittedly may not exist!) an IDE could let you edit the tangled version directly, and put your edits back into the "essay" at the right places, so round-trip editing would be feasible.
    - samatman 7y ago
      Exactly. It's not fair to compare 40(!) years of advances in tooling surrounding the pile-of-files approach to software, to the somewhat withered on the vine approach embodied in literate programming. It's a road not taken, and I think that's a pity, so I'm doing something about it <shrug>
      - gmfawcett 7y ago
        I hope you'll post something about your progress. It's not an easy problem to tackle, but hopefully a fun one. :)
shaggyfrog 7y ago
> perl -a -MText::LevenshteinXS -e 'print distance(@F), "\n"' Step 2: draw the rest of the owl
- jiofih 7y ago
  The rebuttal itself is a lot more comical than any joke: literally countering Knuth’s “code should read and flow like prose for full understanding” with “f you, look at what I can do with my 20 years of hacking and thousands of lines of code I’ve never seen, written by someone else”.
- reggieband 7y ago
  In case anyone doesn't know this meme: https://knowyourmeme.com/memes/how-to-draw-an-owl https://knowyourmeme.com/memes/how-to-draw-an-owl
virtue3 7y ago
He, self admittedly, slightly cheated by evoking perl to do the "hard" bit. TFA is not so much about Knuth but mostly about unix being highly capable.
somesortofsystm 7y ago
I think its entirely fair to say that Knuth was in a frame to demonstrate one thing, and implement something second. That he was ideating about the subject didn't - necessarily - prevent a successful implementation. Certainly, as another set of eyes, the lower character count matters most, though.
robertlagrant 7y ago
If I can use Perl, I can do it in one line of bash.
- jeremyjh 7y ago
  The author did it in one line of Perl, using an existing library. How is that different from using awk? Yes awk is widely deployed but so is CPAN. In any case deployment isn’t part of the argument for using the UNIX philosophy.
  - zpallin 7y ago
    The original bar set by Doug Mcllroy was: > ... [a] more practical, much faster to implement, debug and modify solution of the problem takes only six lines of shell script by reusing standard Unix utilities. Unlike tr, sort, uniq, awk and tr, perl is not a standard Unix utility. Not only that, but Text::LevenshteinXS is a plugin that must be downloaded. It's still far more convenient than Knuth's work, and it follows Spinellis' reasoning about the Unix mindset, but Spinellis' Levenshtein example doesn't actually support Mcllroy's original argument.
  - 08-15 7y ago
    What exactly is the UNIX philosophy? As far as I can see, it's roughly "Data structures are hard, so let's pretend everything is ACSII text. Now we can use a really difficult systems programming language (C) to build functions with weird calling conventions ("tools") and glue them together with an awful scripting language (sh)." 'awk' fits into this framework awkwardly. It implements a restricted pattern (go line-by-line, match actions to lines), it doesn't want to be a full programming language, even though it really is. But 'perl' is a programming language, and it wants to be one. Once you have 'perl', what is the point of using a reasonable scripting language (perl) to build functions with weird calling conventions and gluing them together with an awful scripting language? You're better off writing functions(!) with normal calling conventions (a library) and gluing them together using the good scripting language. That logic taken taken to its conclusion replaces the shell with a clean language, encourages libraries instead of "tools", and embeds the 'awk' pattern into said language instead of relegating it to an incomplete secondary scripting language. In one word: 'scsh'.
scythe 7y ago
To me the question doesn't depend so much on whether Knuth was "framed". The meaningful criticism leveled at Knuth's code was that it was monolithic. It's true that it was long because he wrote it from scratch, but that's not enough to force you to be tightly coupled. Did Knuth try to make his code reusable? Was it reusable? I think those are the key questions.
- gmfawcett 7y ago
  That's not really a meaningful criticism. As others have pointed out, the point of Knuth's exercise was not to optimally solve the technical problem, but to demonstrate the effectiveness of Literate Programming (or the lack thereof). The technical problem was just a strawman, so that Knuth had a non-trivial program to demonstrate. With this in mind, McIlroy's pipe example isn't a critique of LP at all -- if anything, it was just a distracting advertisement for the Unix style of composing programs in the shell. What McIlroy could have examined -- and chose not to at the time -- is whether awk, sed, tr, and friends could themselves be written in a literate style, and whether such a rewrite would have achieved the goals that Knuth was setting out for LP. Knuth could have chosen to break his monolith into multiple, loosely-coupled programs, and then written then all in an LP style. But would that have really made the demonstration any more effective?
  - scythe 7y ago
    > Knuth could have chosen to break his monolith into multiple, loosely-coupled programs, and then written then all in an LP style. But would that have really made the demonstration any more effective? I would say yes. Clearly loose-coupling isn't necessary for a program that small. And no, it isn't always optimal. But I have clear memories of being asked to re-do an intro CS assignment three times because it wasn't in a properly object-oriented style. Modularity is not a necessity all of the time, but it is important sometimes. Demonstrating the potential to write reusable code seems just as important as demonstrating anything else. (If the conclusion is "LP helps clarity, but you can't write libraries", is that even positive?) As far as I can tell, "this code has only one useful point of entry" was a key part of the anti-LP argument leveled by McIlroy. After all, isn't the goal to demonstrate that LP works for people who aren't Donald Knuth?
blackandblue 7y ago
> In contrast, Python encourages the creation of monoliths of the type McIlroy criticized. unfair and gratuitous criticism of python... i have seen and written many small tools you can run using "python -m".
- btilly 7y ago
  As someone who uses both languages extensively, I disagree. You are right that Python is great for writing small tools that you can run, just like Perl. But Python does not lend itself to writing them inline in a command line like was done here. Perl not only does, but has a number of useful features specifically added to fit this common use case. 3 of which were used in this example. (-a for autosplit, -M to load a module, and -e to have the code passed as an argument on the command line rather than having to have it saved to a file.) Secondly, Perl lends itself to being used as a "better shell" while Python does not. What I mean is that anything that can be written in bash can be trivially rewritten in Perl, and the program that you get tends to be substantially more maintainable if the bash script is at all complex. In such a rewrite there usually isn't a good reason to change the structure of the program and make it into a single Perl program. By contrast Python has focused on the "One True Way" to do things, and the plumbing work for calling external commands is just verbose enough that a Python rewrite of a bash script is not necessarily better than the bash script. And furthermore it is much more likely that the Python rewrite of the bash script is much better rewritten as a Python script. The result is that for someone who lives on the Unix command line, Perl integrates into their world better than Python does. If you have never lived on the Unix command line, the objections may sound silly. But spend months typing commands and doing the extra steps that Python requires Every Single Time will get old. (This is historically not surprising. Perl 1 was focused on generating text reports. Perl 2 moved into being a sysadmin tool. Perl wound up as a web language because it is what all of the sysadmins recommended for text manipulation to people writing early CGI scripts.)
  - codetrotter 7y ago
    I use a few different languages, one of which is Python, and I use the command line a lot, and I agree that Python is too verbose for a lot of the things that I do on the command line. Therefore, Python is not something that I reach for when doing simple tasks involving pipelines and/or file operations. I have not yet put time into learning Perl. In no small part because I was intimidated by the weirdness of some of the Perl code that I've seen. The terseness that Perl allows, and which I desire, is at once compelling and scary at the same time. For this Perl also has earned the reputation that it "Write Once, Read Never". But let's assume that I overcome my fear of Perl. Which version of Perl would you recommend that I learn? Perl 5 or the language formerly known as Perl 6?
coolreader18 7y ago
I really like the idea of the Unix model as well, but you're not going to be able to use it effectively to write an actual application. If you're writing a word processor and you need to find the levenshtein distance between the most frequent word pairs (maybe some measure of how alliterative/consonant/assonant your document is?) then you're probably not going to be building the word processor using the Unix model, and even if you are (the closest you can get is probably using Tcl/Tk?) then it's still best to write out what you're doing as clear as possible. Note that it took me about 5 minutes to figure out what the shell pipeline presented in the article actually does, and multiple times my reasoning about it lead me to think "wait, does this actually do what it's supposed to do?"
- iamapipebomb 7y ago
  acme could be seen as a word processor in the unix model. It's easy to pipe text to composable commands. I can put the Levenshtein perl one-liner in the blue bar and middle click it to get the distance of two highlighted words.
  - pjmlp 7y ago
    ACME is a word processor in the Oberon System model, which has very little to do with UNIX model and plenty with Xerox GUI based REPL environments.
- chongli 7y ago
  A word processor is anti-Unix on its face. If you want the Unix equivalent, look no further than vi and TeX. With vi you can pipe your document through a spellchecker such as gnu aspell [1]. [1] http://aspell.net/ http://aspell.net/
pjmlp 7y ago
> In my everyday work, I use Unix commands many times daily to perform diverse and very different tasks. I very rarely encounter tasks that cannot be solved by joining together a couple of commands. Others just use a REPL instead, where tr, sort, uniq, and sed get to be function calls with a threading macro.
- chaps 7y ago
  Meh, a shell is just a REPL and a pipe is just a threading macro. What's your point?
  - pacala 7y ago
    A repl for a sane language...
    - chaps 7y ago
      No need for language snobbery - the sanity of the "language" isn't what's being discussed here. We're talking about the capabilities of of unix tools within domains where they'd be used. If you want to use a repl within your domain, that's your choice, but understand that in doing so, you're working with a relatively limited domain compared those within the reach of unix binaries.
  - pjmlp 7y ago
    The UNIX shell is a primitive REPL, without the capabilities of the REPLs developed at Xerox PARC, TI and Genera, regarding structured data, debugging tools, function composition, inline graphics, ability to directly interact with OS APIs. A chariot in the age of cars.
    - chaps 7y ago
      A car without infrastructure is just a fancy box. A chariot without infrastructure is a rideable horse. A rideable horse in the age of broken, disparate, infrastructure. But, at the end of the day, it all depends on what you're trying to accomplish. I use repls, shells, notebooks, etc, on a regular basis. Unix tools solve some problems. Repls solve other problems. Notebooks another. What's important, to me, is to be able to be able to make the most out of them all, despite their flaws, because they're simply the tools that we have in our toolchain. It would be a shame to not learn our own tools, when they can offer us so, so much.
neves 7y ago
BTW, Spinellis MOOC about unix tools must be great: https://www.edx.org/course/unix-tools-data-software-and-production-engineering https://www.edx.org/course/unix-tools-data-software-and-prod... I'm really a fan of Spinellis, his books are excellent: https://www.spinellis.gr/pubs/index.html#book https://www.spinellis.gr/pubs/index.html#book Effective Debugging book is a must read for any software developer. The Elements of Computing Style is useful for any knowledge worker. Code Reading is probably the only important book in the subject. The only problem with his books is that they are rather expensive, specially for developers who doesn't earn in dollars :-(
- DSpinellis 7y ago
  That is indeed a pity. I try to compensate by making as much material as possible openly available, such as through the MOOC you mentioned (I've been working for five years on it), through my blog, and through open source software and content.
pdpi 7y ago
Somebody, somewhere, wrote the Text::LevenshteinXS module. Somebody, somewhere, had to write sed, awk, head, sort, tr... It's all fine and dandy to say "look at these awesome tools that make tasks like these trivially easy. See how powerful Unix is". But this fails to consider that somebody, somewhere, has to be a tool writer, not just a tool user. Knuth's code was a tool writer's code, exemplifying a technique (Literate Programming) that is aimed at long form code writers in general. As with others before, the author fails to grasp that this is an apples to oranges comparison.
- crystaldev 7y ago
  Just like apples and oranges can be compared as foods (and fruits), bespoke vs. reused solutions can be compared so software. The reused parts are of higher quality than their corresponding bespoke implementations and can be composed to accomplish the same task as well as many others. It's a powerful lesson.
  - JadeNB 7y ago
    > The reused parts are of higher quality than their corresponding bespoke implementations …. I think that stating this as an inevitable and uncontested fact might be pushing the point.
    - crystaldev 7y ago
      Most of the components are multi-platform, partly or fully POSIX-standardized, battle-tested, blazing, fast, etc. It's some of the most widely-used and arguably greatest software ever written.
      - nyberg 7y ago
        And some of it is absolutely terrible with several tools overlapping in use cases along with unclear perfornance hints. Even worse is the combination performing worse which may not matter on small inputs but has a huge impact when you crank up the size. Then there's the issue of understanding why performance suffers and in what cases it's better to roll your own custom solution that better fits the problem. Reuse saves work when your problem maps perfectly to existing tools while what may seem like a minor difference will propagate down your program to end in countless issues from a codebase you don't know.
lotwxyz 7y ago
My Unix philosophy to showing people what a web-based 'ls' command looks like is this: $ import fs && import util && comstr --nowrap ls | pretty | less (Works here: https://dev.lotw.xyz/shell.os https://dev.lotw.xyz/shell.os)
- DSpinellis 7y ago
  This looks intriguing. Can you please explain what I'm looking at? I feel like Bowman waking up at the end of "2001: A Space Odyssey"
jimbokun 7y ago
Did Knuth ever reply personally on whether or not he felt he was framed?
svat 7y ago
I'll avoid repeating my comments from the previous thread (https://news.ycombinator.com/item?id=22406070 https://news.ycombinator.com/item?id=22406070) and say just the ones that I think didn't get enough attention: 1. I can't speak for Hillel Wayne who used the word "framed", but I didn't understand his newsletter post as Bentley having "framed" Knuth -- I understood his post as pointing out that in the popular imagination/folklore, the story had mutated over the course of years from the original setting (a program that Knuth was asked to write in WEB specifically as that was the point, and a review of that program by McIlroy evangelizing the then little-known Unix philosophy) to a "framing" where two people were competing to solve the same problem with the same available resources, and one of them did it in a "worse" way. (Also left this comment on the blog post above.) 2. Here's a comment on the previous thread from someone who says they read the column when it was posted, and their reaction they say was one of cringing -- so at least at that time it probably wasn't perceived that way: https://news.ycombinator.com/item?id=22418721 https://news.ycombinator.com/item?id=22418721 3. Much of the space taken by the literate program is for explaining a very interesting data structure that we could call a hash-packed trie (AFAICT, devised on that occasion for that problem -- a small twist on the packed tries used in TeX for hyphenation, and described in the thesis of Knuth's student Frank Liang). One cannot obtain this data structure by combining other programs, only by combining other ideas. (I mentioned this in the previous thread as well: https://news.ycombinator.com/item?id=22413391 https://news.ycombinator.com/item?id=22413391) 4. So as far as evaluating literate programming goes, the real question (and the answer is not obvious to me!) is: if you're going to write a program that uses a custom data structure (like this), how should you organize that program? Should you write it as Knuth does, or as a conventional program (like I tried to do with my translation: https://codegolf.stackexchange.com/a/197870 https://codegolf.stackexchange.com/a/197870)? And as for estimating the value of a new data structure in the first place: as of now (at that question), solutions based on a trie are about 200 times faster than the shell pipeline, on a large testcase. (The hash-packed trie, which Knuth calls "slow" in his program, is not so bad either, and it does economize on memory a bit.)
- mikekchar 7y ago
  I have my own answer for #4 (which, to me, is the only interesting question about this affair). I've actually done a fair amount of literate programming on my own, although I only have a couple of examples that one can look at these days. Here is a small library for fluent matcher system for Jasmine and React: https://github.com/ygt-mikekchar/react-maybe-matchers/blob/master/src/ReactMaybeMatchers.litcoffee https://github.com/ygt-mikekchar/react-maybe-matchers/blob/m... You will see that I've included yet another monad tutorial :-) I don't link to this as a way of saying that I think this is a good example of LP. It's not really. I was experimenting quite a lot. However, I can tell you one thing about it: it is practically impossible to refactor. As a result, I decided that LP is not particularly good for working on living programs. Or, at least, it is not conducive to my style of programming, which encourages refactoring. Nothing I write is "frozen". It is all in flux and so the value of documentation is transient. Additionally, it is rare that a programmer wishes to read code from the top to the bottom. If they ever do, it's usually the first time they have read the code. After that, they will want fast access to the parts that they want to modify. Sorting out the code from the text becomes difficult. If you make a change, you also have to review all of the text to make sure that you haven't clobber something that is referenced elsewhere. It will work well for something short, but it's not great for large projects. I still do LP style things. Here is an unfinished blog post on ideas about OO: https://github.com/ygt-mikekchar/oojs/blob/master/oojs.org https://github.com/ygt-mikekchar/oojs/blob/master/oojs.org However, to contrast with this, I would invite you to look at https://gitlab.com/mikekchar/testy https://gitlab.com/mikekchar/testy where I put some of those ideas into action (especially see the design.md and coding_standard.md documents to show what constraints I chose in this experiment). Crucially, after this code had run its course, I'd changed a lot of my ideas and never went back to my blog post. For me, the actual code is far more instructive than the blog post ever was. Of course, I'm the author, so I understand what I was trying to say and I only need a quick peek at the code to remind me what I was thinking. For me, that's the dilemma of LP: once you know what you want to know, the text is in the way. New people will benefit from the Gentle Introduction (sorry, couldn't resist the TeX reference...), but 99% of the time nobody will benefit from it. Is the other 1% of the time worth it. It may be, actually, but boy is it hard to convince yourself of that!
musicale 7y ago
Everyone knows Donald Knuth would have finished TAOCP faster if he only understood how to use UNIX pipes.
GedByrne 7y ago
I want to read the follow up article where the challenge is to create a typesetted document. Bentleys criticism includes a single like Shell script invoking Latex.
nixpulvis 7y ago
I find it comedic that someone called LaTeX error handling "phenomenally good".
- svat 7y ago
  It's my comment you're referring to (linked from the post). The full sentence was “BTW, TeX's error handling is phenomenally good IMO; the opposite of the situation with LaTeX.” You've reversed the meaning(!) but I stand by my original comment: I invite you to try plain TeX (instead of LaTeX) for a few weeks/months, and see how you feel about the way it handles errors. Unlike LaTeX, where the (TeX) error messages usually appear arbitrary / incomprehensible / unrelated to what you're doing, in TeX (IMO) all the error messages are very informative and include a lot of information and give you ways to recover from your problem and poke around, get more context, etc. First you'll have to have read a manual (or I recommend A Beginner's Book of TeX by Seroul and Levy), but my claim is about the user experience in the steady state. Of course, part of the reason is that LaTeX is much more complicated than the low-level things one may be doing with plain TeX. Another reason is that the LaTeX authors were working with severe constraints, one of which was of their own choosing: their choice of using TeX macros as a “programming language” (which it was never intended to be, and at which is it horrible). Nevertheless, a big part of the reason is that they were trying so hard to make things "easy" for the user in the typical case that they didn't care as much about ways in which things can go wrong and how surprising errors can be.
j88439h84 7y ago
> In fact, one of the reasons I sometimes prefer using Perl over Python is that it's very easy to incorporate into modular Unix tool pipelines. In contrast, Python encourages the creation of monoliths of the type McIlroy criticized. Python's Mario tool makes it easy to use Python code in pipelines. https://github.com/python-mario/mario https://github.com/python-mario/mario
jeffdavis 7y ago
There's a careful balance when using tools and libraries. It's obvious that they are a good choice sometimes, but I've been surprised at the number of times where a tool/library that looks like a perfect fit is actually not, and the whole problem needs to be reconsidered and I end up writing a lot of original code.
carlsborg 7y ago
The premise was that piping together shell commands was “better engineering” than a computer program that captured the authors thoughts? Which is more robust for debugging, producing diagnostics, error handling and reporting, extending, and code reuse? Laughable and sad at the same time, because ACM would publish that.

130 points by DSpinellis 7y ago

dang 7y ago

The previous episode: https://news.ycombinator.com/item?id=22406070 https://news.ycombinator.com/item?id=22406070.

giancarlostoro 7y ago
According to a sibling comment this is the follow up to this post. Sibling comment here: https://news.ycombinator.com/item?id=22436816 https://news.ycombinator.com/item?id=22436816
- dang 7y ago
  Sorry, I'm afraid I don't understand.

kencausey 7y ago

I feel like 'by Jon Bentley' should be restored to the end of the title to help differentiate it from the earlier posting to which this is a reply.

jagged-chisel 7y ago

“Through this demonstration I haven't proven that Bentley didn't frame Knuth; it seems that at some point McIlroy admitted that the criticism was unfair.“

benchaney 7y ago

Yes he was. The modification to the problem being discussed here is really besides the point.

samatman 7y ago

Thank you for adding to the considerable weight of literature completely missing the point of Dr. Knuth's literate software, which is: tr, sort, uniq, and sed, should all be literate programs. They would be easier to read, reason about, modify, and extend. At this point, tooling for literate programming lags considerably compared to illiterate programming, and that's entirely because of the determination to miss the point exhibited here. Too bad really.

lonelappde 7y ago
Literate programs are essays. They might be easy to read and reason about, but not modify or extend. A computer program is not best understood and managed as a linear artifact. Much of it's power is in graph nature. Documentation comments are great, but that's not the same as literate programming.
- gmfawcett 7y ago
  I don't think a literate program is more or less linear than the source code that is extracted/tangled from it. Both artifacts have a sequence: for a C program, the tangled version would put the #includes before declarations before definitions, for example. In that sense, the LP program is an alternate linearization of the program, in that the authors can choose the order in which to introduce the program. But few LP programs are naively linear -- they typically impose a tree structure on the code, made up of labelled sections and subsections. Readers don't have to start at line 1 of the program/essay, they can navigate from the table of contents to the section of interest. A compelling argument for LP is that it's an additive technology. If you don't want to read the essay, that's fine -- just tangle the code, and read the source-code artifact instead. With the right tooling (which admittedly may not exist!) an IDE could let you edit the tangled version directly, and put your edits back into the "essay" at the right places, so round-trip editing would be feasible.
  - samatman 7y ago
    Exactly. It's not fair to compare 40(!) years of advances in tooling surrounding the pile-of-files approach to software, to the somewhat withered on the vine approach embodied in literate programming. It's a road not taken, and I think that's a pity, so I'm doing something about it <shrug>
    - gmfawcett 7y ago
      I hope you'll post something about your progress. It's not an easy problem to tackle, but hopefully a fun one. :)

shaggyfrog 7y ago

> perl -a -MText::LevenshteinXS -e 'print distance(@F), "\n"' Step 2: draw the rest of the owl

jiofih 7y ago
The rebuttal itself is a lot more comical than any joke: literally countering Knuth’s “code should read and flow like prose for full understanding” with “f you, look at what I can do with my 20 years of hacking and thousands of lines of code I’ve never seen, written by someone else”.
reggieband 7y ago
In case anyone doesn't know this meme: https://knowyourmeme.com/memes/how-to-draw-an-owl https://knowyourmeme.com/memes/how-to-draw-an-owl

virtue3 7y ago

He, self admittedly, slightly cheated by evoking perl to do the "hard" bit. TFA is not so much about Knuth but mostly about unix being highly capable.

somesortofsystm 7y ago

I think its entirely fair to say that Knuth was in a frame to demonstrate one thing, and implement something second. That he was ideating about the subject didn't - necessarily - prevent a successful implementation. Certainly, as another set of eyes, the lower character count matters most, though.

robertlagrant 7y ago

If I can use Perl, I can do it in one line of bash.

jeremyjh 7y ago
The author did it in one line of Perl, using an existing library. How is that different from using awk? Yes awk is widely deployed but so is CPAN. In any case deployment isn’t part of the argument for using the UNIX philosophy.
- zpallin 7y ago
  The original bar set by Doug Mcllroy was: > ... [a] more practical, much faster to implement, debug and modify solution of the problem takes only six lines of shell script by reusing standard Unix utilities. Unlike tr, sort, uniq, awk and tr, perl is not a standard Unix utility. Not only that, but Text::LevenshteinXS is a plugin that must be downloaded. It's still far more convenient than Knuth's work, and it follows Spinellis' reasoning about the Unix mindset, but Spinellis' Levenshtein example doesn't actually support Mcllroy's original argument.
- 08-15 7y ago
  What exactly is the UNIX philosophy? As far as I can see, it's roughly "Data structures are hard, so let's pretend everything is ACSII text. Now we can use a really difficult systems programming language (C) to build functions with weird calling conventions ("tools") and glue them together with an awful scripting language (sh)." 'awk' fits into this framework awkwardly. It implements a restricted pattern (go line-by-line, match actions to lines), it doesn't want to be a full programming language, even though it really is. But 'perl' is a programming language, and it wants to be one. Once you have 'perl', what is the point of using a reasonable scripting language (perl) to build functions with weird calling conventions and gluing them together with an awful scripting language? You're better off writing functions(!) with normal calling conventions (a library) and gluing them together using the good scripting language. That logic taken taken to its conclusion replaces the shell with a clean language, encourages libraries instead of "tools", and embeds the 'awk' pattern into said language instead of relegating it to an incomplete secondary scripting language. In one word: 'scsh'.

scythe 7y ago

To me the question doesn't depend so much on whether Knuth was "framed". The meaningful criticism leveled at Knuth's code was that it was monolithic. It's true that it was long because he wrote it from scratch, but that's not enough to force you to be tightly coupled. Did Knuth try to make his code reusable? Was it reusable? I think those are the key questions.

gmfawcett 7y ago
That's not really a meaningful criticism. As others have pointed out, the point of Knuth's exercise was not to optimally solve the technical problem, but to demonstrate the effectiveness of Literate Programming (or the lack thereof). The technical problem was just a strawman, so that Knuth had a non-trivial program to demonstrate. With this in mind, McIlroy's pipe example isn't a critique of LP at all -- if anything, it was just a distracting advertisement for the Unix style of composing programs in the shell. What McIlroy could have examined -- and chose not to at the time -- is whether awk, sed, tr, and friends could themselves be written in a literate style, and whether such a rewrite would have achieved the goals that Knuth was setting out for LP. Knuth could have chosen to break his monolith into multiple, loosely-coupled programs, and then written then all in an LP style. But would that have really made the demonstration any more effective?
- scythe 7y ago
  > Knuth could have chosen to break his monolith into multiple, loosely-coupled programs, and then written then all in an LP style. But would that have really made the demonstration any more effective? I would say yes. Clearly loose-coupling isn't necessary for a program that small. And no, it isn't always optimal. But I have clear memories of being asked to re-do an intro CS assignment three times because it wasn't in a properly object-oriented style. Modularity is not a necessity all of the time, but it is important sometimes. Demonstrating the potential to write reusable code seems just as important as demonstrating anything else. (If the conclusion is "LP helps clarity, but you can't write libraries", is that even positive?) As far as I can tell, "this code has only one useful point of entry" was a key part of the anti-LP argument leveled by McIlroy. After all, isn't the goal to demonstrate that LP works for people who aren't Donald Knuth?

blackandblue 7y ago

> In contrast, Python encourages the creation of monoliths of the type McIlroy criticized. unfair and gratuitous criticism of python... i have seen and written many small tools you can run using "python -m".

btilly 7y ago
As someone who uses both languages extensively, I disagree. You are right that Python is great for writing small tools that you can run, just like Perl. But Python does not lend itself to writing them inline in a command line like was done here. Perl not only does, but has a number of useful features specifically added to fit this common use case. 3 of which were used in this example. (-a for autosplit, -M to load a module, and -e to have the code passed as an argument on the command line rather than having to have it saved to a file.) Secondly, Perl lends itself to being used as a "better shell" while Python does not. What I mean is that anything that can be written in bash can be trivially rewritten in Perl, and the program that you get tends to be substantially more maintainable if the bash script is at all complex. In such a rewrite there usually isn't a good reason to change the structure of the program and make it into a single Perl program. By contrast Python has focused on the "One True Way" to do things, and the plumbing work for calling external commands is just verbose enough that a Python rewrite of a bash script is not necessarily better than the bash script. And furthermore it is much more likely that the Python rewrite of the bash script is much better rewritten as a Python script. The result is that for someone who lives on the Unix command line, Perl integrates into their world better than Python does. If you have never lived on the Unix command line, the objections may sound silly. But spend months typing commands and doing the extra steps that Python requires Every Single Time will get old. (This is historically not surprising. Perl 1 was focused on generating text reports. Perl 2 moved into being a sysadmin tool. Perl wound up as a web language because it is what all of the sysadmins recommended for text manipulation to people writing early CGI scripts.)
- codetrotter 7y ago
  I use a few different languages, one of which is Python, and I use the command line a lot, and I agree that Python is too verbose for a lot of the things that I do on the command line. Therefore, Python is not something that I reach for when doing simple tasks involving pipelines and/or file operations. I have not yet put time into learning Perl. In no small part because I was intimidated by the weirdness of some of the Perl code that I've seen. The terseness that Perl allows, and which I desire, is at once compelling and scary at the same time. For this Perl also has earned the reputation that it "Write Once, Read Never". But let's assume that I overcome my fear of Perl. Which version of Perl would you recommend that I learn? Perl 5 or the language formerly known as Perl 6?

coolreader18 7y ago

I really like the idea of the Unix model as well, but you're not going to be able to use it effectively to write an actual application. If you're writing a word processor and you need to find the levenshtein distance between the most frequent word pairs (maybe some measure of how alliterative/consonant/assonant your document is?) then you're probably not going to be building the word processor using the Unix model, and even if you are (the closest you can get is probably using Tcl/Tk?) then it's still best to write out what you're doing as clear as possible. Note that it took me about 5 minutes to figure out what the shell pipeline presented in the article actually does, and multiple times my reasoning about it lead me to think "wait, does this actually do what it's supposed to do?"

iamapipebomb 7y ago
acme could be seen as a word processor in the unix model. It's easy to pipe text to composable commands. I can put the Levenshtein perl one-liner in the blue bar and middle click it to get the distance of two highlighted words.
- pjmlp 7y ago
  ACME is a word processor in the Oberon System model, which has very little to do with UNIX model and plenty with Xerox GUI based REPL environments.
chongli 7y ago
A word processor is anti-Unix on its face. If you want the Unix equivalent, look no further than vi and TeX. With vi you can pipe your document through a spellchecker such as gnu aspell [1]. [1] http://aspell.net/ http://aspell.net/

pjmlp 7y ago

> In my everyday work, I use Unix commands many times daily to perform diverse and very different tasks. I very rarely encounter tasks that cannot be solved by joining together a couple of commands. Others just use a REPL instead, where tr, sort, uniq, and sed get to be function calls with a threading macro.

chaps 7y ago
Meh, a shell is just a REPL and a pipe is just a threading macro. What's your point?
- pacala 7y ago
  A repl for a sane language...
  - chaps 7y ago
    No need for language snobbery - the sanity of the "language" isn't what's being discussed here. We're talking about the capabilities of of unix tools within domains where they'd be used. If you want to use a repl within your domain, that's your choice, but understand that in doing so, you're working with a relatively limited domain compared those within the reach of unix binaries.
- pjmlp 7y ago
  The UNIX shell is a primitive REPL, without the capabilities of the REPLs developed at Xerox PARC, TI and Genera, regarding structured data, debugging tools, function composition, inline graphics, ability to directly interact with OS APIs. A chariot in the age of cars.
  - chaps 7y ago
    A car without infrastructure is just a fancy box. A chariot without infrastructure is a rideable horse. A rideable horse in the age of broken, disparate, infrastructure. But, at the end of the day, it all depends on what you're trying to accomplish. I use repls, shells, notebooks, etc, on a regular basis. Unix tools solve some problems. Repls solve other problems. Notebooks another. What's important, to me, is to be able to be able to make the most out of them all, despite their flaws, because they're simply the tools that we have in our toolchain. It would be a shame to not learn our own tools, when they can offer us so, so much.

neves 7y ago

BTW, Spinellis MOOC about unix tools must be great: https://www.edx.org/course/unix-tools-data-software-and-production-engineering https://www.edx.org/course/unix-tools-data-software-and-prod... I'm really a fan of Spinellis, his books are excellent: https://www.spinellis.gr/pubs/index.html#book https://www.spinellis.gr/pubs/index.html#book Effective Debugging book is a must read for any software developer. The Elements of Computing Style is useful for any knowledge worker. Code Reading is probably the only important book in the subject. The only problem with his books is that they are rather expensive, specially for developers who doesn't earn in dollars :-(

DSpinellis 7y ago
That is indeed a pity. I try to compensate by making as much material as possible openly available, such as through the MOOC you mentioned (I've been working for five years on it), through my blog, and through open source software and content.

pdpi 7y ago

Somebody, somewhere, wrote the Text::LevenshteinXS module. Somebody, somewhere, had to write sed, awk, head, sort, tr... It's all fine and dandy to say "look at these awesome tools that make tasks like these trivially easy. See how powerful Unix is". But this fails to consider that somebody, somewhere, has to be a tool writer, not just a tool user. Knuth's code was a tool writer's code, exemplifying a technique (Literate Programming) that is aimed at long form code writers in general. As with others before, the author fails to grasp that this is an apples to oranges comparison.

crystaldev 7y ago
Just like apples and oranges can be compared as foods (and fruits), bespoke vs. reused solutions can be compared so software. The reused parts are of higher quality than their corresponding bespoke implementations and can be composed to accomplish the same task as well as many others. It's a powerful lesson.
- JadeNB 7y ago
  > The reused parts are of higher quality than their corresponding bespoke implementations …. I think that stating this as an inevitable and uncontested fact might be pushing the point.
  - crystaldev 7y ago
    Most of the components are multi-platform, partly or fully POSIX-standardized, battle-tested, blazing, fast, etc. It's some of the most widely-used and arguably greatest software ever written.
    - nyberg 7y ago
      And some of it is absolutely terrible with several tools overlapping in use cases along with unclear perfornance hints. Even worse is the combination performing worse which may not matter on small inputs but has a huge impact when you crank up the size. Then there's the issue of understanding why performance suffers and in what cases it's better to roll your own custom solution that better fits the problem. Reuse saves work when your problem maps perfectly to existing tools while what may seem like a minor difference will propagate down your program to end in countless issues from a codebase you don't know.

lotwxyz 7y ago

My Unix philosophy to showing people what a web-based 'ls' command looks like is this: $ import fs && import util && comstr --nowrap ls | pretty | less (Works here: https://dev.lotw.xyz/shell.os https://dev.lotw.xyz/shell.os)

DSpinellis 7y ago
This looks intriguing. Can you please explain what I'm looking at? I feel like Bowman waking up at the end of "2001: A Space Odyssey"

jimbokun 7y ago

Did Knuth ever reply personally on whether or not he felt he was framed?

svat 7y ago

I'll avoid repeating my comments from the previous thread (https://news.ycombinator.com/item?id=22406070 https://news.ycombinator.com/item?id=22406070) and say just the ones that I think didn't get enough attention: 1. I can't speak for Hillel Wayne who used the word "framed", but I didn't understand his newsletter post as Bentley having "framed" Knuth -- I understood his post as pointing out that in the popular imagination/folklore, the story had mutated over the course of years from the original setting (a program that Knuth was asked to write in WEB specifically as that was the point, and a review of that program by McIlroy evangelizing the then little-known Unix philosophy) to a "framing" where two people were competing to solve the same problem with the same available resources, and one of them did it in a "worse" way. (Also left this comment on the blog post above.) 2. Here's a comment on the previous thread from someone who says they read the column when it was posted, and their reaction they say was one of cringing -- so at least at that time it probably wasn't perceived that way: https://news.ycombinator.com/item?id=22418721 https://news.ycombinator.com/item?id=22418721 3. Much of the space taken by the literate program is for explaining a very interesting data structure that we could call a hash-packed trie (AFAICT, devised on that occasion for that problem -- a small twist on the packed tries used in TeX for hyphenation, and described in the thesis of Knuth's student Frank Liang). One cannot obtain this data structure by combining other programs, only by combining other ideas. (I mentioned this in the previous thread as well: https://news.ycombinator.com/item?id=22413391 https://news.ycombinator.com/item?id=22413391) 4. So as far as evaluating literate programming goes, the real question (and the answer is not obvious to me!) is: if you're going to write a program that uses a custom data structure (like this), how should you organize that program? Should you write it as Knuth does, or as a conventional program (like I tried to do with my translation: https://codegolf.stackexchange.com/a/197870 https://codegolf.stackexchange.com/a/197870)? And as for estimating the value of a new data structure in the first place: as of now (at that question), solutions based on a trie are about 200 times faster than the shell pipeline, on a large testcase. (The hash-packed trie, which Knuth calls "slow" in his program, is not so bad either, and it does economize on memory a bit.)

mikekchar 7y ago
I have my own answer for #4 (which, to me, is the only interesting question about this affair). I've actually done a fair amount of literate programming on my own, although I only have a couple of examples that one can look at these days. Here is a small library for fluent matcher system for Jasmine and React: https://github.com/ygt-mikekchar/react-maybe-matchers/blob/master/src/ReactMaybeMatchers.litcoffee https://github.com/ygt-mikekchar/react-maybe-matchers/blob/m... You will see that I've included yet another monad tutorial :-) I don't link to this as a way of saying that I think this is a good example of LP. It's not really. I was experimenting quite a lot. However, I can tell you one thing about it: it is practically impossible to refactor. As a result, I decided that LP is not particularly good for working on living programs. Or, at least, it is not conducive to my style of programming, which encourages refactoring. Nothing I write is "frozen". It is all in flux and so the value of documentation is transient. Additionally, it is rare that a programmer wishes to read code from the top to the bottom. If they ever do, it's usually the first time they have read the code. After that, they will want fast access to the parts that they want to modify. Sorting out the code from the text becomes difficult. If you make a change, you also have to review all of the text to make sure that you haven't clobber something that is referenced elsewhere. It will work well for something short, but it's not great for large projects. I still do LP style things. Here is an unfinished blog post on ideas about OO: https://github.com/ygt-mikekchar/oojs/blob/master/oojs.org https://github.com/ygt-mikekchar/oojs/blob/master/oojs.org However, to contrast with this, I would invite you to look at https://gitlab.com/mikekchar/testy https://gitlab.com/mikekchar/testy where I put some of those ideas into action (especially see the design.md and coding_standard.md documents to show what constraints I chose in this experiment). Crucially, after this code had run its course, I'd changed a lot of my ideas and never went back to my blog post. For me, the actual code is far more instructive than the blog post ever was. Of course, I'm the author, so I understand what I was trying to say and I only need a quick peek at the code to remind me what I was thinking. For me, that's the dilemma of LP: once you know what you want to know, the text is in the way. New people will benefit from the Gentle Introduction (sorry, couldn't resist the TeX reference...), but 99% of the time nobody will benefit from it. Is the other 1% of the time worth it. It may be, actually, but boy is it hard to convince yourself of that!

musicale 7y ago

Everyone knows Donald Knuth would have finished TAOCP faster if he only understood how to use UNIX pipes.

GedByrne 7y ago

I want to read the follow up article where the challenge is to create a typesetted document. Bentleys criticism includes a single like Shell script invoking Latex.

nixpulvis 7y ago

I find it comedic that someone called LaTeX error handling "phenomenally good".

svat 7y ago
It's my comment you're referring to (linked from the post). The full sentence was “BTW, TeX's error handling is phenomenally good IMO; the opposite of the situation with LaTeX.” You've reversed the meaning(!) but I stand by my original comment: I invite you to try plain TeX (instead of LaTeX) for a few weeks/months, and see how you feel about the way it handles errors. Unlike LaTeX, where the (TeX) error messages usually appear arbitrary / incomprehensible / unrelated to what you're doing, in TeX (IMO) all the error messages are very informative and include a lot of information and give you ways to recover from your problem and poke around, get more context, etc. First you'll have to have read a manual (or I recommend A Beginner's Book of TeX by Seroul and Levy), but my claim is about the user experience in the steady state. Of course, part of the reason is that LaTeX is much more complicated than the low-level things one may be doing with plain TeX. Another reason is that the LaTeX authors were working with severe constraints, one of which was of their own choosing: their choice of using TeX macros as a “programming language” (which it was never intended to be, and at which is it horrible). Nevertheless, a big part of the reason is that they were trying so hard to make things "easy" for the user in the typical case that they didn't care as much about ways in which things can go wrong and how surprising errors can be.

j88439h84 7y ago

> In fact, one of the reasons I sometimes prefer using Perl over Python is that it's very easy to incorporate into modular Unix tool pipelines. In contrast, Python encourages the creation of monoliths of the type McIlroy criticized. Python's Mario tool makes it easy to use Python code in pipelines. https://github.com/python-mario/mario https://github.com/python-mario/mario

jeffdavis 7y ago

There's a careful balance when using tools and libraries. It's obvious that they are a good choice sometimes, but I've been surprised at the number of times where a tool/library that looks like a perfect fit is actually not, and the whole problem needs to be reconsidered and I end up writing a lot of original code.

carlsborg 7y ago

The premise was that piping together shell commands was “better engineering” than a computer program that captured the authors thoughts? Which is more robust for debugging, producing diagnostics, error handling and reporting, extending, and code reuse? Laughable and sad at the same time, because ACM would publish that.