7 ms·

There's another interesting story about Knuth that's worth repeating here. http://www.leancrew.com/all-this/2011/12/more-shell-less-egg/ http://www.leancrew.co

There's another interesting story about Knuth that's worth repeating here. http://www.leancrew.com/all-this/2011/12/more-shell-less-egg/ http://www.leancrew.com/all-this/2011/12/more-shell-less-egg... The tl;dr is that Knuth wrote an elaborate implementation of a program to solve a particular problem, and Doug McIlroy replaced it entirely with a six step shell pipeline. (Knuth's program was written using his literate programming tools, could be typeset in TeX, and involved some precise work with data structures and algorithms.) I love this story as an example both of Knuth's genius and perspective, but also as a way to show what his level of dedication can achieve. It's an amazing intellectual accomplishment. I also love this story as a demonstration what those of us without that skill and dedication can achieve using the advancements built on the work of Knuth and others.

sobellian 8y ago
I love that story, although I think the takeaway needs to be slightly updated. It's quite possible that a sorted word count was easiest to do with bash and unix utilities in 1992. Now, it's easier and more comprehensible in your favorite scripting language (which I'm assuming isn't bash). The real lesson is to program using the most powerful tools at your disposal.
- gmfawcett 8y ago
  That's a good lesson, but we shouldn't stop there. Knuth is a computer scientist, but also an artist: writer, musician, typographer, and creative programmer. His insights into programming-as-literature have infused an often-soulless industry with something soulful. He began with the popular notion that programming should be a creative, playful act, and tried to elevate mere playfulness into the realm of fine art. An incredible detour in an amazing career, and an enriching lesson of a different kind.
  - billfruit 8y ago
    And a theologian, he has has written works on divinity, etc..
- mschaef 8y ago
  > It's quite possible that a sorted word count was easiest to do with bash and unix utilities in 1992. Now, it's easier and more comprehensible in your favorite scripting language It'd be interesting to see how true this is in reality. I'm not at all convinced that most scripting languages can do this quite as concisely as bash and the Unix userland. (But, being honest, this problem seems very well aligned to that tooling's strenghts.)
  - sobellian 8y ago
    Well, what is easy or comprehensible isn't necessarily concise. Bash is great for code golf but I know few people who prefer to code in it.
    - sobellian 8y ago
      And for completeness, a Ruby implementation: puts STDIN.read .split(/[^A-Za-z]+/) .group_by(&:downcase).transform_values(&:size) .sort_by(&:last).reverse .take(ARGV[0].to_i) .map{ |l| l.join("\t") }.join("\n") Wordier than the bash script, but still pretty short and fairly trivial. Anyone could make this in five minutes.
  - btilly 8y ago
    It is easy to do in your favorite scripting language. But is unlikely to be as short. Comprehensibility is in the eye of the beholder. Here is a Perl solution for comparison. use strict; use warnings; my $limit = shift @ARGV; my %count; for my $line (<>) { while ($line =~ /(\w+)/g) { $count{lc $1}++; } } @words = sort {$count{$b} <=> $count{$a} or $a cmp $b} keys %count; $limit = @words if @words < $limit; print "$_\t$count{$_}\n" for @words[0..($limit-1)];
    - mkl 8y ago
      Python is relatively short, batteries included and all that: import re import sys import collections words = re.findall('[a-z]+', sys.stdin.read().lower()) c = collections.Counter(words) for word, count in c.most_common(int(sys.argv[1])): print(f'{count:>7} {word}') Three times the character count of the Unix pipe version, but IMHO a lot more readable (and generalisable).
- flukus 8y ago
  There is no bash (or any scripting) involved, the complete command from the article is: tr -cs A-Za-z '\n' | tr A-Z a-z | sort | uniq -c | sort -rn | sed ${1}q The real benefit is that each is a single command that is easy to test in isolation and it's multi-process. That's not possible in most scripting languages.
  - sobellian 8y ago
    McIlroy literally calls it a script in his review (notice the ${1}). Nothing prevents you from unit testing in scripting languages. Multi-process, sure, but most people aren't looking for that.
    - flukus 8y ago
      > McIlroy literally calls it a script in his review (notice the ${1}). Ok, there's a single shell substitution, if it was fixed would you still call it a script? Technically the result of that is itself a sed script "3q", but if you count either of those then there isn't a lot of wiggle room between script and command, the arguments to tr are by far the most complex "script" involved. > Nothing prevents you from unit testing in scripting languages. That is a world away from what I'm talking about. Each line of that command can be executed on the CLI in isolation, you'd be replicating a lot more in nearly any scripting language, except maybe perl and awk. > Multi-process, sure, but most people aren't looking for that. Neither am I generally, still quite nice when you get it for free though.
      - sobellian 8y ago
        You're really splitting hairs. You can execute a Python command in a REPL. There's little material difference between scripts and commands for our purposes. And scripting languages provide facilities to test functions in isolation.
  - zbentley 8y ago
    > There is no bash (or any scripting) involved What provides the pipe functionality?
gmfawcett 8y ago
To his great credit, Knuth included McIlroy's critique, in full and without response, when he published Literate Programming. Knuth also conceded in an earlier chapter that literate programming was probably a crazy idea and that promoting it was perhaps an act of madness! It's delightful that we live in a world where there's room both for industrial-strength Fabergé eggs and hack-it-together pipelines. :)
vram22 8y ago
That's a good one. It's one of the classic examples of the Unix philosophy and how pipelines can be powerful, although of course not applicable in all cases, and they have their issues too - which often becomes an argument on HN (about text streams vs. object pipes a la PowerShell, etc.). I had seen it a while ago and blogged about it here, with solutions (approx.) in Python and shell: The Bentley-Knuth problem and solutions: https://jugad2.blogspot.com/2012/07/the-bentley-knuth-problem-and-solutions.html https://jugad2.blogspot.com/2012/07/the-bentley-knuth-proble... Also, previous HN thread about it here: Revisiting Knuth and McIlroy's word count programs (2011) (franklinchen.com) https://news.ycombinator.com/item?id=15264997 https://news.ycombinator.com/item?id=15264997
- henrik_w 8y ago
  There is also the book "Exercises in Programming Style" that uses the term frequency problem to illustrate 33 different programming paradigms (including a shell solution). https://www.amazon.com/Exercises-Programming-Style-Cristina-Videira/dp/1482227371/ https://www.amazon.com/Exercises-Programming-Style-Cristina-...
  - vram22 8y ago
    Cool. Will check that out.
svat 8y ago
That was quite unfair criticism, and even Doug McIlroy knew it (as he admitted later). The background is this: - Bentley, the author of the column, invited Knuth to demonstrate literate programming using a program of his choice. - Knuth insisted that to be fair, Bentley ought to specify the program to be written; else someone might object that Knuth chose a program that would be good for literate programming. - Bentley chose (what we'd now call) the term frequency problem (list the top k most frequent words in a text file), and accordingly Knuth wrote a system program for solving just this one particular task. (Did everything from opening the input file to formatting the output, etc.) - Doug McIlroy was asked to “review” this program, the way works of literature are reviewed. He happens to be the inventor of Unix pipes. Towards the end of his review, along with many other points (e.g. Knuth didn't include diagrams in cases where most of us would appreciate them, something I still struggle with when reading TeX and other programs), he used the opportunity to demonstrate his own invention, a shell pipeline using now-standard Unix tools (tr, sort, uniq, sed). There are a few things wrong with this criticism: - The main thing is that DEK wrote the program he was asked to write, so pointing out that he shouldn't have written that program is a criticism of the one who chose the program (Bentley mentioned this when printing McIlroy's review). - At the time, Unix wasn't even widely available outside Bell Labs and a few places; it definitely wasn't available to Knuth or most of the column's readers. - Knuth's program, fine-tuned for the task, is more efficient than the shell pipeline. - Even if you use a shell pipeline, someone has to write the “standard” programs that go into it (the "tr", "sort", "uniq" and "sed" above), and literate programming can be used there. In fact, Knuth did exactly that a few years later, rewriting Unix's “wc” (IIRC) and compared the resulting program with that of Sun Unix's wc, and his LP version had, among things, better error handling. (He's explained it by saying that in conventional programming, if you have a small function and 90% of it is error-checking, it looks like the function is “about” error-checking, so there's a psychological resistance to doing too much of that, while with LP you move the error-handling to a separate section entirely about error-checking and then you tend to do a better job. BTW, TeX's error handling is phenomenally good IMO; the opposite of the situation with LaTeX.) All that said, there is some valid criticism that Knuth prefers to write monolithic programs, but that works for him. He seems not to consider it a problem that to change something you have to understand more-or-less the entire program; he seems to prefer doing that anyway (he reads other people's code a lot: in Peter Seibel's Coders at Work, he was the only person interviewed who read others' programs regularly).
mci 8y ago
> He has fashioned a sort of industrial-strength Fabergé egg—intricate, wonderfully worked, refined beyond all ordinary desires, a museum piece from the start. Curiously, I independently came up with this metaphor in my review of The MMIX Supplement to The Art of Computer Programming [1]: "You can marvel at its intricacies like one marvels at a Fabergé egg." [1] https://www.amazon.com/gp/customer-reviews/R25NHN9UP7PLJI/ https://www.amazon.com/gp/customer-reviews/R25NHN9UP7PLJI/
ineedasername 8y ago
McIlroy's critique is both astute on it's own yet perhaps not relevant in the context of what Knuth had been asked to do, namely present an example of literate programming. That's not to say the McIlroy's script can't have a literate form, only that it wouldn't serve as a good exemplar. It all serves as a good reminder of how Knuth's work should be used & viewed: It shouldn't be taken as insight into how to manage and develop for software projects. It's more useful as teaching tools for how to think about solving problems. In my mind it's the difference between "pure" science and practical engineering, though I'm not sure that's a perfect analogy. To try another analogy, it's like Knuth was asked to make a custom set of clothes. And McIlroy's critique seems like saying "Not everyone can afford custom made clothes, and the GAP produces perfectly serviceable clothes that are far more practical in most situations" It's correct, sure, but rather besides the point in the context of the original request.
- mschaef 8y ago
  > perhaps not relevant in the context of what Knuth had been asked to do, namely present an example of literate programming. Agreed... if that was the goal, then McIlroy's example was terrible (in that it didn't use literate programming at all).
jeklj 8y ago
Yes, I love this story too. It really demonstrates so much. And oh, the hackles it raises. :)
- mschaef 8y ago
  > And oh, the hackles it raises. :) The main reason I like the story is that it's good fodder for discussion about priorities in software development. After all, counting words isn't exactly an interesting problem in 2018, but deciding the effective use of libraries and programmer time is still very interesting.

sobellian 8y ago

I love that story, although I think the takeaway needs to be slightly updated. It's quite possible that a sorted word count was easiest to do with bash and unix utilities in 1992. Now, it's easier and more comprehensible in your favorite scripting language (which I'm assuming isn't bash). The real lesson is to program using the most powerful tools at your disposal.

gmfawcett 8y ago
That's a good lesson, but we shouldn't stop there. Knuth is a computer scientist, but also an artist: writer, musician, typographer, and creative programmer. His insights into programming-as-literature have infused an often-soulless industry with something soulful. He began with the popular notion that programming should be a creative, playful act, and tried to elevate mere playfulness into the realm of fine art. An incredible detour in an amazing career, and an enriching lesson of a different kind.
- billfruit 8y ago
  And a theologian, he has has written works on divinity, etc..
mschaef 8y ago
> It's quite possible that a sorted word count was easiest to do with bash and unix utilities in 1992. Now, it's easier and more comprehensible in your favorite scripting language It'd be interesting to see how true this is in reality. I'm not at all convinced that most scripting languages can do this quite as concisely as bash and the Unix userland. (But, being honest, this problem seems very well aligned to that tooling's strenghts.)
- sobellian 8y ago
  Well, what is easy or comprehensible isn't necessarily concise. Bash is great for code golf but I know few people who prefer to code in it.
  - sobellian 8y ago
    And for completeness, a Ruby implementation: puts STDIN.read .split(/[^A-Za-z]+/) .group_by(&:downcase).transform_values(&:size) .sort_by(&:last).reverse .take(ARGV[0].to_i) .map{ |l| l.join("\t") }.join("\n") Wordier than the bash script, but still pretty short and fairly trivial. Anyone could make this in five minutes.
- btilly 8y ago
  It is easy to do in your favorite scripting language. But is unlikely to be as short. Comprehensibility is in the eye of the beholder. Here is a Perl solution for comparison. use strict; use warnings; my $limit = shift @ARGV; my %count; for my $line (<>) { while ($line =~ /(\w+)/g) { $count{lc $1}++; } } @words = sort {$count{$b} <=> $count{$a} or $a cmp $b} keys %count; $limit = @words if @words < $limit; print "$_\t$count{$_}\n" for @words[0..($limit-1)];
  - mkl 8y ago
    Python is relatively short, batteries included and all that: import re import sys import collections words = re.findall('[a-z]+', sys.stdin.read().lower()) c = collections.Counter(words) for word, count in c.most_common(int(sys.argv[1])): print(f'{count:>7} {word}') Three times the character count of the Unix pipe version, but IMHO a lot more readable (and generalisable).
flukus 8y ago
There is no bash (or any scripting) involved, the complete command from the article is: tr -cs A-Za-z '\n' | tr A-Z a-z | sort | uniq -c | sort -rn | sed ${1}q The real benefit is that each is a single command that is easy to test in isolation and it's multi-process. That's not possible in most scripting languages.
- sobellian 8y ago
  McIlroy literally calls it a script in his review (notice the ${1}). Nothing prevents you from unit testing in scripting languages. Multi-process, sure, but most people aren't looking for that.
  - flukus 8y ago
    > McIlroy literally calls it a script in his review (notice the ${1}). Ok, there's a single shell substitution, if it was fixed would you still call it a script? Technically the result of that is itself a sed script "3q", but if you count either of those then there isn't a lot of wiggle room between script and command, the arguments to tr are by far the most complex "script" involved. > Nothing prevents you from unit testing in scripting languages. That is a world away from what I'm talking about. Each line of that command can be executed on the CLI in isolation, you'd be replicating a lot more in nearly any scripting language, except maybe perl and awk. > Multi-process, sure, but most people aren't looking for that. Neither am I generally, still quite nice when you get it for free though.
    - sobellian 8y ago
      You're really splitting hairs. You can execute a Python command in a REPL. There's little material difference between scripts and commands for our purposes. And scripting languages provide facilities to test functions in isolation.
- zbentley 8y ago
  > There is no bash (or any scripting) involved What provides the pipe functionality?

gmfawcett 8y ago

To his great credit, Knuth included McIlroy's critique, in full and without response, when he published Literate Programming. Knuth also conceded in an earlier chapter that literate programming was probably a crazy idea and that promoting it was perhaps an act of madness! It's delightful that we live in a world where there's room both for industrial-strength Fabergé eggs and hack-it-together pipelines. :)

vram22 8y ago

That's a good one. It's one of the classic examples of the Unix philosophy and how pipelines can be powerful, although of course not applicable in all cases, and they have their issues too - which often becomes an argument on HN (about text streams vs. object pipes a la PowerShell, etc.). I had seen it a while ago and blogged about it here, with solutions (approx.) in Python and shell: The Bentley-Knuth problem and solutions: https://jugad2.blogspot.com/2012/07/the-bentley-knuth-problem-and-solutions.html https://jugad2.blogspot.com/2012/07/the-bentley-knuth-proble... Also, previous HN thread about it here: Revisiting Knuth and McIlroy's word count programs (2011) (franklinchen.com) https://news.ycombinator.com/item?id=15264997 https://news.ycombinator.com/item?id=15264997

henrik_w 8y ago
There is also the book "Exercises in Programming Style" that uses the term frequency problem to illustrate 33 different programming paradigms (including a shell solution). https://www.amazon.com/Exercises-Programming-Style-Cristina-Videira/dp/1482227371/ https://www.amazon.com/Exercises-Programming-Style-Cristina-...
- vram22 8y ago
  Cool. Will check that out.

svat 8y ago

That was quite unfair criticism, and even Doug McIlroy knew it (as he admitted later). The background is this: - Bentley, the author of the column, invited Knuth to demonstrate literate programming using a program of his choice. - Knuth insisted that to be fair, Bentley ought to specify the program to be written; else someone might object that Knuth chose a program that would be good for literate programming. - Bentley chose (what we'd now call) the term frequency problem (list the top k most frequent words in a text file), and accordingly Knuth wrote a system program for solving just this one particular task. (Did everything from opening the input file to formatting the output, etc.) - Doug McIlroy was asked to “review” this program, the way works of literature are reviewed. He happens to be the inventor of Unix pipes. Towards the end of his review, along with many other points (e.g. Knuth didn't include diagrams in cases where most of us would appreciate them, something I still struggle with when reading TeX and other programs), he used the opportunity to demonstrate his own invention, a shell pipeline using now-standard Unix tools (tr, sort, uniq, sed). There are a few things wrong with this criticism: - The main thing is that DEK wrote the program he was asked to write, so pointing out that he shouldn't have written that program is a criticism of the one who chose the program (Bentley mentioned this when printing McIlroy's review). - At the time, Unix wasn't even widely available outside Bell Labs and a few places; it definitely wasn't available to Knuth or most of the column's readers. - Knuth's program, fine-tuned for the task, is more efficient than the shell pipeline. - Even if you use a shell pipeline, someone has to write the “standard” programs that go into it (the "tr", "sort", "uniq" and "sed" above), and literate programming can be used there. In fact, Knuth did exactly that a few years later, rewriting Unix's “wc” (IIRC) and compared the resulting program with that of Sun Unix's wc, and his LP version had, among things, better error handling. (He's explained it by saying that in conventional programming, if you have a small function and 90% of it is error-checking, it looks like the function is “about” error-checking, so there's a psychological resistance to doing too much of that, while with LP you move the error-handling to a separate section entirely about error-checking and then you tend to do a better job. BTW, TeX's error handling is phenomenally good IMO; the opposite of the situation with LaTeX.) All that said, there is some valid criticism that Knuth prefers to write monolithic programs, but that works for him. He seems not to consider it a problem that to change something you have to understand more-or-less the entire program; he seems to prefer doing that anyway (he reads other people's code a lot: in Peter Seibel's Coders at Work, he was the only person interviewed who read others' programs regularly).

mci 8y ago

> He has fashioned a sort of industrial-strength Fabergé egg—intricate, wonderfully worked, refined beyond all ordinary desires, a museum piece from the start. Curiously, I independently came up with this metaphor in my review of The MMIX Supplement to The Art of Computer Programming [1]: "You can marvel at its intricacies like one marvels at a Fabergé egg." [1] https://www.amazon.com/gp/customer-reviews/R25NHN9UP7PLJI/ https://www.amazon.com/gp/customer-reviews/R25NHN9UP7PLJI/

ineedasername 8y ago

McIlroy's critique is both astute on it's own yet perhaps not relevant in the context of what Knuth had been asked to do, namely present an example of literate programming. That's not to say the McIlroy's script can't have a literate form, only that it wouldn't serve as a good exemplar. It all serves as a good reminder of how Knuth's work should be used & viewed: It shouldn't be taken as insight into how to manage and develop for software projects. It's more useful as teaching tools for how to think about solving problems. In my mind it's the difference between "pure" science and practical engineering, though I'm not sure that's a perfect analogy. To try another analogy, it's like Knuth was asked to make a custom set of clothes. And McIlroy's critique seems like saying "Not everyone can afford custom made clothes, and the GAP produces perfectly serviceable clothes that are far more practical in most situations" It's correct, sure, but rather besides the point in the context of the original request.

mschaef 8y ago
> perhaps not relevant in the context of what Knuth had been asked to do, namely present an example of literate programming. Agreed... if that was the goal, then McIlroy's example was terrible (in that it didn't use literate programming at all).

jeklj 8y ago

Yes, I love this story too. It really demonstrates so much. And oh, the hackles it raises. :)

mschaef 8y ago
> And oh, the hackles it raises. :) The main reason I like the story is that it's good fodder for discussion about priorities in software development. After all, counting words isn't exactly an interesting problem in 2018, but deciding the effective use of libraries and programmer time is still very interesting.