5 ms·

More shell, less egg (2011)

37 points by zdw 5y ago

School-Cotton 5y ago
Anyone know where to find Knuth’s version?
- JohnHaugeland 5y ago
  the article says the book's name
- tyingq 5y ago
  Here: https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-1.pdf https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-... https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-2.pdf https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-...
- svat 5y ago
  The original published version (and McIlroy's review) is in the second of these two columns (including the earlier one for context): • Bentley, J., & Knuth, D. (1986). Programming pearls: Literate Programming. Communications of the ACM, 29(5), 384–369. doi:10.1145/5689.315644 = https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-1.pdf https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-... • Bentley, J., Knuth, D., & McIlroy, D. (1986). Programming pearls: A Literate Program. Communications of the ACM, 29(6), 471–483. doi:10.1145/5948.315654 = https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-2.pdf https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-... It has been reprinted (along with the review) in Knuth's books Literate Programming (https://cs.stanford.edu/~knuth/lp.html https://cs.stanford.edu/~knuth/lp.html), one of the volumes of his collected papers (https://cs.stanford.edu/~knuth/selected.html https://cs.stanford.edu/~knuth/selected.html). Of course this post here ("More shell, less egg") is highly misleading, one which I'll post a separate top-level comment.
  - School-Cotton 5y ago
    Thanks!
thriftwy 5y ago
It's not as bad that 10 pages of Pascal take place of a shell one-liner. It's really bad when you have to write those 10 pages over and over again due to insufficient tooling available.
SavantIdiot 5y ago
- morelisp 5y ago
  The unfortunate closing statement is the single reason I didn't include this in our shell programming onboarding documentation as a motivating example of how shell fluency can help solve ad hoc questions.
  - SavantIdiot 5y ago
    And it got flagged. HN: still a bro club after all these years.
- kwhitefoot 5y ago
  Perhaps you could suggest an alternative?
  - morelisp 5y ago
    The piece would be better without the closing section entirely. McIlroy demonstrated a different, not competing and not in any sense "courageous" or "daring", approach to program design.
- kwhitefoot 5y ago
  These days it's not much more gendered than the word guys which is now frequently used to refer to mixed sex groups. And anyway Doug McIlroy is male so what exactly is your complaint?
dang 5y ago
Past related threads: Donald Knuth was framed - https://news.ycombinator.com/item?id=22406070 https://news.ycombinator.com/item?id=22406070 - Feb 2020 (180 comments) More shell, less egg (And Then McIlroy Said Unto Knuth) - https://news.ycombinator.com/item?id=15701452 https://news.ycombinator.com/item?id=15701452 - Nov 2017 (2 comments) More shell, less egg (2011) - https://news.ycombinator.com/item?id=15265000 https://news.ycombinator.com/item?id=15265000 - Sept 2017 (7 comments) More shell, less egg (2011) - https://news.ycombinator.com/item?id=10081539 https://news.ycombinator.com/item?id=10081539 - Aug 2015 (28 comments) More shell, less egg (2011, on McIlroy and Knuth) - https://news.ycombinator.com/item?id=7381900 https://news.ycombinator.com/item?id=7381900 - March 2014 (3 comments) Knuth and the Unix Way - https://news.ycombinator.com/item?id=3329668 https://news.ycombinator.com/item?id=3329668 - Dec 2011 (62 comments)
tpoacher 5y ago
Every time I read in an article (or more typically in a headline) how some guy X "eviscerated" / "destroyed" some other guy Y and their argument, 9 times out of 10 it just means some guy X had a different opinion. Very occasionally their opinion may be backed by an actual counterargument, but this is quite rare, and even then the counterarguments tend to be of varying quality. Rarely is there an actual "evisceration" at play (a term I would have otherwise reserved for someone pointing out basic logical flaws in an argument, which cause it to collapse as completely invalid). I've seen this kind of language used as clickbait more than anything else so many times by now, that by this point whenever I hear "X destroys Y", my knee jerk reaction is that, despite the usually present controversy surrounding the kind of opinions that attract this kind of reaponses, for Y to merit such a vacuous attack in the first place means they're probably right.
- deleted 5y ago
  [deleted]
- michaelcampbell 5y ago
  > Every time I read in an article (or more typically in a headline) how some guy X "eviscerated" / "destroyed" some other guy Y and their argument, 9 times out of 10 it just means some guy X had a different opinion. Neither was said. > And then he calmly and clearly eviscerated the very foundation of Knuth’s program.
  - tpoacher 5y ago
    I fail to see how this is not a perfect example of the above. Just because he supposedly did it "calmly"? How did he "eviscerate" it? Did he have an objective metric where it scored low despite promises? Did he show it kills babies? No. He just wrote a different one which in his opinion was better. The author is simply exaggerating for effect.
ufo 5y ago
I've heard this story before, but I can't shake the impression that a lot of it is just luck that the problem is something that lends itself very well to a shell pipeline. One thing that I'd love to hear more people talk about is how to apply shell-style composition to settings where it's not so obvious, or how to create new programs that lend themselves well to this programming style.
tyingq 5y ago
Was curious how fast the shell was versus something more purpose written. This short bit of (golfed/terse) Perl seems to be 2-3x faster for the inputs I tested. Though it's also 2x more "code", despite golfing and skipping best practice strict/warnings. And both are wasting some time sorting the whole list versus a partial heap sort or similar. #!/usr/bin/perl $l=shift(@ARGV)||1; while (<>) { for (split) { $c{lc($_)}++; } } for (sort { $c{$b} <=> $c{$a} } (keys %c)) { print "$c{$_} $_\n"; last if ++$i == $l; }
- chasil 5y ago
  That would depend upon the shell that you used. GNU Bash in particular is not known for speed, although this is just a question of pipes. I remember that cdrecord has explict advice on raising pipe buffer size to avoid an underun (that will render a CD unusable). Perhaps pipe tuning here might also impact performance. I don't know if dash will increase performance in this particular case. $ rpm -qi dash | tail -3 DASH is a POSIX-compliant implementation of /bin/sh that aims to be as small as possible. It does this without sacrificing speed where possible. In fact, it is significantly faster than bash (the GNU Bourne-Again SHell) for most tasks.
  - tyingq 5y ago
    Looking at the shell script, I think it will always be somewhat slow as it's sorting twice. One pass to sort the words so that "uniq -c" can work correctly, then the reverse numeric sort. I don't think the shell used matters, as most of the cpu time is in the sorting.
- mst 5y ago
  I think I'd prefer populating %c using: $c{lc $_}++ for map split(), <ARGV>; (I pondered 'do { local $/; <ARGV> }' to do a full slurp as well, but I don't think it helps with clarity and I'm not actually golfing here even though the resulting code happens to be shorter) Also, if you have List::UtilsBy handy (which I basically always do), rev_nsort_by { $c{$_} } keys %c could be a nicer way to express the sort, although probably as fast because of the block expr versus anon-sub-via-& prototype part. (I have not executed any of this code, I'm just musing here, so no warranty express or implied ;)
svat 5y ago
There has been much discussion about how this post is highly misleading. See • Hillel Wayne, "Donald Knuth Was Framed": https://buttondown.email/hillelwayne/archive/donald-knuth-was-framed/ https://buttondown.email/hillelwayne/archive/donald-knuth-wa... • me, various comments, including https://news.ycombinator.com/item?id=22407313 https://news.ycombinator.com/item?id=22407313 in the comments on that post, linking to older/other comments.
- dang 5y ago
  I added that one to the list of past threads here: https://news.ycombinator.com/item?id=30039164 https://news.ycombinator.com/item?id=30039164. Thanks!
- CrazyStat 5y ago
  Thank you. This story comes up from time to time and I hate it. Knuth was asked to illustrate literate programming style, and he did so. The request was not "show us how to find the most frequently occurring words in a file." McIlroy's response would be fine for the latter and utterly useless for the former (and actual) request.

37 points by zdw 5y ago

School-Cotton 5y ago

Anyone know where to find Knuth’s version?

JohnHaugeland 5y ago
the article says the book's name
tyingq 5y ago
Here: https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-1.pdf https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-... https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-2.pdf https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-...
svat 5y ago
The original published version (and McIlroy's review) is in the second of these two columns (including the earlier one for context): • Bentley, J., & Knuth, D. (1986). Programming pearls: Literate Programming. Communications of the ACM, 29(5), 384–369. doi:10.1145/5689.315644 = https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-1.pdf https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-... • Bentley, J., Knuth, D., & McIlroy, D. (1986). Programming pearls: A Literate Program. Communications of the ACM, 29(6), 471–483. doi:10.1145/5948.315654 = https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-2.pdf https://www.cs.tufts.edu/~nr/cs257/archive/don-knuth/pearls-... It has been reprinted (along with the review) in Knuth's books Literate Programming (https://cs.stanford.edu/~knuth/lp.html https://cs.stanford.edu/~knuth/lp.html), one of the volumes of his collected papers (https://cs.stanford.edu/~knuth/selected.html https://cs.stanford.edu/~knuth/selected.html). Of course this post here ("More shell, less egg") is highly misleading, one which I'll post a separate top-level comment.
- School-Cotton 5y ago
  Thanks!

thriftwy 5y ago

It's not as bad that 10 pages of Pascal take place of a shell one-liner. It's really bad when you have to write those 10 pages over and over again due to insufficient tooling available.

SavantIdiot 5y ago

morelisp 5y ago
The unfortunate closing statement is the single reason I didn't include this in our shell programming onboarding documentation as a motivating example of how shell fluency can help solve ad hoc questions.
- SavantIdiot 5y ago
  And it got flagged. HN: still a bro club after all these years.
kwhitefoot 5y ago
Perhaps you could suggest an alternative?
- morelisp 5y ago
  The piece would be better without the closing section entirely. McIlroy demonstrated a different, not competing and not in any sense "courageous" or "daring", approach to program design.
kwhitefoot 5y ago
These days it's not much more gendered than the word guys which is now frequently used to refer to mixed sex groups. And anyway Doug McIlroy is male so what exactly is your complaint?

dang 5y ago

Past related threads: Donald Knuth was framed - https://news.ycombinator.com/item?id=22406070 https://news.ycombinator.com/item?id=22406070 - Feb 2020 (180 comments) More shell, less egg (And Then McIlroy Said Unto Knuth) - https://news.ycombinator.com/item?id=15701452 https://news.ycombinator.com/item?id=15701452 - Nov 2017 (2 comments) More shell, less egg (2011) - https://news.ycombinator.com/item?id=15265000 https://news.ycombinator.com/item?id=15265000 - Sept 2017 (7 comments) More shell, less egg (2011) - https://news.ycombinator.com/item?id=10081539 https://news.ycombinator.com/item?id=10081539 - Aug 2015 (28 comments) More shell, less egg (2011, on McIlroy and Knuth) - https://news.ycombinator.com/item?id=7381900 https://news.ycombinator.com/item?id=7381900 - March 2014 (3 comments) Knuth and the Unix Way - https://news.ycombinator.com/item?id=3329668 https://news.ycombinator.com/item?id=3329668 - Dec 2011 (62 comments)

tpoacher 5y ago

Every time I read in an article (or more typically in a headline) how some guy X "eviscerated" / "destroyed" some other guy Y and their argument, 9 times out of 10 it just means some guy X had a different opinion. Very occasionally their opinion may be backed by an actual counterargument, but this is quite rare, and even then the counterarguments tend to be of varying quality. Rarely is there an actual "evisceration" at play (a term I would have otherwise reserved for someone pointing out basic logical flaws in an argument, which cause it to collapse as completely invalid). I've seen this kind of language used as clickbait more than anything else so many times by now, that by this point whenever I hear "X destroys Y", my knee jerk reaction is that, despite the usually present controversy surrounding the kind of opinions that attract this kind of reaponses, for Y to merit such a vacuous attack in the first place means they're probably right.

deleted 5y ago
[deleted]
michaelcampbell 5y ago
> Every time I read in an article (or more typically in a headline) how some guy X "eviscerated" / "destroyed" some other guy Y and their argument, 9 times out of 10 it just means some guy X had a different opinion. Neither was said. > And then he calmly and clearly eviscerated the very foundation of Knuth’s program.
- tpoacher 5y ago
  I fail to see how this is not a perfect example of the above. Just because he supposedly did it "calmly"? How did he "eviscerate" it? Did he have an objective metric where it scored low despite promises? Did he show it kills babies? No. He just wrote a different one which in his opinion was better. The author is simply exaggerating for effect.

ufo 5y ago

I've heard this story before, but I can't shake the impression that a lot of it is just luck that the problem is something that lends itself very well to a shell pipeline. One thing that I'd love to hear more people talk about is how to apply shell-style composition to settings where it's not so obvious, or how to create new programs that lend themselves well to this programming style.

tyingq 5y ago

Was curious how fast the shell was versus something more purpose written. This short bit of (golfed/terse) Perl seems to be 2-3x faster for the inputs I tested. Though it's also 2x more "code", despite golfing and skipping best practice strict/warnings. And both are wasting some time sorting the whole list versus a partial heap sort or similar. #!/usr/bin/perl $l=shift(@ARGV)||1; while (<>) { for (split) { $c{lc($_)}++; } } for (sort { $c{$b} <=> $c{$a} } (keys %c)) { print "$c{$_} $_\n"; last if ++$i == $l; }

chasil 5y ago
That would depend upon the shell that you used. GNU Bash in particular is not known for speed, although this is just a question of pipes. I remember that cdrecord has explict advice on raising pipe buffer size to avoid an underun (that will render a CD unusable). Perhaps pipe tuning here might also impact performance. I don't know if dash will increase performance in this particular case. $ rpm -qi dash | tail -3 DASH is a POSIX-compliant implementation of /bin/sh that aims to be as small as possible. It does this without sacrificing speed where possible. In fact, it is significantly faster than bash (the GNU Bourne-Again SHell) for most tasks.
- tyingq 5y ago
  Looking at the shell script, I think it will always be somewhat slow as it's sorting twice. One pass to sort the words so that "uniq -c" can work correctly, then the reverse numeric sort. I don't think the shell used matters, as most of the cpu time is in the sorting.
mst 5y ago
I think I'd prefer populating %c using: $c{lc $_}++ for map split(), <ARGV>; (I pondered 'do { local $/; <ARGV> }' to do a full slurp as well, but I don't think it helps with clarity and I'm not actually golfing here even though the resulting code happens to be shorter) Also, if you have List::UtilsBy handy (which I basically always do), rev_nsort_by { $c{$_} } keys %c could be a nicer way to express the sort, although probably as fast because of the block expr versus anon-sub-via-& prototype part. (I have not executed any of this code, I'm just musing here, so no warranty express or implied ;)

svat 5y ago

There has been much discussion about how this post is highly misleading. See • Hillel Wayne, "Donald Knuth Was Framed": https://buttondown.email/hillelwayne/archive/donald-knuth-was-framed/ https://buttondown.email/hillelwayne/archive/donald-knuth-wa... • me, various comments, including https://news.ycombinator.com/item?id=22407313 https://news.ycombinator.com/item?id=22407313 in the comments on that post, linking to older/other comments.

dang 5y ago
I added that one to the list of past threads here: https://news.ycombinator.com/item?id=30039164 https://news.ycombinator.com/item?id=30039164. Thanks!
CrazyStat 5y ago
Thank you. This story comes up from time to time and I hate it. Knuth was asked to illustrate literate programming style, and he did so. The request was not "show us how to find the most frequently occurring words in a file." McIlroy's response would be fine for the latter and utterly useless for the former (and actual) request.