5 ms·

McIlroy's basic point is that, instead of building single-use programs, we should be directing our effort towards building reusable components, i.e. Unix utilit

McIlroy's basic point is that, instead of building single-use programs, we should be directing our effort towards building reusable components, i.e. Unix utilities. Then, over time, most programs could be built as a simple piping together of high-level components. This vision seems to have failed for about 5 decades now, but it is still a popular one. After no one was able to build serious applications by gluing together Unix utilities, it was thought that we would instead build serious applications by gluing together objects. This vision has also failed (or at least, not succeeded in a very apparent way). In the mean time there have been other promised types of "componentry", like COM, CORBA, WS-*, ..., all more or less complete failures (or at least, egregiously unapparent successes). So, in my view, it is Knuth who has the last laugh.

angersock 15y ago
What's somewhat amusing, though, is that there doesn't seem to be a great deal of wholly bespoke software being written in certain industries (games, a lot of the Web stuff). It's pretty rare that a thick monolithic client of any real seriousness is still being produced, In a way, a lot of folks are just plumbing together little utilities and libraries to do what they could be programming themselves. jQuery, Rails, node.js, redis, sqllite, etc. are the components for larger plumbed systems. So, with the overwhelming reliance on frameworks and third-party libraries in modern software development, perhaps McIlroy may be shown right yet.
- deleted 15y ago
  [deleted]
tmhedberg 15y ago
I strongly disagree with the notion that the composition of small programs to build larger ones is a "failed vision". Shell scripting is an invaluable tool in the daily-use toolbox for a huge number of people, myself included. And what is a shell, besides a programming language for gluing together other programs? Now, if you mean that no one is writing full-scale professional desktop or web applications by piping together sed and awk, then of course you are correct. But many projects come closer to this than you might think. Git, for instance, was originally just a large collection of shell scripts, which has over time been gradually rewritten in C. Many pieces of it are, even today, still written in the POSIX Bourne shell. To me, the principal reason for the astounding longevity of Unix is its emphasis on composability. The tiny, ubiquitous Unix utilities are ancient compared to virtually every other piece of software in common usage, yet are no less useful today than when ken and dmr first conceived of them. By making the system modular and composable at a very fundamental level, they ensured (perhaps as much by accident as by intentional design) that users far into the future would be able to continue using their tools for purposes not yet dreamed of. And indeed, this is very much the case, and is likely to continue that way for some time to come. Whatever the eventual successor to Unix turns out to be, it's likely to have the same emphasis on small building blocks with enormous synergy when composed. Likewise, when you say that the vision of "building serious applications by gluing together objects" has "not succeeded in a very apparent way", I frankly have no idea what you're talking about. OOP is the dominant paradigm in modern programming, bar none. You might argue that some other paradigm (FP, perhaps?) would have served us better in retrospect, but the last thing anyone can truthfully claim about the object-oriented approach is that it has not been successful. It's hard to think of even one popular programming language that doesn't borrow at least a few ideas from that school of thought. I do agree with you about the "enterprisey" component approach being largely a failure. Thankfully, CORBA, SOAP, et al. seem to be mostly behind us, or at least rapidly receding.
- nwmcsween 15y ago
  Composition should be of libraries or algorithms not arbitrary black box programs with n different options. Shell scripting for anything non-trivial is a pain as nothing is portable or reusable. Linux has extensions to posixx, Freebsd as well and so do their specific core utilities.
  - deleted 15y ago
    [deleted]
- grannyg00se 15y ago
  "Git, for instance, was originally just a large collection of shell scripts, which has over time been gradually rewritten in C. " Doesn't that prove the parent's point that the collection of supposedly reusable components could not be glued together to form GIT? There must be a reason why it was rewritten in C. And if it had to be rewritten then I don't think we can call the "large collection of shell scripts" a complete success. "OOP is the dominant paradigm in modern programming, bar none." There could be many reasons for that but it does not address the point that the actual gluing together of objects has somehow succeeded. How much of that OO work is actually shared between projects? How much of it is simply OOP for the sake of it because the language was chosen for convenience? To me success would be using objects from one project without modification in a completely separate project. I just don't see that happening very often. If the objects are simple enough you might as well just rewrite them. If they are complicated enough to be of value to import, they usually require modification.
  - pmarin 15y ago
    Doesn't that prove the parent's point that the collection of supposedly reusable components could not be glued together to form GIT? There must be a reason why it was rewritten in C. Ironically the main reason for the rewriting was the poor performance of the fork system call in Windows (Cygwin) which make the system slow particularly for shell scripting. The problem was Windows, not Git.
    - 15y ago
sedq 15y ago
When I read comments like this I'm left with two questions: 1. What are "serious applications"? 2. How does this person define "failed"? This task of counting word frequencies looks like an ideal job for awk. But there are many ways to do it, and with different UNIX's utilities. UNIX is remarkably flexible. When awk was introduced it was not imagined that people would try to write 10 page programs with it. But of course, they did. Are these the so-called "serious applications" that some people want to write? If UNIX utilities and pipes (including the pipe function in C) are a "failure", why are they still with us after so many years? If that's failure, then what is "success"? I think it comes down to what you're trying to do. For processing text, such as the task discussed in the article, I find UNIX utilities to be enough. I still have no idea what "serious applications" are. Is text processing "serious"? Much of UNIX's userland was intended for text processing. And for that it works very well.
derleth 15y ago
As sedq, who is dead for some reason, said: > When I read comments like this I'm left with two questions: 1. What are "serious applications"? 2. How does this person define "failed"? > This task of counting word frequencies looks like an ideal job for awk. But there are many ways to do it, and with different UNIX's utilities. UNIX is remarkably flexible. > When awk was introduced it was not imagined that people would try to write 10 page programs with it. But of course, they did. Are these the so-called "serious applications" that some people want to write? > If UNIX utilities and pipes (including the pipe function in C) are a "failure", why are they still with us after so many years? If that's failure, then what is "success"? > I think it comes down to what you're trying to do. For processing text, such as the task discussed in the article, I find UNIX utilities to be enough. > I still have no idea what "serious applications" are. Is text processing "serious"? > Much of UNIX's userland was intended for text processing. And for that it works very well.
- wmf 15y ago
  If UNIX utilities and pipes (including the pipe function in C) are a "failure", why are they still with us after so many years? I guess "us" means 1% of the population, since Mac/Windows users generally don't use the command line and the apps they do use aren't held together with pipes.
  - sedq 15y ago
    True. And that may be what he meant by failure. If "everyone" isn't using them, then he considers them a "failure". What I observe however is that these utilities are not going to disappear, they are not going to be replaced; they will remain part of numerous operating systems "everyone" continues to use (maybe they'll be buried beneath the surface but they will still be there). And therefore they will be "with us" for a long time, because the people that _do_ use them are the ones that build and administer the systems for "everyone" else to use. If these utilities served no useful purpose, then I think we would get rid of them. Interesting about git. I was not aware it was built using the shell. CVS also began as a set of shell scripts. For me, nothing beats the shell for one-liners. I don't think of this as "prototyping", but I certainly wouldn't mind having some of my better scripts re-written in C. And given the choice I'd rather hack together a "prototype" using a shell than using some other scripting language, Java or whatever is the "most popular" prototyping environment.
zohebv 15y ago
I think functional programmers would disagree with your thesis :-) As an example http://xmonad.org/manpage.html http://xmonad.org/manpage.html - Window manager in 200 lines of code http://www.haskell.org/haskellwiki/Xmonad/Screenshots http://www.haskell.org/haskellwiki/Xmonad/Screenshots Or FRAN http://conal.net/papers/icfp97/ http://conal.net/papers/icfp97/ I am sure there are pretty cool smalltalk and lips examples too. Knuth is an algorithms engineer and his solution reflects his background. McIlroy's invention of pipes was seminal, but the state of the art has moved on much further. I think David Turner's contributions via SASL have never been fully acknowledged. I would posit that we now know how to go about achieving the composability techniques required to build large programs easily. The answer seems to lie embedded somewhere deep inside category theory! The ideas are slowly getting popular and seeping over to mainstream languages. At least for me, the way I program today is fundamentally different from 12 years ago.
derefr 15y ago
I would say that in the modern day, we seem to be building plenty of serious applications by gluing together web service APIs.

angersock 15y ago

What's somewhat amusing, though, is that there doesn't seem to be a great deal of wholly bespoke software being written in certain industries (games, a lot of the Web stuff). It's pretty rare that a thick monolithic client of any real seriousness is still being produced, In a way, a lot of folks are just plumbing together little utilities and libraries to do what they could be programming themselves. jQuery, Rails, node.js, redis, sqllite, etc. are the components for larger plumbed systems. So, with the overwhelming reliance on frameworks and third-party libraries in modern software development, perhaps McIlroy may be shown right yet.

deleted 15y ago
[deleted]

tmhedberg 15y ago

I strongly disagree with the notion that the composition of small programs to build larger ones is a "failed vision". Shell scripting is an invaluable tool in the daily-use toolbox for a huge number of people, myself included. And what is a shell, besides a programming language for gluing together other programs? Now, if you mean that no one is writing full-scale professional desktop or web applications by piping together sed and awk, then of course you are correct. But many projects come closer to this than you might think. Git, for instance, was originally just a large collection of shell scripts, which has over time been gradually rewritten in C. Many pieces of it are, even today, still written in the POSIX Bourne shell. To me, the principal reason for the astounding longevity of Unix is its emphasis on composability. The tiny, ubiquitous Unix utilities are ancient compared to virtually every other piece of software in common usage, yet are no less useful today than when ken and dmr first conceived of them. By making the system modular and composable at a very fundamental level, they ensured (perhaps as much by accident as by intentional design) that users far into the future would be able to continue using their tools for purposes not yet dreamed of. And indeed, this is very much the case, and is likely to continue that way for some time to come. Whatever the eventual successor to Unix turns out to be, it's likely to have the same emphasis on small building blocks with enormous synergy when composed. Likewise, when you say that the vision of "building serious applications by gluing together objects" has "not succeeded in a very apparent way", I frankly have no idea what you're talking about. OOP is the dominant paradigm in modern programming, bar none. You might argue that some other paradigm (FP, perhaps?) would have served us better in retrospect, but the last thing anyone can truthfully claim about the object-oriented approach is that it has not been successful. It's hard to think of even one popular programming language that doesn't borrow at least a few ideas from that school of thought. I do agree with you about the "enterprisey" component approach being largely a failure. Thankfully, CORBA, SOAP, et al. seem to be mostly behind us, or at least rapidly receding.

nwmcsween 15y ago
Composition should be of libraries or algorithms not arbitrary black box programs with n different options. Shell scripting for anything non-trivial is a pain as nothing is portable or reusable. Linux has extensions to posixx, Freebsd as well and so do their specific core utilities.
- deleted 15y ago
  [deleted]
grannyg00se 15y ago
"Git, for instance, was originally just a large collection of shell scripts, which has over time been gradually rewritten in C. " Doesn't that prove the parent's point that the collection of supposedly reusable components could not be glued together to form GIT? There must be a reason why it was rewritten in C. And if it had to be rewritten then I don't think we can call the "large collection of shell scripts" a complete success. "OOP is the dominant paradigm in modern programming, bar none." There could be many reasons for that but it does not address the point that the actual gluing together of objects has somehow succeeded. How much of that OO work is actually shared between projects? How much of it is simply OOP for the sake of it because the language was chosen for convenience? To me success would be using objects from one project without modification in a completely separate project. I just don't see that happening very often. If the objects are simple enough you might as well just rewrite them. If they are complicated enough to be of value to import, they usually require modification.
- pmarin 15y ago
  Doesn't that prove the parent's point that the collection of supposedly reusable components could not be glued together to form GIT? There must be a reason why it was rewritten in C. Ironically the main reason for the rewriting was the poor performance of the fork system call in Windows (Cygwin) which make the system slow particularly for shell scripting. The problem was Windows, not Git.
  - 15y ago

sedq 15y ago

When I read comments like this I'm left with two questions: 1. What are "serious applications"? 2. How does this person define "failed"? This task of counting word frequencies looks like an ideal job for awk. But there are many ways to do it, and with different UNIX's utilities. UNIX is remarkably flexible. When awk was introduced it was not imagined that people would try to write 10 page programs with it. But of course, they did. Are these the so-called "serious applications" that some people want to write? If UNIX utilities and pipes (including the pipe function in C) are a "failure", why are they still with us after so many years? If that's failure, then what is "success"? I think it comes down to what you're trying to do. For processing text, such as the task discussed in the article, I find UNIX utilities to be enough. I still have no idea what "serious applications" are. Is text processing "serious"? Much of UNIX's userland was intended for text processing. And for that it works very well.

derleth 15y ago

As sedq, who is dead for some reason, said: > When I read comments like this I'm left with two questions: 1. What are "serious applications"? 2. How does this person define "failed"? > This task of counting word frequencies looks like an ideal job for awk. But there are many ways to do it, and with different UNIX's utilities. UNIX is remarkably flexible. > When awk was introduced it was not imagined that people would try to write 10 page programs with it. But of course, they did. Are these the so-called "serious applications" that some people want to write? > If UNIX utilities and pipes (including the pipe function in C) are a "failure", why are they still with us after so many years? If that's failure, then what is "success"? > I think it comes down to what you're trying to do. For processing text, such as the task discussed in the article, I find UNIX utilities to be enough. > I still have no idea what "serious applications" are. Is text processing "serious"? > Much of UNIX's userland was intended for text processing. And for that it works very well.

wmf 15y ago
If UNIX utilities and pipes (including the pipe function in C) are a "failure", why are they still with us after so many years? I guess "us" means 1% of the population, since Mac/Windows users generally don't use the command line and the apps they do use aren't held together with pipes.
- sedq 15y ago
  True. And that may be what he meant by failure. If "everyone" isn't using them, then he considers them a "failure". What I observe however is that these utilities are not going to disappear, they are not going to be replaced; they will remain part of numerous operating systems "everyone" continues to use (maybe they'll be buried beneath the surface but they will still be there). And therefore they will be "with us" for a long time, because the people that _do_ use them are the ones that build and administer the systems for "everyone" else to use. If these utilities served no useful purpose, then I think we would get rid of them. Interesting about git. I was not aware it was built using the shell. CVS also began as a set of shell scripts. For me, nothing beats the shell for one-liners. I don't think of this as "prototyping", but I certainly wouldn't mind having some of my better scripts re-written in C. And given the choice I'd rather hack together a "prototype" using a shell than using some other scripting language, Java or whatever is the "most popular" prototyping environment.

zohebv 15y ago

I think functional programmers would disagree with your thesis :-) As an example http://xmonad.org/manpage.html http://xmonad.org/manpage.html - Window manager in 200 lines of code http://www.haskell.org/haskellwiki/Xmonad/Screenshots http://www.haskell.org/haskellwiki/Xmonad/Screenshots Or FRAN http://conal.net/papers/icfp97/ http://conal.net/papers/icfp97/ I am sure there are pretty cool smalltalk and lips examples too. Knuth is an algorithms engineer and his solution reflects his background. McIlroy's invention of pipes was seminal, but the state of the art has moved on much further. I think David Turner's contributions via SASL have never been fully acknowledged. I would posit that we now know how to go about achieving the composability techniques required to build large programs easily. The answer seems to lie embedded somewhere deep inside category theory! The ideas are slowly getting popular and seeping over to mainstream languages. At least for me, the way I program today is fundamentally different from 12 years ago.

derefr 15y ago

I would say that in the modern day, we seem to be building plenty of serious applications by gluing together web service APIs.