6 ms·

The Unix Command Language (1976)

147 points by susam 6y ago

susam 6y ago
Direct links to PDF (scanned images) and HTML (generated from Markdown): PDF: https://susam.github.io/tucl/the-unix-command-language.pdf https://susam.github.io/tucl/the-unix-command-language.pdf HTML: https://susam.github.io/tucl/the-unix-command-language.html https://susam.github.io/tucl/the-unix-command-language.html The PDF document is a scanned copy of the first ever paper that was published on the Unix shell. The paper was written by Ken Thompson and published in Structured Programming (Infotech state of the art report). The scanned copy was first obtained from Ken Thompson by wesleyneo and shared on archive.org. I combined the scanned images into a single PDF document and shared it here. Also, wesleyneo transcribed the paper to text format and I further edited and reformatted it to Markdown format. The HTML document is automatically generated from the Markdown file. All of these files are shared on the Internet with permission from Ken Thompson.
- userbinator 6y ago
  A bit of proofreading caught some small errors which were not in the original: - "This this invokes" -> "Thus this invokes" - "the standard output had occurred" -> "the standard output that has occurred" - "without typing up a console" -> "without tying up a console" - "The Shell, as command" -> "The Shell, as a command" - "late a night" -> "late at night" - "source programs has something" -> "source program has something" - "which states its contents" -> "which states it contents" [sic] - "In sincerely" -> "I sincerely"
  - susam 6y ago
    Fixed in https://github.com/susam/tucl/commit/b2abf75eefb34601bc38618e46794e3b72de68df https://github.com/susam/tucl/commit/b2abf75eefb34601bc38618... (commit b2abf75). Thank you for proofreading the Markdown document.
    - dmckeon 6y ago
      In the HTML version: s/21:26:39/21:36:39/ to agree with the PDF & spoken text.
      - susam 6y ago
        Fixed in commit https://github.com/susam/tucl/commit/56429d7fa941670cd6ff7c49e005c39d840d7a8d https://github.com/susam/tucl/commit/56429d7fa941670cd6ff7c4... (commit 56429d7). Thank you for reporting this error.
nsajko 6y ago
Thank you! I've searched for this three years ago, and I think it wasn't available on the Internet back then.
userbinator 6y ago
The "ls >output" example exposes a subtle artifact of the implementation: the output file will contain its own name.
- codetrotter 6y ago
  It’s not really subtle I think. When you redirect to a file, the file is opened for writing by the shell before it runs the program that you are invoking. This is also the same reason that if you try something like the following: grep whatever somefile >somefile You always end up with somefile being empty, whereas what you thought you were doing was replacing the contents of the file with only the lines in the file that contain the word “whatever”. But instead the shell will truncate the file first and then grep is searching through an already empty file and therefore there is nothing to find and the file ends up empty.
- rootbear 6y ago
  Back in my BSD-on-a-VAX days, I once typed grep string * > out and wondered why it was taking so long. The shell (probably csh) had created the file "out" before it expanded the *, so my grep command was forever finding "string" in it's own output. It filled the disk before I realized what was happening. I just did a quick test and bash seems to create the output file after doing the globbing, and under tcsh, grep gave a warning that the output file was also an input file and it did not search it.
jsz0 6y ago
It's always fascinating to read this type of stuff with the knowledge the authors influenced the way people use technology in such a fundamental way. It's like the inventor of the screwdriver modestly writing about how their curious little tool can be used.
quietbritishjim 6y ago
I'd be curious to know what some of "missing" concepts were. Obviously they turned out not to be necessary. > Many familiar computing ‘concepts’ are missing from UNIX. Files have no records. There are no access methods. User programs contain no system buffers. There are no file types.
- jcelerier 6y ago
  > Obviously they turned out not to be necessary. why would you believe that ? maybe computing would be 10x less painful today than it is if these missing concepts had been implemented. Or maybe 10x more.
  - kps 6y ago
    There exist people who have used both.
    - taffer 6y ago
      And these people were not always happy[1] with the way things are in UNIX. An example of something missing from UNIX is a service or job subsystem, which is why SMF, launchd and systemd were invented. [1] https://web.mit.edu/~simsong/www/ugh.pdf https://web.mit.edu/~simsong/www/ugh.pdf
      - generalizations 6y ago
        And it seems that a lot of people aren't happy with it, either.
- ryl00 6y ago
  Probably referring to examples like VAX/VMS filesystems? I never worked on them much, and only very shallowly. Fortran file I/O has vestiges of record-oriented interfaces still in the standard, I'm assuming from co-evolution with file systems of the time.
- rst 6y ago
asicsp 6y ago
Related: The Origin of the Shell, discussed two days back https://news.ycombinator.com/item?id=25207957 https://news.ycombinator.com/item?id=25207957
oliviergg 6y ago
The clarity of this documentation is really impressive. It’s like I was rediscovering the shell again.
beny23 6y ago
I think the clarity and brevity of this is amazing. Should be mandatory reading for teaching to people starting in computing. I’ve always thought that the concept of piping input to output and the shell is on of the most powerful “programming environments” around and to see its origins is humbling!
deleted 6y ago
[deleted]
hi41 6y ago
Such clarity and beauty in writing! I pray that in my lifetime I may be granted just one percent of that.
jonahx 6y ago
The first paragraph is a gem: "A program is generally exponentially complicated by the number of notions that it invents for itself. To reduce this complication to a minimum, you have to make the number of notions zero or one, which are two numbers that can be raised to any power without disturbing this concept. Since you cannot achieve much with zero notions, it is my belief that you should base systems on a single notion."
smlckz 6y ago
In the section THE SHELL AS A COMMAND, shouldn't the sh >file be sh <file ?
- susam 6y ago
  This was fixed a few minutes ago: https://github.com/susam/tucl/pull/7 https://github.com/susam/tucl/pull/7. Please see the PDF document at https://susam.github.io/tucl/the-unix-command-language.pdf https://susam.github.io/tucl/the-unix-command-language.pdf for a scanned copy of the original paper. If you find an error in the Markdown or HTML transcript, please create an issue or send a pull request to https://github.com/susam/tucl https://github.com/susam/tucl.
nerdponx 6y ago
All these demonstrations of piping make me wonder how and why POSIX file naming conventions became so loose (https://dwheeler.com/essays/fixing-unix-linux-filenames.html https://dwheeler.com/essays/fixing-unix-linux-filenames.html). Were things like whitespace in filenames just completely unimaginable at the time? Was colon-separated PATH not yet conceived of?
bewuethr 6y ago
A few things that stood out to me: Apart from the "goto" command and a different "if" statement syntax, all of the examples would run in a modern Bash shell; not much has changed fundamentally. There seem to be no comments, and the argument for the no-op ":" is used for this purpose instead. Using the argument of ":" as the label for the "goto" command reminded me of sed, which does exactly the same (and probably was influenced by it): sed ': label s/x/y/ /x/b label' loops over the "s/x/y/" command until the pattern space doesn't match "/x/" any longer. (Just for illustration - this could be replaced by a single global substitution, of course.) "goto" and labels are also the replacement for all the looping constructs we're used to today. The differentiation between source/filter/sink types of programs makes so much sense, and while maybe obvious, I've never seen it described this clearly. Descriptions of the composability of Unix programs often focus on the filter aspect of programs.
- asicsp 6y ago
  you can also use `t` command instead of `/x/b` or `//b` $ echo 'coffining' | sed ':d s/fin//; td' cog $ echo 'coffining' | sed ':d s/fin//; //b d' cog
- deleted 6y ago
  [deleted]
jolmg 6y ago
> The command > goto argument > 'rewinds' its standard input. It then reads the standard input looking for the syntax of a : command with an argument matching it own. It leaves the standard input positioned after the : command. When control is returned to the shell, execution continues where the standard input was positioned. (This implies certain things about file positioning and open file sharing that will not be discussed here.) Oh man, `:` makes so much sense now. It was originally meant to be like a label.
- userbinator 6y ago
  DOS' COMMAND.COM and later Windows' cmd.exe works the same way.
ncmncm 6y ago
Around 1978, while in high school, I spent many afternoons at the local university library skimming through six-inch-thick bound stacks of computer software journals. (That was how people communicated, before Internets.) I distinctly remember encountering exactly this paper, as my first exposure to Unix. It made a deep, deep impression. Pipe composition is just unaccountably powerful. What might not be so easy for people to understand today is how profoundly shocking it was to see computer commands and results expressed in lower case. I had never seen it done. Ever. It didn't seem possible, at first. After a few seconds, it was obvious, but still oddly liberating. You don't know computers are shouting at you until, suddenly, they aren't.

147 points by susam 6y ago

Direct links to PDF (scanned images) and HTML (generated from Markdown): PDF: https://susam.github.io/tucl/the-unix-command-language.pdf https://susam.github.io/tucl/the-unix-command-language.pdf HTML: https://susam.github.io/tucl/the-unix-command-language.html https://susam.github.io/tucl/the-unix-command-language.html The PDF document is a scanned copy of the first ever paper that was published on the Unix shell. The paper was written by Ken Thompson and published in Structured Programming (Infotech state of the art report). The scanned copy was first obtained from Ken Thompson by wesleyneo and shared on archive.org. I combined the scanned images into a single PDF document and shared it here. Also, wesleyneo transcribed the paper to text format and I further edited and reformatted it to Markdown format. The HTML document is automatically generated from the Markdown file. All of these files are shared on the Internet with permission from Ken Thompson.

userbinator 6y ago
A bit of proofreading caught some small errors which were not in the original: - "This this invokes" -> "Thus this invokes" - "the standard output had occurred" -> "the standard output that has occurred" - "without typing up a console" -> "without tying up a console" - "The Shell, as command" -> "The Shell, as a command" - "late a night" -> "late at night" - "source programs has something" -> "source program has something" - "which states its contents" -> "which states it contents" [sic] - "In sincerely" -> "I sincerely"
- susam 6y ago
  Fixed in https://github.com/susam/tucl/commit/b2abf75eefb34601bc38618e46794e3b72de68df https://github.com/susam/tucl/commit/b2abf75eefb34601bc38618... (commit b2abf75). Thank you for proofreading the Markdown document.
  - dmckeon 6y ago
    In the HTML version: s/21:26:39/21:36:39/ to agree with the PDF & spoken text.
    - susam 6y ago
      Fixed in commit https://github.com/susam/tucl/commit/56429d7fa941670cd6ff7c49e005c39d840d7a8d https://github.com/susam/tucl/commit/56429d7fa941670cd6ff7c4... (commit 56429d7). Thank you for reporting this error.

nsajko 6y ago

Thank you! I've searched for this three years ago, and I think it wasn't available on the Internet back then.

userbinator 6y ago

The "ls >output" example exposes a subtle artifact of the implementation: the output file will contain its own name.

codetrotter 6y ago
It’s not really subtle I think. When you redirect to a file, the file is opened for writing by the shell before it runs the program that you are invoking. This is also the same reason that if you try something like the following: grep whatever somefile >somefile You always end up with somefile being empty, whereas what you thought you were doing was replacing the contents of the file with only the lines in the file that contain the word “whatever”. But instead the shell will truncate the file first and then grep is searching through an already empty file and therefore there is nothing to find and the file ends up empty.
rootbear 6y ago
Back in my BSD-on-a-VAX days, I once typed grep string * > out and wondered why it was taking so long. The shell (probably csh) had created the file "out" before it expanded the *, so my grep command was forever finding "string" in it's own output. It filled the disk before I realized what was happening. I just did a quick test and bash seems to create the output file after doing the globbing, and under tcsh, grep gave a warning that the output file was also an input file and it did not search it.

jsz0 6y ago

It's always fascinating to read this type of stuff with the knowledge the authors influenced the way people use technology in such a fundamental way. It's like the inventor of the screwdriver modestly writing about how their curious little tool can be used.

quietbritishjim 6y ago

I'd be curious to know what some of "missing" concepts were. Obviously they turned out not to be necessary. > Many familiar computing ‘concepts’ are missing from UNIX. Files have no records. There are no access methods. User programs contain no system buffers. There are no file types.

jcelerier 6y ago
> Obviously they turned out not to be necessary. why would you believe that ? maybe computing would be 10x less painful today than it is if these missing concepts had been implemented. Or maybe 10x more.
- kps 6y ago
  There exist people who have used both.
  - taffer 6y ago
    And these people were not always happy[1] with the way things are in UNIX. An example of something missing from UNIX is a service or job subsystem, which is why SMF, launchd and systemd were invented. [1] https://web.mit.edu/~simsong/www/ugh.pdf https://web.mit.edu/~simsong/www/ugh.pdf
    - generalizations 6y ago
      And it seems that a lot of people aren't happy with it, either.
ryl00 6y ago
Probably referring to examples like VAX/VMS filesystems? I never worked on them much, and only very shallowly. Fortran file I/O has vestiges of record-oriented interfaces still in the standard, I'm assuming from co-evolution with file systems of the time.
rst 6y ago

asicsp 6y ago

Related: The Origin of the Shell, discussed two days back https://news.ycombinator.com/item?id=25207957 https://news.ycombinator.com/item?id=25207957

oliviergg 6y ago

The clarity of this documentation is really impressive. It’s like I was rediscovering the shell again.

beny23 6y ago

I think the clarity and brevity of this is amazing. Should be mandatory reading for teaching to people starting in computing. I’ve always thought that the concept of piping input to output and the shell is on of the most powerful “programming environments” around and to see its origins is humbling!

deleted 6y ago

[deleted]

hi41 6y ago

Such clarity and beauty in writing! I pray that in my lifetime I may be granted just one percent of that.

jonahx 6y ago

The first paragraph is a gem: "A program is generally exponentially complicated by the number of notions that it invents for itself. To reduce this complication to a minimum, you have to make the number of notions zero or one, which are two numbers that can be raised to any power without disturbing this concept. Since you cannot achieve much with zero notions, it is my belief that you should base systems on a single notion."

smlckz 6y ago

In the section THE SHELL AS A COMMAND, shouldn't the sh >file be sh <file ?

susam 6y ago
This was fixed a few minutes ago: https://github.com/susam/tucl/pull/7 https://github.com/susam/tucl/pull/7. Please see the PDF document at https://susam.github.io/tucl/the-unix-command-language.pdf https://susam.github.io/tucl/the-unix-command-language.pdf for a scanned copy of the original paper. If you find an error in the Markdown or HTML transcript, please create an issue or send a pull request to https://github.com/susam/tucl https://github.com/susam/tucl.

nerdponx 6y ago

All these demonstrations of piping make me wonder how and why POSIX file naming conventions became so loose (https://dwheeler.com/essays/fixing-unix-linux-filenames.html https://dwheeler.com/essays/fixing-unix-linux-filenames.html). Were things like whitespace in filenames just completely unimaginable at the time? Was colon-separated PATH not yet conceived of?

bewuethr 6y ago

A few things that stood out to me: Apart from the "goto" command and a different "if" statement syntax, all of the examples would run in a modern Bash shell; not much has changed fundamentally. There seem to be no comments, and the argument for the no-op ":" is used for this purpose instead. Using the argument of ":" as the label for the "goto" command reminded me of sed, which does exactly the same (and probably was influenced by it): sed ': label s/x/y/ /x/b label' loops over the "s/x/y/" command until the pattern space doesn't match "/x/" any longer. (Just for illustration - this could be replaced by a single global substitution, of course.) "goto" and labels are also the replacement for all the looping constructs we're used to today. The differentiation between source/filter/sink types of programs makes so much sense, and while maybe obvious, I've never seen it described this clearly. Descriptions of the composability of Unix programs often focus on the filter aspect of programs.

asicsp 6y ago
you can also use `t` command instead of `/x/b` or `//b` $ echo 'coffining' | sed ':d s/fin//; td' cog $ echo 'coffining' | sed ':d s/fin//; //b d' cog
deleted 6y ago
[deleted]

jolmg 6y ago

> The command > goto argument > 'rewinds' its standard input. It then reads the standard input looking for the syntax of a : command with an argument matching it own. It leaves the standard input positioned after the : command. When control is returned to the shell, execution continues where the standard input was positioned. (This implies certain things about file positioning and open file sharing that will not be discussed here.) Oh man, `:` makes so much sense now. It was originally meant to be like a label.

userbinator 6y ago
DOS' COMMAND.COM and later Windows' cmd.exe works the same way.

ncmncm 6y ago

Around 1978, while in high school, I spent many afternoons at the local university library skimming through six-inch-thick bound stacks of computer software journals. (That was how people communicated, before Internets.) I distinctly remember encountering exactly this paper, as my first exposure to Unix. It made a deep, deep impression. Pipe composition is just unaccountably powerful. What might not be so easy for people to understand today is how profoundly shocking it was to see computer commands and results expressed in lower case. I had never seen it done. Ever. It didn't seem possible, at first. After a few seconds, it was obvious, but still oddly liberating. You don't know computers are shouting at you until, suddenly, they aren't.