9 ms·

Brian Kernighan gives credit to Doug McIlroy for coming up with the idea in 1964. Quoting from _UNIX A History and a Memoir_, page 68: "To put my strongest co

Brian Kernighan gives credit to Doug McIlroy for coming up with the idea in 1964. Quoting from _UNIX A History and a Memoir_, page 68: "To put my strongest concerns in a nutshell: 1. we should have some way of coupling programs like garden hose..." from a Bell Labs internal memo. Kernighan's book is a great read, BTW. Highly recommend to all UNIX nerds.

all2 7y ago
Does anyone know how the pipe idea compares to Erlang messaging? As far as I can tell the implementation of each might look like the other in some ways (a queue of some kind with MutEx guaranteed somehow). Does anyone have the knowledge to write a little on this? Or maybe point to a similar article for that style of messaging?
- Matthias247 7y ago
  Pipes are bytes Streams. Communication between actors as well as CSP channels are object streams. Pipes exercise backpressure on the writer - if the pipe is full the writer is blocked. Actor systems mostly use seemingly unbounded queues. The sender will not get blocked.
  - toast0 7y ago
    > Pipes exercise backpressure on the writer - if the pipe is full the writer is blocked. Actor systems mostly use seemingly unbounded queues. The sender will not get blocked. Erlang does suspend (block) processes that send to ports or nodes when the buffers for that get full; but not when sending to local processes. There used to be an optional reduction count punishment for senders when sending to messages with larger mailboxes, but it seems that may have been removed. I don't think it would be too hard to add a feature where sending to a local mailbox over a specified size caused the sender to be suspended, but tracking it might be a little difficult.
- wahern 7y ago
  Here's a short 2014 paper from McIlroy himself discussing his early thinking back then: https://www.cs.dartmouth.edu/~doug/sieve/sieve.pdf https://www.cs.dartmouth.edu/~doug/sieve/sieve.pdf TL;DR: McIlroy was applying the concept of coroutines, which was described by Melvin Conway in 1963. Two processes communicating over a pipe are basically coroutines, except instead of passing structured data they're just passing bytes. A Unix pipeline is a set of multiple coroutines. In fact, Tony Hoare's 1978 Communicating Sequential Processes paper cites the UNIX shell[1] for the concept of coroutines, and discussion of coroutines figures prominently in that paper. See https://www.cs.cmu.edu/~crary/819-f09/Hoare78.pdf https://www.cs.cmu.edu/~crary/819-f09/Hoare78.pdf CSP basically models the behavior of a large set of coroutines. AFAIU, Erlang was partly inspired by CSP. You can draw a straight line from coroutines, through Unix pipelines and CSP, to Erlang's processes. [1] Specifically, Ken Thompson's 1976 paper, "The UNIX command language." See https://archive.org/details/theunixcommandlanguage https://archive.org/details/theunixcommandlanguage
  - TheGrassyKnoll 7y ago
    "...a practical exploration of using Python coroutines..." http://www.dabeaz.com/coroutines/index.html
jjice 7y ago
+1 for Kernighan's book. I'm not one for impulse purchases, but as when I saw a link on HN, I immediately bought it and read through it. I lent it to a professor of mine and he's using as a text for his history course next semester.
leoc 7y ago
Here's a nice McIlroy interview with broken formatting: https://www.princeton.edu/~hos/mike/transcripts/mcilroy.htm https://www.princeton.edu/~hos/mike/transcripts/mcilroy.htm .
yesenadam 7y ago
A scan of the original 1964 memo: https://www.oreilly.com/library/view/making-sense-of/9781492042563/assets/msos_0405.png https://www.oreilly.com/library/view/making-sense-of/9781492...
- amenonsen 7y ago
  Thank you very much for digging this up, and to all the other commenters here for the useful references.
  - tlamponi 7y ago
    FYI: it's literally in the linked article from this HN post, maybe start reading that, not just the headline and then commenting...
    - amenonsen 7y ago
      FYI: I'm the author of the linked article, and I added a link to the image after it was posted here.
      - combatentropy 7y ago
        This has got to be one of the funniest exchanges I have ever seen on Hacker News.
      - tlamponi 7y ago
        Thanks for adding context, that wasn't clear at all to me, it seemed like the classical "Reads headline and then some comments, and post 'smart' stuff which the original article read anyway" response to me. Sorry for the "accusation" then.
- aasasd 7y ago
  Now I'm confused and curious as to what the ‘library filing scheme’ means. Does it mean shared libraries? IIRC Unix stuff was statically-linked at first. And then again, what are ‘indexing’ and ‘data path switching’?

all2 7y ago

Does anyone know how the pipe idea compares to Erlang messaging? As far as I can tell the implementation of each might look like the other in some ways (a queue of some kind with MutEx guaranteed somehow). Does anyone have the knowledge to write a little on this? Or maybe point to a similar article for that style of messaging?

Matthias247 7y ago
Pipes are bytes Streams. Communication between actors as well as CSP channels are object streams. Pipes exercise backpressure on the writer - if the pipe is full the writer is blocked. Actor systems mostly use seemingly unbounded queues. The sender will not get blocked.
- toast0 7y ago
  > Pipes exercise backpressure on the writer - if the pipe is full the writer is blocked. Actor systems mostly use seemingly unbounded queues. The sender will not get blocked. Erlang does suspend (block) processes that send to ports or nodes when the buffers for that get full; but not when sending to local processes. There used to be an optional reduction count punishment for senders when sending to messages with larger mailboxes, but it seems that may have been removed. I don't think it would be too hard to add a feature where sending to a local mailbox over a specified size caused the sender to be suspended, but tracking it might be a little difficult.
wahern 7y ago
Here's a short 2014 paper from McIlroy himself discussing his early thinking back then: https://www.cs.dartmouth.edu/~doug/sieve/sieve.pdf https://www.cs.dartmouth.edu/~doug/sieve/sieve.pdf TL;DR: McIlroy was applying the concept of coroutines, which was described by Melvin Conway in 1963. Two processes communicating over a pipe are basically coroutines, except instead of passing structured data they're just passing bytes. A Unix pipeline is a set of multiple coroutines. In fact, Tony Hoare's 1978 Communicating Sequential Processes paper cites the UNIX shell[1] for the concept of coroutines, and discussion of coroutines figures prominently in that paper. See https://www.cs.cmu.edu/~crary/819-f09/Hoare78.pdf https://www.cs.cmu.edu/~crary/819-f09/Hoare78.pdf CSP basically models the behavior of a large set of coroutines. AFAIU, Erlang was partly inspired by CSP. You can draw a straight line from coroutines, through Unix pipelines and CSP, to Erlang's processes. [1] Specifically, Ken Thompson's 1976 paper, "The UNIX command language." See https://archive.org/details/theunixcommandlanguage https://archive.org/details/theunixcommandlanguage
- TheGrassyKnoll 7y ago
  "...a practical exploration of using Python coroutines..." http://www.dabeaz.com/coroutines/index.html

jjice 7y ago

+1 for Kernighan's book. I'm not one for impulse purchases, but as when I saw a link on HN, I immediately bought it and read through it. I lent it to a professor of mine and he's using as a text for his history course next semester.

leoc 7y ago

Here's a nice McIlroy interview with broken formatting: https://www.princeton.edu/~hos/mike/transcripts/mcilroy.htm https://www.princeton.edu/~hos/mike/transcripts/mcilroy.htm .

yesenadam 7y ago

A scan of the original 1964 memo: https://www.oreilly.com/library/view/making-sense-of/9781492042563/assets/msos_0405.png https://www.oreilly.com/library/view/making-sense-of/9781492...

amenonsen 7y ago
Thank you very much for digging this up, and to all the other commenters here for the useful references.
- tlamponi 7y ago
  FYI: it's literally in the linked article from this HN post, maybe start reading that, not just the headline and then commenting...
  - amenonsen 7y ago
    FYI: I'm the author of the linked article, and I added a link to the image after it was posted here.
    - combatentropy 7y ago
      This has got to be one of the funniest exchanges I have ever seen on Hacker News.
    - tlamponi 7y ago
      Thanks for adding context, that wasn't clear at all to me, it seemed like the classical "Reads headline and then some comments, and post 'smart' stuff which the original article read anyway" response to me. Sorry for the "accusation" then.
aasasd 7y ago
Now I'm confused and curious as to what the ‘library filing scheme’ means. Does it mean shared libraries? IIRC Unix stuff was statically-linked at first. And then again, what are ‘indexing’ and ‘data path switching’?