9 ms·

Beej's Guide to Unix IPC (2015)

220 points by qntty 5y ago

bool3max 5y ago
I personally dislike Beej's writing style. I tried to like his Network Programming guide but I found that most of what it conveys can be learned more quickly by simply reading the relevant POSIX/Linux manpages.
- beej71 5y ago
  It's tough to target all audiences, and as such the guide certainly isn't for everyone.
colbyhub 5y ago
I freaking love his guides, I'm currently reading through his C guide: https://beej.us/guide/bgc/ https://beej.us/guide/bgc/ It's very entertaining to read though which holds my attention well, and is well suited to people with existing programming experience. I've tried to read K&R but I keep coming back to Beej's.
- shric 5y ago
  I really want to like Beej, and I get that he wants to make things easy for people to understand, but his guides are full of unnecessary shortcuts which are actually wrong. For example, he doesn't seem to understand the difference between arrays and pointers -- at [1] he says: You can’t [get the length of an array] ish. C doesn’t record this information [footnote 50]. You have to manage it separately in another variable. [footnote 50] says "Since arrays are just pointers to the first element of the array under the hood, there’s no additional information recording the length" And then later he says: There is a trick to get the number of elements in an array in the scope in which an array is declared. But, generally speaking, this won’t work the way you want if you pass the array to a function [footnote 51] [footnote 51] says "Because when you pass an array to a function, you’re actually just passing a pointer to the first element of that array, not the “entire” array." Footnote 51 is correct, and contradicts footnote 50. There is a simple, and correct, way to explain this: Arrays and pointers are distinct types in C but arrays are converted to a pointer to their first element, except when it is the operand of the sizeof operator, the _Alignof operator, or the unary & operator, or is a string literal used to initialize an array. Yes, it's more wordy, but it's what's actually going on. Arrays aren't pointers, but he insists on claiming they are throughout. Footnote 51 shouldn't be a footnote, it should be prominent in the text. [1] https://beej.us/guide/bgc/html/split/arrays.html#fnref50 https://beej.us/guide/bgc/html/split/arrays.html#fnref50 P.S. Not going to go into it here, but his "unix" network programming guides used to be full of linux-specific things (without pointing this out), but I think it's improved over the last 10 or so years.
nathias 5y ago
Beej really has the best guides.
synergy20 5y ago
The IPC is in SystemV flavor but I think these days we normally should adopt Posix IPC instead especially for Linux/BSD, am I missing something? "On Linux and FreeBSD there is big advantage of posix queues, as handler given by mq_open are basically file descriptor which can be polled/epolled/selected/kqueued" "all POSIX IPC is thread-safe, while most SysV IPC is NOT"
- beej71 5y ago
  It hasn't been appreciably touched in decades. It could definitely use an update. And I need to port it to the new pandoc toolchain.
  - synergy20 5y ago
    Sounds great. It's hard to find nice tutorials for posix-ipc on the internet.
    - chasil 5y ago
      Fun fact: The more elaborate portions of System V IPC trace their origins to "Columbus UNIX" that was developed so a database could run on the platform. https://en.wikipedia.org/wiki/CB_UNIX https://en.wikipedia.org/wiki/CB_UNIX
  - pure_simplicity 5y ago
    If you use pandoc, have you considered epub as a target format? I am getting good results going from latex to epub using pandoc. Not sure what your sources and toolchain are like, though. Thanks for the great guides. :)
    - systemvoltage 5y ago
      Disagree. ePub (or any flowable medium) sucks when code blocks are involved. eBooks are great for novels and long form prose. Not so much for technical books. Why? Because inevitably the code is going to be either wrapped, too tiny or overflowing. Furthermore, syntax highlighting is a complete utter mess in ebooks. Reviews don’t lie. People don’t like it. And it’s obvious where ebooks shine and where they don’t. Personally: I want proper typesetting and constancy in the layout. But with eReaders, ebooks work pretty well and PDFs don’t. With iPads, PDFs are better IMO. But for technical books PDFs set by a professional typesetter like Springer is fantastic. Great typography and layout.
DataDaoDe 5y ago
Beej's guides are amazing with a memorable and entertaining writing style. I remember reading his network guide almost two decades ago when I was a kid, had just heard of this thing called a socket, but the only connotation I had for one at the time was a wall socket. His guide helped me understand networking but more importantly it made it a fun experience for a programmer just starting out.
- systemvoltage 5y ago
  I have to point this one out from his guide to C: https://beej.us/guide/bgc/html/split/hello-world.html https://beej.us/guide/bgc/html/split/hello-world.html > Before we go on, why would I even begin to bother pointing out that a pound sign is called an octothorpe? The answer is simple: I think the word octothorpe is so excellently funny, I have to gratuitously spread its name around whenever I get the opportunity. Octothorpe. Octothorpe, octothorpe, octothorpe. I can't find it right now but somewhere in the guide in a different chapter, he randomly inserts "Octothorpe"... good comedy. I think this kind of writing is exceedingly rare and immensely enjoyable.
- ironman1478 5y ago
  I remember in college in my networking class beej's guide was recommended by the professor and its stuck with me for 7 years. I've written a lot of socket based programs over the years and I still revisit it from time to time. Its the perfect guide. I also just point people to it whenever they ask about socket programming and basic networking because the guide explains it so much better than I ever could.
- jjice 5y ago
  I've literally laughed aloud reading his guides, I'm a huge fan. It's honestly one of the best beginning programming-related books in my opinion.
- enahs-sf 5y ago
  Been at it again with the amazing content. Love everything this guy does. He has taught me so much about network programming and C in a very accessible way.
- 5y ago
Const-me 5y ago
fork() is rarely ideal, unless followed by exec() to launch another processes. That copy-on-write thing often wastes too much physical memory. I think threads are usually better, the shared memory allows to marshal large volumes of data without making copies. I like poll() much better than signals. Signals are too limiting, and too arcane. It's much easier to use poll() and dispatch things manually based on the signaled handles, at least there's a guarantee your code won't be interrupted by another event. Pretty much all modern kernel things are compatible with poll. There's eventfd() which acts as an event or semaphore, message queues (but not the ones from the article, better ones, created with mq_open not msgget), Unix domain sockets for cross-process messaging. There's more, poll() can track termination of processes (see pidfd_open), page flip events in a GPU with DRM, consumed or available video/audio samples with V4L2 or ALSA.
- kragen 5y ago
  I partly agree, but a lot of this stuff is very context-dependent. Signals are interrupts, which are faster than an event loop but very bug-prone, especially in C (few C library functions are safe in interrupts), but shared-memory threads are too, for almost exactly the same reasons. If you want to share memory between processes, it's easy enough to use mmap(). Both threads and signals are somewhat tamer in Python, but lose some of their advantages on the way. Interrupts are really needed for three reasons: device driver latency and buffering (multithreaded CPUs probably being a superior alternative), killing infinite loops, and userspace virtual memory. Unix signals are useless for the first and arguably so bad at the other two that Unix would have been better off without them. Generally running multiple processes of the same program with different exec() calls uses more memory than forking it at startup, not less. It's true that threads use less still, and event loops less than that, when you don't have a multicore scaling problem. If you're looking at writing your own event loop, select() is more convenient than poll(), and io_uring is worth checking out. kqueue, iocp, and epoll are faster than poll(). Also though maybe just try libev, libevent, or libuv instead of writing your own.
  - IiydAbITMvJkqKf 5y ago
    NEVER use select(). It's like gets() - an API that should never be called in new code. poll is the standardized alternative (epoll/kqueue are more performant with large numbers of fds, but are os-dependent). The biggest problem with select() is that it will corrupt your stack if you use it with an fd with a value >= 1024. Since you can't easily control fd values, you must assume that calling select() is UB unless explicitly proven otherwise. There are also performance problems due to having to set up the sets before each call, but these are minor in comparison.
dang 5y ago
Past related threads: Beej's Guide to Unix IPC (2010) - https://news.ycombinator.com/item?id=9619375 https://news.ycombinator.com/item?id=9619375 - May 2015 (31 comments) Beejs Guide to Unix IPC - https://news.ycombinator.com/item?id=1525227 https://news.ycombinator.com/item?id=1525227 - July 2010 (19 comments)
_8j50 5y ago
Any reason to use these in new code instead of say.. dbus?
- trissylegs 5y ago
  dbus only covers a few use cases covered by all of these. Things like signals, file locking and mmap are still pretty important. Even if your using dbus. (Eg use dbus to share a file and mmap it)
- colonwqbang 5y ago
  Dbus is a good place to start. Raw sockets and shared memory are still necessary to get good enough performance in some cases. Dbus can be very slow! If something needs to happen several times per second, or big amounts of binary data should be transferred, dbus can become a bottleneck in embedded applications. I think dbus works great for service discovery and for low-frequency low-bandwidth communication. For other cases, people just exchange a file descriptor over dbus and proceed to talk directly to each other.
biohax2015 5y ago
Is this still up-to-date?
- beej71 5y ago
  Ish. In that APIs never die, yes. :) I really do need to update it since significant advances have been made since it was written nearly 25 (!!) years ago.
alexander1100 5y ago
Been really is one of the greatest unofficial mentors of the greater development community
- jwmcq 5y ago
  I don't know C very well, but what little I know is from Beej. I don't know network programming very well, but what little I know was learned from Beej. It's actually been quite rare in my professional programming career that I've had to use any of these things that I learned from Beej but... when I needed them.... I knew where to look and, knowing even just that, it's made me look like a wizard. Honestly, I can't see a 'donate' link here, but I really feel that I should.

220 points by qntty 5y ago

I personally dislike Beej's writing style. I tried to like his Network Programming guide but I found that most of what it conveys can be learned more quickly by simply reading the relevant POSIX/Linux manpages.

beej71 5y ago
It's tough to target all audiences, and as such the guide certainly isn't for everyone.

colbyhub 5y ago

I freaking love his guides, I'm currently reading through his C guide: https://beej.us/guide/bgc/ https://beej.us/guide/bgc/ It's very entertaining to read though which holds my attention well, and is well suited to people with existing programming experience. I've tried to read K&R but I keep coming back to Beej's.

shric 5y ago
I really want to like Beej, and I get that he wants to make things easy for people to understand, but his guides are full of unnecessary shortcuts which are actually wrong. For example, he doesn't seem to understand the difference between arrays and pointers -- at [1] he says: You can’t [get the length of an array] ish. C doesn’t record this information [footnote 50]. You have to manage it separately in another variable. [footnote 50] says "Since arrays are just pointers to the first element of the array under the hood, there’s no additional information recording the length" And then later he says: There is a trick to get the number of elements in an array in the scope in which an array is declared. But, generally speaking, this won’t work the way you want if you pass the array to a function [footnote 51] [footnote 51] says "Because when you pass an array to a function, you’re actually just passing a pointer to the first element of that array, not the “entire” array." Footnote 51 is correct, and contradicts footnote 50. There is a simple, and correct, way to explain this: Arrays and pointers are distinct types in C but arrays are converted to a pointer to their first element, except when it is the operand of the sizeof operator, the _Alignof operator, or the unary & operator, or is a string literal used to initialize an array. Yes, it's more wordy, but it's what's actually going on. Arrays aren't pointers, but he insists on claiming they are throughout. Footnote 51 shouldn't be a footnote, it should be prominent in the text. [1] https://beej.us/guide/bgc/html/split/arrays.html#fnref50 https://beej.us/guide/bgc/html/split/arrays.html#fnref50 P.S. Not going to go into it here, but his "unix" network programming guides used to be full of linux-specific things (without pointing this out), but I think it's improved over the last 10 or so years.

nathias 5y ago

Beej really has the best guides.

synergy20 5y ago

The IPC is in SystemV flavor but I think these days we normally should adopt Posix IPC instead especially for Linux/BSD, am I missing something? "On Linux and FreeBSD there is big advantage of posix queues, as handler given by mq_open are basically file descriptor which can be polled/epolled/selected/kqueued" "all POSIX IPC is thread-safe, while most SysV IPC is NOT"

beej71 5y ago
It hasn't been appreciably touched in decades. It could definitely use an update. And I need to port it to the new pandoc toolchain.
- synergy20 5y ago
  Sounds great. It's hard to find nice tutorials for posix-ipc on the internet.
  - chasil 5y ago
    Fun fact: The more elaborate portions of System V IPC trace their origins to "Columbus UNIX" that was developed so a database could run on the platform. https://en.wikipedia.org/wiki/CB_UNIX https://en.wikipedia.org/wiki/CB_UNIX
- pure_simplicity 5y ago
  If you use pandoc, have you considered epub as a target format? I am getting good results going from latex to epub using pandoc. Not sure what your sources and toolchain are like, though. Thanks for the great guides. :)
  - systemvoltage 5y ago
    Disagree. ePub (or any flowable medium) sucks when code blocks are involved. eBooks are great for novels and long form prose. Not so much for technical books. Why? Because inevitably the code is going to be either wrapped, too tiny or overflowing. Furthermore, syntax highlighting is a complete utter mess in ebooks. Reviews don’t lie. People don’t like it. And it’s obvious where ebooks shine and where they don’t. Personally: I want proper typesetting and constancy in the layout. But with eReaders, ebooks work pretty well and PDFs don’t. With iPads, PDFs are better IMO. But for technical books PDFs set by a professional typesetter like Springer is fantastic. Great typography and layout.

DataDaoDe 5y ago

Beej's guides are amazing with a memorable and entertaining writing style. I remember reading his network guide almost two decades ago when I was a kid, had just heard of this thing called a socket, but the only connotation I had for one at the time was a wall socket. His guide helped me understand networking but more importantly it made it a fun experience for a programmer just starting out.

systemvoltage 5y ago
I have to point this one out from his guide to C: https://beej.us/guide/bgc/html/split/hello-world.html https://beej.us/guide/bgc/html/split/hello-world.html > Before we go on, why would I even begin to bother pointing out that a pound sign is called an octothorpe? The answer is simple: I think the word octothorpe is so excellently funny, I have to gratuitously spread its name around whenever I get the opportunity. Octothorpe. Octothorpe, octothorpe, octothorpe. I can't find it right now but somewhere in the guide in a different chapter, he randomly inserts "Octothorpe"... good comedy. I think this kind of writing is exceedingly rare and immensely enjoyable.
ironman1478 5y ago
I remember in college in my networking class beej's guide was recommended by the professor and its stuck with me for 7 years. I've written a lot of socket based programs over the years and I still revisit it from time to time. Its the perfect guide. I also just point people to it whenever they ask about socket programming and basic networking because the guide explains it so much better than I ever could.
jjice 5y ago
I've literally laughed aloud reading his guides, I'm a huge fan. It's honestly one of the best beginning programming-related books in my opinion.
enahs-sf 5y ago
Been at it again with the amazing content. Love everything this guy does. He has taught me so much about network programming and C in a very accessible way.
5y ago

Const-me 5y ago

fork() is rarely ideal, unless followed by exec() to launch another processes. That copy-on-write thing often wastes too much physical memory. I think threads are usually better, the shared memory allows to marshal large volumes of data without making copies. I like poll() much better than signals. Signals are too limiting, and too arcane. It's much easier to use poll() and dispatch things manually based on the signaled handles, at least there's a guarantee your code won't be interrupted by another event. Pretty much all modern kernel things are compatible with poll. There's eventfd() which acts as an event or semaphore, message queues (but not the ones from the article, better ones, created with mq_open not msgget), Unix domain sockets for cross-process messaging. There's more, poll() can track termination of processes (see pidfd_open), page flip events in a GPU with DRM, consumed or available video/audio samples with V4L2 or ALSA.

kragen 5y ago
I partly agree, but a lot of this stuff is very context-dependent. Signals are interrupts, which are faster than an event loop but very bug-prone, especially in C (few C library functions are safe in interrupts), but shared-memory threads are too, for almost exactly the same reasons. If you want to share memory between processes, it's easy enough to use mmap(). Both threads and signals are somewhat tamer in Python, but lose some of their advantages on the way. Interrupts are really needed for three reasons: device driver latency and buffering (multithreaded CPUs probably being a superior alternative), killing infinite loops, and userspace virtual memory. Unix signals are useless for the first and arguably so bad at the other two that Unix would have been better off without them. Generally running multiple processes of the same program with different exec() calls uses more memory than forking it at startup, not less. It's true that threads use less still, and event loops less than that, when you don't have a multicore scaling problem. If you're looking at writing your own event loop, select() is more convenient than poll(), and io_uring is worth checking out. kqueue, iocp, and epoll are faster than poll(). Also though maybe just try libev, libevent, or libuv instead of writing your own.
- IiydAbITMvJkqKf 5y ago
  NEVER use select(). It's like gets() - an API that should never be called in new code. poll is the standardized alternative (epoll/kqueue are more performant with large numbers of fds, but are os-dependent). The biggest problem with select() is that it will corrupt your stack if you use it with an fd with a value >= 1024. Since you can't easily control fd values, you must assume that calling select() is UB unless explicitly proven otherwise. There are also performance problems due to having to set up the sets before each call, but these are minor in comparison.

dang 5y ago

Past related threads: Beej's Guide to Unix IPC (2010) - https://news.ycombinator.com/item?id=9619375 https://news.ycombinator.com/item?id=9619375 - May 2015 (31 comments) Beejs Guide to Unix IPC - https://news.ycombinator.com/item?id=1525227 https://news.ycombinator.com/item?id=1525227 - July 2010 (19 comments)

_8j50 5y ago

Any reason to use these in new code instead of say.. dbus?

trissylegs 5y ago
dbus only covers a few use cases covered by all of these. Things like signals, file locking and mmap are still pretty important. Even if your using dbus. (Eg use dbus to share a file and mmap it)
colonwqbang 5y ago
Dbus is a good place to start. Raw sockets and shared memory are still necessary to get good enough performance in some cases. Dbus can be very slow! If something needs to happen several times per second, or big amounts of binary data should be transferred, dbus can become a bottleneck in embedded applications. I think dbus works great for service discovery and for low-frequency low-bandwidth communication. For other cases, people just exchange a file descriptor over dbus and proceed to talk directly to each other.

biohax2015 5y ago

Is this still up-to-date?

beej71 5y ago
Ish. In that APIs never die, yes. :) I really do need to update it since significant advances have been made since it was written nearly 25 (!!) years ago.

alexander1100 5y ago

Been really is one of the greatest unofficial mentors of the greater development community

jwmcq 5y ago
I don't know C very well, but what little I know is from Beej. I don't know network programming very well, but what little I know was learned from Beej. It's actually been quite rare in my professional programming career that I've had to use any of these things that I learned from Beej but... when I needed them.... I knew where to look and, knowing even just that, it's made me look like a wizard. Honestly, I can't see a 'donate' link here, but I really feel that I should.