5 ms·

Thanks for the reply. My use case is very simple: a stateless "something happened" notification, which can be delivered asynchronously. Coalescing or even occas

Thanks for the reply. My use case is very simple: a stateless "something happened" notification, which can be delivered asynchronously. Coalescing or even occasional drops are fine. I did originally use a Unix domain socket server, but that added a lot of complexity: one has to arrange for it to be launched, guard against the possibility that it gets stuck, version it, deal with permissions, etc. My new solution on Linux is a total hack: there's a FIFO, and to post a notification, you write to it. Clients see that the FIFO became readable, and that change represents the notification. The sender then drains the data it wrote, so that the FIFO becomes unreadable again. This is a total abuse of FIFOs, but it's proven to be much simpler than trying to manage a separate server. I've never heard of TIPC. From a little searching it looks like it's very capable but geared towards clusters, and is overkill for my use case. What do you think?

bdash 11y ago
We had a similar problem to solve for iOS and OS X and settled on the using FIFOs in that manner as well. A colleague wrote up a blog post about the various alternatives that were evaluated before settling on that approach: https://realm.io/news/thomas-goyne-fast-inter-process-communication/ https://realm.io/news/thomas-goyne-fast-inter-process-commun...
- ridiculous_fish 11y ago
  Ha ha ha, that's great! If we are sent to programmer purgatory, at least we'll have each other.
pdkl95 11y ago
If drops are fine, TIPC is probably overkill. I would probably just wrap something generic using UNIX domain sockets up into a library and re-use that as needed. Depending on your permissions requirements[1], and if you really only need a signaling flag, have you considered the filesystem? Just touch a file in a well-defined directory named after the event that happened, and poll it periodically. Removing the file clears the flag. Signals can coalesce, but you should never drop any. You can poll a directory (that will normally be empty) without much CPU load (the directory inode will be cached most of the time). You could setup a multiple listeners by giving them their own "inbox" directory, like. e.g.: # send a notification touch "${HOME}/.${app_name}rc/messages/${destination}/${signal_flag_name}" Using the filesystem opens up the possibility of a message sender being anything that can generate - even indirectly - an open(O_CREAT). Your signals also persist across programs shutdowns and crashes - you can send and receive even when the other side isn't running - and your state can persist across reboots. Also, you can leverage some of the guarantees provided by the kenel's vfs layer. For example, rename(2) is atomic, so you can send small data payloads by writing to a different name first. FLAG_PATH="${HOME}/.${app_name}rc/messages/${destination}/${signal_flag_name}" # using the PID ($$) to not collide with other message senders TEMP_PATH="${FLAG_PATH}-new-$$" echo -e "foo=bar\nbaz=quux\ncount=42" > "${TEMP_PATH}" mv "${TEMP_PATH}" "${FLAG_PATH}" As an optional linux-specific feature, you can extend that technique to be event-driven (no polling loop) by telling the kernel to notify you about file-create events by listening to the directory (NOT the file) with inotify(7) for the IN_CREATE messages. Those events can be received either in a simple blocking style by letting poll(2) wake-up your process. Alternatively, you can receive events in in a non-blocking style with poll(2) if you give inotify_init1(2) the IN_NONBLOCK flag. The man oage inotify(7) should have an example. [1] this can get annoyingly complicated - but certainly not impossible - if you have to care about user/group permissions, esp. on the directory. Making a group specific to the message sending can help.
- ridiculous_fish 11y ago
  Thank you for this thoughtful reply. There's a variety of options if I'm willing to poll, including shared memory or the filesystem idea you outline, but I hope to avoid polling for hygienic reasons. I also explored inotify but found it to be unreliable (https://github.com/travis-ci/travis-ci/issues/2342 https://github.com/travis-ci/travis-ci/issues/2342).
  - pdkl95 11y ago
    That's why I like inotify in blocking mode - the call to poll() is just to wake up the process (I think you could just blocking-read the inotify file handle? I haven't tried it directly). The point of using inotify is that you don't need to poll, because the kernel send your process reliable events instead over a file handle. The use of poll(2) is just a consequence of the interface using a file handle. As, I originally said, though, there is certainly no one-size-fits-all solution, these are just a few of the available options, which may not be apropriate for your situation.
    - ridiculous_fish 11y ago
      I like blocking inotify in principle - the problem is that it just didn't work! I think there is a gap in the Linux APIs in this area. Its multicast IPC mechanisms are just too heavyweight.
teddyh 11y ago
> there's a FIFO, and to post a notification, you write to it. Clients see that the FIFO became readable, and that change represents the notification. The sender then drains the data it wrote, so that the FIFO becomes unreadable again. Beware: I tried that once, and it was unreliable. Only some clients woke up.

bdash 11y ago

We had a similar problem to solve for iOS and OS X and settled on the using FIFOs in that manner as well. A colleague wrote up a blog post about the various alternatives that were evaluated before settling on that approach: https://realm.io/news/thomas-goyne-fast-inter-process-communication/ https://realm.io/news/thomas-goyne-fast-inter-process-commun...

ridiculous_fish 11y ago
Ha ha ha, that's great! If we are sent to programmer purgatory, at least we'll have each other.

pdkl95 11y ago

If drops are fine, TIPC is probably overkill. I would probably just wrap something generic using UNIX domain sockets up into a library and re-use that as needed. Depending on your permissions requirements[1], and if you really only need a signaling flag, have you considered the filesystem? Just touch a file in a well-defined directory named after the event that happened, and poll it periodically. Removing the file clears the flag. Signals can coalesce, but you should never drop any. You can poll a directory (that will normally be empty) without much CPU load (the directory inode will be cached most of the time). You could setup a multiple listeners by giving them their own "inbox" directory, like. e.g.: # send a notification touch "${HOME}/.${app_name}rc/messages/${destination}/${signal_flag_name}" Using the filesystem opens up the possibility of a message sender being anything that can generate - even indirectly - an open(O_CREAT). Your signals also persist across programs shutdowns and crashes - you can send and receive even when the other side isn't running - and your state can persist across reboots. Also, you can leverage some of the guarantees provided by the kenel's vfs layer. For example, rename(2) is atomic, so you can send small data payloads by writing to a different name first. FLAG_PATH="${HOME}/.${app_name}rc/messages/${destination}/${signal_flag_name}" # using the PID ($$) to not collide with other message senders TEMP_PATH="${FLAG_PATH}-new-$$" echo -e "foo=bar\nbaz=quux\ncount=42" > "${TEMP_PATH}" mv "${TEMP_PATH}" "${FLAG_PATH}" As an optional linux-specific feature, you can extend that technique to be event-driven (no polling loop) by telling the kernel to notify you about file-create events by listening to the directory (NOT the file) with inotify(7) for the IN_CREATE messages. Those events can be received either in a simple blocking style by letting poll(2) wake-up your process. Alternatively, you can receive events in in a non-blocking style with poll(2) if you give inotify_init1(2) the IN_NONBLOCK flag. The man oage inotify(7) should have an example. [1] this can get annoyingly complicated - but certainly not impossible - if you have to care about user/group permissions, esp. on the directory. Making a group specific to the message sending can help.

ridiculous_fish 11y ago
Thank you for this thoughtful reply. There's a variety of options if I'm willing to poll, including shared memory or the filesystem idea you outline, but I hope to avoid polling for hygienic reasons. I also explored inotify but found it to be unreliable (https://github.com/travis-ci/travis-ci/issues/2342 https://github.com/travis-ci/travis-ci/issues/2342).
- pdkl95 11y ago
  That's why I like inotify in blocking mode - the call to poll() is just to wake up the process (I think you could just blocking-read the inotify file handle? I haven't tried it directly). The point of using inotify is that you don't need to poll, because the kernel send your process reliable events instead over a file handle. The use of poll(2) is just a consequence of the interface using a file handle. As, I originally said, though, there is certainly no one-size-fits-all solution, these are just a few of the available options, which may not be apropriate for your situation.
  - ridiculous_fish 11y ago
    I like blocking inotify in principle - the problem is that it just didn't work! I think there is a gap in the Linux APIs in this area. Its multicast IPC mechanisms are just too heavyweight.

teddyh 11y ago

> there's a FIFO, and to post a notification, you write to it. Clients see that the FIFO became readable, and that change represents the notification. The sender then drains the data it wrote, so that the FIFO becomes unreadable again. Beware: I tried that once, and it was unreliable. Only some clients woke up.