4 ms·

> If you want to share memory between processes, it's easy enough to use mmap() It's only easy when all of the following is true: (1) sharing one time only (2)

> If you want to share memory between processes, it's easy enough to use mmap() It's only easy when all of the following is true: (1) sharing one time only (2) Sharing exactly one continuous buffer, as opposed to anything more complicated with pointers inside (3) Sharing between processes one-to-one. When any of these things is false, mmap becomes hard. For instance, despite C++ std::vector is a continuous buffer, when sharing a mutable one that grow/reallocate thing gonna cause issues. > select() is more convenient than poll() Strongly disagree. These macros and the bitmap is pure BS. For some handle types like sockets, for hardware reasons (multicore CPUs) Linux no longer guarantees sequential file descriptors. Even the manual page says "WARNING: select() can monitor only file descriptors numbers that are less than FD_SETSIZE (1024) - an unreasonably low limit for many modern applications - and this limitation will not change. All modern applications should instead use poll(2) or epoll(7)" poll() is awesome in comparison, the API with that array of simple structures is perfect for the job. At least in terms of usability, AFAIK epoll does better performance-wise. > io_uring is worth checking out Right, but the use case is different, it's only for I/O. > kqueue, iocp Not available on Linux? > epoll Yeah, but I think to truly benefit from this one, need multithreading. Multithreaded async I/O is hard, I think people only doing that when they absolutely have to for performance reasons. > just try libev, libevent, or libuv instead of writing your own My use cases are probably different. I rarely using poll() for network sockets, it's usually other things: process and threads synchronization, timers with timerfd(), hardware devices (GPU, sound card, hardware video codecs).

Hello71 5y ago
> For some handle types like sockets, for hardware reasons (multicore CPUs) Linux no longer guarantees sequential file descriptors. huh? this would be a severe POSIX violation, which says that for socket and accept, "The file descriptor shall be allocated as described in File Descriptor Allocation.": "All functions that open one or more file descriptors shall, unless specified otherwise, atomically allocate the lowest numbered available (that is, not already open in the calling process) file descriptor at the time of each allocation."
kragen 5y ago
You can have the same file mmap()ed at the same address in two separate processes, so putting pointers in it is a valid thing to do. In most languages doing dynamic allocation in shared memory is a world of pain, but maybe the STL allocator approach makes it a reasonable thing to do in C++? Sharing a complex linked data structure between different threads is a somewhat tricky thing to do in any case, though in some cases it may be the only reasonable solution. In many cases something like FlatBuffers may be a better solution. I'm not sure what you mean by "sharing one time only" or "sharing between processes one-to-one"; can you elaborate? It isn't really true that select() can only monitor file descriptor numbers less than FD_SETSIZE; you just have to allocate the bitmaps differently instead of using the fd_set type. Libevent does this, for example. io_uring isn't really only for I/O; you can use it for madvise(), poll(), and timers as well. Normally I'd be comfortable with saying that's "I/O" but in this context poll() and timers are the things you're implicitly opposing to "I/O". Yes, kqueue and iocp are not available on Linux. But io_uring and epoll aren't available on other operating systems, so I thought it was worthwhile mentioning the corresponding facilities there. Why do you say you need multithreading to truly benefit from epoll? If you have an I/O-bound process that has a lot of file descriptors open, running in an event loop, epoll is a faster alternative to select() (or, as you say, poll()), isn't it? This thing about nonsequential socket file descriptors is news to me; I hadn't heard anything about it, though I admit I haven't been paying attention. Where can I find out more?
- Const-me 5y ago
  > I'm not sure what you mean by "sharing one time only" or "sharing between processes one-to-one" By one time only, I meant that you only want to pass the data once, as opposed to sharing the data and being able to update it on one side and see updates on the other one. It’s not impossible, but practically hard to implement growable/shrinkable shared memory areas, the communicating processes need to unmap and remap the virtual memory regions. It’s similar with more than 2 processes sharing the memory. Technically doable but hard in practice, for instance it’s easy to leak these memfd handles. None of these issues are present for threads sharing the address space. > you just have to allocate the bitmaps differently instead of using the fd_set type When really unlucky, with these bitmaps one gonna need to waste kilobytes of memory to represent a set with only a couple of handles. The RAM itself is usually cheap at that scale, the problem is the performance overhead of searching for the set bits. > epoll is a faster alternative to select() (or, as you say, poll()), isn't it? Right, but for IO bound things, epoll() + multithreading is even faster. Modern network cards are aware of multi-core CPUs on the other side of the PCIe bus. Faster cards like 10 Gbit/second have multiple transmit/receive queues designed to interact with different CPU cores. OTOH, when things aren’t that I/O bound, the single dispatcher thread is adequate, and the API overhead of poll() is reasonable too, given the usability win. > Where can I find out more? Yeah, you’re right about that, accept() never does that, according to specs. However, note the F_DUPFD parameter of fcntl: duplicate the file descriptor using the lowest-numbered available file descriptor greater than or equal to arg. This alone means that for libraries or long-lived projects, relying on file handles to be small numbers is fundamentally unreliable.

Hello71 5y ago

> For some handle types like sockets, for hardware reasons (multicore CPUs) Linux no longer guarantees sequential file descriptors. huh? this would be a severe POSIX violation, which says that for socket and accept, "The file descriptor shall be allocated as described in File Descriptor Allocation.": "All functions that open one or more file descriptors shall, unless specified otherwise, atomically allocate the lowest numbered available (that is, not already open in the calling process) file descriptor at the time of each allocation."

kragen 5y ago

You can have the same file mmap()ed at the same address in two separate processes, so putting pointers in it is a valid thing to do. In most languages doing dynamic allocation in shared memory is a world of pain, but maybe the STL allocator approach makes it a reasonable thing to do in C++? Sharing a complex linked data structure between different threads is a somewhat tricky thing to do in any case, though in some cases it may be the only reasonable solution. In many cases something like FlatBuffers may be a better solution. I'm not sure what you mean by "sharing one time only" or "sharing between processes one-to-one"; can you elaborate? It isn't really true that select() can only monitor file descriptor numbers less than FD_SETSIZE; you just have to allocate the bitmaps differently instead of using the fd_set type. Libevent does this, for example. io_uring isn't really only for I/O; you can use it for madvise(), poll(), and timers as well. Normally I'd be comfortable with saying that's "I/O" but in this context poll() and timers are the things you're implicitly opposing to "I/O". Yes, kqueue and iocp are not available on Linux. But io_uring and epoll aren't available on other operating systems, so I thought it was worthwhile mentioning the corresponding facilities there. Why do you say you need multithreading to truly benefit from epoll? If you have an I/O-bound process that has a lot of file descriptors open, running in an event loop, epoll is a faster alternative to select() (or, as you say, poll()), isn't it? This thing about nonsequential socket file descriptors is news to me; I hadn't heard anything about it, though I admit I haven't been paying attention. Where can I find out more?

Const-me 5y ago
> I'm not sure what you mean by "sharing one time only" or "sharing between processes one-to-one" By one time only, I meant that you only want to pass the data once, as opposed to sharing the data and being able to update it on one side and see updates on the other one. It’s not impossible, but practically hard to implement growable/shrinkable shared memory areas, the communicating processes need to unmap and remap the virtual memory regions. It’s similar with more than 2 processes sharing the memory. Technically doable but hard in practice, for instance it’s easy to leak these memfd handles. None of these issues are present for threads sharing the address space. > you just have to allocate the bitmaps differently instead of using the fd_set type When really unlucky, with these bitmaps one gonna need to waste kilobytes of memory to represent a set with only a couple of handles. The RAM itself is usually cheap at that scale, the problem is the performance overhead of searching for the set bits. > epoll is a faster alternative to select() (or, as you say, poll()), isn't it? Right, but for IO bound things, epoll() + multithreading is even faster. Modern network cards are aware of multi-core CPUs on the other side of the PCIe bus. Faster cards like 10 Gbit/second have multiple transmit/receive queues designed to interact with different CPU cores. OTOH, when things aren’t that I/O bound, the single dispatcher thread is adequate, and the API overhead of poll() is reasonable too, given the usability win. > Where can I find out more? Yeah, you’re right about that, accept() never does that, according to specs. However, note the F_DUPFD parameter of fcntl: duplicate the file descriptor using the lowest-numbered available file descriptor greater than or equal to arg. This alone means that for libraries or long-lived projects, relying on file handles to be small numbers is fundamentally unreliable.