4 ms·

Good article. However, it would be interesting to compare these two things against better system calls. > the data referred to by these pointers must be copied

Good article. However, it would be interesting to compare these two things against better system calls. > the data referred to by these pointers must be copied into the kernel Modern OS kernels avoid that when they can. The article is about Linux, modern ones have io_uring: https://unixism.net/loti/what_is_io_uring.html https://unixism.net/loti/what_is_io_uring.html In short, that thing allows to setup these pointers in advance, and make the memory visible to both user-mode code and OS kernel at the same time. This way no need to copy the data. Unfortunately, implementing that thing is way more complicated than calling read() or setting up memory mapping. For optimal performance you gonna need to zero-copy all the way up to whatever code is consuming the data read from these files.

tinus_hn 6y ago
How do they avoid race conditions if the kernel can use these pointers while userspace can modify them at the same time?
- Const-me 6y ago
  The common pattern with any I/O, race conditions are avoided by writing in the documentation "please don't modify the memory after the I/O is initiated and before it's completed". Enforcing that would cause too much overhead.
  - tinus_hn 6y ago
    Wouldn’t that violate the rule that userspace can’t crash the kernel? On Windows this is solved by using the memory protection provided by the cpu and Structured Exception Handling; I am curious if Linux has something similar.
    - labawi 6y ago
      The kernel likely keeps it's own copies of necessary validated commands and pointers, but not of data buffers. If you read/write in an inappropriate time, you are likely to read/write garbage, get misinterpreted commands or segfaults, but that's your problem. It shouldn't affect the kernel, aside from bugs present.
      - Const-me 6y ago
        Yes. And in case of io_uring, the pointers to the buffers are provided well in advance, with IORING_REGISTER_BUFFERS opcode of io_uring_register API, to be reused across I/O system calls. AFAIK, Windows only has an equivalent for sockets, called Registered Input/Output (RIO). However, RIO was introduced in Windows 8 (2012) while io_uring is much newer, introduced in Linux kernel 5.1 (2019).
    - Const-me 6y ago
      > Wouldn’t that violate the rule that userspace can’t crash the kernel? A program that accesses same memory location from different cores at the same time won’t crash, as long as the address is good. You’ll just get random garbage in your memory. Concurrent updates by CPU and DMA devices is equally fine, again garbage out is the only result you’ll get. If you corrupt things like that due to a concurrency bug in your userspace code, you may get garbage in your memory, garbage in your files, but I don’t see why would it crash the kernel? > On Windows this is solved by using the memory protection provided by the cpu and Structured Exception Handling Switching memory protection is relatively expensive. Also memory protection doesn’t have byte granularity: you can allocate a 4kb buffer (a single page), view it as a circular buffer with four 1kb-long pieces, and use that circular buffer for streamed I/O. It’s the same on Windows. Check the docs for WriteFileEx, it says “This buffer must remain valid for the duration of the write operation. The caller must not use this buffer until the write operation is completed.” but I don’t think you gonna get any SEH if you ignore what’s written there, and use the buffer in the meantime.

tinus_hn 6y ago

How do they avoid race conditions if the kernel can use these pointers while userspace can modify them at the same time?

Const-me 6y ago
The common pattern with any I/O, race conditions are avoided by writing in the documentation "please don't modify the memory after the I/O is initiated and before it's completed". Enforcing that would cause too much overhead.
- tinus_hn 6y ago
  Wouldn’t that violate the rule that userspace can’t crash the kernel? On Windows this is solved by using the memory protection provided by the cpu and Structured Exception Handling; I am curious if Linux has something similar.
  - labawi 6y ago
    The kernel likely keeps it's own copies of necessary validated commands and pointers, but not of data buffers. If you read/write in an inappropriate time, you are likely to read/write garbage, get misinterpreted commands or segfaults, but that's your problem. It shouldn't affect the kernel, aside from bugs present.
    - Const-me 6y ago
      Yes. And in case of io_uring, the pointers to the buffers are provided well in advance, with IORING_REGISTER_BUFFERS opcode of io_uring_register API, to be reused across I/O system calls. AFAIK, Windows only has an equivalent for sockets, called Registered Input/Output (RIO). However, RIO was introduced in Windows 8 (2012) while io_uring is much newer, introduced in Linux kernel 5.1 (2019).
  - Const-me 6y ago
    > Wouldn’t that violate the rule that userspace can’t crash the kernel? A program that accesses same memory location from different cores at the same time won’t crash, as long as the address is good. You’ll just get random garbage in your memory. Concurrent updates by CPU and DMA devices is equally fine, again garbage out is the only result you’ll get. If you corrupt things like that due to a concurrency bug in your userspace code, you may get garbage in your memory, garbage in your files, but I don’t see why would it crash the kernel? > On Windows this is solved by using the memory protection provided by the cpu and Structured Exception Handling Switching memory protection is relatively expensive. Also memory protection doesn’t have byte granularity: you can allocate a 4kb buffer (a single page), view it as a circular buffer with four 1kb-long pieces, and use that circular buffer for streamed I/O. It’s the same on Windows. Check the docs for WriteFileEx, it says “This buffer must remain valid for the duration of the write operation. The caller must not use this buffer until the write operation is completed.” but I don’t think you gonna get any SEH if you ignore what’s written there, and use the buffer in the meantime.