4 ms·

This is a well-written article with excellent explanations and I thoroughly enjoyed it. However, none of the variants using vmsplice (i.e., all but the slowest

This is a well-written article with excellent explanations and I thoroughly enjoyed it. However, none of the variants using vmsplice (i.e., all but the slowest) are safe. When you gift [1] pages to the kernel there is no reliable general purpose way to know when the pages are safe to reuse again. This post (and the earlier FizzBuzz variant) try to get around this by assuming the pages are available again after "pipe size" bytes have been written after the gift, _but this is not true in general_. For example, the read side may also use splice-like calls to move the pages to another pipe or IO queue in zero-copy way so the lifetime of the page can extend beyond the original pipe. This will show up as race conditions and spontaneously changing data where a downstream consumer sees the page suddenly change as it it overwritten by the original process. The author of these splice methods, Jens Axboe, had proposed a mechanism which enabled you to determine when it was safe to reuse the page, but as far as I know nothing was ever merged. So the scenarios where you can use this are limited to those where you control both ends of the pipe and can be sure of the exact page lifetime. --- [1] Specifically, using SPLICE_F_GIFT.

rostayob 4y ago
(I am the author of the post) I haven't digested this comment fully yet, but just to be clear, I am _not_ using SPLICE_F_GIFT (and I don't think the fizzbuzz program is either). However I think what you're saying makes sense in general, SPLICE_F_GIFT or not. Are you sure this unsafety depends on SPLICE_F_GIFT? Also, do you have a reference to the discussions regarding this (presumably on LKML)?
- BeeOnRope 4y ago
  Yeah my mention of gift was a red herring: I had assumed gift was being used but the same general problem (the "page garbage collection issue") crops up regardless. If you don't use gift, you never know when the pages are free to use again, so in principle you need to keep writing to new buffers indefinitely. One "solution" to this problem is to gift the pages, in which case the kernel does the GC for you, but you need to churn through new pages constantly because you've gifted the old ones. Gift is especially useful when the page gifted can be used directly in the page cache (i.e., writing a file, not a pipe). Without gift some consumption patterns may be safe but I think they are exactly those which involve a copy (not using gift means that a copy will occur for additional read-side scenarios). Ultimately the problem is that if some downstream process is able to get a zero-copy view of a page from an upstream writer, how can this be safe to concurrently modification? The pipe size trick is one way it could work, but it doesn't pan out because the pages may live beyond the immediately pipe (this is actually alluded in the FizzBuzz article where they mentioned things blew up if more than one pipe was involved).
  - rostayob 4y ago
    Yes, this all makes sense, although like everything splicing-related, it is very subtle. Maybe I should have mentioned the subtleness and dangerousness of splicing at the beginning, rather than at the end. I still think the man page of vmsplice is quite misleading! Specifically: SPLICE_F_GIFT The user pages are a gift to the kernel. The application may not modify this memory ever, otherwise the page cache and on-disk data may differ. Gifting pages to the kernel means that a subsequent splice(2) SPLICE_F_MOVE can successfully move the pages; if this flag is not speci‐ fied, then a subsequent splice(2) SPLICE_F_MOVE must copy the pages. Data must also be properly page aligned, both in memory and length. To me, this indicates that if we're _not_ using SPLICE_F_GIFT downstream splices will be automatically taken care of, safety-wise.
    - scottlamb 4y ago
      Hmm, reading this side-by-side with a paragraph from BeeOnRope's comment: > This post (and the earlier FizzBuzz variant) try to get around this by assuming the pages are available again after "pipe size" bytes have been written after the gift, _but this is not true in general_. For example, the read side may also use splice-like calls to move the pages to another pipe or IO queue in zero-copy way so the lifetime of the page can extend beyond the original pipe. The paragraph you quoted says that the "splice-like calls to move the pages" actually copy when SPLICE_F_GIFT is not specified. So perhaps the combination of not using SPLICE_F_GIFT and waiting until "pipe size" bytes have been written is safe.
      - BeeOnRope 4y ago
        Yes it is not clear to me when the copy actually happens but I had assumed the > 30 GB/s result after read was changed to use splice must imply zero copy.
        rostayob 4y ago
        It could be that when splicing to /dev/null (which I'm doing), the kernel knows that they their content is never witnessed, and therefore no copy is required. But I haven't verified that
- scottlamb 4y ago
  I think you're right that the same problem applies without SPLICE_F_GIFT. One of the other fizzbuzz code golfers discusses that here: https://codegolf.stackexchange.com/a/239848 https://codegolf.stackexchange.com/a/239848 I wonder if io_uring handles this (yet). io_uring is a newer async IO mechanism by the same author which tells you when your IOs have completed. So you might think it would: * But from a quick look, I think its vmsplice equivalent operation just tells you when the syscall would have returned, so maybe not. [edit: actually, looks like there's not even an IORING_OP_VMSPLICE operation in the latest mainline tree yet, just drafts on lkml. Maybe if/when the vmsplice op is added, it will wait to return for the right time.] * And in this case (no other syscalls or work to perform while waiting) I don't see any advantage in io_uring's read/write operations over just plain synchronous read/write.
  - Matthias247 4y ago
    uring only really applies for async IO - and would tell you when an otherwise blocking syscall would have finished. Since the benchmark here uses blocking calls, there shouldn’t be any change in behavior. The lifetime of the buffer is an orthogonal concern to the lifetime of the operation. Even if the kernel knows when the operation is done inside the kernel it wouldn’t have a way to know whether the consuming application is done with it.
    - scottlamb 4y ago
      > uring only really applies for async IO - and would tell you when an otherwise blocking syscall would have finished. Since the benchmark here uses blocking calls, there shouldn’t be any change in behavior. The lifetime of the buffer is an orthogonal concern to the lifetime of the operation. Even if the kernel knows when the operation is done inside the kernel it wouldn’t have a way to know whether the consuming application is done with it. That doesn't match what I've read. E.g. https://lwn.net/Articles/810414/ https://lwn.net/Articles/810414/ opens with "At its core, io_uring is a mechanism for performing asynchronous I/O, but it has been steadily growing beyond that use case and adding new capabilities." More precisely: * While most/all ops are async IO now, is there any reason to believe folks won't want to extend it to batch basically any hot-path non-vDSO syscall? As I said, batching doesn't help here, but it does in a lot of other scenarios. * Several IORING_OP_s seem to be growing capabilities that aren't matched by like-named syscalls. E.g. IO without file descriptors, registered buffers, automatic buffer selection, multishot, and (as of a month ago) "ring mapped supplied buffers". Beyond the individual operation level, support for chains. Why not a mechanism that signals completion when the buffer passed to vmsplice is available for reuse? (Maybe by essentially delaying the vmsplice syscall'ss return [1], maybe by a second command, maybe by some extra completion event from the same command, details TBD.) [1] edit: although I guess that's not ideal. The reader side could move the page and want to examine following bytes, but those won't get written until the writer sees the vmsplice return and issues further writes.
- rostayob 4y ago
  Actually, from re-reading the man page for vmsplice, it seems like it _should_ depend on SPLICE_F_GIFT (or in other words, it should be safe without it). But from what I know about how vmsplice is implemented, gifting or not, it sounds like it should be unsafe anyhow.
- DerSaidin 4y ago
  Hello https://mazzo.li/posts/fast-pipes.html#what-are-pipes-made-of https://mazzo.li/posts/fast-pipes.html#what-are-pipes-made-o... I think the diagram near the start of this section has "head" and "tail" swapped. Edit: Nevermind, I didn't read far enough.
haberman 4y ago
What if the writer frees the memory entirely? Can you segv the reader? That would be quite a dangerous pattern.
- BeeOnRope 4y ago
  No, because "freeing" memory makes sense within a single process and really means "removing the V->P mapping for the page", or more accurately something like "telling the malloc() implementation this pointer (implying a range of virtual memory) is free, which may or may not end up unmapping the V range from the process (or doing things like M_ADV_DONTNEED))". None of those would impact the same physical page mapped into another process with its own V->P mapping.
robocat 4y ago
> However, none of the variants using vmsplice (i.e., all but the slowest) are safe. When you gift [1] pages to the kernel there is no reliable general purpose way to know when the pages are safe to reuse again. [snip] This will show up as race conditions and spontaneously changing data where a downstream consumer sees the page suddenly change as it it overwritten by the original process. That sounds like a security issue - the ability of an upstream generator process to write into the memory of a downstream reader process, or more perverser vice versa is even worser. I presume that the Linux kernel only lets this happen (zero copy) when the two processes are running as the same user?
- hamandcheese 4y ago
  It’s not clear to me that the kernel allows the receiving process to write instead of just read. But also, if you are sending data, why would you later read/process that send buffer? The only attack vector I could imagine would be if one sender was splicing the same memory to two or more receivers. A malicious receiver with write access to the spliced memory could compromise other readers.

rostayob 4y ago

(I am the author of the post) I haven't digested this comment fully yet, but just to be clear, I am _not_ using SPLICE_F_GIFT (and I don't think the fizzbuzz program is either). However I think what you're saying makes sense in general, SPLICE_F_GIFT or not. Are you sure this unsafety depends on SPLICE_F_GIFT? Also, do you have a reference to the discussions regarding this (presumably on LKML)?

BeeOnRope 4y ago
Yeah my mention of gift was a red herring: I had assumed gift was being used but the same general problem (the "page garbage collection issue") crops up regardless. If you don't use gift, you never know when the pages are free to use again, so in principle you need to keep writing to new buffers indefinitely. One "solution" to this problem is to gift the pages, in which case the kernel does the GC for you, but you need to churn through new pages constantly because you've gifted the old ones. Gift is especially useful when the page gifted can be used directly in the page cache (i.e., writing a file, not a pipe). Without gift some consumption patterns may be safe but I think they are exactly those which involve a copy (not using gift means that a copy will occur for additional read-side scenarios). Ultimately the problem is that if some downstream process is able to get a zero-copy view of a page from an upstream writer, how can this be safe to concurrently modification? The pipe size trick is one way it could work, but it doesn't pan out because the pages may live beyond the immediately pipe (this is actually alluded in the FizzBuzz article where they mentioned things blew up if more than one pipe was involved).
- rostayob 4y ago
  Yes, this all makes sense, although like everything splicing-related, it is very subtle. Maybe I should have mentioned the subtleness and dangerousness of splicing at the beginning, rather than at the end. I still think the man page of vmsplice is quite misleading! Specifically: SPLICE_F_GIFT The user pages are a gift to the kernel. The application may not modify this memory ever, otherwise the page cache and on-disk data may differ. Gifting pages to the kernel means that a subsequent splice(2) SPLICE_F_MOVE can successfully move the pages; if this flag is not speci‐ fied, then a subsequent splice(2) SPLICE_F_MOVE must copy the pages. Data must also be properly page aligned, both in memory and length. To me, this indicates that if we're _not_ using SPLICE_F_GIFT downstream splices will be automatically taken care of, safety-wise.
  - scottlamb 4y ago
    Hmm, reading this side-by-side with a paragraph from BeeOnRope's comment: > This post (and the earlier FizzBuzz variant) try to get around this by assuming the pages are available again after "pipe size" bytes have been written after the gift, _but this is not true in general_. For example, the read side may also use splice-like calls to move the pages to another pipe or IO queue in zero-copy way so the lifetime of the page can extend beyond the original pipe. The paragraph you quoted says that the "splice-like calls to move the pages" actually copy when SPLICE_F_GIFT is not specified. So perhaps the combination of not using SPLICE_F_GIFT and waiting until "pipe size" bytes have been written is safe.
    - BeeOnRope 4y ago
      Yes it is not clear to me when the copy actually happens but I had assumed the > 30 GB/s result after read was changed to use splice must imply zero copy.
      - rostayob 4y ago
        It could be that when splicing to /dev/null (which I'm doing), the kernel knows that they their content is never witnessed, and therefore no copy is required. But I haven't verified that
scottlamb 4y ago
I think you're right that the same problem applies without SPLICE_F_GIFT. One of the other fizzbuzz code golfers discusses that here: https://codegolf.stackexchange.com/a/239848 https://codegolf.stackexchange.com/a/239848 I wonder if io_uring handles this (yet). io_uring is a newer async IO mechanism by the same author which tells you when your IOs have completed. So you might think it would: * But from a quick look, I think its vmsplice equivalent operation just tells you when the syscall would have returned, so maybe not. [edit: actually, looks like there's not even an IORING_OP_VMSPLICE operation in the latest mainline tree yet, just drafts on lkml. Maybe if/when the vmsplice op is added, it will wait to return for the right time.] * And in this case (no other syscalls or work to perform while waiting) I don't see any advantage in io_uring's read/write operations over just plain synchronous read/write.
- Matthias247 4y ago
  uring only really applies for async IO - and would tell you when an otherwise blocking syscall would have finished. Since the benchmark here uses blocking calls, there shouldn’t be any change in behavior. The lifetime of the buffer is an orthogonal concern to the lifetime of the operation. Even if the kernel knows when the operation is done inside the kernel it wouldn’t have a way to know whether the consuming application is done with it.
  - scottlamb 4y ago
    > uring only really applies for async IO - and would tell you when an otherwise blocking syscall would have finished. Since the benchmark here uses blocking calls, there shouldn’t be any change in behavior. The lifetime of the buffer is an orthogonal concern to the lifetime of the operation. Even if the kernel knows when the operation is done inside the kernel it wouldn’t have a way to know whether the consuming application is done with it. That doesn't match what I've read. E.g. https://lwn.net/Articles/810414/ https://lwn.net/Articles/810414/ opens with "At its core, io_uring is a mechanism for performing asynchronous I/O, but it has been steadily growing beyond that use case and adding new capabilities." More precisely: * While most/all ops are async IO now, is there any reason to believe folks won't want to extend it to batch basically any hot-path non-vDSO syscall? As I said, batching doesn't help here, but it does in a lot of other scenarios. * Several IORING_OP_s seem to be growing capabilities that aren't matched by like-named syscalls. E.g. IO without file descriptors, registered buffers, automatic buffer selection, multishot, and (as of a month ago) "ring mapped supplied buffers". Beyond the individual operation level, support for chains. Why not a mechanism that signals completion when the buffer passed to vmsplice is available for reuse? (Maybe by essentially delaying the vmsplice syscall'ss return [1], maybe by a second command, maybe by some extra completion event from the same command, details TBD.) [1] edit: although I guess that's not ideal. The reader side could move the page and want to examine following bytes, but those won't get written until the writer sees the vmsplice return and issues further writes.
rostayob 4y ago
Actually, from re-reading the man page for vmsplice, it seems like it _should_ depend on SPLICE_F_GIFT (or in other words, it should be safe without it). But from what I know about how vmsplice is implemented, gifting or not, it sounds like it should be unsafe anyhow.
DerSaidin 4y ago
Hello https://mazzo.li/posts/fast-pipes.html#what-are-pipes-made-of https://mazzo.li/posts/fast-pipes.html#what-are-pipes-made-o... I think the diagram near the start of this section has "head" and "tail" swapped. Edit: Nevermind, I didn't read far enough.

haberman 4y ago

What if the writer frees the memory entirely? Can you segv the reader? That would be quite a dangerous pattern.

BeeOnRope 4y ago
No, because "freeing" memory makes sense within a single process and really means "removing the V->P mapping for the page", or more accurately something like "telling the malloc() implementation this pointer (implying a range of virtual memory) is free, which may or may not end up unmapping the V range from the process (or doing things like M_ADV_DONTNEED))". None of those would impact the same physical page mapped into another process with its own V->P mapping.

robocat 4y ago

> However, none of the variants using vmsplice (i.e., all but the slowest) are safe. When you gift [1] pages to the kernel there is no reliable general purpose way to know when the pages are safe to reuse again. [snip] This will show up as race conditions and spontaneously changing data where a downstream consumer sees the page suddenly change as it it overwritten by the original process. That sounds like a security issue - the ability of an upstream generator process to write into the memory of a downstream reader process, or more perverser vice versa is even worser. I presume that the Linux kernel only lets this happen (zero copy) when the two processes are running as the same user?

hamandcheese 4y ago
It’s not clear to me that the kernel allows the receiving process to write instead of just read. But also, if you are sending data, why would you later read/process that send buffer? The only attack vector I could imagine would be if one sender was splicing the same memory to two or more receivers. A malicious receiver with write access to the spliced memory could compromise other readers.