3 ms·

Don't kTLS sockets[0] with crypto offloading[1], sendfile/vmsplice, device-to-device DMA transfers[2] and possibly io_uring solve all those things on linux? Gra

Don't kTLS sockets[0] with crypto offloading[1], sendfile/vmsplice, device-to-device DMA transfers[2] and possibly io_uring solve all those things on linux? Granted, they're not POSIX, but they're incremental extensions. Netflix implemented similar extensions in freebsd[3] [0] https://www.kernel.org/doc/Documentation/networking/tls.txt https://www.kernel.org/doc/Documentation/networking/tls.txt [1] https://lwn.net/Articles/734030/ https://lwn.net/Articles/734030/ [2] https://lwn.net/Articles/767281/ https://lwn.net/Articles/767281/ [3] https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf

loeg 8y ago
Not really. These are all incremental performance improvements on POSIX but don't address the author's concerns / desires in the paper. All of them continue to require the kernel to mediate IO between userspace and the hardware. For some reason the author is fixated on direct user access to partitioned hardware queues. Netflix's CDN operating system is based on FreeBSD, and they did add a kind of kTLS implementation, but they did not add it to FreeBSD upstream for a host of reasons.
- the8472 8y ago
  The means may be different, but the ends they aim for seem to be similar. The ends here are improved latency, throughput, parallelism, non-blocking APIs and security. The above-mentioned improvements aim to address the first four without completely bypassing the kernel, instead changing APIs so they step out of the way most of the time, limiting them to coordination tasks and then either offloading to the hardware or directly piping the data to/from userspace mappings without additional context switches where needed. VMs using SR-IOV address the security aspect. It's basically the difference between a green field design and tacking all those innovations onto the glueball that is linux. The latter may be ugly and complex, but it has the advantage of being backwards-compatible.
  - loeg 8y ago
    > It's basically the difference between a green field design and tacking all those innovations onto the glueball that is linux. The latter may be ugly and complex, but it has the advantage of being backwards-compatible. I'm definitely not trying to argue in favor of the paper's opinion. :-) I just believe the author of the article would disagree that the glueball your earlier remarks describe addresses the author's concerns. They specifically say that Linux cannot be fixed to their satisfaction, or at least argue that idea. Look for the section on page 5 headed, "Why not use kernel-bypass techniques on Linux?"
    - the8472 8y ago
      Well, and all I am saying is that the paper, including the section you mention, don't take these latest developments into account. They're not kernel bypasses. They're driver level optimizations and traditional APIs operating on file descriptors and virtual memory aiming to solve similar problems. This stuff is very new of course, so we have to wait for something to actually integrate all those pieces and then for benchmarks.
- drewg123 8y ago
  The host of reasons was mostly one reason: "lack of time". Now that we have to deal with multiple vendors of inline hardware TLS offload solutions, it is more critical to get it upstream, and so it is being upstreamed as we speak. The first piece of it (fixing send tags so they are reliable and can be used for inline hw tls in addition to hardware pacing) is up for review right now: https://reviews.freebsd.org/D20117 https://reviews.freebsd.org/D20117

loeg 8y ago

Not really. These are all incremental performance improvements on POSIX but don't address the author's concerns / desires in the paper. All of them continue to require the kernel to mediate IO between userspace and the hardware. For some reason the author is fixated on direct user access to partitioned hardware queues. Netflix's CDN operating system is based on FreeBSD, and they did add a kind of kTLS implementation, but they did not add it to FreeBSD upstream for a host of reasons.

the8472 8y ago
The means may be different, but the ends they aim for seem to be similar. The ends here are improved latency, throughput, parallelism, non-blocking APIs and security. The above-mentioned improvements aim to address the first four without completely bypassing the kernel, instead changing APIs so they step out of the way most of the time, limiting them to coordination tasks and then either offloading to the hardware or directly piping the data to/from userspace mappings without additional context switches where needed. VMs using SR-IOV address the security aspect. It's basically the difference between a green field design and tacking all those innovations onto the glueball that is linux. The latter may be ugly and complex, but it has the advantage of being backwards-compatible.
- loeg 8y ago
  > It's basically the difference between a green field design and tacking all those innovations onto the glueball that is linux. The latter may be ugly and complex, but it has the advantage of being backwards-compatible. I'm definitely not trying to argue in favor of the paper's opinion. :-) I just believe the author of the article would disagree that the glueball your earlier remarks describe addresses the author's concerns. They specifically say that Linux cannot be fixed to their satisfaction, or at least argue that idea. Look for the section on page 5 headed, "Why not use kernel-bypass techniques on Linux?"
  - the8472 8y ago
    Well, and all I am saying is that the paper, including the section you mention, don't take these latest developments into account. They're not kernel bypasses. They're driver level optimizations and traditional APIs operating on file descriptors and virtual memory aiming to solve similar problems. This stuff is very new of course, so we have to wait for something to actually integrate all those pieces and then for benchmarks.
drewg123 8y ago
The host of reasons was mostly one reason: "lack of time". Now that we have to deal with multiple vendors of inline hardware TLS offload solutions, it is more critical to get it upstream, and so it is being upstreamed as we speak. The first piece of it (fixing send tags so they are reliable and can be used for inline hw tls in addition to hardware pacing) is up for review right now: https://reviews.freebsd.org/D20117 https://reviews.freebsd.org/D20117