3 ms·
How important is it to bypass the kernel if the kernel doesn't get to see/handle each packet individually? As soon as you get the hardware to handle TCP reasse
by londons_explore 7y ago
How important is it to bypass the kernel if the kernel doesn't get to see/handle each packet individually?
As soon as you get the hardware to handle TCP reassembly and just wake the kernel up once per few megabytes of data sent/received, things scale well again.
There's work to do though - there are no systems around today that I'm aware of which can send data from SSD to a TCP socket (common use case for cache server) without the data itself going through the CPU (despite most chipsets allowing the network card to be sent data directly from a PCIE-connected SSD).
- q3k 7y ago> As soon as you get the hardware to handle TCP reassembly and just wake the kernel up once per few megabytes of data sent/received, things scale well again. Not everything in this world is TCP. TFA mentions DNS, for instance (which as of this time is still mostly UDP based). > There's work to do though - there are no systems around today that I'm aware of which can send data from SSD to a TCP socket (common use case for cache server) without the data itself going through the CPU (despite most chipsets allowing the network card to be sent data directly from a PCIE-connected SSD). They certainly exist, they are just not available or known to the general public (hint: look for hyperscalers that manufacture custom network cards).
- nine_k 7y agoI suppose that CPU would schedule a DMA transfer from SSD to RAM, and then from RAM to a NIC. For NICs that have memory-mapped buffers, it could be just one transfer. What am I missing?
- londons_explore 7y agoNothing. That's how it would work. But today's Linux kernel doesn't do that (even when you use the sendfile() API)
- navaati 7y ago> But today's Linux kernel doesn't do that (even when you use the sendfile() API) Oh ? I’m quite disappointed, I thought that was the whole point of sendfile() ! What does it do if not that ?
- wmf 7y agosendfile() has done zero-copy networking for a while (the data still goes through main memory and it gets DMAed twice, but the cores don't touch it). More recently, NVMe controller memory buffers should allow data to be DMAed from the SSD to the NIC without hitting RAM and it looks like Linux is adding support for it using the p2pdma subsystem.
- scurvy 7y agoIt's pretty effective for things like Intrusion Detection Systems (IDS), at least in my experience. I don't know enough about all of the glue in-between, but things like Suricata scream when combined with dedicated kernel bypass cards.
- Nextgrid 7y agoHow would your “SSD to TCP” solution handle TLS which is pretty ubiquitous? What about the SSD being encrypted or using a complex file system (ZFS, etc) where data is stored in non-continuous blocks?
- londons_explore 7y agoPass TLS session key to hardware. Hardware does encryption. Where the data to be sent is non-contiguous in storage, the OS would probably have to make two requests (although those two requests could still be made simultaneously, so not requiring an extra wake-up/interrupt)
- vardump 7y ago> As soon as you get the hardware to handle TCP reassembly and just wake the kernel up once per few megabytes of data sent/received, things scale well again. Not a generic solution, because it increases latency. Which you could avoid by doing everything in user or kernel space.