4 ms·

Could it perhaps be even faster just to build the entire stack in hardware perhaps with an FPGA accelerator or even a custom card? I know TOE[1] is a thing, but

Could it perhaps be even faster just to build the entire stack in hardware perhaps with an FPGA accelerator or even a custom card? I know TOE[1] is a thing, but from what I read Linux doesn't really support it. [1]https://en.m.wikipedia.org/wiki/TCP_offload_engine https://en.m.wikipedia.org/wiki/TCP_offload_engine

dijit 6y ago
That is the case, but you need some DMA in the OS, so it’s up to the operating system driver to expose some portion of memory to data coming in with these means. The difference between non-DMA and DMA is staggering.
notacoward 6y ago
This has been done many, many times. It works extremely well for some applications, but also often becomes an operational nightmare. Let's say that you do manage to create a hardware implementation that's fully equivalent to the kernel one at some point in time, even in terms of things like configurability and monitoring. Something that has never happened before AFAIK. Then technology changes, your needs change, or there's a security issue. Now you have to go through a special update process - assuming an update is even available - to keep up. This might not be a problem for a small deployment, but in a small deployment it probably wasn't worth going the specialized route anyway. In a large deployment, anything that doesn't fit into the common update/remediation workflow is going to require special accommodation in code. Is the engineering cost plus the hardware cost worth it to the customer? Sometimes still yes, but more often no. There are many examples of companies who found out the hard way that the market for this kind of thing isn't big enough to recoup their own development and other costs. P.S. This is very similar to the arguments for/against hardware RAID controllers. For whatever reasons, rightly or wrongly, those are also steadily losing popularity. Software really is eating the world. P.P.S. In some cases, e.g. Amazon, the "smart NIC" approach is the common workflow, so the color of this argument changes. OTOH, it's also worth noting that the kinds of network filtering/virtualization/whatever that Amazon does is very specific to them and has nothing to do with any standard. They dedicate staff to support it. Bespoke ASIC/FPGA approaches aren't the same as a market in which you can sell them.
- regularfry 6y ago
  If there was something that looked like a standardised FPGA coprocessor on the NIC that the kernel could write microcode to on boot, or on hot-plug, updating the FPGA would be part of a kernel upgrade.
maven29 6y ago
https://www.theregister.com/2020/09/25/smartnic_dpu/ https://www.theregister.com/2020/09/25/smartnic_dpu/ I was under the impression that these "SmartNICs" were already commonplace in prominent public clouds. Is this a completely unrelated use-case?
- wmf 6y ago
  Most SmartNICs do not offload TCP; they're used for other purposes.
luizfelberti 6y ago
I think this means that the Linux kernel only doesn't support the proprietary vendor-specific ASIC offloading mechanisms. Linux already supports offloading even arbitrary eBPF/XDP programs to compatible NICs, which is already extensively used for DDoS mitigation (Cloudflare) or even Load Balancing (Facebook). I'd be surprised if it didn't leverage these mechanisms to offload the kernel's own network stack as well, but haven't really checked to confirm...
tsimionescu 6y ago
I think implementing a general-purpose TCP/IP stack entirely in an FPGA is not likely to be very good return on investment, especially if you'll want stuff like filtering, (selective) packet capture etc. However, specialized stacks for certain purposes are a relatively common thing. For example, most L23 network testing solutions use FPGAs to generate and receive massive amounts of (almost) stateless traffic (think line rate for 400GE using just 2 machines), while performing certain kinds of analysis on it (latency, loss). But these are usually just Ethernet frames or maybe IP datagrams with random payloads.

dijit 6y ago

That is the case, but you need some DMA in the OS, so it’s up to the operating system driver to expose some portion of memory to data coming in with these means. The difference between non-DMA and DMA is staggering.

notacoward 6y ago

This has been done many, many times. It works extremely well for some applications, but also often becomes an operational nightmare. Let's say that you do manage to create a hardware implementation that's fully equivalent to the kernel one at some point in time, even in terms of things like configurability and monitoring. Something that has never happened before AFAIK. Then technology changes, your needs change, or there's a security issue. Now you have to go through a special update process - assuming an update is even available - to keep up. This might not be a problem for a small deployment, but in a small deployment it probably wasn't worth going the specialized route anyway. In a large deployment, anything that doesn't fit into the common update/remediation workflow is going to require special accommodation in code. Is the engineering cost plus the hardware cost worth it to the customer? Sometimes still yes, but more often no. There are many examples of companies who found out the hard way that the market for this kind of thing isn't big enough to recoup their own development and other costs. P.S. This is very similar to the arguments for/against hardware RAID controllers. For whatever reasons, rightly or wrongly, those are also steadily losing popularity. Software really is eating the world. P.P.S. In some cases, e.g. Amazon, the "smart NIC" approach is the common workflow, so the color of this argument changes. OTOH, it's also worth noting that the kinds of network filtering/virtualization/whatever that Amazon does is very specific to them and has nothing to do with any standard. They dedicate staff to support it. Bespoke ASIC/FPGA approaches aren't the same as a market in which you can sell them.

regularfry 6y ago
If there was something that looked like a standardised FPGA coprocessor on the NIC that the kernel could write microcode to on boot, or on hot-plug, updating the FPGA would be part of a kernel upgrade.

maven29 6y ago

https://www.theregister.com/2020/09/25/smartnic_dpu/ https://www.theregister.com/2020/09/25/smartnic_dpu/ I was under the impression that these "SmartNICs" were already commonplace in prominent public clouds. Is this a completely unrelated use-case?

wmf 6y ago
Most SmartNICs do not offload TCP; they're used for other purposes.

luizfelberti 6y ago

I think this means that the Linux kernel only doesn't support the proprietary vendor-specific ASIC offloading mechanisms. Linux already supports offloading even arbitrary eBPF/XDP programs to compatible NICs, which is already extensively used for DDoS mitigation (Cloudflare) or even Load Balancing (Facebook). I'd be surprised if it didn't leverage these mechanisms to offload the kernel's own network stack as well, but haven't really checked to confirm...

tsimionescu 6y ago

I think implementing a general-purpose TCP/IP stack entirely in an FPGA is not likely to be very good return on investment, especially if you'll want stuff like filtering, (selective) packet capture etc. However, specialized stacks for certain purposes are a relatively common thing. For example, most L23 network testing solutions use FPGAs to generate and receive massive amounts of (almost) stateless traffic (think line rate for 400GE using just 2 machines), while performing certain kinds of analysis on it (latency, loss). But these are usually just Ethernet frames or maybe IP datagrams with random payloads.