7 ms·
I think once you get to a certain traffic level you are forced to do kernel-bypass stuff like DPDK. Regardless of Linux or FreeBSD being the kernel.
by GhettoMaestro 7y ago
I think once you get to a certain traffic level you are forced to do kernel-bypass stuff like DPDK. Regardless of Linux or FreeBSD being the kernel.
- the8472 7y agoIf it's just serving files you don't necessarily need DPDK/XDP. For server-grade hardware there now is P2P-DMA and TLS accelerators which can offload everything to peripherials while still using normal socket APIs. You get NVMe -(PCIe)-> crypto accelerator -(PCIe)-> ethernet for the bulk of the data. Neither CPU nor main memory see any of the network packets as long as they stay on the happy path. Only connection setup, DMA orchestration and occasional TLS renegotiation have to be handled.
- drewg123 7y agoIn the work we've done, the TLS crypto of bulk data is handled entirely in the CPU by the kernel via ktls. Please see the slides, specifically the data flow diagrams.
- cthalupa 7y agoI think he's talking about the general case (and since XDP is in the discussion, probably the general case for Linux) and not trying to speak specifically to what Netflix is doing for their CDN.
- floatboth 7y agoI know Chelsio has crypto directly on the NIC, but are dedicated crypto accelerator cards a thing and are they ever worth it? Why leave the CPU idle when the CPU itself is a good crypto accelerator (AES-NI, ARMv8 crypto)? AMD Ryzen has a built-in crypto "decelerator" — a FreeBSD driver was written for the crypto engine, but it's disabled by default because it made everything slower than AES-NI. (Though I guess it would be funny to use it to mine bitcoin, since it supports SHA256. AMD — Advanced Mining Devices!)
- loeg 7y agoIntel has a product line called QAT ("QuickAssisT"?) that does crypto acceleration, as well as compression. I don't know how performant it is. There are definitely several older crypto accelerators that were faster than CPUs of the time; I don't know if any of them (outside of QAT) is still relevant. The AMD Zen1 Crypto Co-Processor is indeed slower than AESNI; I think it's mostly used by stuff like SecureBoot, TPM, etc, and also used internally by the CPU to generate RDRAND/RDSEED data. It was probably never intended to be used by OS drivers and certainly not intended to be any kind of accelerator.
- shaklee3 7y agoSupposedly QAT is built into the chipsets of Skylake and above now. I've never seen anyone try it, though.
- loeg 7y agoBy chipset, you mean northbridge? Or the CPU? The part I know of that is built into the CPU is a DMA engine called I/OAT; it just does DMA and maybe basic checksum and RAID transformations. It is sometimes confused with QAT (I've personally confused the two...): https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html https://www.intel.com/content/www/us/en/wireless-network/acc...
- shaklee3 7y agoThe northbridge. my understanding is that they no longer sell the discrete cards to perform these tasks, and instead offload it to chips that come on the boards.
- deleted 7y ago[deleted]
- loeg 7y agoThat might be true, but you can still buy the cards from 3rd party resellers, e.g., https://www.newegg.com/p/2AS-006R-00046 https://www.newegg.com/p/2AS-006R-00046 .
- toast0 7y agoNetflix is not bypassing the FreeBSD kernel. They bypass userland for bulk data though --- with sendfile, and kTLS, their webserver (nginx) has to negotiate TLS and parse requests and open files, but then it's pretty simple.
- drewg123 7y agoAuthor here.. We're doing ~200Gb/s through the kernel.
- GhettoMaestro 7y agoThat's why I carefully inserted my weasel words of "I think" ;) Thank you for your contributions. Great information.
- drewg123 7y agoThanks! I often do the same :)