5 ms·
Is it still accurate that for network bound servers that FreeBSD still outperforms Linux?
by mikece 7y ago
Is it still accurate that for network bound servers that FreeBSD still outperforms Linux?
- monocasa 7y agoStock, probably not. FreeBSD is a nicer platform to modify for your use case though.
- tieze 7y agoAs I understood Linux made a lot of progress in the last couple of years with say zero-copy patches, etc. I thought they reached some kind of parity.
- whalesalad 7y agoAFAIK Linux is actually more performant these days, network-wise.
- GhettoMaestro 7y agoI think once you get to a certain traffic level you are forced to do kernel-bypass stuff like DPDK. Regardless of Linux or FreeBSD being the kernel.
- the8472 7y agoIf it's just serving files you don't necessarily need DPDK/XDP. For server-grade hardware there now is P2P-DMA and TLS accelerators which can offload everything to peripherials while still using normal socket APIs. You get NVMe -(PCIe)-> crypto accelerator -(PCIe)-> ethernet for the bulk of the data. Neither CPU nor main memory see any of the network packets as long as they stay on the happy path. Only connection setup, DMA orchestration and occasional TLS renegotiation have to be handled.
- drewg123 7y agoIn the work we've done, the TLS crypto of bulk data is handled entirely in the CPU by the kernel via ktls. Please see the slides, specifically the data flow diagrams.
- cthalupa 7y agoI think he's talking about the general case (and since XDP is in the discussion, probably the general case for Linux) and not trying to speak specifically to what Netflix is doing for their CDN.
- floatboth 7y agoI know Chelsio has crypto directly on the NIC, but are dedicated crypto accelerator cards a thing and are they ever worth it? Why leave the CPU idle when the CPU itself is a good crypto accelerator (AES-NI, ARMv8 crypto)? AMD Ryzen has a built-in crypto "decelerator" — a FreeBSD driver was written for the crypto engine, but it's disabled by default because it made everything slower than AES-NI. (Though I guess it would be funny to use it to mine bitcoin, since it supports SHA256. AMD — Advanced Mining Devices!)
- loeg 7y agoIntel has a product line called QAT ("QuickAssisT"?) that does crypto acceleration, as well as compression. I don't know how performant it is. There are definitely several older crypto accelerators that were faster than CPUs of the time; I don't know if any of them (outside of QAT) is still relevant. The AMD Zen1 Crypto Co-Processor is indeed slower than AESNI; I think it's mostly used by stuff like SecureBoot, TPM, etc, and also used internally by the CPU to generate RDRAND/RDSEED data. It was probably never intended to be used by OS drivers and certainly not intended to be any kind of accelerator.
- shaklee3 7y agoSupposedly QAT is built into the chipsets of Skylake and above now. I've never seen anyone try it, though.
- loeg 7y agoBy chipset, you mean northbridge? Or the CPU? The part I know of that is built into the CPU is a DMA engine called I/OAT; it just does DMA and maybe basic checksum and RAID transformations. It is sometimes confused with QAT (I've personally confused the two...): https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html https://www.intel.com/content/www/us/en/wireless-network/acc...
- shaklee3 7y agoThe northbridge. my understanding is that they no longer sell the discrete cards to perform these tasks, and instead offload it to chips that come on the boards.
- toast0 7y agoNetflix is not bypassing the FreeBSD kernel. They bypass userland for bulk data though --- with sendfile, and kTLS, their webserver (nginx) has to negotiate TLS and parse requests and open files, but then it's pretty simple.
- drewg123 7y agoAuthor here.. We're doing ~200Gb/s through the kernel.
- GhettoMaestro 7y agoThat's why I carefully inserted my weasel words of "I think" ;) Thank you for your contributions. Great information.
- drewg123 7y agoThanks! I often do the same :)
- big_chungus 7y agoIt depends on how exactly each is tuned. Cache and buffer sizes, drivers, TCP congestion control algorithms, and more can affect this hugely, and are different. The defaults on linux (and probably on freebsd) also change reasonably often, so you can expect linux 4.x to have singificant differences from 5.x.
- drewg123 7y agoAuthor here.. It depends heavily on the workload. We have man years of optimizations for our CDN workload that we have been upstreaming to FreeBSD. See, for example, the other talk I co-presented regarding Kernel TLS on FreeBSD. https://people.freebsd.org/~gallatin/talks/euro2019-ktls.pdf https://people.freebsd.org/~gallatin/talks/euro2019-ktls.pdf
- continuations 7y agoWhat made Netflix pick FreeBSD over Linux in the first place?
- o-__-o 7y agoBSDs unwavering commitment to reliability. A Linux kernel crashes. A lot. FreeBSD ran cdrom.com for 365+ days without rebooting. Just saying
- yjftsjthsd-h 7y agoWhat's "a lot" here, and are you sure that's still true? Because my suspicion is that it might have been true before FAANG sunk billons of R&D into Linux, but I would be shocked if Linux kernels cash at any rare worth noticing today. Although it's splitting hairs, really; when is the last time you saw any major kernel crash?
- drewg123 7y agoI crash FreeBSD every day :)
- o-__-o 7y agoyou're making me want to start building some BSD boxes
- throw0101a 7y agoEvery year I still see a Linux system or two live-lock. When I was still doing Solaris and FreeBSD admining that never happened. However most systems aren't pushed to extremes, so these corner cases aren't seen very often, regardless of OS.
- loeg 7y agoI think it's safe to say any kind of sweeping generalization like this is unlikely to be broadly true (in either direction). Both work quite well.