9 ms·
To be clear, was FreeBSD used because of historical reasons or because similar performance can't be/harder to achieve on Linux?
by GrigoriyMikh 5y ago
To be clear, was FreeBSD used because of historical reasons or because similar performance can't be/harder to achieve on Linux?
- smoldesu 5y agoI'd imagine scaling, licensing and overhead all had something to do with it, too.
- GrigoriyMikh 5y agoInteresting. Are there any benchmarks that you would recommend to look at, regarding FreeBSD vs Linux networking performance?
- GrigoriyMikh 5y agoFor anyone interested, here is benchs from late 2018, comparing Fedora and FreeBSD performance: https://matteocroce.medium.com/linux-and-freebsd-networking-cbadcdb15ddd https://matteocroce.medium.com/linux-and-freebsd-networking-...
- trasz 5y agoIt’s probably worth noting that there have been huge scalability improvements - including introduction of epochs (+/- RCU) - in FreeBSD over the last few years, for both networking and VFS.
- lostmsu 5y agoWhy does the author put so much effort into testing VMs? Bare metal installations aren't even tried, so the article won't represent more typical setup (unless you want to run in cloud, in which case it would make sense to test in cloud).
- genewitch 5y agoIf given the choice I'd never run anything on bare metal again. Let's say we have some service we want to run on a bare metal server. For not very much more hardware money, amortized, I can set up two or three VMs to do the same server, duplicated. Then if any subset of that metal goes bad, a replica/duplicate is already ready to go. There's no network overhead, etc. I've been doing this for stuff like SMTPE authority servers and ntpd and things that absolutely cannot go down, for over a decade.
- smoldesu 5y agoThat doesn't really matter for benchmarking purposes. I think the parent comment was emphasizing that syscall benchmarks don't make sense when you're running through a hypervisor, since you're running tacitly different instructions than would be run on a bare-metal or provisioned server.
- teknoraver 5y agoBecause the cards were in PCI passthrough, so the performance was exactly the same of a physical system
- lostmsu 5y agoAuthor mentions as much as to say that there's was some indirectness at least in the interrupts. There are also VirtIO drivers involved, and according to the article, they had effect too.
- crest 5y agoNginx and OpenSSL are open source. Give it a try and reproduce their results with Linux ;-).
- betaby 5y agosendfile + kTLS. I'm unaware of the in-kernel TLS implementation for Linux. Is there any around?
- merb 5y agohttps://www.kernel.org/doc/html/latest/networking/tls-offload.html https://www.kernel.org/doc/html/latest/networking/tls-offloa... https://docs.mellanox.com/display/OFEDv521040/Kernel+Transport+Layer+Security+%28kTLS%29+Offloads https://docs.mellanox.com/display/OFEDv521040/Kernel+Transpo...
- drewg123 5y agoYes, Linux has kTLS. When I tried to use it, it was horribly broken, so my fear is that its not well used/tested, but it exists. Mellanox, for example, developed their inline hardware kTLS offload on linux.
- Matthias247 5y ago> When I tried to use it, it was horribly broken, so my fear is that its not well used/tested, but it exists. Do you have any additional references around this? I'm aware that most rarely used functionality is often broken and therefore usually don't recommend people to use it, but would like to learn about kTLS in particular. I think for Linux OpenSSL 3 now added support for it in userspace. But there's also the kernel components as well as drivers - all of them could have their set of issues.
- drewg123 5y agoI recall that simple transmits from offset 0..N in a file worked. But range requests of the form N..N+2MB lead to corrupt data. Its been 2+ years, and I heard it was later fixed in Linux.
- Thaxll 5y agoI mean most CDN and FANG run on Linux, I think in that case it's kTLS that makes a big difference the rest not much.
- drewg123 5y agoAsync sendfile is also an advantage for FreeBSD. It is specific to FreeBSD. It allows an nginx worker to send from a file that's cold on disk without blocking, and without resorting to threadpools using a thread-per-file. The gist is that the sendfile() call stages the pages waiting to be read in the socket buffer, and marks the mbufs with M_NOTREADY (so they cannot be sent by TCP). When the disk read completes, a sendfile callback happens in the context of the disk ithread. This clears the M_NOTREADY flag and tells TCP they are ready to be sent. See https://www.nginx.com/blog/nginx-and-netflix-contribute-new-sendfile2-to-freebsd/ https://www.nginx.com/blog/nginx-and-netflix-contribute-new-...
- Thaxll 5y agosendfile() with splice and io_uring is similar? I know that this is very experimental on Linux. The overall idea is to copy bytes from disk to the socket with almost no allocation and not blocking, this is the idea right?
- Matthias247 5y agoMaybe. A few things in io-uring are implemented by letting a kernel task/thread block on doing the actual work. Which calls that are seems to change in every version, and might be tricky to find out without reading the kernel code.