11 ms·

FreeBSD has lower latency, and Linux has faster application speeds

134 points by krn 8y ago

rurban 8y ago
He forgot to say that Dave Miller brought in siphash for recent networking hashes on Linux, which should bring down performance of the Linux networking stack closer to Ruby/Python levels. About 20 slower. There's nothing slower than this, whilst not improving security, but increased the accompanied security theatre. It's a mess over there.
rb2k_ 8y ago
No sources or benchmarks? Also: > But unless you’re using Fedora, your servers aren’t gonna taste it soon. Just by bad timing, the new Ubuntu LTS (18.04) due out next week will still use 4.15, which it will support for 5 years. Since CentOS and Debian are even further behind on kernels, you won’t see a lot of “fast Linux” in production until April 2020 when Ubuntu 20.04 LTS comes out. If you are at a point where the differences would matter to you, you can (and will) probably just install a newer kernel and have the improvements available today. Just because upstream doesn't have it doesn't mean you couldn't compile it yourself or use backports.
- corv 8y ago
  > No sources or benchmarks? There was this in TFA: https://www.phoronix.com/scan.php?item=netperf-bsd-linux&num=3&page=article https://www.phoronix.com/scan.php?item=netperf-bsd-linux&num...
  - r1ch 8y ago
    This benchmark is kind of meaningless. What is a "TCP request response" actually measuring? Any Linux server can certainly handle more than 340 HTTP reqs/sec for example. The "netperf" benchmarking software appears to be from 1993 (complete with a webpage from that era! https://hewlettpackard.github.io/netperf/ https://hewlettpackard.github.io/netperf/) so I have a lot of doubts that it is using modern networking APIs or taking advantage of modern hardware.
    - drewg123 8y ago
      Netperf is probably the most widely adopted network benchmark out there. Almost every company I've seen has used it for performance and Q/A testing. For what it does, it is quite efficient. I remember when I was doing 10GbE drivers in the mid 2000s, people would complain of terrible performance on "modern" tools like iperf, but would see 10Gb/s using netperf. This is because "modern" tools did things like gettimeofday() around every socket read or write, making them basically gettomeofday() benchmarks at high message rates, but netperf did the gettimeofday() around the entire test. Netperf's problem has always been that it is single connection, single-threaded. Most people use it coupled with patches or scripts to run many copies of netperf in parallel. I suspect what happened in these tests is that the author just ran a single copy of the tests, and did not bother to adjust the interrupt coalesing settings. So what he really measured was different interrupt coalescing settings in the FreeBSD and Linux driver. If he'd have run 500 or 1000 copies, I'll bet he'd see vastly different results. BTW, if you're looking for a modern network benchmark, check out uperf (http://uperf.org/ http://uperf.org/) It seems to have been largely abandoned after the Oracle acquisition, which is a shame, because it was just plane awesome. It could replicate many scenarios very realistically, supported multi-threading, etc.
      - rjmcmahon 8y ago
        Iperf 2.0.10+ uses clock_gettime() when a timestamps is needed. For TCP and no interval reporting the only calls needed are at the beginning and end of the test. The performance problem we hit with 2.0.5 had to do with insufficient shared memory between traffic threads and the reporter thread.
  - jasondclinton 8y ago
    The benchmark results at that link are from machines with different processor clocks. The results are invalid.
- ams6110 8y ago
  > Just because upstream doesn't have it doesn't mean you couldn't compile it yourself True, but if you are managing production machines, compiling kernels for every errata and security update is pretty low on the list of things you get enthusiastic about. Also a lot of commercial or scientific linux software is only "certified" and supported on stock RHEL kernels.
  - cozzyd 8y ago
    There's always http://elrepo.org/tiki/kernel-ml http://elrepo.org/tiki/kernel-ml (and also kernel-lt) so you don't have to compile things yourself. But things like VMWare and various proprietary device drivers will break.
  - geofft 8y ago
    On the other hand, if you're making or losing money on latency (e.g., you're a high-frequency trader), compiling kernels for every potential performance improvement is a thing you're enthusiastic about, a lot of security fixes (local privilege escalations, drivers for desktop hardware, etc.) don't apply to you, and you don't care about things being "certified" because you have the expertise to fix problems in-house as necessary. And from the other direction, if you're making or losing money on security, expecting distro kernels to turn around security fixes in a timely manner seems like a mistake. At least know how to rebuild your distro kernel with a local patch.
  - rb2k_ 8y ago
    The amount of filesystem/networking issues I've seen fixed and all the performance improvements I got from just updating a kernel probably make up for the time that it caused issues. I guess it needs a critical mass + certain infrastructure maturity that running on whatever the latest kernel is makes sense. As for the 'certified on stock RHEL kernel' software... I try to keep away from those, but I'm picky and pretty lucky when it comes to job selection :)
- fulafel 8y ago
  It's also incorrect even for LTS supported kernels: LTS releases incorporate new kernel versions along the way as part of the hardware enablement work, though they are not used by default. For example, here's 4.13 in the previous LTS (16.04), up from the default 4.4: http://packages.ubuntu.com/linux-signed-image-generic-hwe-16.04 http://packages.ubuntu.com/linux-signed-image-generic-hwe-16...
sliken 8y ago
The post seems to be confusing FreeBSD userspace (which OSX adopted parts of) and the FreeBSD kernel (which has the mentioned lower network latency). OSX didn't adopt the FreeBSD kernel, but instead started with the Mach microkernel and has been going in their own direction since.
- 4ad 8y ago
  Not that it matters (the post is mostly just unfounded opinion), but: > OSX didn't adopt the FreeBSD kernel, but instead started with the Mach microkernel and has been going in their own direction since. MacOS kernel uses a combination of Mach and BSD kernel code. It started with 4.4BSD, and at some point the bulk of BSD code was updated with code from FreeBSD 5. After that, it was piecewise updated with more modern stuff, the MAC framework (used to implement sandboxing on iOS and macOS) and DTrace from newer FreeBSD kernels.
  - hestefisk 8y ago
    There is a great talk on OS X internals from CCC here: https://events.ccc.de/congress/2007/Fahrplan/attachments/1053_inside-macosx-kernel.pdf https://events.ccc.de/congress/2007/Fahrplan/attachments/105... It particularly talks about the unique hybrid kernel approach and how there is nothing like it.
    - loeg 8y ago
      BSD itself took the VM subsystem from Mach (so BSD is kind of an interesting hybrid as well). While it has been heavily modified in the last 25+ years, the VM portion of the FreeBSD kernel is still stylistically distinct from BSD portions of the kernel source code.
- trasz 8y ago
  Not just userspace - OSX adopted large pieces of FreeBSD kernel, including the syscall layer and the network stack. Also, Mach isn't technically a microkernel - the whole kernel runs in Ring 0. But yeah, OSX kernel is vastly different from FreeBSD - first, there were huge differences to start with, only some pieces of kernel were adopted, and finally there were thousands of man-hours put into each system afterwards, diverging them even more.
snvzz 8y ago
Doesn't Dragonfly (freebsd fork as of 2003) have even lower latency?
- hyper2grams 8y ago
  DragonFly BSD is a phenomenal project and deserves more recognition imo.
TorKlingberg 8y ago
With such a long post, I would have expected some technical details for why FreeBSD has lower latency, and Linux has faster application speeds.
j16sdiz 8y ago
The benchmark use outdated API. Misleading on distro release a date. No explanation on the improvements. I would vote down this post if I could.
- josersanchez 8y ago
  You can down vote on quora.
- deleted 8y ago
  [deleted]
stefan_ 8y ago
Latency and throughput ("application speeds"), certainly in computing, are competing qualities.
api 8y ago
A bit of a tangent but: take another look at FreeBSD folks. If we re-did our infrastructure from scratch we'd look at either FreeBSD or Alpine Linux. Both would offer an escape hatch from the needless bloat that mainstream Linux has become.
- hpcjoe 8y ago
  Dependency radii of packages in CentOS are horrible. Your minimal installs are hovering above 1GB. Debian are better at 400MB or so. Alpine is great, in that you can get this down to well under 100MB. This has been a long standing complaint of mine[1], and an argument I've had with many proponents of "software must come packaged only from the provider of the distro" people. Physical and logical footprint is a real thing. Ridiculous secondary/tertiary dependencies are a waste of that precious resource, and an unneeded/unwanted installation of useless bits. [1] http://scalability.org/2018/04/distribution-package-dependency-radii-or-why-distros-may-be-doomed/ http://scalability.org/2018/04/distribution-package-dependen...
  - mmt 8y ago
    Although I agree with you that this bloat is undesirable, I'm not convinced it's actually a problem with the distro. (My primary experience is with Ubuntu, and to a lesser extent, mostly by extension, Debian) Rather, I think it's a problem with the package maintainers themselves. I certainly admit that for very many packages, this is a distinction without a difference. However, my point is that this dependency bloat isn't based on some kind of policy that the distro is encouraging or even enforcing. Rather, the problem is merely that too many source packages create too few targets, perhaps because the original packagers never foresaw the need (for, e.g., both an X version of some graphics or GPU-related library and a non-X one for pure server work). I've occasionally gotten around this problem by custom-building a local version of the package that avoids the dependency bloat. That violates the "software must come packaged only from the provider of the distro", unless one considers the local superset a forked distro. > Physical and logical footprint is a real thing. Ultimately, I think you and I are in a vanishingly small minority here. Virtualization (even the full, pre-container kind, with a full copy of the kernel on every instance) gained a startling level of a popularity, and was even lionized as a money-saving tool (which, of course, it was, for some "IT" shops).
drewg123 8y ago
Speaking as somebody who works on the FreeBSD kernel at Netflix, the whole article is kind of nonsensical. We don't use FreeBSD because of "latency". We serve clients over https that could be tens or hundreds of ms away using the kernel TCP stack. The folks who care about latency are high frequency traders, who care about every nanosecond. They tend to use userspace / hardware offloaded solutions and do things like busy-waiting for messages.
- tiffanyh 8y ago
  Drew Have you ever consider using Dragonfly. It appears to dramatically beat in perf test FreeBSD (2017). https://leaf.dragonflybsd.org/~sephe/perf_cmp.pdf https://leaf.dragonflybsd.org/~sephe/perf_cmp.pdf I’m curious to hear your take on Dragonfly since you’re a kernel developer. Thanks in advance for all your work. Edit: more recent links (2018) with even higher perf https://leaf.dragonflybsd.org/~sephe/dfly1.pdf https://leaf.dragonflybsd.org/~sephe/dfly1.pdf https://leaf.dragonflybsd.org/~sephe/dfly1_p.pdf https://leaf.dragonflybsd.org/~sephe/dfly1_p.pdf
  - drewg123 8y ago
    I have looked at Dragonfly with interest, due to how they manage VM. I really like how they shard the page queues to be per-cpu. With that said, a lot of work has been done by jeffr in the last 6 months to a year to improve NUMA performance (which we partially sponsored) which has dramatically improved VM scalability even on single socket boxes, and which has allowed us to remove a lot of our local hacks. I considered giving Dragonfly a try in the past, but just never had the time. There would be a depressing amount of work before we could even consider Dragonfly, mostly centered around our async sendfile (which is now upstream in FreeBSD), our TCP changes: BBR (not yet upstream), RACK (now upstream) and TCP pacing (now upstream). Not to mention unmapped mbufs (not yet upstream) and kernel TLS (also not yet upstream). Also, the last time I looked, Dragonfly did not have drivers for Chelsio or Mellanox 10/25/40/100GbE NICs. Without even just one of these things, performance will be so bad that any comparison testing would be meaningless.
    - 8y ago
parvenu74 8y ago
> "There are some 'BSD-inspired' networking improvements to Linux starting with the 4.16." By "inspired by BSD" do the author mean the code has been copied and pasted from FreeBSD to Linux, something which cannot be done in reverse because of the encumbrance of the GPL?
- zeth___ 8y ago
  BSD is free to adopt the GPL at their leisure. Who knows, companies might then start contributing to the project instead of building OS's on top of it, looking at you osx.
  - parvenu74 8y ago
    GPL supporters are all about freedom -- until someone advocates for the freedom to make a proprietary product on top of the open source base.
    - cozzyd 8y ago
      Yes... that is the point.
      - parvenu74 8y ago
        Oh I forgot -- at HN you never question the GPL. And FreeBSD sux because it's not GPL. Got it. I'll stop wasting my time even mentioning it again.
        jhasse 8y ago
        You didn't question the GPL, you just explained how it works.
    - bonzini 8y ago
      More specifically it's about freedom for all users, transitively.
vermaden 8y ago
FreeBSD was used in 1999 to render The Matrix on 32 Pentium II boxes because the software in Linux Compatibility mode on FreeBSD was faster then natively on Linux, that is a fact: https://www.freebsd.org/news/press-rel-1.html https://www.freebsd.org/news/press-rel-1.html FreeBSD can be several times faster then Linux when it comes to network stack: https://pbs.twimg.com/media/CzFfTSRUQAATwaq.jpg https://pbs.twimg.com/media/CzFfTSRUQAATwaq.jpg But often Linux is faster, you just need to find benchmark that favorites one or another, both are fast in general.
- ajross 8y ago
  Why on earth would you link to a Phoronix screenshot instead of the article that explains it?! Those results look suspicious enough[1] that I wanted to look it up. But of course I can't, and Google isn't helping me much. Shame! Shame, shame, shame! [1] Factor of 3+ differences not just between BSD and Linux but between otherwise very comparable linux distros. Something's weird with the setup there, like it's measuring default firewalling overhead or something and not kernel behavior.
  - loeg 8y ago
    > Phoronix > Something's weird with the setup there, like it's measuring default firewalling overhead or something and not kernel behavior. This is something you come to expect if you've seen enough of what Phoronix publishes.
  - buzer 8y ago
    Looks like source is https://www.phoronix.com/scan.php?page=article&item=netperf-bsd-linux&num=1 https://www.phoronix.com/scan.php?page=article&item=netperf-... I guess they have everything just set to default disto settings which likely in some cases include firewall while others don't. They also don't mention which NIC they are using, the motherboard has two different NICs (i218-LM & I210-AT).
    - jasondclinton 8y ago
      Those benchmarks were run on machines with different processor clocks and different motherboard chipsets. The results are invalid.
HippoBaro 8y ago
Woudn't anyone really into IP latency optimization be using something like DPDK on Linux? If you want high-performance networking, then using the kernel isn't the best idea IMO.
rjmcmahon 8y ago
Well, here's just one measurement using iperf 2.0.12. All clocks are synchronized to a GPS disciplined oven controlled oscillator (OCXO). Connected via 1Gbs. FreeBSD latency is significantly better. Source fedora 28 xeon machine, receivers FreeBSD 11.1 and Fedora 25 (same brix platform) connected via a Cisco 300 switch. RX: FreeBSD: root@zeus:/usr/local/src/iperf2-code # iperf -s -u -e --udp-histogram=10u,100000 --realtime ------------------------------------------------------------ Server listening on UDP port 5001 with pid 53703 Receiving 1470 byte datagrams UDP buffer size: 41.1 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.100.34 port 5001 connected with 192.168.100.55 port 52536 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Latency avg/min/max/stdev PPS [ 3] 0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 0.004 ms 0/ 892 (0%) 0.086/ 0.060/ 0.150/ 0.014 ms 89 pps [ 3] 0.00-10.00 sec T8(f)-PDF: bin(w=10us):cnt(892)=7:160,8:143,9:181,10:159,11:242,12:6,16:1 (5/95%=7/11,Outliers=0,obl/obu=0/0) Fedora 25: [root@hera iperf2-code]# iperf -s -u -e --udp-histogram=10u,10000 --realtime ------------------------------------------------------------ Server listening on UDP port 5001 with pid 16669 Receiving 1470 byte datagrams UDP buffer size: 208 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.100.33 port 5001 connected with 192.168.100.55 port 35894 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Latency avg/min/max/stdev PPS [ 3] 0.00-9.99 sec 1.25 MBytes 1.05 Mbits/sec 0.010 ms 0/ 892 (0%) 0.261/ 0.098/ 0.319/ 0.021 ms 89 pps [ 3] 0.00-9.99 sec T8(f)-PDF: bin(w=10us):cnt(892)=10:1,21:1,23:6,24:107,25:141,26:209,27:151,28:124,29:81,30:47,31:19,32:5 (5/95%=24/30,Outliers=0,obl/obu=0/0) TX: [root@rjm-clubhouse-28 rjmcmahon]# iperf -s -e -u --udp-histogram=10u,100000 --realtime ------------------------------------------------------------ Server listening on UDP port 5001 with pid 26016 Receiving 1470 byte datagrams UDP buffer size: 208 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.100.55 port 5001 connected with 192.168.100.34 port 20343 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Latency avg/min/max/stdev PPS [ 3] 0.00-10.01 sec 1.25 MBytes 1.05 Mbits/sec 0.010 ms 0/ 893 (0%) 0.094/ 0.064/ 0.476/ 0.023 ms 89 pps [ 3] 0.00-10.01 sec T8(f)-PDF: bin(w=10us):cnt(893)=7:12,8:38,9:291,10:311,11:198,12:28,13:10,14:1,15:1,42:1,45:1,48:1 (5/95%=8/11,Outliers=3,obl/obu=0/0) [ 4] local 192.168.100.55 port 5001 connected with 192.168.100.33 port 58391 [ 4] 0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 0.013 ms 0/ 892 (0%) 0.079/ 0.048/ 0.127/ 0.009 ms 89 pps [ 4] 0.00-10.00 sec T8(f)-PDF: bin(w=10us):cnt(892)=5:1,6:10,7:115,8:265,9:425,10:51,11:18,12:3,13:4 (5/95%=7/10,Outliers=0,obl/obu=0/0)
ksec 8y ago
I reckon if someone make Java a first class citizen on FreeBSD, it may actually get a larger amount of usage.

134 points by krn 8y ago

He forgot to say that Dave Miller brought in siphash for recent networking hashes on Linux, which should bring down performance of the Linux networking stack closer to Ruby/Python levels. About 20 slower. There's nothing slower than this, whilst not improving security, but increased the accompanied security theatre. It's a mess over there.

rb2k_ 8y ago

No sources or benchmarks? Also: > But unless you’re using Fedora, your servers aren’t gonna taste it soon. Just by bad timing, the new Ubuntu LTS (18.04) due out next week will still use 4.15, which it will support for 5 years. Since CentOS and Debian are even further behind on kernels, you won’t see a lot of “fast Linux” in production until April 2020 when Ubuntu 20.04 LTS comes out. If you are at a point where the differences would matter to you, you can (and will) probably just install a newer kernel and have the improvements available today. Just because upstream doesn't have it doesn't mean you couldn't compile it yourself or use backports.

corv 8y ago
> No sources or benchmarks? There was this in TFA: https://www.phoronix.com/scan.php?item=netperf-bsd-linux&num=3&page=article https://www.phoronix.com/scan.php?item=netperf-bsd-linux&num...
- r1ch 8y ago
  This benchmark is kind of meaningless. What is a "TCP request response" actually measuring? Any Linux server can certainly handle more than 340 HTTP reqs/sec for example. The "netperf" benchmarking software appears to be from 1993 (complete with a webpage from that era! https://hewlettpackard.github.io/netperf/ https://hewlettpackard.github.io/netperf/) so I have a lot of doubts that it is using modern networking APIs or taking advantage of modern hardware.
  - drewg123 8y ago
    Netperf is probably the most widely adopted network benchmark out there. Almost every company I've seen has used it for performance and Q/A testing. For what it does, it is quite efficient. I remember when I was doing 10GbE drivers in the mid 2000s, people would complain of terrible performance on "modern" tools like iperf, but would see 10Gb/s using netperf. This is because "modern" tools did things like gettimeofday() around every socket read or write, making them basically gettomeofday() benchmarks at high message rates, but netperf did the gettimeofday() around the entire test. Netperf's problem has always been that it is single connection, single-threaded. Most people use it coupled with patches or scripts to run many copies of netperf in parallel. I suspect what happened in these tests is that the author just ran a single copy of the tests, and did not bother to adjust the interrupt coalesing settings. So what he really measured was different interrupt coalescing settings in the FreeBSD and Linux driver. If he'd have run 500 or 1000 copies, I'll bet he'd see vastly different results. BTW, if you're looking for a modern network benchmark, check out uperf (http://uperf.org/ http://uperf.org/) It seems to have been largely abandoned after the Oracle acquisition, which is a shame, because it was just plane awesome. It could replicate many scenarios very realistically, supported multi-threading, etc.
    - rjmcmahon 8y ago
      Iperf 2.0.10+ uses clock_gettime() when a timestamps is needed. For TCP and no interval reporting the only calls needed are at the beginning and end of the test. The performance problem we hit with 2.0.5 had to do with insufficient shared memory between traffic threads and the reporter thread.
- jasondclinton 8y ago
  The benchmark results at that link are from machines with different processor clocks. The results are invalid.
ams6110 8y ago
> Just because upstream doesn't have it doesn't mean you couldn't compile it yourself True, but if you are managing production machines, compiling kernels for every errata and security update is pretty low on the list of things you get enthusiastic about. Also a lot of commercial or scientific linux software is only "certified" and supported on stock RHEL kernels.
- cozzyd 8y ago
  There's always http://elrepo.org/tiki/kernel-ml http://elrepo.org/tiki/kernel-ml (and also kernel-lt) so you don't have to compile things yourself. But things like VMWare and various proprietary device drivers will break.
- geofft 8y ago
  On the other hand, if you're making or losing money on latency (e.g., you're a high-frequency trader), compiling kernels for every potential performance improvement is a thing you're enthusiastic about, a lot of security fixes (local privilege escalations, drivers for desktop hardware, etc.) don't apply to you, and you don't care about things being "certified" because you have the expertise to fix problems in-house as necessary. And from the other direction, if you're making or losing money on security, expecting distro kernels to turn around security fixes in a timely manner seems like a mistake. At least know how to rebuild your distro kernel with a local patch.
- rb2k_ 8y ago
  The amount of filesystem/networking issues I've seen fixed and all the performance improvements I got from just updating a kernel probably make up for the time that it caused issues. I guess it needs a critical mass + certain infrastructure maturity that running on whatever the latest kernel is makes sense. As for the 'certified on stock RHEL kernel' software... I try to keep away from those, but I'm picky and pretty lucky when it comes to job selection :)
fulafel 8y ago
It's also incorrect even for LTS supported kernels: LTS releases incorporate new kernel versions along the way as part of the hardware enablement work, though they are not used by default. For example, here's 4.13 in the previous LTS (16.04), up from the default 4.4: http://packages.ubuntu.com/linux-signed-image-generic-hwe-16.04 http://packages.ubuntu.com/linux-signed-image-generic-hwe-16...

sliken 8y ago

The post seems to be confusing FreeBSD userspace (which OSX adopted parts of) and the FreeBSD kernel (which has the mentioned lower network latency). OSX didn't adopt the FreeBSD kernel, but instead started with the Mach microkernel and has been going in their own direction since.

4ad 8y ago
Not that it matters (the post is mostly just unfounded opinion), but: > OSX didn't adopt the FreeBSD kernel, but instead started with the Mach microkernel and has been going in their own direction since. MacOS kernel uses a combination of Mach and BSD kernel code. It started with 4.4BSD, and at some point the bulk of BSD code was updated with code from FreeBSD 5. After that, it was piecewise updated with more modern stuff, the MAC framework (used to implement sandboxing on iOS and macOS) and DTrace from newer FreeBSD kernels.
- hestefisk 8y ago
  There is a great talk on OS X internals from CCC here: https://events.ccc.de/congress/2007/Fahrplan/attachments/1053_inside-macosx-kernel.pdf https://events.ccc.de/congress/2007/Fahrplan/attachments/105... It particularly talks about the unique hybrid kernel approach and how there is nothing like it.
  - loeg 8y ago
    BSD itself took the VM subsystem from Mach (so BSD is kind of an interesting hybrid as well). While it has been heavily modified in the last 25+ years, the VM portion of the FreeBSD kernel is still stylistically distinct from BSD portions of the kernel source code.
trasz 8y ago
Not just userspace - OSX adopted large pieces of FreeBSD kernel, including the syscall layer and the network stack. Also, Mach isn't technically a microkernel - the whole kernel runs in Ring 0. But yeah, OSX kernel is vastly different from FreeBSD - first, there were huge differences to start with, only some pieces of kernel were adopted, and finally there were thousands of man-hours put into each system afterwards, diverging them even more.

snvzz 8y ago

Doesn't Dragonfly (freebsd fork as of 2003) have even lower latency?

hyper2grams 8y ago
DragonFly BSD is a phenomenal project and deserves more recognition imo.

TorKlingberg 8y ago

With such a long post, I would have expected some technical details for why FreeBSD has lower latency, and Linux has faster application speeds.

j16sdiz 8y ago

The benchmark use outdated API. Misleading on distro release a date. No explanation on the improvements. I would vote down this post if I could.

josersanchez 8y ago
You can down vote on quora.
deleted 8y ago
[deleted]

stefan_ 8y ago

Latency and throughput ("application speeds"), certainly in computing, are competing qualities.

api 8y ago

A bit of a tangent but: take another look at FreeBSD folks. If we re-did our infrastructure from scratch we'd look at either FreeBSD or Alpine Linux. Both would offer an escape hatch from the needless bloat that mainstream Linux has become.

hpcjoe 8y ago
Dependency radii of packages in CentOS are horrible. Your minimal installs are hovering above 1GB. Debian are better at 400MB or so. Alpine is great, in that you can get this down to well under 100MB. This has been a long standing complaint of mine[1], and an argument I've had with many proponents of "software must come packaged only from the provider of the distro" people. Physical and logical footprint is a real thing. Ridiculous secondary/tertiary dependencies are a waste of that precious resource, and an unneeded/unwanted installation of useless bits. [1] http://scalability.org/2018/04/distribution-package-dependency-radii-or-why-distros-may-be-doomed/ http://scalability.org/2018/04/distribution-package-dependen...
- mmt 8y ago
  Although I agree with you that this bloat is undesirable, I'm not convinced it's actually a problem with the distro. (My primary experience is with Ubuntu, and to a lesser extent, mostly by extension, Debian) Rather, I think it's a problem with the package maintainers themselves. I certainly admit that for very many packages, this is a distinction without a difference. However, my point is that this dependency bloat isn't based on some kind of policy that the distro is encouraging or even enforcing. Rather, the problem is merely that too many source packages create too few targets, perhaps because the original packagers never foresaw the need (for, e.g., both an X version of some graphics or GPU-related library and a non-X one for pure server work). I've occasionally gotten around this problem by custom-building a local version of the package that avoids the dependency bloat. That violates the "software must come packaged only from the provider of the distro", unless one considers the local superset a forked distro. > Physical and logical footprint is a real thing. Ultimately, I think you and I are in a vanishingly small minority here. Virtualization (even the full, pre-container kind, with a full copy of the kernel on every instance) gained a startling level of a popularity, and was even lionized as a money-saving tool (which, of course, it was, for some "IT" shops).

drewg123 8y ago

Speaking as somebody who works on the FreeBSD kernel at Netflix, the whole article is kind of nonsensical. We don't use FreeBSD because of "latency". We serve clients over https that could be tens or hundreds of ms away using the kernel TCP stack. The folks who care about latency are high frequency traders, who care about every nanosecond. They tend to use userspace / hardware offloaded solutions and do things like busy-waiting for messages.

tiffanyh 8y ago
Drew Have you ever consider using Dragonfly. It appears to dramatically beat in perf test FreeBSD (2017). https://leaf.dragonflybsd.org/~sephe/perf_cmp.pdf https://leaf.dragonflybsd.org/~sephe/perf_cmp.pdf I’m curious to hear your take on Dragonfly since you’re a kernel developer. Thanks in advance for all your work. Edit: more recent links (2018) with even higher perf https://leaf.dragonflybsd.org/~sephe/dfly1.pdf https://leaf.dragonflybsd.org/~sephe/dfly1.pdf https://leaf.dragonflybsd.org/~sephe/dfly1_p.pdf https://leaf.dragonflybsd.org/~sephe/dfly1_p.pdf
- drewg123 8y ago
  I have looked at Dragonfly with interest, due to how they manage VM. I really like how they shard the page queues to be per-cpu. With that said, a lot of work has been done by jeffr in the last 6 months to a year to improve NUMA performance (which we partially sponsored) which has dramatically improved VM scalability even on single socket boxes, and which has allowed us to remove a lot of our local hacks. I considered giving Dragonfly a try in the past, but just never had the time. There would be a depressing amount of work before we could even consider Dragonfly, mostly centered around our async sendfile (which is now upstream in FreeBSD), our TCP changes: BBR (not yet upstream), RACK (now upstream) and TCP pacing (now upstream). Not to mention unmapped mbufs (not yet upstream) and kernel TLS (also not yet upstream). Also, the last time I looked, Dragonfly did not have drivers for Chelsio or Mellanox 10/25/40/100GbE NICs. Without even just one of these things, performance will be so bad that any comparison testing would be meaningless.
  - 8y ago

parvenu74 8y ago

> "There are some 'BSD-inspired' networking improvements to Linux starting with the 4.16." By "inspired by BSD" do the author mean the code has been copied and pasted from FreeBSD to Linux, something which cannot be done in reverse because of the encumbrance of the GPL?

zeth___ 8y ago
BSD is free to adopt the GPL at their leisure. Who knows, companies might then start contributing to the project instead of building OS's on top of it, looking at you osx.
- parvenu74 8y ago
  GPL supporters are all about freedom -- until someone advocates for the freedom to make a proprietary product on top of the open source base.
  - cozzyd 8y ago
    Yes... that is the point.
    - parvenu74 8y ago
      Oh I forgot -- at HN you never question the GPL. And FreeBSD sux because it's not GPL. Got it. I'll stop wasting my time even mentioning it again.
      - jhasse 8y ago
        You didn't question the GPL, you just explained how it works.
  - bonzini 8y ago
    More specifically it's about freedom for all users, transitively.

vermaden 8y ago

FreeBSD was used in 1999 to render The Matrix on 32 Pentium II boxes because the software in Linux Compatibility mode on FreeBSD was faster then natively on Linux, that is a fact: https://www.freebsd.org/news/press-rel-1.html https://www.freebsd.org/news/press-rel-1.html FreeBSD can be several times faster then Linux when it comes to network stack: https://pbs.twimg.com/media/CzFfTSRUQAATwaq.jpg https://pbs.twimg.com/media/CzFfTSRUQAATwaq.jpg But often Linux is faster, you just need to find benchmark that favorites one or another, both are fast in general.

ajross 8y ago
Why on earth would you link to a Phoronix screenshot instead of the article that explains it?! Those results look suspicious enough[1] that I wanted to look it up. But of course I can't, and Google isn't helping me much. Shame! Shame, shame, shame! [1] Factor of 3+ differences not just between BSD and Linux but between otherwise very comparable linux distros. Something's weird with the setup there, like it's measuring default firewalling overhead or something and not kernel behavior.
- loeg 8y ago
  > Phoronix > Something's weird with the setup there, like it's measuring default firewalling overhead or something and not kernel behavior. This is something you come to expect if you've seen enough of what Phoronix publishes.
- buzer 8y ago
  Looks like source is https://www.phoronix.com/scan.php?page=article&item=netperf-bsd-linux&num=1 https://www.phoronix.com/scan.php?page=article&item=netperf-... I guess they have everything just set to default disto settings which likely in some cases include firewall while others don't. They also don't mention which NIC they are using, the motherboard has two different NICs (i218-LM & I210-AT).
  - jasondclinton 8y ago
    Those benchmarks were run on machines with different processor clocks and different motherboard chipsets. The results are invalid.

HippoBaro 8y ago

Woudn't anyone really into IP latency optimization be using something like DPDK on Linux? If you want high-performance networking, then using the kernel isn't the best idea IMO.

rjmcmahon 8y ago

Well, here's just one measurement using iperf 2.0.12. All clocks are synchronized to a GPS disciplined oven controlled oscillator (OCXO). Connected via 1Gbs. FreeBSD latency is significantly better. Source fedora 28 xeon machine, receivers FreeBSD 11.1 and Fedora 25 (same brix platform) connected via a Cisco 300 switch. RX: FreeBSD: root@zeus:/usr/local/src/iperf2-code # iperf -s -u -e --udp-histogram=10u,100000 --realtime ------------------------------------------------------------ Server listening on UDP port 5001 with pid 53703 Receiving 1470 byte datagrams UDP buffer size: 41.1 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.100.34 port 5001 connected with 192.168.100.55 port 52536 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Latency avg/min/max/stdev PPS [ 3] 0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 0.004 ms 0/ 892 (0%) 0.086/ 0.060/ 0.150/ 0.014 ms 89 pps [ 3] 0.00-10.00 sec T8(f)-PDF: bin(w=10us):cnt(892)=7:160,8:143,9:181,10:159,11:242,12:6,16:1 (5/95%=7/11,Outliers=0,obl/obu=0/0) Fedora 25: [root@hera iperf2-code]# iperf -s -u -e --udp-histogram=10u,10000 --realtime ------------------------------------------------------------ Server listening on UDP port 5001 with pid 16669 Receiving 1470 byte datagrams UDP buffer size: 208 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.100.33 port 5001 connected with 192.168.100.55 port 35894 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Latency avg/min/max/stdev PPS [ 3] 0.00-9.99 sec 1.25 MBytes 1.05 Mbits/sec 0.010 ms 0/ 892 (0%) 0.261/ 0.098/ 0.319/ 0.021 ms 89 pps [ 3] 0.00-9.99 sec T8(f)-PDF: bin(w=10us):cnt(892)=10:1,21:1,23:6,24:107,25:141,26:209,27:151,28:124,29:81,30:47,31:19,32:5 (5/95%=24/30,Outliers=0,obl/obu=0/0) TX: [root@rjm-clubhouse-28 rjmcmahon]# iperf -s -e -u --udp-histogram=10u,100000 --realtime ------------------------------------------------------------ Server listening on UDP port 5001 with pid 26016 Receiving 1470 byte datagrams UDP buffer size: 208 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.100.55 port 5001 connected with 192.168.100.34 port 20343 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Latency avg/min/max/stdev PPS [ 3] 0.00-10.01 sec 1.25 MBytes 1.05 Mbits/sec 0.010 ms 0/ 893 (0%) 0.094/ 0.064/ 0.476/ 0.023 ms 89 pps [ 3] 0.00-10.01 sec T8(f)-PDF: bin(w=10us):cnt(893)=7:12,8:38,9:291,10:311,11:198,12:28,13:10,14:1,15:1,42:1,45:1,48:1 (5/95%=8/11,Outliers=3,obl/obu=0/0) [ 4] local 192.168.100.55 port 5001 connected with 192.168.100.33 port 58391 [ 4] 0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 0.013 ms 0/ 892 (0%) 0.079/ 0.048/ 0.127/ 0.009 ms 89 pps [ 4] 0.00-10.00 sec T8(f)-PDF: bin(w=10us):cnt(892)=5:1,6:10,7:115,8:265,9:425,10:51,11:18,12:3,13:4 (5/95%=7/10,Outliers=0,obl/obu=0/0)

ksec 8y ago

I reckon if someone make Java a first class citizen on FreeBSD, it may actually get a larger amount of usage.