4 ms·

Speaking as somebody who works on the FreeBSD kernel at Netflix, the whole article is kind of nonsensical. We don't use FreeBSD because of "latency". We serv

Speaking as somebody who works on the FreeBSD kernel at Netflix, the whole article is kind of nonsensical. We don't use FreeBSD because of "latency". We serve clients over https that could be tens or hundreds of ms away using the kernel TCP stack. The folks who care about latency are high frequency traders, who care about every nanosecond. They tend to use userspace / hardware offloaded solutions and do things like busy-waiting for messages.

tiffanyh 8y ago
Drew Have you ever consider using Dragonfly. It appears to dramatically beat in perf test FreeBSD (2017). https://leaf.dragonflybsd.org/~sephe/perf_cmp.pdf https://leaf.dragonflybsd.org/~sephe/perf_cmp.pdf I’m curious to hear your take on Dragonfly since you’re a kernel developer. Thanks in advance for all your work. Edit: more recent links (2018) with even higher perf https://leaf.dragonflybsd.org/~sephe/dfly1.pdf https://leaf.dragonflybsd.org/~sephe/dfly1.pdf https://leaf.dragonflybsd.org/~sephe/dfly1_p.pdf https://leaf.dragonflybsd.org/~sephe/dfly1_p.pdf
- drewg123 8y ago
  I have looked at Dragonfly with interest, due to how they manage VM. I really like how they shard the page queues to be per-cpu. With that said, a lot of work has been done by jeffr in the last 6 months to a year to improve NUMA performance (which we partially sponsored) which has dramatically improved VM scalability even on single socket boxes, and which has allowed us to remove a lot of our local hacks. I considered giving Dragonfly a try in the past, but just never had the time. There would be a depressing amount of work before we could even consider Dragonfly, mostly centered around our async sendfile (which is now upstream in FreeBSD), our TCP changes: BBR (not yet upstream), RACK (now upstream) and TCP pacing (now upstream). Not to mention unmapped mbufs (not yet upstream) and kernel TLS (also not yet upstream). Also, the last time I looked, Dragonfly did not have drivers for Chelsio or Mellanox 10/25/40/100GbE NICs. Without even just one of these things, performance will be so bad that any comparison testing would be meaningless.
  - tiffanyh 8y ago
    Thanks for the thoughtful reply. Just a thought, the Linux/BSD community might hugely benefit if someone with your deep knowledge released a set of perf test scripts that anyone can run locally to regression test network perf. That way, the OSS community can integrate those perf test scripts into their commit/regression test pipeline.
    - toast0 8y ago
      It would be great to have this kind of test, but often the regression test is going to be run the system on production load. Benchmarks have a way of testing something, but not quite what you need to test. The interactions between the whole system are important. For the Netflix boxes, they're pushing 40gbps+, not a lot of the community is going to be to be able to test that, unless they have fairly expensive networks laying around.
      - drewg123 8y ago
        You're spot on. We don't really have good benchmarks that approximate our production traffic. BTW, 40Gb/s was so 2015. I have a box serving at 156Gb/s :)
        jacquesm 8y ago
        As someone who grew up around Telex machines and current loop interfaces: consider me impressed!
        toast0 8y ago
        > BTW, 40Gb/s was so 2015. I have a box serving at 156Gb/s :) That's right, I wanted to put 100+, but I wasn't totally sure. I stopped counting when you got way beyond the 20G connectivity on the servers I manage.
        ksec 8y ago
        Oh mine! I record there was an article about pushing 100Gb/s not long ago and hitting memory bandwidth limit. Now it is 150Gb/s already ? Are you guys going to try something crazy like 400Gb/s ? At this rate Netflix could start selling their Appliance as another business.
kierenj 8y ago
If I may ask, why do you use FreeBSD in particular then?
- mobilemidget 8y ago
  Native (properly working) ZFS, I don't need any other reason :) It works on linux, but to be able too boot from it, still a huge hassle.
  - vbezhenar 8y ago
    I tried to intall Ubuntu into root ZFS and it wasn't much of a hassle, just simple howto, everything worked.
    - xenophonf 8y ago
      I've been running root on ZFS (GELI- or LUKS-encrypted, no less) since FreeBSD 7 and Ubuntu 12, and to be frank the story on Linux just isn't that great. Documentation for root on ZFS only exists for Arch, Debian, Gentoo, and Ubuntu, with only Debian and Ubuntu having documentation for encrypted root ZFS pools. Last I checked, not a single Linux distribution's installer supported ZFS, so if you want root on ZFS, you must install Linux by hand. That's great if you have unlimited time, but it sucks for any kind of deployment at scale. Contrast with FreeBSD, with first-class support for installing root on ZFS since 10.0-RELEASE, four years ago.
    - drb91 8y ago
      I’m going go out on a limb abd speculate there might be different hurdles for a single installation vs many installations.
    - kzrdude 8y ago
      Does it have he same level of QA?
deleted 8y ago
[deleted]

tiffanyh 8y ago

Drew Have you ever consider using Dragonfly. It appears to dramatically beat in perf test FreeBSD (2017). https://leaf.dragonflybsd.org/~sephe/perf_cmp.pdf https://leaf.dragonflybsd.org/~sephe/perf_cmp.pdf I’m curious to hear your take on Dragonfly since you’re a kernel developer. Thanks in advance for all your work. Edit: more recent links (2018) with even higher perf https://leaf.dragonflybsd.org/~sephe/dfly1.pdf https://leaf.dragonflybsd.org/~sephe/dfly1.pdf https://leaf.dragonflybsd.org/~sephe/dfly1_p.pdf https://leaf.dragonflybsd.org/~sephe/dfly1_p.pdf

drewg123 8y ago
I have looked at Dragonfly with interest, due to how they manage VM. I really like how they shard the page queues to be per-cpu. With that said, a lot of work has been done by jeffr in the last 6 months to a year to improve NUMA performance (which we partially sponsored) which has dramatically improved VM scalability even on single socket boxes, and which has allowed us to remove a lot of our local hacks. I considered giving Dragonfly a try in the past, but just never had the time. There would be a depressing amount of work before we could even consider Dragonfly, mostly centered around our async sendfile (which is now upstream in FreeBSD), our TCP changes: BBR (not yet upstream), RACK (now upstream) and TCP pacing (now upstream). Not to mention unmapped mbufs (not yet upstream) and kernel TLS (also not yet upstream). Also, the last time I looked, Dragonfly did not have drivers for Chelsio or Mellanox 10/25/40/100GbE NICs. Without even just one of these things, performance will be so bad that any comparison testing would be meaningless.
- tiffanyh 8y ago
  Thanks for the thoughtful reply. Just a thought, the Linux/BSD community might hugely benefit if someone with your deep knowledge released a set of perf test scripts that anyone can run locally to regression test network perf. That way, the OSS community can integrate those perf test scripts into their commit/regression test pipeline.
  - toast0 8y ago
    It would be great to have this kind of test, but often the regression test is going to be run the system on production load. Benchmarks have a way of testing something, but not quite what you need to test. The interactions between the whole system are important. For the Netflix boxes, they're pushing 40gbps+, not a lot of the community is going to be to be able to test that, unless they have fairly expensive networks laying around.
    - drewg123 8y ago
      You're spot on. We don't really have good benchmarks that approximate our production traffic. BTW, 40Gb/s was so 2015. I have a box serving at 156Gb/s :)
      - jacquesm 8y ago
        As someone who grew up around Telex machines and current loop interfaces: consider me impressed!
      - toast0 8y ago
        > BTW, 40Gb/s was so 2015. I have a box serving at 156Gb/s :) That's right, I wanted to put 100+, but I wasn't totally sure. I stopped counting when you got way beyond the 20G connectivity on the servers I manage.
      - ksec 8y ago
        Oh mine! I record there was an article about pushing 100Gb/s not long ago and hitting memory bandwidth limit. Now it is 150Gb/s already ? Are you guys going to try something crazy like 400Gb/s ? At this rate Netflix could start selling their Appliance as another business.

kierenj 8y ago

If I may ask, why do you use FreeBSD in particular then?

mobilemidget 8y ago
Native (properly working) ZFS, I don't need any other reason :) It works on linux, but to be able too boot from it, still a huge hassle.
- vbezhenar 8y ago
  I tried to intall Ubuntu into root ZFS and it wasn't much of a hassle, just simple howto, everything worked.
  - xenophonf 8y ago
    I've been running root on ZFS (GELI- or LUKS-encrypted, no less) since FreeBSD 7 and Ubuntu 12, and to be frank the story on Linux just isn't that great. Documentation for root on ZFS only exists for Arch, Debian, Gentoo, and Ubuntu, with only Debian and Ubuntu having documentation for encrypted root ZFS pools. Last I checked, not a single Linux distribution's installer supported ZFS, so if you want root on ZFS, you must install Linux by hand. That's great if you have unlimited time, but it sucks for any kind of deployment at scale. Contrast with FreeBSD, with first-class support for installing root on ZFS since 10.0-RELEASE, four years ago.
  - drb91 8y ago
    I’m going go out on a limb abd speculate there might be different hurdles for a single installation vs many installations.
  - kzrdude 8y ago
    Does it have he same level of QA?

deleted 8y ago

[deleted]