7 ms·

I always heard that FreeBSD has unparalleled networking Does it mean that it'd be worth picking FreeBSD over Linux for my C# crud app if it had to handle a lot

I always heard that FreeBSD has unparalleled networking Does it mean that it'd be worth picking FreeBSD over Linux for my C# crud app if it had to handle a lot of requests/sec? (let's ignore db for the moment)

deleted 5y ago
[deleted]
kd913 5y ago
I don't think it would be worthwhile. If you want better latency and throughput, you wouldn't be using the kernel network stack and instead be opting for some userspace networking stack like DPDK or onload. Depends obviously on what the bottlenecks of your application are, your NIC and the characteristics of your hardware as well.
- crest 5y ago
  FreeBSD has netmap for fast userspace packet processing.
  - kd913 5y ago
    True, and the Linux kernel has zero-copy AF_XDP that enable memory to be shared with userspace. However, low-latency networking is a lot more than just simple kernel bypass. It's things like pinning cpu cores dedicated for networking, disabling C-states, epolling and being able to utilize bespoke firmware interfaces designed for smartnics. Also application protocol, ie using features like TCP checksum offload and TSO. Heck the application would also need to be adjusted for a low-latency environment via probably a custom JVM and doing things like reading data structures/variables to ensure they are in CPU cache. Frankly I would recommend trying openonload which at least is compatible with native Linux socket programming unlike DPDK.
- almalkemqq 5y ago
  I loved the simple explanation :)
toast0 5y ago
As with all things, you would really need to benchmark the system, preferably with real load, both ways to know for sure. But that takes a lot of time, especially if you're going to put in the time to tweak both systems. People can do amazing stuff with enough time in both FreeBSD and Linux. I honestly think most server applications wouldn't be held back by either OS. You need your application to be really lightweight and focused before the OS makes a big difference, and even then, the differences only show if you're maxing out the hardware. I worked at WhatsApp, and enjoyed working with FreeBSD there, and clearly it worked for us. Linux in FB datacenters also worked, but the server components were a lot different so there was never an apples to apples comparison. I run FreeBSD on my personal servers because I enjoyed working with it at Yahoo and then WhatsApp; but my personal servers don't have any performance needs. Sure, the networking stuff is nice (and it was nice to work with in the kernel), but what I like most about FreeBSD is the lack of churn. I can look at old administrative recipies and all the commands still work. I can expect (and mostly get) that when I upgrade, everything will keep working, and maybe a little better; occassionally, a lot better.
- waynesonfire 5y ago
  > but what I like most about FreeBSD is the lack of churn I agree completely. As I mature in this field this becomes an ever important characteristic of the technology I adopt. Erlang also shares this property.
  - toast0 5y ago
    I feel like Erlang churn is more acceptable because they're usually honest in naming. You get pg2 which does what pg did, but differntly. And now pg (which could have been pg3, but the old pg had been gone long enough to reclaim the name). Similar with phash2, although phash may live for much longer.
- tiffanyh 5y ago
  Hi toast0 IIRC, I saw a presentation by someone (Rick?) where at your previous employer - you guys slimmed FreeBSD down to be unbelievably minimal such that only 2-3 total services ran on the entire server. Was that done for performance reason? Or for hardening reasons? If someone wanted to do that today with FreeBSD: would you recommend it and how would you go about doing it (NanoBSD)?
adrian_b 5y ago
There was a time when FreeBSD had much better networking performance than Linux, but that was many years ago. Now, on supported hardware, their performance should be similar. However, Linux has drivers for much more varied networking hardware. FreeBSD has very good support for Intel NIC's but some of the hardware from other vendors may happen to be not supported. FreeBSD has a few nicer kernel features for those who develop themselves a networking application, but more networking libraries useful for high performance applications are available for Linux, even if DPDK, which was mentioned in another reply, is available for both Linux and FreeBSD. So, while I am a very satisfied FreeBSD user, I would recommend to someone with less experience to use Linux, as there are more resources readily available. On the other hand, for someone who wants to learn more about the implementation of networking applications, it can be useful to also try FreeBSD, to understand more about alternative solutions.
0xdeadb00f 5y ago
FreeBSD (and all BSDs for that matter) have pretty good networking stacks besides any new recent-gen WiFi.
deleted 5y ago
[deleted]
loeg 5y ago
C# support isn't great on FreeBSD yet, so probably not.
- tester756 5y ago
  but if we assume that it works fine? AFAIK there's ongoing and active work for FreeBSD https://github.com/dotnet/runtime/issues/14537 https://github.com/dotnet/runtime/issues/14537
  - loeg 5y ago
    That issue has been continuously open since 2015. Assuming it works, sure, but I would not hold my breath.

deleted 5y ago

[deleted]

kd913 5y ago

I don't think it would be worthwhile. If you want better latency and throughput, you wouldn't be using the kernel network stack and instead be opting for some userspace networking stack like DPDK or onload. Depends obviously on what the bottlenecks of your application are, your NIC and the characteristics of your hardware as well.

crest 5y ago
FreeBSD has netmap for fast userspace packet processing.
- kd913 5y ago
  True, and the Linux kernel has zero-copy AF_XDP that enable memory to be shared with userspace. However, low-latency networking is a lot more than just simple kernel bypass. It's things like pinning cpu cores dedicated for networking, disabling C-states, epolling and being able to utilize bespoke firmware interfaces designed for smartnics. Also application protocol, ie using features like TCP checksum offload and TSO. Heck the application would also need to be adjusted for a low-latency environment via probably a custom JVM and doing things like reading data structures/variables to ensure they are in CPU cache. Frankly I would recommend trying openonload which at least is compatible with native Linux socket programming unlike DPDK.
almalkemqq 5y ago
I loved the simple explanation :)

toast0 5y ago

As with all things, you would really need to benchmark the system, preferably with real load, both ways to know for sure. But that takes a lot of time, especially if you're going to put in the time to tweak both systems. People can do amazing stuff with enough time in both FreeBSD and Linux. I honestly think most server applications wouldn't be held back by either OS. You need your application to be really lightweight and focused before the OS makes a big difference, and even then, the differences only show if you're maxing out the hardware. I worked at WhatsApp, and enjoyed working with FreeBSD there, and clearly it worked for us. Linux in FB datacenters also worked, but the server components were a lot different so there was never an apples to apples comparison. I run FreeBSD on my personal servers because I enjoyed working with it at Yahoo and then WhatsApp; but my personal servers don't have any performance needs. Sure, the networking stuff is nice (and it was nice to work with in the kernel), but what I like most about FreeBSD is the lack of churn. I can look at old administrative recipies and all the commands still work. I can expect (and mostly get) that when I upgrade, everything will keep working, and maybe a little better; occassionally, a lot better.

waynesonfire 5y ago
> but what I like most about FreeBSD is the lack of churn I agree completely. As I mature in this field this becomes an ever important characteristic of the technology I adopt. Erlang also shares this property.
- toast0 5y ago
  I feel like Erlang churn is more acceptable because they're usually honest in naming. You get pg2 which does what pg did, but differntly. And now pg (which could have been pg3, but the old pg had been gone long enough to reclaim the name). Similar with phash2, although phash may live for much longer.
tiffanyh 5y ago
Hi toast0 IIRC, I saw a presentation by someone (Rick?) where at your previous employer - you guys slimmed FreeBSD down to be unbelievably minimal such that only 2-3 total services ran on the entire server. Was that done for performance reason? Or for hardening reasons? If someone wanted to do that today with FreeBSD: would you recommend it and how would you go about doing it (NanoBSD)?

adrian_b 5y ago

There was a time when FreeBSD had much better networking performance than Linux, but that was many years ago. Now, on supported hardware, their performance should be similar. However, Linux has drivers for much more varied networking hardware. FreeBSD has very good support for Intel NIC's but some of the hardware from other vendors may happen to be not supported. FreeBSD has a few nicer kernel features for those who develop themselves a networking application, but more networking libraries useful for high performance applications are available for Linux, even if DPDK, which was mentioned in another reply, is available for both Linux and FreeBSD. So, while I am a very satisfied FreeBSD user, I would recommend to someone with less experience to use Linux, as there are more resources readily available. On the other hand, for someone who wants to learn more about the implementation of networking applications, it can be useful to also try FreeBSD, to understand more about alternative solutions.

0xdeadb00f 5y ago

FreeBSD (and all BSDs for that matter) have pretty good networking stacks besides any new recent-gen WiFi.

deleted 5y ago

[deleted]

loeg 5y ago

C# support isn't great on FreeBSD yet, so probably not.

tester756 5y ago
but if we assume that it works fine? AFAIK there's ongoing and active work for FreeBSD https://github.com/dotnet/runtime/issues/14537 https://github.com/dotnet/runtime/issues/14537
- loeg 5y ago
  That issue has been continuously open since 2015. Assuming it works, sure, but I would not hold my breath.