4 ms·

Using the new bridges of FreeBSD 15

113 points by vermaden 8mo ago

shashasha2 8mo ago
Bhyve bridges are inefficient: every packet traverses NIC → CPU → bridge → VM, adding unnecessary copies that kill throughput. Switching to SR-IOV eliminated that overhead and I saturated the 10 GbE link.
- kev009 8mo ago
  It would benefit from a batching mechanism.
- assimpleaspossi 8mo ago
  You used the new optimized bridges on FreeBSD 15?
  - crest 8mo ago
    The bridge driver gained features (vlan filtering) not performance.
- gigatexal 8mo ago
  On Linux?
- Veserv 8mo ago
  I do not see how that follows. Memory bandwidth is measured in the hundreds of Gb/s. You can issue tens of unnecessary full memory copies before you bottleneck at a paltry 10 Gb/s. It is much more likely there is something else terribly wrong in a network stack if it can not even drive a measly 10 Gb/s.
  - stingraycharles 8mo ago
    That assumes memory bandwidth is the issue, and not latency and/or CPU.
    - simoncion 8mo ago
      My stupid Zen 3 Frankenrouter absolutely saturates both directions of a 10Gbit symmetric link, and it's using Linux software bridges, software firewalling, and software routing. At ~400usec at idle, latency is low, but twice that of a host system that has no software bridges. [0] Some tiny, underpowered ARM box wouldn't have the power to do all that in software, but you're not going to be running VMs on a tiny, underpowered ARM box. [0] However, the fully-loaded latency is far better than the system with no software bridges; ~1200usec vs ~7200usec. One might conclude that factors other than the software bridges, firewalls, and routing are the significant components of the latency figures.
ggm 8mo ago
When this settles down, I look forward to all of jail/iojail, Sylve, Bastille, Bhyve documenting this in a mutually consistent manner. As it stands, I have managed to completely knot my brain over the abstractions, what is happening. It's me, not the systems, but I think there is a little bit of "meh, I understand it, so it must be obvious to anyone smart" going on, and alas, I am not smart, and I get confused easily. I'm in bastille atm, but have been in all of them and TrueNAS core. and libvirt over on the other unix.
- _0xdd 8mo ago
  Honestly, best thing I did was ditch all that and just read the handbook, specificially chapter 17 [1]. All of my jails are now set up manually, initially using /etc/jail.conf, and now individual jail configs in /etc/jail.conf.d/. I still use vm-bhyve [2] for my Bhyve virtual machines, but that's been rock solid for me for years. [1] https://docs.freebsd.org/en/books/handbook/jails/ https://docs.freebsd.org/en/books/handbook/jails/ [2] https://github.com/freebsd/vm-bhyve https://github.com/freebsd/vm-bhyve
waynesonfire 8mo ago
> -tso4 -tso6 -vlanhwfilter -vlanmtu -vlanhwtso -vlanhwtag -vlanhwcsum -lro Whys the author disabling tso and lro? Whats the motivation? I'm not familiar with the other flags.
- j16sdiz 8mo ago
  Looks like TSO does not support VLAN. Not sure about lro.
- kev009 8mo ago
  People found this worked in the past and it gets copied around. There is no reason to disable some of this. Bridge will automatically disable LRO and find the common set of other offloads. TSO is not useful for a bridged guest.
- crest 8mo ago
  LRO because the bridge has to forward the real frames. TSO because it’s fairly useless now.
- rballpug 8mo ago
  TCP is able to use server port 53 (decimal) for two-byte low-level intervals between 2-5 seconds. What is subdomain label in lro?
bzmrgonz 8mo ago
I for one welcome and applaud any progress on the bsd front,and this seems to be huge.
j16sdiz 8mo ago
Why sudden surge of FreeBSD-related posts? Did anything special or new happened on FreeBSD land?
- kev009 8mo ago
  15.0 was released a couple months ago, hence the title.
  - j16sdiz 8mo ago
    We have three (including this) FreeBSD posts in the past two days. Back to FreeBSD: Part 1 (hypha.pub) https://news.ycombinator.com/item?id=47108989 https://news.ycombinator.com/item?id=47108989 Linuxulator on FreeBSD Feels Like Magic (hayzam.com) https://news.ycombinator.com/item?id=47113527 https://news.ycombinator.com/item?id=47113527
    - kev009 8mo ago
      Generally people get more excited any time a major release of anything comes out. But FWIW HN has always had favorable front paging for anything related to FreeBSD and OpenBSD.
      - shevy-java 8mo ago
        Not disagreeing, but if the release was a few months ago, then the poster is quite correct - there is a recent "surge" of FreeBSD related posts. And these are not quite ... how shall I word it somewhat nicely ... not quite as fascinating to, say, a wider linux community as such. With that I don't mean "because we use linux, we are snobs", but that what the FreeBSD guys talk about, seems a little bit ... outdated. The heavy use of shell scripts for instance in this post here - I never understood that focus on shell scripts in general, including on Linux. I transitioned into using ruby (or python, but mostly ruby) to replace all shell script needs a long time ago. Every time I am assumed to have to write a shell script I wonder why I would want to cripple myself when I can use a better programming language instead. Many of these shown "innovations" are also not really groundbreaking. To me it seems as if there is a distinct lack of FreeBSD users out there compared to Linux users. As a consequence Linux simply has a lot more information and news (a lot of which is also low quality of course; I am not saying it is all pancakes and sunflowers in the Linux ecosystem either).
simoncion 8mo ago
From TFA: They have also soft-deprecated the ability to have any layer 3 addresses on member interfaces which makes it behave like a real hardware switch. The net.link.bridge.member_ifaddrs sysctl controls this behavior and it will be removed in FreeBSD 16.0-RELEASE, same as if set to zero. I'm a little bit uncertain. This means that the bridge may have one or more L3 addresses assigned to it, but the interfaces attached to that bridge may not, right? If that's right, how does that interact with things like Linux's veth pairs? [0] Can the half of the pair that's not a member of the bridge have an IP address? [0] I assume something like that exists in FreeBSD-land.

113 points by vermaden 8mo ago

Bhyve bridges are inefficient: every packet traverses NIC → CPU → bridge → VM, adding unnecessary copies that kill throughput. Switching to SR-IOV eliminated that overhead and I saturated the 10 GbE link.

kev009 8mo ago
It would benefit from a batching mechanism.
assimpleaspossi 8mo ago
You used the new optimized bridges on FreeBSD 15?
- crest 8mo ago
  The bridge driver gained features (vlan filtering) not performance.
gigatexal 8mo ago
On Linux?
Veserv 8mo ago
I do not see how that follows. Memory bandwidth is measured in the hundreds of Gb/s. You can issue tens of unnecessary full memory copies before you bottleneck at a paltry 10 Gb/s. It is much more likely there is something else terribly wrong in a network stack if it can not even drive a measly 10 Gb/s.
- stingraycharles 8mo ago
  That assumes memory bandwidth is the issue, and not latency and/or CPU.
  - simoncion 8mo ago
    My stupid Zen 3 Frankenrouter absolutely saturates both directions of a 10Gbit symmetric link, and it's using Linux software bridges, software firewalling, and software routing. At ~400usec at idle, latency is low, but twice that of a host system that has no software bridges. [0] Some tiny, underpowered ARM box wouldn't have the power to do all that in software, but you're not going to be running VMs on a tiny, underpowered ARM box. [0] However, the fully-loaded latency is far better than the system with no software bridges; ~1200usec vs ~7200usec. One might conclude that factors other than the software bridges, firewalls, and routing are the significant components of the latency figures.

ggm 8mo ago

When this settles down, I look forward to all of jail/iojail, Sylve, Bastille, Bhyve documenting this in a mutually consistent manner. As it stands, I have managed to completely knot my brain over the abstractions, what is happening. It's me, not the systems, but I think there is a little bit of "meh, I understand it, so it must be obvious to anyone smart" going on, and alas, I am not smart, and I get confused easily. I'm in bastille atm, but have been in all of them and TrueNAS core. and libvirt over on the other unix.

_0xdd 8mo ago
Honestly, best thing I did was ditch all that and just read the handbook, specificially chapter 17 [1]. All of my jails are now set up manually, initially using /etc/jail.conf, and now individual jail configs in /etc/jail.conf.d/. I still use vm-bhyve [2] for my Bhyve virtual machines, but that's been rock solid for me for years. [1] https://docs.freebsd.org/en/books/handbook/jails/ https://docs.freebsd.org/en/books/handbook/jails/ [2] https://github.com/freebsd/vm-bhyve https://github.com/freebsd/vm-bhyve

waynesonfire 8mo ago

> -tso4 -tso6 -vlanhwfilter -vlanmtu -vlanhwtso -vlanhwtag -vlanhwcsum -lro Whys the author disabling tso and lro? Whats the motivation? I'm not familiar with the other flags.

j16sdiz 8mo ago
Looks like TSO does not support VLAN. Not sure about lro.
kev009 8mo ago
People found this worked in the past and it gets copied around. There is no reason to disable some of this. Bridge will automatically disable LRO and find the common set of other offloads. TSO is not useful for a bridged guest.
crest 8mo ago
LRO because the bridge has to forward the real frames. TSO because it’s fairly useless now.
rballpug 8mo ago
TCP is able to use server port 53 (decimal) for two-byte low-level intervals between 2-5 seconds. What is subdomain label in lro?

bzmrgonz 8mo ago

I for one welcome and applaud any progress on the bsd front,and this seems to be huge.

j16sdiz 8mo ago

Why sudden surge of FreeBSD-related posts? Did anything special or new happened on FreeBSD land?

kev009 8mo ago
15.0 was released a couple months ago, hence the title.
- j16sdiz 8mo ago
  We have three (including this) FreeBSD posts in the past two days. Back to FreeBSD: Part 1 (hypha.pub) https://news.ycombinator.com/item?id=47108989 https://news.ycombinator.com/item?id=47108989 Linuxulator on FreeBSD Feels Like Magic (hayzam.com) https://news.ycombinator.com/item?id=47113527 https://news.ycombinator.com/item?id=47113527
  - kev009 8mo ago
    Generally people get more excited any time a major release of anything comes out. But FWIW HN has always had favorable front paging for anything related to FreeBSD and OpenBSD.
    - shevy-java 8mo ago
      Not disagreeing, but if the release was a few months ago, then the poster is quite correct - there is a recent "surge" of FreeBSD related posts. And these are not quite ... how shall I word it somewhat nicely ... not quite as fascinating to, say, a wider linux community as such. With that I don't mean "because we use linux, we are snobs", but that what the FreeBSD guys talk about, seems a little bit ... outdated. The heavy use of shell scripts for instance in this post here - I never understood that focus on shell scripts in general, including on Linux. I transitioned into using ruby (or python, but mostly ruby) to replace all shell script needs a long time ago. Every time I am assumed to have to write a shell script I wonder why I would want to cripple myself when I can use a better programming language instead. Many of these shown "innovations" are also not really groundbreaking. To me it seems as if there is a distinct lack of FreeBSD users out there compared to Linux users. As a consequence Linux simply has a lot more information and news (a lot of which is also low quality of course; I am not saying it is all pancakes and sunflowers in the Linux ecosystem either).

simoncion 8mo ago

From TFA: They have also soft-deprecated the ability to have any layer 3 addresses on member interfaces which makes it behave like a real hardware switch. The net.link.bridge.member_ifaddrs sysctl controls this behavior and it will be removed in FreeBSD 16.0-RELEASE, same as if set to zero. I'm a little bit uncertain. This means that the bridge may have one or more L3 addresses assigned to it, but the interfaces attached to that bridge may not, right? If that's right, how does that interact with things like Linux's veth pairs? [0] Can the half of the pair that's not a member of the bridge have an IP address? [0] I assume something like that exists in FreeBSD-land.