3 ms·

Been running Bhyve on FreeBSD (technically FreeNAS). Found PCIe pass-through of NMVe drives was fairly straight forward once the correct incantations were found

Been running Bhyve on FreeBSD (technically FreeNAS). Found PCIe pass-through of NMVe drives was fairly straight forward once the correct incantations were found, but network speed to host has been fairly abysmal. On my admittedly aging Threadripper 1920X, I can only get ~2-3 Gbps peak from a Linux guest. That's with virtio, the virtual intel "card" is even slower. They went with Illumos though, so curious if the poor performance is a FreeBSD-specific thing.

bitfilped 2y ago
It's been a minute since I messed with bhyve on FreeBSD, but I'm pretty sure you have to switch out the networking stack to something like Netgraph if you intend to use fast networking.
- craftkiller 2y ago
  Hmmm I'm not the OP, but I run my personal site on a kubernetes cluster hosted in bhyve VMs running Debian on a FreeBSD machine using netgraph for the networking. I just tested by launching iperf3 on the FreeBSD host and launching an alpine linux pod in the cluster, and I only got ~4Gbit/s. This is surprising to me since netgraph is supposed to be capable of much faster networking but I guess this is going through multiple additional layers that may have slowed it down (off the top of my head: kubernetes with flannel, iptables in the VM, bhyve, and pf on the FreeBSD host).
  - bitfilped 2y ago
    Do you know if you're still using if_bridge? I remembered this article from klara that goes a bit more into the details. https://klarasystems.com/articles/using-netgraph-for-freebsds-bhyve-networking/ https://klarasystems.com/articles/using-netgraph-for-freebsd...
    - craftkiller 2y ago
      Thanks, but I am not using if_bridge. I am creating a netgraph bridge[0] which is connected to the host via a netgraph eiface. Then on the host, the packet passes to my real physical interface because I have gateway_enable and I have pf perform NAT[1]. It looks like that blog post connected the netgraph bridge directly to the external interface, so my guess is my slowdown is from either pf performing NAT or the packet forwarding from gateway_enable. [0] https://code.fizz.buzz/talexander/machine_setup/src/commit/20768edcf69eddae5cf65e30a0bde869f9ddd19b/ansible/roles/bhyve/files/bhyve_netgraph_bridge.bash#L214 [1] https://code.fizz.buzz/talexander/machine_setup/src/commit/20768edcf69eddae5cf65e30a0bde869f9ddd19b/ansible/roles/firewall/files/mrmanager_pf.conf#L35
ComputerGuru 2y ago
I just spun up a VNET jail (so it should be essentially using the same network stack and networking isolation level as a bhyve guest would) and tested with iperf3 and without any tweaking or optimization and without even using jumbo frames I'm able to get 24+ Gbps with iperf3 (32k window size, tcp, single stream) between host/guest over the bridged and virtualized network interface. My test hardware is older than yours, it's a Xeon E5-1650 v3 and this is even with nested virtualization since the "host" is actually an ESXi guest running pf! But I think you might be right about something because, playing with it some more, I'm seeing an asymmetry in network I/O speeds; when I use `iperf3 -R` from the VNET jail to make the host connect to the guest and send data instead of the other way around, I get very inconsistent results with bursts of 2 Gbps traffic and then entire seconds without any data transferred (regardless of buffer size). I'd need to do a packet capture to figure out what is happening but it doesn't look like the default configuration performs very well at all!

bitfilped 2y ago

It's been a minute since I messed with bhyve on FreeBSD, but I'm pretty sure you have to switch out the networking stack to something like Netgraph if you intend to use fast networking.

craftkiller 2y ago
Hmmm I'm not the OP, but I run my personal site on a kubernetes cluster hosted in bhyve VMs running Debian on a FreeBSD machine using netgraph for the networking. I just tested by launching iperf3 on the FreeBSD host and launching an alpine linux pod in the cluster, and I only got ~4Gbit/s. This is surprising to me since netgraph is supposed to be capable of much faster networking but I guess this is going through multiple additional layers that may have slowed it down (off the top of my head: kubernetes with flannel, iptables in the VM, bhyve, and pf on the FreeBSD host).
- bitfilped 2y ago
  Do you know if you're still using if_bridge? I remembered this article from klara that goes a bit more into the details. https://klarasystems.com/articles/using-netgraph-for-freebsds-bhyve-networking/ https://klarasystems.com/articles/using-netgraph-for-freebsd...
  - craftkiller 2y ago
    Thanks, but I am not using if_bridge. I am creating a netgraph bridge[0] which is connected to the host via a netgraph eiface. Then on the host, the packet passes to my real physical interface because I have gateway_enable and I have pf perform NAT[1]. It looks like that blog post connected the netgraph bridge directly to the external interface, so my guess is my slowdown is from either pf performing NAT or the packet forwarding from gateway_enable. [0] https://code.fizz.buzz/talexander/machine_setup/src/commit/20768edcf69eddae5cf65e30a0bde869f9ddd19b/ansible/roles/bhyve/files/bhyve_netgraph_bridge.bash#L214 [1] https://code.fizz.buzz/talexander/machine_setup/src/commit/20768edcf69eddae5cf65e30a0bde869f9ddd19b/ansible/roles/firewall/files/mrmanager_pf.conf#L35

ComputerGuru 2y ago

I just spun up a VNET jail (so it should be essentially using the same network stack and networking isolation level as a bhyve guest would) and tested with iperf3 and without any tweaking or optimization and without even using jumbo frames I'm able to get 24+ Gbps with iperf3 (32k window size, tcp, single stream) between host/guest over the bridged and virtualized network interface. My test hardware is older than yours, it's a Xeon E5-1650 v3 and this is even with nested virtualization since the "host" is actually an ESXi guest running pf! But I think you might be right about something because, playing with it some more, I'm seeing an asymmetry in network I/O speeds; when I use `iperf3 -R` from the VNET jail to make the host connect to the guest and send data instead of the other way around, I get very inconsistent results with bursts of 2 Gbps traffic and then entire seconds without any data transferred (regardless of buffer size). I'd need to do a packet capture to figure out what is happening but it doesn't look like the default configuration performs very well at all!