4 ms·

> During those tests, we noticed the machines were randomly freezing after some time, so we decided to upgrade the firmware of the network cards, Reminds me of

> During those tests, we noticed the machines were randomly freezing after some time, so we decided to upgrade the firmware of the network cards, Reminds me of the various i40e Tx freezes I debugged while at Joyent. Granted, this is the illumos driver, not Intel's, but basically there were issues with the programming guide that I had to figure out the hard way. The 700-series controllers have not been the easiest to work with. https://smartos.org/bugview/OS-7492 https://smartos.org/bugview/OS-7492 [Tx freeze when b_cont chain exceeds 8 descriptors] https://smartos.org/bugview/OS-7457 https://smartos.org/bugview/OS-7457 [i40e Tx freezes on zero descriptors]

drewg123 5y ago
This 8 descriptor per packet limit is HORRIFIC. I debugged and fixed this issue on FreeBSD when we first moved to the new iflib based ixl (i40e's name on FreeBSD). They had a routine (ixl_tso_detect_sparse()) which was AFU. I wrote a userspace unit test that proved it was AFU, and then fixed it. I fed them back the routine & the unit test, and they hilariously left my commented debug prints in the routine. https://github.com/freebsd/freebsd-src/blob/412b5e40a721430adba1b4abae210641f733f976/sys/dev/ixl/ixl_txrx.c#L185 https://github.com/freebsd/freebsd-src/blob/412b5e40a721430a... And their 100GbE NIC has the same limit, which is just so sad. All these fancy features, and they cannot handle 8 segments per emitted packet on the wire.
- comex 5y ago
  AFU?
  - phab 5y ago
    all fucked up
userbinator 5y ago
From your first one: Malicious Driver Detection My reaction upon reading that line was "WTF." I haven't touched NIC drivers beyond the classic NE2000s, common Realteks, and the Intel 8254x, but it seems strange to have some sort of... antimalware feature in a NIC? Reminds me of the old BIOSes with "boot sector antivirus".
- rzezeski 5y ago
  Probably more to do with the fact that everything is moving towards virtualization. Oftentimes these NICs dole out VFs directly to VMs via SR-IOV, in which case I imagine the NIC controller has some safeguards to keep the host and the rest of the guest's safe from denial-of-service and other attacks from a malicious guest driver.
sofixa 5y ago
It was the same with Intel's drivers. We had the same issues happen on ESXi servers with X710 NICs, with VMware's repackaged and then Intel's original i40e driver, and it worked terribly, either kernel panicking or just freezing the NICs. It was a fun one to debug, but thankfully we only had to wait a few months ( the issue was known for a year) for Intel to come up with the fixed driver. The bastards at VMware kept the buggy driver on their hardware compatibility list and kept shipping it for multiple versions probably a year later.
- SteveNuts 5y ago
  Was the ixgbe driver not available?
  - sofixa 5y ago
    IIRC the options were i40e and i40en, the latter resulting in daily crashes, so i40e it was :)

drewg123 5y ago

This 8 descriptor per packet limit is HORRIFIC. I debugged and fixed this issue on FreeBSD when we first moved to the new iflib based ixl (i40e's name on FreeBSD). They had a routine (ixl_tso_detect_sparse()) which was AFU. I wrote a userspace unit test that proved it was AFU, and then fixed it. I fed them back the routine & the unit test, and they hilariously left my commented debug prints in the routine. https://github.com/freebsd/freebsd-src/blob/412b5e40a721430adba1b4abae210641f733f976/sys/dev/ixl/ixl_txrx.c#L185 https://github.com/freebsd/freebsd-src/blob/412b5e40a721430a... And their 100GbE NIC has the same limit, which is just so sad. All these fancy features, and they cannot handle 8 segments per emitted packet on the wire.

comex 5y ago
AFU?
- phab 5y ago
  all fucked up

userbinator 5y ago

From your first one: Malicious Driver Detection My reaction upon reading that line was "WTF." I haven't touched NIC drivers beyond the classic NE2000s, common Realteks, and the Intel 8254x, but it seems strange to have some sort of... antimalware feature in a NIC? Reminds me of the old BIOSes with "boot sector antivirus".

rzezeski 5y ago
Probably more to do with the fact that everything is moving towards virtualization. Oftentimes these NICs dole out VFs directly to VMs via SR-IOV, in which case I imagine the NIC controller has some safeguards to keep the host and the rest of the guest's safe from denial-of-service and other attacks from a malicious guest driver.

sofixa 5y ago

It was the same with Intel's drivers. We had the same issues happen on ESXi servers with X710 NICs, with VMware's repackaged and then Intel's original i40e driver, and it worked terribly, either kernel panicking or just freezing the NICs. It was a fun one to debug, but thankfully we only had to wait a few months ( the issue was known for a year) for Intel to come up with the fixed driver. The bastards at VMware kept the buggy driver on their hardware compatibility list and kept shipping it for multiple versions probably a year later.

SteveNuts 5y ago
Was the ixgbe driver not available?
- sofixa 5y ago
  IIRC the options were i40e and i40en, the latter resulting in daily crashes, so i40e it was :)