5 ms·

Intel I219-LM Running at ~60% of Maximum Speed Due to Linux Driver Bug

105 points by zdw 4y ago

jmclnx 4y ago
I wonder if this is an issue with the BSDs too. Sometimes they peek at Linux for these things.
- wmf 4y ago
  This is a hardware bug so I would expect it to affect all OSes. Turning off TSO is counterintuitive so drivers tend to turn it on by default.
- AdrianB1 4y ago
  Probably not, BSD network drivers were a bit more advanced than Linux ones as some companies (like Netflix) invested time and effort in optimizations and monitoring. I am not saying BDS is better, just a few things are quite different vs Linux.
  - betaby 4y ago
    > a bit more advanced than Linux ones and then we look at the wifi stack in BSD(s)...
    - AdrianB1 4y ago
      I don't think that companies like Netflix are interested in WiFi to serve massive amounts of video :)
  - ilyt 4y ago
    > Probably not, BSD network drivers were a bit more advanced than Linux ones as some companies (like Netflix) invested time and effort in optimizations and monitoring. for those 3 nics they use. I doubt some random 2.5Gbit nic is on their list of stuff to fix
- KyleSanderson 4y ago
  It isn't. It's a pure driver bug with all of these. There's a pending patch to clear the MSI registers on disconnect that I'm planning on driving to testing tomorrow. I hounded them for 4 months by sending full stacks with symbols and they don't have any hardware in the lab (which is very common now, unfortunately)... If you want to recover, set a ifdown script to `rmmod igc && modprobe igc` and you will never panic or hang, just have a longer interface bounce time. I'm running this on a 6 nic system which is completely egregious, but it hasn't crashed / hung / panic'd in the field since doing that.
legulere 4y ago
That makes me wonder how much waste there is, just because there is something broken without anyone noticing. Recently a customer had two nearly identical systems behaving completely different in regards to performance. It turned out it simply was some bios setting.
- magicalhippo 4y ago
  > That makes me wonder how much waste there is, just because there is something broken without anyone noticing. I'd say a lot. Not seldom when I go over to a colleague I notice their laptop is breathing hard. "Oh yeah, so annoying! It's been doing that for a while now". Check Task Manager and sure enough, some process is stuck sucking 100% of a core... 5 CPU days worth. Even I find it difficult to spot this, with my 16 cores. I got an efficient cooler, so no fan spins up noticeably. Randomly check Task Manager, oh explorer.exe is pegging a core, and has been using 15 days worth of CPU... Gee thanks!
LinuxBender 4y ago
It's nice to see them giving this some attention. I hope they also look into the I225-V at the myriad of different negotiation bugs that change with each firmware revision on 2.5gb networks that also vary with different offloading features enabled/disabled including EEE. Each 2.5gb switch also exhibits different behavior which is probably not Intel's fault.
- rasz 4y ago
  > is probably not Intel's fault consensus is Intel went down the shitter quality wise, just one example https://www.youtube.com/watch?v=DXNyHFOWx_k https://www.youtube.com/watch?v=DXNyHFOWx_k
  - LinuxBender 4y ago
    Could be. Most of the folks I interacted with in the past have retired. I think I am just going to revert back to full tower PC's for anything I do, even firewalls and appliances so that I am not locked into what may be abandonware. I was hoping to use less power and take up less space with the mini-PC's but in hind-sight that may have been a mistake.
    - rubatuga 4y ago
      Fujitsu S920 seems to be a good option for a mini PC that is cheap and reliable. Not very fast, but has usb 3.0 and a x4 pcie slot. It has been very stable as my wifi access point. First heard about it on Wolfgang's channel.
- snvzz 4y ago
  I got tired of fighting that NIC, and got a Realtek 2.5g one. All my problems gone, life is good.
  - LinuxBender 4y ago
    I would totally do that but I have a few mini-PC's that have the I225-V and I can't swap them out. I've used USB3.2->Ethernet dongles as work arounds until I could find a switch that was happy with it. Same with the latest Protectli firewalls, they also have the I225-V. For those I have to use ethtool to force a specific speed which is not ideal. I'm thinking mini-PC's are maybe a mistake in this generation of NICs.
malkia 4y ago
Not a network guy, but couldn't this be detected (for any other card too) by running some mandatory tests?
andix 4y ago
One would expect Intel to do periodic regression tests on Linux kernel releases on their hardware. A lot of Intel customers use Linux, and they may switch to other vendors if Intel hardware has bad Linux support.
- arghwhat 4y ago
  They do, and they provide some of the most extensive regression testing capabilities provides to kernel development. But to be fair, this is an old NIC with a hardware bug that requires an odd workaround disabling an otherwise very useful performance feature. Missing that edge case is understandable.
  - justinclift 4y ago
    Hmmm, that doesn't really seem to gel though, unless their extensive regression testing is a recent development? Saying that because at some point, that NIC was a new thing, not an old one. So extensive regression testing should have picked it up then, yeah?
    - arghwhat 4y ago
      https://www.intel.com/content/www/us/en/developer/topic-technology/open/linux-kernel-performance/overview.html https://www.intel.com/content/www/us/en/developer/topic-tech... It is not a new thing, but no test is exhaustive and they must prioritize hardware tested for practical reasons. This particular test would need to check if the working NIC could reach full gigabit speeds in TX, as everything else checked out. Regardless, it cannot be said that they do not have regression testing or are irresponsible in that sense, as they provide important services to the community in exactly this area.
      - justinclift 4y ago
        > This particular test would need to check if the working NIC could reach full gigabit speeds in TX, as everything else checked out. To me, that sounds weird. My expectation is that every non-dodgy manufacturer of a network device would test the features they're developing. So, when (say) offloading support of some variety is newly added to a chip set, whoever is adding that support would be testing it at the full speed of the device. If the developers aren't verifying the things they're adding work (somehow), there's something wrong. :(
fulafel 4y ago
Interesting that in a era of 100+ Gbit Ethernet we seem to need the offload features working just right to get even 1 Gbit.
- arghwhat 4y ago
  Well in this case the bug is that the feature doesn't work and needs to be disabled. But generally speaking you want all the offload you can get. For 1G, it allows pushing power efficiency much further. For 100G, it allows you to spend time on something other than parsing and writing packets.
userbinator 4y ago
you will want to look forward to upgrading your Linux kernel build soon... [...] due to a regression introduced back in 2020. If anything, that message seems to be implying to not upgrade if everything is already working well. A bit tangential, but ever since they came out with the '217 I've thought this is one of the worst-named products and could never remember what the actual letter is. Here it's an uppercase I, but even Intel seems to think it's a lowercase L (for LAN?) sometimes: https://www.intel.com/content/www/us/en/support/articles/000023129/ethernet-products/gigabit-ethernet-adapters-up-to-2-5gbe.html https://www.intel.com/content/www/us/en/support/articles/000...
dzdt 4y ago
I have an Intel I219-V adaptor on a Windows desktop. I had a hell of a time trying to figure out the problem when I upgraded to Verizon FIOS internet; upload speeds were abysmal. The chipset has a hardware bug making it incompatible with the Verizon Optical Network Terminal. The workaround is to disable checksum offloading in the network driver advanced settings. It sounds like this is the linux version of the same thing.
timnetworks 4y ago
This dumb chip caused myself and my colleagues some headaches when the driver kept getting deleted as suspected-malicious but reinstalled right away by Windows, causing the whole lot to bounce like on some giant trampoline. Reading the comments here, we aren't the only ones taken by surprise and/or aback.
herpderperator 4y ago
Doesn't disabling offloading (of anything) result in increased CPU usage when transferring lots of data? Is it now that it's faster but uses more CPU, or is it actually faster with the same CPU usage (a "proper" fix)?
- majke 4y ago
  There are probably a couple of independent things: (1) HW GRO, most often LRO (TCP) (2) SW GRO aka GRO Hardware GRO was typically called LRO for TCP and is often disabled in practice. One reason is XDP, with XDP some offloads need to stay off. Another is device quirks: while idea of waiting for more data, and passing larger packets into kernel soiunds nice, it also means that device must do packet concatenation. Think about ECN or selective-acks. The concatentation is lossy and unless the flow is perfect, it might not be possible to concatenate packets. Furthermore devices often had issues with being too aggrressive and loosing important data. In Linux there is a software approach that does kindof the same thing, but in software, called GRO, and this is often sufficient. It also supports more than just TCP. In other words - having LRO disabled is common.
herpderperator 4y ago
Does this mean if someone with an I219-LM did a regular file transfer they would only get 67MiB/s instead of 112MiB/s? Surely this would be immediately noticeable and not take 3 years to fix? Was it just a throughput bug or a packets-per-second bug? Did any other conditions need to be present for it to trigger the slowdown? So many questions... I just don't understand how this can go unnoticed. If you buy a 1Gbps NIC and it performs <1Gbps for a simple file transfer, you would think that's not something Intel would miss, or ignore if reported given how easy it would be to reproduce.

105 points by zdw 4y ago

jmclnx 4y ago

I wonder if this is an issue with the BSDs too. Sometimes they peek at Linux for these things.

wmf 4y ago
This is a hardware bug so I would expect it to affect all OSes. Turning off TSO is counterintuitive so drivers tend to turn it on by default.
AdrianB1 4y ago
Probably not, BSD network drivers were a bit more advanced than Linux ones as some companies (like Netflix) invested time and effort in optimizations and monitoring. I am not saying BDS is better, just a few things are quite different vs Linux.
- betaby 4y ago
  > a bit more advanced than Linux ones and then we look at the wifi stack in BSD(s)...
  - AdrianB1 4y ago
    I don't think that companies like Netflix are interested in WiFi to serve massive amounts of video :)
- ilyt 4y ago
  > Probably not, BSD network drivers were a bit more advanced than Linux ones as some companies (like Netflix) invested time and effort in optimizations and monitoring. for those 3 nics they use. I doubt some random 2.5Gbit nic is on their list of stuff to fix
KyleSanderson 4y ago
It isn't. It's a pure driver bug with all of these. There's a pending patch to clear the MSI registers on disconnect that I'm planning on driving to testing tomorrow. I hounded them for 4 months by sending full stacks with symbols and they don't have any hardware in the lab (which is very common now, unfortunately)... If you want to recover, set a ifdown script to `rmmod igc && modprobe igc` and you will never panic or hang, just have a longer interface bounce time. I'm running this on a 6 nic system which is completely egregious, but it hasn't crashed / hung / panic'd in the field since doing that.

legulere 4y ago

That makes me wonder how much waste there is, just because there is something broken without anyone noticing. Recently a customer had two nearly identical systems behaving completely different in regards to performance. It turned out it simply was some bios setting.

magicalhippo 4y ago
> That makes me wonder how much waste there is, just because there is something broken without anyone noticing. I'd say a lot. Not seldom when I go over to a colleague I notice their laptop is breathing hard. "Oh yeah, so annoying! It's been doing that for a while now". Check Task Manager and sure enough, some process is stuck sucking 100% of a core... 5 CPU days worth. Even I find it difficult to spot this, with my 16 cores. I got an efficient cooler, so no fan spins up noticeably. Randomly check Task Manager, oh explorer.exe is pegging a core, and has been using 15 days worth of CPU... Gee thanks!

LinuxBender 4y ago

It's nice to see them giving this some attention. I hope they also look into the I225-V at the myriad of different negotiation bugs that change with each firmware revision on 2.5gb networks that also vary with different offloading features enabled/disabled including EEE. Each 2.5gb switch also exhibits different behavior which is probably not Intel's fault.

rasz 4y ago
> is probably not Intel's fault consensus is Intel went down the shitter quality wise, just one example https://www.youtube.com/watch?v=DXNyHFOWx_k https://www.youtube.com/watch?v=DXNyHFOWx_k
- LinuxBender 4y ago
  Could be. Most of the folks I interacted with in the past have retired. I think I am just going to revert back to full tower PC's for anything I do, even firewalls and appliances so that I am not locked into what may be abandonware. I was hoping to use less power and take up less space with the mini-PC's but in hind-sight that may have been a mistake.
  - rubatuga 4y ago
    Fujitsu S920 seems to be a good option for a mini PC that is cheap and reliable. Not very fast, but has usb 3.0 and a x4 pcie slot. It has been very stable as my wifi access point. First heard about it on Wolfgang's channel.
snvzz 4y ago
I got tired of fighting that NIC, and got a Realtek 2.5g one. All my problems gone, life is good.
- LinuxBender 4y ago
  I would totally do that but I have a few mini-PC's that have the I225-V and I can't swap them out. I've used USB3.2->Ethernet dongles as work arounds until I could find a switch that was happy with it. Same with the latest Protectli firewalls, they also have the I225-V. For those I have to use ethtool to force a specific speed which is not ideal. I'm thinking mini-PC's are maybe a mistake in this generation of NICs.

malkia 4y ago

Not a network guy, but couldn't this be detected (for any other card too) by running some mandatory tests?

andix 4y ago

One would expect Intel to do periodic regression tests on Linux kernel releases on their hardware. A lot of Intel customers use Linux, and they may switch to other vendors if Intel hardware has bad Linux support.

arghwhat 4y ago
They do, and they provide some of the most extensive regression testing capabilities provides to kernel development. But to be fair, this is an old NIC with a hardware bug that requires an odd workaround disabling an otherwise very useful performance feature. Missing that edge case is understandable.
- justinclift 4y ago
  Hmmm, that doesn't really seem to gel though, unless their extensive regression testing is a recent development? Saying that because at some point, that NIC was a new thing, not an old one. So extensive regression testing should have picked it up then, yeah?
  - arghwhat 4y ago
    https://www.intel.com/content/www/us/en/developer/topic-technology/open/linux-kernel-performance/overview.html https://www.intel.com/content/www/us/en/developer/topic-tech... It is not a new thing, but no test is exhaustive and they must prioritize hardware tested for practical reasons. This particular test would need to check if the working NIC could reach full gigabit speeds in TX, as everything else checked out. Regardless, it cannot be said that they do not have regression testing or are irresponsible in that sense, as they provide important services to the community in exactly this area.
    - justinclift 4y ago
      > This particular test would need to check if the working NIC could reach full gigabit speeds in TX, as everything else checked out. To me, that sounds weird. My expectation is that every non-dodgy manufacturer of a network device would test the features they're developing. So, when (say) offloading support of some variety is newly added to a chip set, whoever is adding that support would be testing it at the full speed of the device. If the developers aren't verifying the things they're adding work (somehow), there's something wrong. :(

fulafel 4y ago

Interesting that in a era of 100+ Gbit Ethernet we seem to need the offload features working just right to get even 1 Gbit.

arghwhat 4y ago
Well in this case the bug is that the feature doesn't work and needs to be disabled. But generally speaking you want all the offload you can get. For 1G, it allows pushing power efficiency much further. For 100G, it allows you to spend time on something other than parsing and writing packets.

userbinator 4y ago

you will want to look forward to upgrading your Linux kernel build soon... [...] due to a regression introduced back in 2020. If anything, that message seems to be implying to not upgrade if everything is already working well. A bit tangential, but ever since they came out with the '217 I've thought this is one of the worst-named products and could never remember what the actual letter is. Here it's an uppercase I, but even Intel seems to think it's a lowercase L (for LAN?) sometimes: https://www.intel.com/content/www/us/en/support/articles/000023129/ethernet-products/gigabit-ethernet-adapters-up-to-2-5gbe.html https://www.intel.com/content/www/us/en/support/articles/000...

dzdt 4y ago

I have an Intel I219-V adaptor on a Windows desktop. I had a hell of a time trying to figure out the problem when I upgraded to Verizon FIOS internet; upload speeds were abysmal. The chipset has a hardware bug making it incompatible with the Verizon Optical Network Terminal. The workaround is to disable checksum offloading in the network driver advanced settings. It sounds like this is the linux version of the same thing.

timnetworks 4y ago

This dumb chip caused myself and my colleagues some headaches when the driver kept getting deleted as suspected-malicious but reinstalled right away by Windows, causing the whole lot to bounce like on some giant trampoline. Reading the comments here, we aren't the only ones taken by surprise and/or aback.

herpderperator 4y ago

Doesn't disabling offloading (of anything) result in increased CPU usage when transferring lots of data? Is it now that it's faster but uses more CPU, or is it actually faster with the same CPU usage (a "proper" fix)?

majke 4y ago
There are probably a couple of independent things: (1) HW GRO, most often LRO (TCP) (2) SW GRO aka GRO Hardware GRO was typically called LRO for TCP and is often disabled in practice. One reason is XDP, with XDP some offloads need to stay off. Another is device quirks: while idea of waiting for more data, and passing larger packets into kernel soiunds nice, it also means that device must do packet concatenation. Think about ECN or selective-acks. The concatentation is lossy and unless the flow is perfect, it might not be possible to concatenate packets. Furthermore devices often had issues with being too aggrressive and loosing important data. In Linux there is a software approach that does kindof the same thing, but in software, called GRO, and this is often sufficient. It also supports more than just TCP. In other words - having LRO disabled is common.

herpderperator 4y ago

Does this mean if someone with an I219-LM did a regular file transfer they would only get 67MiB/s instead of 112MiB/s? Surely this would be immediately noticeable and not take 3 years to fix? Was it just a throughput bug or a packets-per-second bug? Did any other conditions need to be present for it to trigger the slowdown? So many questions... I just don't understand how this can go unnoticed. If you buy a 1Gbps NIC and it performs <1Gbps for a simple file transfer, you would think that's not something Intel would miss, or ignore if reported given how easy it would be to reproduce.