3 ms·

I ran into a similar problem with an Intel motherboard about 10 years ago. We had problems when some NFS traffic would end up getting stalled. Our NFS server w

I ran into a similar problem with an Intel motherboard about 10 years ago. We had problems when some NFS traffic would end up getting stalled. Our NFS server would use UDP packets larger than the MTU and they would end up getting fragmented. Turns out the NIC would not look at the fragmentation headers of the IP packet and always assume a UDP header was present. From time to time, the payload of the NFS packet would have user data that matched the UDP port number the NIC would scan for to determine if the packet should be forwarded to the BMC. This motherboard had no BMC but it was configured as if it did have one. It would time out after a second or so but in the meantime drop a bunch of packets. The NFS server would retransmit the packet but since the payload didn't change, the NIC would reliably drop the rest of the fragments of the packet. Of course Intel claimed it wasn't their bug ("it's a bug in the Linux NFS implementation") but they quickly changed their tune when I coded up a sample program that would send one packet a second and reliably cause the NIC to drop 99% of packets received. While it turned out to be a fairly lame implementation problem on Intel's part (both by ignoring the fragmentation headers and the poor implementation of the motherboard) I have to say it was very satisfying to solve the mystery.

EvanAnderson 14y ago
Reading about the OP's issue got me to a doc from Intel (http://www.intel.com/content/dam/doc/application-note/sideband-technology-appl-note.pdf http://www.intel.com/content/dam/doc/application-note/sideba...) re: the "NC Sideband Interface", which sounds like the place where the bug that bit you "lives". Reading over that doc made me shudder a few times, thinking about the complexity and, thus, potential bugs that could be lurking there. I wonder if the OP's bug was related, too. Having the NIC inspecting incoming frames and potentially diverting them to the management controller sounds like a scary proposition. I'd almost rather just have dedicated Ethernet hardware for the management controller. The decrease in switch ports needed is certainly seductive, but I wonder if it's worth the risk. (Do you happen to recall which Intel motherboard this bit you on? I was just getting out of whitebox Intel motherboard-based server builds about the time you're describing, but I'm just curious if only for the nostalgia.)
- jevinskie 14y ago
  "IPMI operates independently of the OS and allows administrators to manage a system remotely even without an OS, system management software, and even if the monitored system is powered off (along as it is connected to a power source). IPMI can also function after an OS has started, offering enhanced features when used with system management software." Yikes! Sounds like system management mode in a BIOS!
  - EvanAnderson 14y ago
    It's worse than that. It's not BIOS-- it's a freestanding computer. You'll enjoy this (or be horrified by it): http://fish2.com/ipmi/itrain.html http://fish2.com/ipmi/itrain.html
    - rosser 14y ago
      But it's a freestanding computer that means that I don't need to go to the data center at two in the morning to bring up a box that's kernel panicked. Yeah, be careful with it. Firewall it silly. But recognize that it's a tool that can be very useful.
      - EvanAnderson 14y ago
        They're definitely useful tools-- don't get me wrong about that. The fear is that they're controlled, essentially, by an in-band signaling mechanism. Being grafted onto the same NICs on the server computer that you might potentially expose to the Internet makes firewalling them a more difficult proposition. I get a lot of piece of mind from having management interfaces on an an out-of-band control network whenever possible.
        rosser 14y ago
        If you use your BMC's "pass-through" capability to push its traffic through your regular NICs, you're Doing It Wrong. In fact, if you're not running a separate physical network (not merely separate VLANs) that is, to the extent possible, air-gapped from the rest of the world for your IPMI traffic, you're probably still Doing It Wrong.

EvanAnderson 14y ago

Reading about the OP's issue got me to a doc from Intel (http://www.intel.com/content/dam/doc/application-note/sideband-technology-appl-note.pdf http://www.intel.com/content/dam/doc/application-note/sideba...) re: the "NC Sideband Interface", which sounds like the place where the bug that bit you "lives". Reading over that doc made me shudder a few times, thinking about the complexity and, thus, potential bugs that could be lurking there. I wonder if the OP's bug was related, too. Having the NIC inspecting incoming frames and potentially diverting them to the management controller sounds like a scary proposition. I'd almost rather just have dedicated Ethernet hardware for the management controller. The decrease in switch ports needed is certainly seductive, but I wonder if it's worth the risk. (Do you happen to recall which Intel motherboard this bit you on? I was just getting out of whitebox Intel motherboard-based server builds about the time you're describing, but I'm just curious if only for the nostalgia.)

jevinskie 14y ago
"IPMI operates independently of the OS and allows administrators to manage a system remotely even without an OS, system management software, and even if the monitored system is powered off (along as it is connected to a power source). IPMI can also function after an OS has started, offering enhanced features when used with system management software." Yikes! Sounds like system management mode in a BIOS!
- EvanAnderson 14y ago
  It's worse than that. It's not BIOS-- it's a freestanding computer. You'll enjoy this (or be horrified by it): http://fish2.com/ipmi/itrain.html http://fish2.com/ipmi/itrain.html
  - rosser 14y ago
    But it's a freestanding computer that means that I don't need to go to the data center at two in the morning to bring up a box that's kernel panicked. Yeah, be careful with it. Firewall it silly. But recognize that it's a tool that can be very useful.
    - EvanAnderson 14y ago
      They're definitely useful tools-- don't get me wrong about that. The fear is that they're controlled, essentially, by an in-band signaling mechanism. Being grafted onto the same NICs on the server computer that you might potentially expose to the Internet makes firewalling them a more difficult proposition. I get a lot of piece of mind from having management interfaces on an an out-of-band control network whenever possible.
      - rosser 14y ago
        If you use your BMC's "pass-through" capability to push its traffic through your regular NICs, you're Doing It Wrong. In fact, if you're not running a separate physical network (not merely separate VLANs) that is, to the extent possible, air-gapped from the rest of the world for your IPMI traffic, you're probably still Doing It Wrong.