5 ms·

I wonder how many bugs there are like this in low-level networking infrastructure that cause issues which go undiagnosed for years before someone with enough kn

I wonder how many bugs there are like this in low-level networking infrastructure that cause issues which go undiagnosed for years before someone with enough knowledge to diagnose them figures it out. I keep reading about weird networking glitches triggered by specific bit patterns, but certainly if I ever encountered such a think I don't think I'd be able to diagnose it.

qb45 10y ago
One WTF I've seen: a consumer router which failed to NAT outgoing packets if their endian-swapped destination IP matched the LAN subnet, i.e. anything sent to X.1.168.192 would be forwarded to WAN with 192.168.1.Y source IP and get lost. Took me a few weeks to figure out why certain random websites don't work.
- heywire 10y ago
  Reminds me of an issue [1] I ran into with Symantec Firewall looking at UDP fragments as if they were full packets, and blocking them if the value where the port number would be on a full packet matched a rule. [1] https://www.symantec.com/connect/forums/firewall-incorrectly-blocks-fragments-udp-traffic https://www.symantec.com/connect/forums/firewall-incorrectly...
Rapzid 10y ago
Not a network code issue but my favorite anecdote for this type of thing. There was a Xen hypervisor bug for YEARS that would cause the clock to jump 20(?) minutes occasionally. The root cause was found in inline assembly due to a compiler oddity. It was found during some bug day/week thing by somebody with the assembly knowledge who just happened to look at that code and see an issue. He had no idea of the significance of what he found :D
- simcop2387 10y ago
  I'd love to read more about that. Got a link or anything more to google on?
  - Rapzid 10y ago
    It was actually 50 minutes. Here is what I could turn up: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=599161 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=599161 https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=599161;filename=x86-time-scale-asm.patch;msg=234 https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=5991... I remember it was patched and out by debian right around Christmas; quite the present. This bug made me feel very helpless. I recall there being more information around the circumstances of its discovery on the Xen mailing list.
disposablezero 10y ago
There were issues with Cisco switches and G2/G3 HP Proliant servers which caused duplex flapping. We had to force it to 100/Full on both servers and switches to fix the issue. Source: worked at Stanford in IT
digi_owl 10y ago
I seem to recall that a history passed through HN a while back that was about someone diagnosing a fault in a router beyond their ISP. This because their VPN or some such would fail when it got routed through that specific path.
acranox 10y ago
At the university I used to work out, the swapped out the Cisco firewalls for Junipers. One of my applications immediately started having random errors. A week of troubleshooting with the network team, and Juniper support, concluded that the routers were silently dropping some UDP packets. No logs for it, and no rules to explain it, and no pattern to it either. It took days to just convince the network guys that it really might be the firewall, since it wasn't reporting any issues or any reason for the packet loss. Also no fix for it. We had to switch the application to TCP in order to work around the problem.

qb45 10y ago

One WTF I've seen: a consumer router which failed to NAT outgoing packets if their endian-swapped destination IP matched the LAN subnet, i.e. anything sent to X.1.168.192 would be forwarded to WAN with 192.168.1.Y source IP and get lost. Took me a few weeks to figure out why certain random websites don't work.

heywire 10y ago
Reminds me of an issue [1] I ran into with Symantec Firewall looking at UDP fragments as if they were full packets, and blocking them if the value where the port number would be on a full packet matched a rule. [1] https://www.symantec.com/connect/forums/firewall-incorrectly-blocks-fragments-udp-traffic https://www.symantec.com/connect/forums/firewall-incorrectly...

Rapzid 10y ago

Not a network code issue but my favorite anecdote for this type of thing. There was a Xen hypervisor bug for YEARS that would cause the clock to jump 20(?) minutes occasionally. The root cause was found in inline assembly due to a compiler oddity. It was found during some bug day/week thing by somebody with the assembly knowledge who just happened to look at that code and see an issue. He had no idea of the significance of what he found :D

simcop2387 10y ago
I'd love to read more about that. Got a link or anything more to google on?
- Rapzid 10y ago
  It was actually 50 minutes. Here is what I could turn up: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=599161 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=599161 https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=599161;filename=x86-time-scale-asm.patch;msg=234 https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=5991... I remember it was patched and out by debian right around Christmas; quite the present. This bug made me feel very helpless. I recall there being more information around the circumstances of its discovery on the Xen mailing list.

disposablezero 10y ago

There were issues with Cisco switches and G2/G3 HP Proliant servers which caused duplex flapping. We had to force it to 100/Full on both servers and switches to fix the issue. Source: worked at Stanford in IT

digi_owl 10y ago

I seem to recall that a history passed through HN a while back that was about someone diagnosing a fault in a router beyond their ISP. This because their VPN or some such would fail when it got routed through that specific path.

acranox 10y ago

At the university I used to work out, the swapped out the Cisco firewalls for Junipers. One of my applications immediately started having random errors. A week of troubleshooting with the network team, and Juniper support, concluded that the routers were silently dropping some UDP packets. No logs for it, and no rules to explain it, and no pattern to it either. It took days to just convince the network guys that it really might be the firewall, since it wasn't reporting any issues or any reason for the packet loss. Also no fix for it. We had to switch the application to TCP in order to work around the problem.