8 ms·

Anyone ever had arp storms on a home network? I've had something trigger every device into asking 'who has xyz ip tell [Mac]'. It made the network unusable and

Anyone ever had arp storms on a home network? I've had something trigger every device into asking 'who has xyz ip tell [Mac]'. It made the network unusable and even rebooting systems it would come back since as soon as one device asked the question (sent the broken arp packet) all other devices decided they too needed to know. The solution that worked was to flip the circuit breaker for the whole house and reset every network device at once.

klysm 4y ago
I haven’t had a problem where rebooting the house has fixed it thankfully
- somat 4y ago
  I laughed, but it made me think, with "smart" houses becoming a thing, this will one day become a common occurrence. sigh
mh- 4y ago
Yes, generated by AT&T's wireless STBs (cable boxes) back around 2014-15. It was a nightmare to figure out, as I recall.
dylan604 4y ago
That takes did you turn it off and on again to an entirely different level. What if we need to do this city wide?
- wizofaus 4y ago
  That sounds like a good plot for a movie - the day we had to reboot the entire internet...
  - Lammy 4y ago
    "South Park already did it": https://en.wikipedia.org/wiki/Over_Logging https://en.wikipedia.org/wiki/Over_Logging
    - wizofaus 4y ago
      Hah, yeah I did originally think something along those lines, but I wonder if you could actually do it non-comedically (e.g. somehow every node connected to the internet has to all be switched off at the same time and restarted to restore connectivity). Most likely it's already been tried too I guess.
      - dylan604 4y ago
        usually, it's a nefarious evil doer that is threatening this action in the movies vs we gotta reboot the city. there's been reboot the sun plot. there's been reboot the earth's core plot. there's been reboot the machine running the city plot.
        wizofaus 4y ago
        I was thinking more along the lines of a rogue self-replicating packet where every last instance of had to die before routers and switches etc. would start working again.
- gnopgnip 4y ago
  Spanning tree protocol will prevent broadcast/arp storms by preventing loops
eikenberry 4y ago
I have had 2 external USB-power-passthrough laptop doc/hub things with an Ethernet port. They both cause a packet storm on the network if you unplug the computer and leave the ethernet and power plugged in. Causes all my crappy realtek NICs to overheat and flake out. Not exactly the same but super annoying.
- jfim 4y ago
  I had the same issue with an USB Ethernet adapter plugged into a powered USB hub, disconnecting the computer would make the network crap out. Kind of defeats the purpose of having a docked setup with wired Ethernet if one can't undock it or needs to unplug tons of cables each time.
  - smashed 4y ago
    See Ethernet pause frames Usb ethernet adapters will send these pause frames, waiting for the computer to reconnect.. but they end up flooding dumb switches.. https://news.ycombinator.com/item?id=31268134 https://news.ycombinator.com/item?id=31268134
xoa 4y ago
Depending on what network gear you're using (I'm now switching to Omada, previously/still on some site son UniFi, but lots of even 'prosumer' stuff does this) there are specific mitigations available. All of this falls under the heading of "managing broadcast traffic", which is very important even for smaller networks. The three major categories of traffic on a network are unicast, broadcast, and multicast. Unicast is the normal case of one device talking to a single address. Broadcast involves sending a packet to every single possible recipient in the entire broadcast domain (almost always the subnet). ARP fits in here. Multicast is essentially in between, more efficient then broadcast, can still talk to multiple devices that have signed up to hear it. Obviously an actual broadcast storm can take down an entire network, but excessive broadcast traffic on WiFi specifically can also suck up a huge amount of airtime for little bandwidth. Every single device has to go to the slowest speed and stop what they're doing to listen and make sure no one is left out. Using STP/RSTP with proper values set and LACP for aggregated interfaces can help prevent inadvertent network loops. Many switches also support some kind of port isolation and explicit per port storm control restricting max numbers of packets/second for unicast/broadcast/multicast traffic. WiFi APs can use proxy ARP to cut it in their domain too. The WAP already of course knows the MACs of every device connected to it by definition, so there isn't generally any reason not to have it answer ARP requests on their behalf then forward the traffic itself.
- donmcronald 4y ago
  Why the switch to Omada? I’m looking at doing the same due to supply issues with the UniFi stuff. Any tips or opinions on Omada vs UniFi?
  - xoa 4y ago
    Opinions hooboy do I have them ;), I was just asked that exact question on HN a couple of weeks ago in fact. Gave it a shot in a response here [0] which still applies. But basically Ubiquiti has become a toxic dumpster fire of a company and their product lines (UniFi in particular) on a downward trajectory in terms of performance, features and stability for quite a while. I had a certain amount invested in UniFi (think the final total will end up as a few hundred devices) so it's been a staged switch, with a total change of all routing/gateway/security functions to OPNsense completed first. That bought a lot more runway, it's always been the weakest and most neglected area in the ecosystem while obviously also being pretty critical. Yet the Ubiquiti debacle has served to underline for me how valuable self-host is, I've been able to have a nice slow ramp and deal with their implosion precisely because UniFi/UNMS/UISP and all the hardware is fully under my control. So I've been hoping someone would come along and see the potential of the UniFi niche of the networking market and basically copy it without all the junk. Which seems to basically be Omada to a tee. I'd actually originally (and still at many sites) intended to hold off and wait for WiFi 7 gear, because at that point a bunch of clients (and myself for that matter) will be interested in replacing WAPs anyway which is a very natural point to consider changing manufacturer as well. But a breaking point has come at a few places with a final feature which is PPSK, allowing the system to have many different passwords for an SSID that can be assigned different tags. Basically it allows having many of the benefits of WPA-Enterprise in terms of segmenting different clients onto different VLANs and revoking credentials and the like with more security and less manual work than MAB (MAC bypass) while still looking like a normal PSK scheme, which means the vast universe of brand new stuff which doesn't support 802.1x and never will works with it happily (by the same token none of that is going to play directly with using a secure virtual network or other better systems either sadly). Lower overhead and better compatibility than captive portals for non or semi-interactive devices as well. Someone hacked together a demo showing this could work on UniFi WAPs like four freaking years ago and Ubiquiti never did anything with it in favor of endless bikeshedding GUI changes to add more white space and hide important features and information (yes I'm a touch bitter). So I'm not in the position of wholeheartedly recommending Omada yet, I don't have years under my belt there and it's relatively speaking fairly new. It has its own warts and rough edges for sure, from the software to the hardware physical design. But it can be self-hosted and the trajectory looks massively better, has already had more meaningful improvement in months than UniFi has had in years, seems to perform much better so far as well. Of course the Venn diagram of self-hosting, herding lots of hardware with single pane, fully networking features, ecosystem richness and so on is pretty minimal in the overlap. Take away any one or multiple of those and options expand a lot, Aruba InstantOn for example. And welp, this didn't end up "basically" at all did it, sorry about that. I am bummed by the sheer wasted potential with Ubiquiti. So it goes in tech over and over again though, we've all seen this movie many, many times. As far as tips, I would suggest if you plan to stay on the managing-your-own-networks route to very strongly consider having the router/gateway stuff be separate and fully open source as I ended up. Doesn't have to be OPNsense, could be VyOS or plain OpenBSD or whatever else you're most comfortable with and depending on how you want to manage stuff and what needs there are for others to take over. But it's very, very pleasant to have the full spectrum of quality PC hardware available, you can get far more power for less, and you're never stuck with a critical aspect. I'd still suggest generally running that on metal rather than virtualizing it in a (semi)production network, but opinions vary there. ---- 0: https://news.ycombinator.com/item?id=32297556 https://news.ycombinator.com/item?id=32297556
neurostimulant 4y ago
Not sure if it's something similar but I had issue where attempting to setup a wifi smart plug locked up the router for a minute (until the smart plug gave up trying to connect to the router). Wired ethernet still works but the routers show 100% cpu usage on its management interface and the 2.4ghz wifi stopped working (didn't check the 5ghz one). I didn't dig in more because my wife was in a zoom meeting.
comprambler 4y ago
Similar, but not exact, are deb, and likely others, with failed storage will continually dhcprenew as they can’t write the leasefile for the interface. Near ddos as far as windows dhcp server is concerned. Required an arp table trace to find. Bad_Address is usually the symptom.
evulhotdog 4y ago
Yep. There’s actually a known issue with the CalDigit USB C Pro dock that causes this. It’s when the laptop comes up from or goes to sleep and it takes down the whole network. Fortunately there’s a firmware fix for it but it took a while to put two and two together.
- zamadatix 4y ago
  I thought the dock thing was related to excessively sending pause frames to switches that didn't know to not forward them.
  - evulhotdog 4y ago
    The official documentation is here: https://www.caldigit.com/the-usb-c-pro-dock-takes-down-my-network/ https://www.caldigit.com/the-usb-c-pro-dock-takes-down-my-ne...
- iforgotpassword 4y ago
  This freaking issue! This has plagued humanity for decades now I think. I had an older ThinkPad at work once that did that when it went to sleep. Fun for the whole fami^Wgroup of colleagues sharing the switch. Pause frames must die, leave flow control to a higher layer.
- microtonal 4y ago
  I think more adapters/docks with an RTL8153 suffer from this issue. E.g., I had a Lenovo USB-C Gen 2 dock that would fairly reliably take down our office network after suspend-wake. Just avoid Realtek NICs like the plague.
- Abishek_Muthian 4y ago
  I had a Apple USB 2.0 USB dongle which would crash the OSX when the router (DDWRT) is rebooted and sometimes would crash the router when the system was rebooted. Didn't do that with other combinations of OS/Router. Bug bounty opportunity missed?
  - 4y ago
Abishek_Muthian 4y ago
Not ARP storms per se, But I've noticed new Apple devices don't like router forwarding port 53 to pihole, resulting in bonjour storm and crashing pihole instances(Loosing DNS for the entire network).
- ungamed 4y ago
  If you can crash dns this way, please report it, its worthwhile fixing.
  - Abishek_Muthian 4y ago
    I'll try, Didn't have good experience with my earlier bug reports to Apple and now I'm completely out of that ecosystem. In fact I found this issue when my guests have iOS devices. Anyways here's the regex blacklist I use in PiHole to prevent Apple's reverse DNS spam - ^(?:(?:\d+\.)*\d+(?:-in\.addr\.arpa\.)?)$

klysm 4y ago

I haven’t had a problem where rebooting the house has fixed it thankfully

somat 4y ago
I laughed, but it made me think, with "smart" houses becoming a thing, this will one day become a common occurrence. sigh

mh- 4y ago

Yes, generated by AT&T's wireless STBs (cable boxes) back around 2014-15. It was a nightmare to figure out, as I recall.

dylan604 4y ago

That takes did you turn it off and on again to an entirely different level. What if we need to do this city wide?

wizofaus 4y ago
That sounds like a good plot for a movie - the day we had to reboot the entire internet...
- Lammy 4y ago
  "South Park already did it": https://en.wikipedia.org/wiki/Over_Logging https://en.wikipedia.org/wiki/Over_Logging
  - wizofaus 4y ago
    Hah, yeah I did originally think something along those lines, but I wonder if you could actually do it non-comedically (e.g. somehow every node connected to the internet has to all be switched off at the same time and restarted to restore connectivity). Most likely it's already been tried too I guess.
    - dylan604 4y ago
      usually, it's a nefarious evil doer that is threatening this action in the movies vs we gotta reboot the city. there's been reboot the sun plot. there's been reboot the earth's core plot. there's been reboot the machine running the city plot.
      - wizofaus 4y ago
        I was thinking more along the lines of a rogue self-replicating packet where every last instance of had to die before routers and switches etc. would start working again.
gnopgnip 4y ago
Spanning tree protocol will prevent broadcast/arp storms by preventing loops

eikenberry 4y ago

I have had 2 external USB-power-passthrough laptop doc/hub things with an Ethernet port. They both cause a packet storm on the network if you unplug the computer and leave the ethernet and power plugged in. Causes all my crappy realtek NICs to overheat and flake out. Not exactly the same but super annoying.

jfim 4y ago
I had the same issue with an USB Ethernet adapter plugged into a powered USB hub, disconnecting the computer would make the network crap out. Kind of defeats the purpose of having a docked setup with wired Ethernet if one can't undock it or needs to unplug tons of cables each time.
- smashed 4y ago
  See Ethernet pause frames Usb ethernet adapters will send these pause frames, waiting for the computer to reconnect.. but they end up flooding dumb switches.. https://news.ycombinator.com/item?id=31268134 https://news.ycombinator.com/item?id=31268134

xoa 4y ago

Depending on what network gear you're using (I'm now switching to Omada, previously/still on some site son UniFi, but lots of even 'prosumer' stuff does this) there are specific mitigations available. All of this falls under the heading of "managing broadcast traffic", which is very important even for smaller networks. The three major categories of traffic on a network are unicast, broadcast, and multicast. Unicast is the normal case of one device talking to a single address. Broadcast involves sending a packet to every single possible recipient in the entire broadcast domain (almost always the subnet). ARP fits in here. Multicast is essentially in between, more efficient then broadcast, can still talk to multiple devices that have signed up to hear it. Obviously an actual broadcast storm can take down an entire network, but excessive broadcast traffic on WiFi specifically can also suck up a huge amount of airtime for little bandwidth. Every single device has to go to the slowest speed and stop what they're doing to listen and make sure no one is left out. Using STP/RSTP with proper values set and LACP for aggregated interfaces can help prevent inadvertent network loops. Many switches also support some kind of port isolation and explicit per port storm control restricting max numbers of packets/second for unicast/broadcast/multicast traffic. WiFi APs can use proxy ARP to cut it in their domain too. The WAP already of course knows the MACs of every device connected to it by definition, so there isn't generally any reason not to have it answer ARP requests on their behalf then forward the traffic itself.

donmcronald 4y ago
Why the switch to Omada? I’m looking at doing the same due to supply issues with the UniFi stuff. Any tips or opinions on Omada vs UniFi?
- xoa 4y ago
  Opinions hooboy do I have them ;), I was just asked that exact question on HN a couple of weeks ago in fact. Gave it a shot in a response here [0] which still applies. But basically Ubiquiti has become a toxic dumpster fire of a company and their product lines (UniFi in particular) on a downward trajectory in terms of performance, features and stability for quite a while. I had a certain amount invested in UniFi (think the final total will end up as a few hundred devices) so it's been a staged switch, with a total change of all routing/gateway/security functions to OPNsense completed first. That bought a lot more runway, it's always been the weakest and most neglected area in the ecosystem while obviously also being pretty critical. Yet the Ubiquiti debacle has served to underline for me how valuable self-host is, I've been able to have a nice slow ramp and deal with their implosion precisely because UniFi/UNMS/UISP and all the hardware is fully under my control. So I've been hoping someone would come along and see the potential of the UniFi niche of the networking market and basically copy it without all the junk. Which seems to basically be Omada to a tee. I'd actually originally (and still at many sites) intended to hold off and wait for WiFi 7 gear, because at that point a bunch of clients (and myself for that matter) will be interested in replacing WAPs anyway which is a very natural point to consider changing manufacturer as well. But a breaking point has come at a few places with a final feature which is PPSK, allowing the system to have many different passwords for an SSID that can be assigned different tags. Basically it allows having many of the benefits of WPA-Enterprise in terms of segmenting different clients onto different VLANs and revoking credentials and the like with more security and less manual work than MAB (MAC bypass) while still looking like a normal PSK scheme, which means the vast universe of brand new stuff which doesn't support 802.1x and never will works with it happily (by the same token none of that is going to play directly with using a secure virtual network or other better systems either sadly). Lower overhead and better compatibility than captive portals for non or semi-interactive devices as well. Someone hacked together a demo showing this could work on UniFi WAPs like four freaking years ago and Ubiquiti never did anything with it in favor of endless bikeshedding GUI changes to add more white space and hide important features and information (yes I'm a touch bitter). So I'm not in the position of wholeheartedly recommending Omada yet, I don't have years under my belt there and it's relatively speaking fairly new. It has its own warts and rough edges for sure, from the software to the hardware physical design. But it can be self-hosted and the trajectory looks massively better, has already had more meaningful improvement in months than UniFi has had in years, seems to perform much better so far as well. Of course the Venn diagram of self-hosting, herding lots of hardware with single pane, fully networking features, ecosystem richness and so on is pretty minimal in the overlap. Take away any one or multiple of those and options expand a lot, Aruba InstantOn for example. And welp, this didn't end up "basically" at all did it, sorry about that. I am bummed by the sheer wasted potential with Ubiquiti. So it goes in tech over and over again though, we've all seen this movie many, many times. As far as tips, I would suggest if you plan to stay on the managing-your-own-networks route to very strongly consider having the router/gateway stuff be separate and fully open source as I ended up. Doesn't have to be OPNsense, could be VyOS or plain OpenBSD or whatever else you're most comfortable with and depending on how you want to manage stuff and what needs there are for others to take over. But it's very, very pleasant to have the full spectrum of quality PC hardware available, you can get far more power for less, and you're never stuck with a critical aspect. I'd still suggest generally running that on metal rather than virtualizing it in a (semi)production network, but opinions vary there. ---- 0: https://news.ycombinator.com/item?id=32297556 https://news.ycombinator.com/item?id=32297556

neurostimulant 4y ago

Not sure if it's something similar but I had issue where attempting to setup a wifi smart plug locked up the router for a minute (until the smart plug gave up trying to connect to the router). Wired ethernet still works but the routers show 100% cpu usage on its management interface and the 2.4ghz wifi stopped working (didn't check the 5ghz one). I didn't dig in more because my wife was in a zoom meeting.

comprambler 4y ago

Similar, but not exact, are deb, and likely others, with failed storage will continually dhcprenew as they can’t write the leasefile for the interface. Near ddos as far as windows dhcp server is concerned. Required an arp table trace to find. Bad_Address is usually the symptom.

evulhotdog 4y ago

Yep. There’s actually a known issue with the CalDigit USB C Pro dock that causes this. It’s when the laptop comes up from or goes to sleep and it takes down the whole network. Fortunately there’s a firmware fix for it but it took a while to put two and two together.

zamadatix 4y ago
I thought the dock thing was related to excessively sending pause frames to switches that didn't know to not forward them.
- evulhotdog 4y ago
  The official documentation is here: https://www.caldigit.com/the-usb-c-pro-dock-takes-down-my-network/ https://www.caldigit.com/the-usb-c-pro-dock-takes-down-my-ne...
iforgotpassword 4y ago
This freaking issue! This has plagued humanity for decades now I think. I had an older ThinkPad at work once that did that when it went to sleep. Fun for the whole fami^Wgroup of colleagues sharing the switch. Pause frames must die, leave flow control to a higher layer.
microtonal 4y ago
I think more adapters/docks with an RTL8153 suffer from this issue. E.g., I had a Lenovo USB-C Gen 2 dock that would fairly reliably take down our office network after suspend-wake. Just avoid Realtek NICs like the plague.
Abishek_Muthian 4y ago
I had a Apple USB 2.0 USB dongle which would crash the OSX when the router (DDWRT) is rebooted and sometimes would crash the router when the system was rebooted. Didn't do that with other combinations of OS/Router. Bug bounty opportunity missed?
- 4y ago

Abishek_Muthian 4y ago

Not ARP storms per se, But I've noticed new Apple devices don't like router forwarding port 53 to pihole, resulting in bonjour storm and crashing pihole instances(Loosing DNS for the entire network).

ungamed 4y ago
If you can crash dns this way, please report it, its worthwhile fixing.
- Abishek_Muthian 4y ago
  I'll try, Didn't have good experience with my earlier bug reports to Apple and now I'm completely out of that ecosystem. In fact I found this issue when my guests have iOS devices. Anyways here's the regex blacklist I use in PiHole to prevent Apple's reverse DNS spam - ^(?:(?:\d+\.)*\d+(?:-in\.addr\.arpa\.)?)$