11 ms·

Automatically fixing packet loss on my connection

23 points by robotmay 6y ago

mike_d 6y ago
If you are willing to throw a bit of money rather than code at the problem: Ubiquiti has a power strip in beta now that uses statistics from their routers to decide when to power cycle upstream devices. You can also find "Web Power Switches" (https://www.amazon.com/dp/B0765NCB2L https://www.amazon.com/dp/B0765NCB2L) under $200 that are scriptable, so when a ping to a specific site fails it cycles power to a set of devices.
- robotmay 6y ago
  Ooh I didn’t know about that Ubiquiti strip. I must admit that I didn’t even look about for existing solutions (honestly I just wanted to finally have a use for these plugs I had lying about).
  - thisisnico 6y ago
    Working in IT as a Systems Engineer, My thought process is always first -> Existing solution or Tool, if not exist, Patching existing tools together, if not exist, code our own solution.
    - robotmay 6y ago
      Hah yes in this case I already had most of the tools lying about. In fact the hardest part was figuring out why the hell the barrel-jack didn’t fit in the modem. Ended up finding an Amazon review with the correct connector size.
- readmodifywrite 6y ago
  That's kind of ironic, considering they don't provide a scheduled reboot feature for their own gear (which often needs it for IoT usage).
whoisburbansky 6y ago
This is incredibly cool! I’ve never worked with Z-Wave plugs before, the only home automation stuff I have set up involves WiFi connected smart plugs running Tasmota. How do you get the Z-Wave ones to talk to your Home Assistant setup?
- tamu_nerd 6y ago
  Typically this is done with something like an Aeotec Z-Stick Gen5.
  - robotmay 6y ago
    This is what I’m using. The plugs are also Aeotec and work well, although there is some minor high audio frequency when very close to them. Mine are a few years old though. The Z-Wave integration for Home Assistant is basically plug and play with this stick. I run HA in docker and just pass the device through, works nicely.
    - whoisburbansky 6y ago
      Ah, cool, that's good to know. Another piece of kit for the raspberry Pi, glad to hear it's relatively straightforward.
    - tamu_nerd 6y ago
      One word of caution, the world of Z-Wave with Home Assistant is about to be flipped on its head due to some developer personal constraints. https://github.com/home-assistant/architecture/issues/483 https://github.com/home-assistant/architecture/issues/483
tareqak 6y ago
Why does this packet loss happen, and why does restarting the router fix it?
- dilyevsky 6y ago
  Because isp modems are almost universally garbage. I just shutdown -r mine at 4am using cronjob+ssh
  - sq_ 6y ago
    Even more annoyingly, if you get fed up with their crappy hardware and get your own, any and all future flakiness will be attributed by ISP support to using nonstandard hardware.
    - dilyevsky 6y ago
      A lot of us isps straight up won’t allow byo routers so that they can collect $10/mo “equipment fee”. Looking at you att
    - Nextgrid 6y ago
      Except his ISP is explicitly not like that and allows people to use whatever equipment they want.
- robotmay 6y ago
  A&A actually have a nice explanation on their wiki: https://support.aa.net.uk/Packet_Loss https://support.aa.net.uk/Packet_Loss
lrossi 6y ago
Basically restart the modem by cutting the power every time the connection is flaky. With a very complicated setup. Cutting power repeatedly a large number of times is a great way of eventually bricking the box. This modem has an admin web interface. The author could have used it to trigger a restart. The whole thing could have been automated with just a pi at 100 times less cost, effort and risks.
- lrossi 6y ago
  On the upside, if he manages to brick it, the ISP will replace it and maybe his problem will be fixed.
  - robotmay 6y ago
    I’ve never had a device brick by removing the power. The only button on the Vigor 130 immediately cuts the power too: there’s no “shutdown” button.
    - lrossi 6y ago
      Your device’s firmware is not even able to keep a connection up for a few hours. Which is its main function. Do you trust that they tested its code for recovering partitions that were not unmounted cleanly?
      - robotmay 6y ago
        There’s no problem with the device: I triggered the failures/packet loss manually by pulling the cable to test the system. The actual packet loss is intermittent and reasonably rare, and is I suspect a fault at the line level with BT/OpenReach. A&A didn’t provide me with the Vigor 130 btw, it’s my own device. In general I find DrayTek devices very solid and I’ve not had any issues so far when powercycling it over the past few years. It’s pretty much a stateless device, as the actual connection is established from the router on my side of it.
- robotmay 6y ago
kraftman 6y ago
Won't this cause issues with retraining? You'll end up with a high SNR and lower sync speed, which may solve the packet loss problem but isn't ideal. It would be better to either work out whether the router is the issue and replace it or work out if the line is a problem and get an engineer booked out to investigate it.
- robotmay 6y ago
  It’s quite a peculiar issue when it does happen (which is rarely), in that the only way I’ve found of resolving it myself is by dropping the line at that modem. I’m always hesitant to have the line inspected in the UK because there’s every chance OpenReach will come around and balls it up further. I’ve been doing this manually for the past couple of years without any negative effects that I’ve seen. I’d say it tends to happen maybe once every two to three months, which makes me think it’s more likely an issue further up the chain. A&A are pretty spry at resolving issues but often I just drop the line myself to sort it out quicker.
  - philjohn 6y ago
    Have you ruled out the Draytek?
    - robotmay 6y ago
      I don’t have anything to swap it out with unfortunately, and due to the infrequent nature of the issue it’d be pretty hard to predict. Intermittent packet loss is something I’ve experienced everywhere I’ve lived in the UK and with different devices. The only constant has been the BT/OpenReach network on which many of our ISPs are based :|
      - bcrl 6y ago
        ADSL modems cost less than $10 used. Replacing the modem should have been done well before trying this hairbrained workaround that doesn't actually fix the real problem.
      - 6y ago
lupinglade 6y ago
PingDoctor is a great tool for locating and reporting these issues: https://geo.itunes.apple.com/us/app/pingdoctor/id1350044974?mt=12&app=apps https://geo.itunes.apple.com/us/app/pingdoctor/id1350044974?...
Nextgrid 6y ago
This seems like an over-engineered, half-assed solution like putting tape on a check engine light. There is no reason why properly working equipment should need a periodic restart. Either his modem is bad and needs to be replaced or the other end (the DSLAM) is bad and bouncing the link temporarily fixes the problem as a side-effect.
- tinus_hn 6y ago
  Many of these devices are indeed bad and your choice is to replace them with another one that’s exactly as bad.
- robotmay 6y ago
  The modem is fine; this is a very intermittent issue, like once every two months, if that, and caused further up the line. In the UK most of our internet infrastructure is, frankly, a bit shit, and maintained by a small subset of companies. These issues occur on every ISP I've had here, as unfortunately most rely on OpenReach's infrastructure. The reason I chose to automate it is that when I'm away from the house I want the connection to self-repair where possible, and resetting the modem is something basically everyone in the UK has to do sometimes to fix their line (seriously). Yes I could probably have A&A investigate it, but at some point, if it is an identifiable issue, they'd likely have to request an OpenReach engineer to work on something and then there's just the risk of it being made even worse. RevK's blog has plenty of interesting anecdotes about the issues with our internet infrastructure here: https://www.revk.uk/search/label/BT https://www.revk.uk/search/label/BT
cyounkins 6y ago
As A&A's wiki (https://support.aa.net.uk/Packet_Loss https://support.aa.net.uk/Packet_Loss) describes, you can see packet loss at link saturation. There is another option, buffering, which causes high latency. The intermediate device must either buffer (causing "buffer bloat" = increased latency) or drop the packet. If increased packet loss is occurring seemingly randomly, it could be something on the network is updating/syncing and saturating the link. It could be Steam games, desktop OS updates, mobile device updates, podcast downloading, game console updates, etc.
- robotmay 6y ago
  Yeah I have run into this, notably with the Xbox Live app for Windows, which totally saturates the line and generates something like 15% packet loss reports. However that's a different event to the occasional lower but prolonged packet loss which occurs infrequently and usually on a more idle line; which I believe to almost certainly be caused by something beyond my equipment. Knowing how OpenReach operate (who manage our largest internet infrastructure) it's a fairly safe assumption that it's a fault they've introduced somewhere.

23 points by robotmay 6y ago

If you are willing to throw a bit of money rather than code at the problem: Ubiquiti has a power strip in beta now that uses statistics from their routers to decide when to power cycle upstream devices. You can also find "Web Power Switches" (https://www.amazon.com/dp/B0765NCB2L https://www.amazon.com/dp/B0765NCB2L) under $200 that are scriptable, so when a ping to a specific site fails it cycles power to a set of devices.

robotmay 6y ago
Ooh I didn’t know about that Ubiquiti strip. I must admit that I didn’t even look about for existing solutions (honestly I just wanted to finally have a use for these plugs I had lying about).
- thisisnico 6y ago
  Working in IT as a Systems Engineer, My thought process is always first -> Existing solution or Tool, if not exist, Patching existing tools together, if not exist, code our own solution.
  - robotmay 6y ago
    Hah yes in this case I already had most of the tools lying about. In fact the hardest part was figuring out why the hell the barrel-jack didn’t fit in the modem. Ended up finding an Amazon review with the correct connector size.
readmodifywrite 6y ago
That's kind of ironic, considering they don't provide a scheduled reboot feature for their own gear (which often needs it for IoT usage).

whoisburbansky 6y ago

This is incredibly cool! I’ve never worked with Z-Wave plugs before, the only home automation stuff I have set up involves WiFi connected smart plugs running Tasmota. How do you get the Z-Wave ones to talk to your Home Assistant setup?

tamu_nerd 6y ago
Typically this is done with something like an Aeotec Z-Stick Gen5.
- robotmay 6y ago
  This is what I’m using. The plugs are also Aeotec and work well, although there is some minor high audio frequency when very close to them. Mine are a few years old though. The Z-Wave integration for Home Assistant is basically plug and play with this stick. I run HA in docker and just pass the device through, works nicely.
  - whoisburbansky 6y ago
    Ah, cool, that's good to know. Another piece of kit for the raspberry Pi, glad to hear it's relatively straightforward.
  - tamu_nerd 6y ago
    One word of caution, the world of Z-Wave with Home Assistant is about to be flipped on its head due to some developer personal constraints. https://github.com/home-assistant/architecture/issues/483 https://github.com/home-assistant/architecture/issues/483

tareqak 6y ago

Why does this packet loss happen, and why does restarting the router fix it?

dilyevsky 6y ago
Because isp modems are almost universally garbage. I just shutdown -r mine at 4am using cronjob+ssh
- sq_ 6y ago
  Even more annoyingly, if you get fed up with their crappy hardware and get your own, any and all future flakiness will be attributed by ISP support to using nonstandard hardware.
  - dilyevsky 6y ago
    A lot of us isps straight up won’t allow byo routers so that they can collect $10/mo “equipment fee”. Looking at you att
  - Nextgrid 6y ago
    Except his ISP is explicitly not like that and allows people to use whatever equipment they want.
robotmay 6y ago
A&A actually have a nice explanation on their wiki: https://support.aa.net.uk/Packet_Loss https://support.aa.net.uk/Packet_Loss

lrossi 6y ago

Basically restart the modem by cutting the power every time the connection is flaky. With a very complicated setup. Cutting power repeatedly a large number of times is a great way of eventually bricking the box. This modem has an admin web interface. The author could have used it to trigger a restart. The whole thing could have been automated with just a pi at 100 times less cost, effort and risks.

lrossi 6y ago
On the upside, if he manages to brick it, the ISP will replace it and maybe his problem will be fixed.
- robotmay 6y ago
  I’ve never had a device brick by removing the power. The only button on the Vigor 130 immediately cuts the power too: there’s no “shutdown” button.
  - lrossi 6y ago
    Your device’s firmware is not even able to keep a connection up for a few hours. Which is its main function. Do you trust that they tested its code for recovering partitions that were not unmounted cleanly?
    - robotmay 6y ago
      There’s no problem with the device: I triggered the failures/packet loss manually by pulling the cable to test the system. The actual packet loss is intermittent and reasonably rare, and is I suspect a fault at the line level with BT/OpenReach. A&A didn’t provide me with the Vigor 130 btw, it’s my own device. In general I find DrayTek devices very solid and I’ve not had any issues so far when powercycling it over the past few years. It’s pretty much a stateless device, as the actual connection is established from the router on my side of it.
robotmay 6y ago

kraftman 6y ago

Won't this cause issues with retraining? You'll end up with a high SNR and lower sync speed, which may solve the packet loss problem but isn't ideal. It would be better to either work out whether the router is the issue and replace it or work out if the line is a problem and get an engineer booked out to investigate it.

robotmay 6y ago
It’s quite a peculiar issue when it does happen (which is rarely), in that the only way I’ve found of resolving it myself is by dropping the line at that modem. I’m always hesitant to have the line inspected in the UK because there’s every chance OpenReach will come around and balls it up further. I’ve been doing this manually for the past couple of years without any negative effects that I’ve seen. I’d say it tends to happen maybe once every two to three months, which makes me think it’s more likely an issue further up the chain. A&A are pretty spry at resolving issues but often I just drop the line myself to sort it out quicker.
- philjohn 6y ago
  Have you ruled out the Draytek?
  - robotmay 6y ago
    I don’t have anything to swap it out with unfortunately, and due to the infrequent nature of the issue it’d be pretty hard to predict. Intermittent packet loss is something I’ve experienced everywhere I’ve lived in the UK and with different devices. The only constant has been the BT/OpenReach network on which many of our ISPs are based :|
    - bcrl 6y ago
      ADSL modems cost less than $10 used. Replacing the modem should have been done well before trying this hairbrained workaround that doesn't actually fix the real problem.
    - 6y ago

lupinglade 6y ago

PingDoctor is a great tool for locating and reporting these issues: https://geo.itunes.apple.com/us/app/pingdoctor/id1350044974?mt=12&app=apps https://geo.itunes.apple.com/us/app/pingdoctor/id1350044974?...

Nextgrid 6y ago

This seems like an over-engineered, half-assed solution like putting tape on a check engine light. There is no reason why properly working equipment should need a periodic restart. Either his modem is bad and needs to be replaced or the other end (the DSLAM) is bad and bouncing the link temporarily fixes the problem as a side-effect.

tinus_hn 6y ago
Many of these devices are indeed bad and your choice is to replace them with another one that’s exactly as bad.
robotmay 6y ago
The modem is fine; this is a very intermittent issue, like once every two months, if that, and caused further up the line. In the UK most of our internet infrastructure is, frankly, a bit shit, and maintained by a small subset of companies. These issues occur on every ISP I've had here, as unfortunately most rely on OpenReach's infrastructure. The reason I chose to automate it is that when I'm away from the house I want the connection to self-repair where possible, and resetting the modem is something basically everyone in the UK has to do sometimes to fix their line (seriously). Yes I could probably have A&A investigate it, but at some point, if it is an identifiable issue, they'd likely have to request an OpenReach engineer to work on something and then there's just the risk of it being made even worse. RevK's blog has plenty of interesting anecdotes about the issues with our internet infrastructure here: https://www.revk.uk/search/label/BT https://www.revk.uk/search/label/BT

cyounkins 6y ago

As A&A's wiki (https://support.aa.net.uk/Packet_Loss https://support.aa.net.uk/Packet_Loss) describes, you can see packet loss at link saturation. There is another option, buffering, which causes high latency. The intermediate device must either buffer (causing "buffer bloat" = increased latency) or drop the packet. If increased packet loss is occurring seemingly randomly, it could be something on the network is updating/syncing and saturating the link. It could be Steam games, desktop OS updates, mobile device updates, podcast downloading, game console updates, etc.

robotmay 6y ago
Yeah I have run into this, notably with the Xbox Live app for Windows, which totally saturates the line and generates something like 15% packet loss reports. However that's a different event to the occasional lower but prolonged packet loss which occurs infrequently and usually on a more idle line; which I believe to almost certainly be caused by something beyond my equipment. Knowing how OpenReach operate (who manage our largest internet infrastructure) it's a fairly safe assumption that it's a fault they've introduced somewhere.