7 ms·
Someone needs to explain why it took 18 months to figure this out considering how simple it was to detect. You have a problem that's repeatable right on schedu
by Nextgrid 6y ago
Someone needs to explain why it took 18 months to figure this out considering how simple it was to detect.
You have a problem that's repeatable right on schedule, that presumably shows up in the logs and affects multiple customers. It should be a no-brainer to send someone with test equipment out there to measure things in the field while the problem was happening.
- bayindirh 6y ago> that presumably shows up in the logs Ah, no. SNR problems in the shared medium is not something wathced or logged. I remember losing internet connection on heavy rains sometimes because an odd cable shorted at an underground gallery. It wasn't FTTX then, so it was not easy to find. > You have a problem that's repeatable right on schedule. Similar problems rooted to bandwidth scarcity happened all the time, so in a village I'm not sure there's a lot of people who can discern the difference between, ah it's slow again and Oh, I lost connection, look at that SNR value. I better write a technical report to ISP. > It should be a no-brainer to send someone with test equipment out there to measure things in the field while the problem was happening. Even the simplest problem, if it's happening 1 in 10.000 is not a no-brainer because, you won't think that'd happen. Especially because of an old TV.
- Nextgrid 6y ago> SNR problems in the shared medium is not something wathced or logged This is a problem then and should be addressed. > problems rooted to bandwidth scarcity happened all the time But the ISP should be able to tell whether their uplink is saturated (and actually upgrade it if it happens too often), so if they're consistently getting complaints around a certain timeframe and the uplink is not saturated during those timeframes then it must be something else. > Especially because of an old TV But the diagnostic procedure had nothing to do whether it's an old TV or something else. It was caused by EMI and is detectable with readily-available test equipment which indeed detected it immediately. Furthermore monitoring packet loss at the PPP level can be done (in fact, Andrews & Arnold - a small ISP in the UK - does it) so if an ISP doesn't have this in place to detect widespread faults then something is also wrong there - why are they operating with zero visibility on their network?
- bayindirh 6y ago> But the diagnostic procedure had nothing to do whether it's an old TV or something else. It was caused by EMI and is detectable with readily-available test equipment which indeed detected it immediately. They concluded that something is fishy because normal procedures didn't fix anything. I'm not sure that this is a problem they diagnose 5 times a week. > But the ISP should be able to tell whether their uplink is saturated (and actually upgrade it if it happens too often), so if they're consistently getting complaints around a certain timeframe and the uplink is not saturated during those timeframes then it must be something else. Good luck escalating this problem from a village to correct people in a big-shiny-ISP(TM). They're too busy(!) to look for 10 customers' problem. I accept that 18 months in unacceptable but, this is the reality we're in. Support and Ops teams are always overworked because management thinks that "the load just doesn't justify more hands in that department". > This is a problem then and should be addressed. The backbone operators and ISPs doesn't monitor SNR levels because they're transient and doesn't provide meaningful data. If you want to do it, it's millions of connections. Also, Not everyone use the standard equipment provided by the ISP so, they cannot get info from all devices (e.g.: from mine). So, endpoint ops are complex if you're working on a WAN with random equipment. Instead, using bandwidth graphs to find problems is more practical and reliable. Since the problem starts at 07:00, it shows like "They're not using their connections much, eh?" in the said graphs. Disclosure: My workplace also happens to be the national academic network operator. I'm two hops away from a country-wide core router.
- Nextgrid 6y ago> because normal procedures didn't fix anything But why did it take 18 months to realize that? You've got an issue that occurs consistently and is preventing customers from getting the service they're paying for (and are presumably in contracts so they can't even cancel without a major fight). > They're too busy(!) to look for 10 customers' problem So we're in agreement that this is unacceptable and resolving this problem has been grossly mismanaged and someone should be held accountable? > doesn't monitor SNR levels because they're transient and doesn't provide meaningful data This seems weird to me. You've got a nationwide network designed to run on old, poor quality wiring (the article doesn't specify whether it's DSL or coax, but it would apply to both). Wiring issues should be expected and automated processes in place to monitor it and quickly detect issues like this one. > Not everyone use the standard equipment provided by the ISP Can't this be collected at the DSLAM or CMTS level? Or in case of DSL, even at the PPP level? (Andrews & Arnold - a small DSL ISP in the UK - measures packet loss at that level and while it's not true SNR, it would be enough for these purposes as localized packet loss would pretty much always mean bad SNR or failing equipment)
- C1sc0cat 6y agoThis is openretch (openreach) not Virgin so not sure where you get shared medium from I do know people who worked in radio interference and they told me one time they traced a fault to a Budgie cage. The Budgie had a swing and when it was in use it caused a short which effected the subs (customer) line.
- bayindirh 6y agoIf you're running network over coax alongside cable TV, it's shared medium. I'm not in the UK or US though, so regardless of their name, every ISP is equal and is indeed a bog standard ISP from my vantage point.
- milankragujevic 6y agoBT openreach mainly does DSL and FTTH, AFAIK. edit: Just to clarify, FTTH is by definition a shared medium, but the huge capacity in practice means that there is little congestion usually. Also, since glass fiber is a dielectric, it is difficult to get noise in the signal, and it is immune to EMI. DSL is by definition a dedicated medium, having a twisted pair wire from the DSLAM at the central office or street cabinet to the user premises into the [splitter and] modem. One user per twisted pair only. However, the wires are not shielded and thus are susceptible to both crosstalk from other pairs and also EMI given that it's using electrical signals to transfer data. And of course, the speeds are low for most users due to the local loop length and in general VDSL2 35b maxes out at 300/100 Mbps at around ~80 meters, and VDSL2 17a maxes out at 100/50 Mbps at around 100 meters.
- LargoLasskhyfv 6y agoBy that you mean GPON, I guess? Not so here, where it is PTP. And an ISP with excellent peering.
- milankragujevic 6y agoEdit: disregard my comment, I mistook PTP to mean Ethernet while in reality it's Point to point Fiber. In any case my point about noise still stands. But PTP Fiber is really rare in the world, given how many people are stuck on coax. Yes, with FTTH (or FTTP) I meant GPON. Refarding PTP, do you mean a residential ethernet network? (Technically FTTB, copper / ethernet to the premises) Branding anything other than FTTH as "Fiber" is misleading. It's like saying DOCSIS 3.0 is Fiber because its a Hybrid Fiber-Coaxial network, while in reality all cable internet networks are HFC, the difference is the number of nodes serviced by a CMTS, which when reduced by a node split will result in higher speeds and more capacity. Or saying DSL is "Fiber" because uplink to the DSLAM is Fiber. All DSLAM uplinks are Fiber, difference is how many street cabinets are there because DSL speed is dictated by the local loop length, and with more street cabinets there is higher chance one gets built closer to you enabling you faster speeds.
- lima 6y ago> Ah, no. SNR problems in the shared medium is not something wathced or logged. A good ISP definitely does monitor SNR. It's a basic metric available from each individual customer device.
- bayindirh 6y ago> It's a basic metric available from each individual customer device. Not mine. Good luck accessing to the modem you provided which I reconfigured to be a dumb bridge and terminated the ADSL connection on my router. Oh, the router neither see the POTS line (it only gets Ethernet) nor have remove management capabilities. Where endpoints are heterogeneous, there's guaranteed, real fun (Even more so if you make assumptions).
- deleted 6y ago[deleted]
- lima 6y agoThey still have SNR metrics on their end of the connection. And this is why providers hate heterogeneous networks :-) Mine politely asks you to plug in the original modem before they'll help you debug a connectivity issue. But even then, there are sometimes standards like TR-069 that apply across vendors.
- JosephRedfern 6y ago> Ah, no. SNR problems in the shared medium is not something wathced or logged. I remember losing internet connection on heavy rains sometimes because an odd cable shorted at an underground gallery. It wasn't FTTX then, so it was not easy to find. Even if SNR itself isn't logged, dynamic line management would surely result in dropped speeds, which would be logged, and would stand out as anomalous?
- tonyb 6y agoThis isn’t accurate. The amount of data available on a DOCSIS network is incredible. Per modem SNR, power levels, preEq data, etc. If the data is used properly it cans find a loose screw on a tap or a damaged cable down the the foot. That being said impulse noise is one of the harder things to track down. If the modern tools don’t work most MSOs would end up doing the “unplug a leg and see if the SNR gets better” method until they found the offending house.
- bryanrasmussen 6y agoconsidering the thousands of bad ux experiences to outright bugs I encounter every day, considering that some of them are bugs that have lasted for more than a year, considering I just wrote up a 20 page proposal on fixing the search for a pretty big Scandinavian site is obviously costing them money by not having it fixed on stuff that has been the same way for years, considering the governmental sites I am forced to use that are abusively bad and that companies put up job advertisements bragging about how they made these sites and that if you come work for them you will get to work on those sites as well... considering all that, why do they need to explain something took 18 months to fix that affected a few people again? Because I'm starting to think you must live in some sort of paradise I have been cast out of, and it's not fair!
- aussieguy1234 6y agoIt's not surprising. As a software engineer I've seen bugs that took days to track down which were fixed with a single line of code.
- jacquesm 6y agoWith removing a single line of code, and in some cases even just a spurious period or comma.
- kn100 6y agoYou'd be surprised how hard it is to identify SHINE. In places like this, you'll probably find the fastest broadband available is actually ADSL2+ - which is highly susceptible to noise, and in some places in the UK the lines which run to peoples houses aren't even copper, they're ALU - which is obviously not ideal. Even the tiniest of things can have an enormous impact on line performance. I worked at a UK ISP a couple years back and lets just say the 'monitoring' that ISPS have is extremely limited. Openreach (the company that manages the phone lines, which ISPs lease) almost certainly has more monitoring but the problem is the customer calls the ISP, and the ISP then tells Openreach - which means there's a fairly huge disconnect between the customer and the engineers fixing the line.
- tialaramex 6y ago> I worked at a UK ISP a couple years back and lets just say the 'monitoring' that ISPS have is extremely limited. That's a choice. Presumably a cost conscious choice, not a limit imposed by Openreach. A&A presents each customer with a live updated chart of bandwidth usage, latency, and low-level polling of the DSL modem in the customer's home. If your line drops even if you aren't "using" it for anything, you can opt to get a SMS about that, and choose whether this applies at weekends or overnight. They even plot the rate of "login"/ "logout" events for the modems, both as a whole and mapped onto exchange locations so that you can see if there's a problem beyond just you got unlucky this time.
- donalhunt 6y agoMany years ago when I was having cable modem issues, I used to scrape the SNMP off the modem and map stuff like that locally. Was very helpful when diagnosing issues with Level 1 support. "yes... it happens 20 times a day at these times..." Since then the firmware on devices seems to have either dropped those types of features (cost reduction) or restricted them to an ISP-accessible interface ("security"). :(
- dwd 6y agoInterference is probably way down the list of potential causes. The first customer complaint was likely just one customer to begin with and could have reported that the issue was occurring at a different time of day depending on when they used the Internet. It wouldn't have been until they had multiple complaints and eventually a user who could pinpoint losing connection at 7:00am that they looked at it being something external to the individual customers connection.
- xenadu02 6y agoIt says they did a cable replacement. I suspect that took some time and they thought it would resolve the problem. They were probably quite surprised when it didn't.
- mannykannot 6y agoI guess someone who thinks 'fixing' a problem, without finding out what is causing it, has a high probability of success, would be surprised when it did not work. They are probably surprised every time it doesn't work.
- aaron695 6y agoCan you explain what has happened here at a technical level? What was the TV interfering with exactly? The ISP's equipment coming into the village? I don't understand how you know it was simple. It's not clear to me what happened in the BBC article. > You have a problem that's repeatable right on schedule This is incorrect, the TV had to be turned on. Hyperbolic quotes on BBC like "It turned out that at 7am every morning " are not actually going to be true in the field.
- stoobs 6y agoShort answer - it's BT. From personal experience it probably took most of that time to escalate the issue high enough for someone to take it seriously, then a few visits to get the right people onsite. It would have probably been quicker to migrate the village to FTTP
- memsom 6y agoNo it is worse than that, it is Openreach. BT spun them off, and they are not the same company anymore - this was to avoid being told they had a monopoly on the lines. But in truth, BT get a lot more priority with Openreach than any other provider, but the quality of the service Openreach provides is inadequate at best anyway. Relevant story - we moved in to a new build and waited 3 months to get ADSL. Not anything fancy, just plain old ADSL. The provider we went with was Talk Talk, and they basically had to go to Openreach and ask them to sort out the issue (turned out to be something to do with over capacity at the exchange.) This dragged on for months. I spent hours on the phone talking to someone in India, who then talked to someone in the UK who then got on to Openreach. It was a wholly terrible experience. I finally lost my proverbial s??t and complained directly to Openreach (and believe it or not actually got a response with the real reason for the delay - some infrastructure through which caballing went through had collapsed and they needed to schedule it to be fixed, but as it was under a dual carriage way, that was taking some time. Still possibly a lie, but better than the nonsense I was getting from the guy in India.) So as they had been lying to me constantly, I gave up and cancelled the contract, and we went with BT instead because we could then use the FON wireless of our neighbours in the interim. Well, one week later the ADSL came on. This was at least 2 weeks before it was scheduled by Openreach and BT, who were aware of the issue. So all we can assume is someone magically fixed it. To this day I don't know if BT and Openreach had some special relationship or if we were just lucky and BT juggled something in the exchange, but - yeah, 4 years later we have cruddy ADSL still, and no Fibre or Cable.
- sgt101 6y agoYou get what you pay for from Openreach - or more precisely the service provider pays little, and tells you it's Openreach while you pay lots. Sky and BT retail definitely pay for the top options of services.
- andybak 6y agoIt probably takes 18 months to get a support case escalated to the point where a real engineer looks at it.
- jve 6y agoProbably. From the wording in article they BEGAN solving the issue after 18 months. But then, it was only after some cable has been replaced - that could have taken a while to plan, budget and actually do, then again collect feedback. > After 18 months engineers began an investigation after a cable replacement programme failed to fix the issue.
- Macha 6y agoIn Ireland, not the UK, but from what I hear the ISP market is pretty similar. 1. You have an issue with your broadband in a rural area. You call up your ISP. You get some handwavy answer about contention and they ask you to restart your router and call them back if it happens again. 2. The next time they agree to have an engineer take a look. You hear nothing back. You contact them again, they say their engineer looked at the exchange and found no issues. 3. The next time you ring them and they agree to send an engineer to your house. The engineer (contracted out) arrives at 2pm, they check your modem, rant a bit off the record about how they don't fund shit, then determines that everything's working fine at 2pm, and it's the ISP's problem that their network collapses at 6pm because they oversold the rate of your village. 4. You contact the ISP. They read the engineer's report, saw it was working fine at 2pm, and decided they didn't need to take action. They suggest using the internet at another time of day, such as when you and most of your neighbours are away from the house and at work. 5. If enough people in the area decide they complain they might finally deign to do something about their service in a few months. More likely is some other ISP comes around and signs up all the customers so pissed off with the last one, until most of the village is now on this ISP and the same problems reocur. 6. After like 4 or so years you've looped through all the ISPs, maybe there's a new generation of tech which promises higher speeds but they can't be bothered funding anything locally so your actual speed and reliability is unchanged.
- awalton 6y ago> You have a problem that's repeatable right on schedule, that presumably shows up in the logs and affects multiple customers. It should be a no-brainer to send someone with test equipment out there to measure things in the field while the problem was happening. You say that, but the San Antonio Caltrain to California Avenue Caltrain stretch still has an enormous T-Mobile deadzone, even after I twice reported it to them and both times they said "they'd send someone to take a look."
- unnouinceput 6y agoHere is your explanation. They didn't care.