15 ms·
BT fault hits broadband users and banks
- voltagex_ 10y agoUKNOF's mailing list is private so it's hard to get a handle on what's going on, but from what I've heard, a DC went down and BT's infrastructure hasn't recovered.
- charlieegan3 10y agoIt'd be nice if we got email notifications about this. I wasted time trying to work out what was going on (and if the issue was in our local network).
- bonaldi 10y agoYou want to them to email you to tell you your internet service is down? Er ...
- Swizec 10y agoMost people have redundancy. Computer has wifi, phone has LTE. Usually different providers. I would love an email thst my internet service is down. Or text.
- dingaling 10y agoMany of the niche ISPs such as AAISP provide line status notifications.
- vidarh 10y agoIncidentally BT also has infrastructure or delivering SMS to landline voice numbers via text to speech - or at least had as of a couple of years ago - so as lon as it's the broadband service and not the end user lines that were down they'd still be able to deliver it.
- 8draco8 10y agoYes, because email can be received on mobile phone which is outside of BT
- charlieegan3 10y agoLots of people get email on their phones these days via a mobile data connection. Our DSL connection isn't 'down' - just really, really slow.
- pmlnr 10y agoThe mobile connection still goes through an ISP...
- simonswords82 10y agoI see the point you're making (and I did laugh) but a lot of us get e-mails on our phones and would still be able to pick them up if our ISP went down.
- _puk 10y agoFunnily enough I can get emails on my mobile, even when my landline is down.. Not sure if it is a complete outage, I'm having issues with many sites, but HN works fine - which usually points to a DNS issue cough
- billpg 10y agoI noticed this morning that HTTPS was failing but HTTP was fine, but it "fixed itself" after ten minutes. No idea if this was related.
- tehbeard 10y agoHad issues yesterday where BT was complaining that my machine was using non BT DNS (all our machines at home use google's 8.8.8.8 / 8.8.4.4) Slightly concerning since we never enabled parental controls (Which the message implied was the reason it wanted us to use BT DNS) and now hearing that somehow only https traffic was affected by an issue.
- alex_hitchins 10y agoThis sounds very troubling. I hope it was just a coincidence.
- redcalx 10y agoAs an aside, I've found BT's DNS to be very flaky; sometimes down (or unreachable at least) for many hours at a time. I switched to google's and never had any problems since.
- 8draco8 10y agoThis is probably the reason why linux.dropbox.com was down for brief time today
- pbhjpbhj 10y agoOut of all the sites you could have mentioned why did you think that (the deb repo for Dropbox's packages) was pertinent to this story?
- 8draco8 10y agoBecause it was. This and ubuntu repo was not working on my PC during BT outage, other sites worked. After BT sorted it out I didn't have any problems. I asumme that it was the issue with some BT connection between me and dropbox server
- chriscoffee 10y agoApparently down to Telicity Group DC having power failure which BT heavily rely on
- dijit 10y agoWhich DC, there are many. Two London ones I can think of is north of the city (Oliver's yard) and docklands.
- ropiku 10y agoArstechnica says Telehouse North Docklands http://arstechnica.co.uk/business/2016/07/bt-isps-telehouse-north-major-outage/ http://arstechnica.co.uk/business/2016/07/bt-isps-telehouse-...
- mrseb 10y agoWe've updated the story - did some more investigation. Definitely looks like it was Telecity.
- plq 10y agoMight be a better organized coup attempt from the remain supporters :)
- J_Darnley 10y agoMiddle and upper class folks cutting off their Netflix and Spotify entertainment while also leaving them unable to tweet incessantly about it. I don't think so.
- CM30 10y agoHuh, so that's why my internet connection was timing out on a lot of sites earlier today.
- thomseddon 10y agoIt's a power outage in Telecity Harbour Exchange, from the LINX mailing list this morning: 08:12 - We are currently experiencing an outage impacting some of our equipment located at Telecity/Equinix Harbour Exchange in London. It is impacting both the Juniper and Extreme LANs. - We are still investigating to get further information, but we have lost all management access to the devices at the moment. 08:45 - We have had confirmation from Telecity/Equinix that there was indeed a partial loss of power impacting one of the power feeds. - Power failed at 07:55 BST and was restored at 08:17 BST for the majority of our equipment. - Unfortunately one of our edge routers (edge4-tch) on the Juniper LAN is still down at the moment. 09:33 - Power to edge4-tch has also been restored now. The router has been back up since 9:15 BST. - All member services should be restored. - We have an engineer on site and are working with Telecity/Equinix to understand why we lost all power to that device. - We are also waiting for a full report from Telecity/Equinix about the cause of the power failure. --- We don't peer with BT so ou traffic was OK but we did notice loss of service from AWS to BTNet
- mattlondon 10y agoGreat update thanks. Perhaps I am being naive, but I thought data centers had back-up power to cover this exact problem? I.e. battery power until the generators kick in... or is this how it used to be done and these days no one bothers and just relies on fail-over to other sites????
- pyvpx 10y agodespite best efforts, sometimes the generators don't kick in. sometimes not all of them kick in. sometimes one ATS in one feed in one section that just happens to house one half of an important switch chassis doesn't work as it did during tests last quarter.
- snowy 10y agoTrue. But Data centers usually have an A and B power systems independent of each other for redundancy . Equipment is usually connected to both. The failure of either should not cause an outage. (Unless some one is stupid enough to connect core network equipment to only A or B power)
- forcer 10y agoWe run one the most popular speed test site in the UK at http://www.broadbandspeedchecker.co.uk http://www.broadbandspeedchecker.co.uk . We are getting 20x more people testing today than usual. Also, because Ookla is down.
- anexprogrammer 10y agoI find the Talktalk chairman's quoted comments hilarious, though I assume they're not specifically in relation to this outage. If it were Talktalk's network spending would drop about 90% and complaints rise a similar amount. It's not like TT would be unaffected by a Telecity outage. I don;t think TT ever cared what their customers think, just reducing prices and service to the bone.
- pjc50 10y agoOpenReach own all the ADSL wiring - that's what TT have to buy from them. Most consumer ISPs are quite 'virtual' and dependent on BT for actual provisioning. The manager of a tiny geek-friendly ISP, Andrews&Arnold, frequently complains about this on his blog: http://www.revk.uk/ http://www.revk.uk/
- anexprogrammer 10y agoWell they own exchange to home. Since unbundling it's a bit fuzzy as most of them have their DSLAM in the exchange and pay rents for space in exchange and run virtual over BT's backbone and so forth. But at least it freed up a lot more competition. Hence Sky asking for Fibre cabinet unbundling recently. If it was A&A or Zen's chairman quoted there'd have been no need for snark. ;)
- topbanana 10y agoI'm not a TalkTalk retail customer, but I do use their backhaul through another ISP, and it seems very reliable.
- anexprogrammer 10y agoI expect the wholesale service is a very different animal to retail. I'd expect them to lose custom very quickly if they weren't reliability competitive. The customer is usually going to blame whoever they pay the adsl bill to, even when they're not the cause. TT often topped the retail broadband complaints charts. Though BT "won" the last one I saw with the EE merger.
- 10y ago
- stevesimmons 10y agoMy BT Infinity home broadband in south London is affected. 'mtr' shows any traffic via the BT-owned IP address 31.55.187.177 gets 88% packet loss when it hits that router. Web sites using other routes are ok.
- shocks 10y agoGraphs here: http://www.ukinternetreport.co.uk/?p=d&id=215 http://www.ukinternetreport.co.uk/?p=d&id=215
- anon4711 10y agoTheir customers should rightly be outaged.
- chrisstu 10y agoThe infuriating this is that whenever this happens at BT, they say nothing about the cause and nothing about how they will prevent recurrence. As a BT Infinity customer, I've had a total loss of internet several times over the last couple of years and never any feedback from BT afterwards.
- tonylemesmer 10y agoThey seem like one of those obnoxious organisations that think if they provide any information then it will only invite a shitstorm. Unfortunately too many people still chose them as a provider. I nearly. Nearly switched to them last month because of their prices and BT Sport package. But fortunately I'm stuck on a 12 month contract with IDNet. Shame.
- radiowave 10y agoAs a colleague of mine used to say when I worked there: "It's a good thing we're not a communications company."
- jayflux 10y agoIs there a reason there are so many Data Centres in Docklands? Is there something about that area that makes it great for datacentres?
- jasonkester 10y agoLiving in England, you really need a downtime strategy for your internet connection. While it's seldom a complete country-wide thing like this one, you can still expect your broadband to drop out for up to an hour at a time at least once every day. BT is just that incapable of providing DSL to residences. I live in a tiny village in rural France half the time, and never really think about the internet connection. Just like you never worry that water won't come out of the kitchen sink when you turn it on. But then I come back to England for the summer and there at the front of my mind several times a day is the Internet connection. Why is it super slow all of a sudden? Hey, it dropped again. I hope it comes back soon because I need to check stuff in. I have a personal hotspot set up on my phone that gets used within my house on a weekly basis. There's an OpenReach van parked out back a couple times a year fixing yet another fault on our line. Nowhere else I've lived has it been anywhere near this bad. I look forward to the day where BT has some actual competition.
- jon-wood 10y agoI think to an extent it depends where you are. While I'm far from BT's biggest fan, the actual internet connection they provide is pretty reliable here. The piece of crap router on the other hand needs rebooting to bring the wireless networking back up every few days, and has absolutely no logging to diagnose why that's the case.
- ChrisMeek 10y agoI've been with BT for a very long time (5+ years) and other than during unseasonably violent storms this is the first time that I've suffered any kind of outage at all. Not only that but I'm in the middle of nowhere and still get fibre (to cab at least)
- Nexxxeh 10y agoThen you're been very lucky. http://www.theregister.co.uk/2016/02/03/bt_blames_faulty_router_for_mass_internet_outage/ http://www.theregister.co.uk/2016/02/03/bt_blames_faulty_rou... http://www.theregister.co.uk/2015/11/17/bt_outage_three_days_fix/ http://www.theregister.co.uk/2015/11/17/bt_outage_three_days... http://www.theregister.co.uk/2015/07/29/bt_broadband_in_broadbased_brownout_and_titsup_incidents/ http://www.theregister.co.uk/2015/07/29/bt_broadband_in_broa... http://www.theregister.co.uk/2014/12/12/bt_infinity_working_to_fix_the_problem_after_three_days_of_outages/ http://www.theregister.co.uk/2014/12/12/bt_infinity_working_... http://www.theregister.co.uk/2014/06/30/bt_major_dns_outage/ http://www.theregister.co.uk/2014/06/30/bt_major_dns_outage/ http://www.theregister.co.uk/2014/02/25/bt_cable_theft/ http://www.theregister.co.uk/2014/02/25/bt_cable_theft/ And that's in just over two years.
- Fifer82 10y agoThe best thing about being BT is that you own everything so you don't have to justify anything to anyone. Since half of the lords have long term shares in the monopoly the moment that it went private, then your back is covered as far as law is concerned so long as you pay your dividends. You can just laugh as the customer goes somewhere else, because you will still be picking up 75% of the broadband bill. I also love how BT pays out so much to shareholders, and the public purse bankrolls the infrastructure, then everyone pays BT wholesale. What an outstanding deal. BT is corrupt and I would like to see it fragmented.
- pyvpx 10y agosounds a bit like Deutsche Telekom.
- spacecowboy_lon 10y agoBT doesn't own Telecity I suspect that if they did it wouldn't have happened. And BT pays out around 3% dividend to its owners which is not excessive a lot of BT's detractors have an axe to grind (Sky, Murdoch and Virgin) and want to cherry pick customers and to hell with the USO.
- jsingleton 10y agoAnd again the next day: https://news.ycombinator.com/item?id=12135794 https://news.ycombinator.com/item?id=12135794