16 ms·
East Coast Internet Outage
Anyone else noticing a widespread internet outage on the East Coast of the US?
- imwally 6y agoYes, sites are loading very slowly, if at all. Thought it was AWS at first but Google services are having the same issue.
- TimC123456 6y agoYes, lots of issues connecting to various cloud services here in DC.
- TimC123456 6y agoOn Verizon FIOS here, perhaps this is another Verizon outage?
- tlunter 6y agoExperiencing it from Verizon ISP. Any others that are also experiencing this not on Verizon?
- evanc123 6y agoyep, i'm in DC area and having issues across multiple sites. Fios Fiber here
- beckingz 6y agoSame thing here on the MD side on Fios.
- zenexer 6y agoObserving the same issue from various East Coast locations, all on Verizon Fios.
- PStamatiou 6y agoSame thing in NYC. on Verizon FiOS fiber
- cshift 6y agoSpotify is loading weird for me (New England).
- aaomidi 6y agoFios as well.
- ixtli 6y agoVerizon in NYC acting funny.
- Wistar 6y agoOn Fios in Seattle. Seems good here.
- wronglebowski 6y agoFIOS Here in the Philadelphia area, having issues hitting AWS as well.
- jgrahamc 6y agoLooks like it's Verizon having trouble.
- silveira 6y agoSpikes everywhere https://downdetector.com/ https://downdetector.com/
- dx87 6y agoYep, on verizon fios just outside baltimore. Can't ping 8.8.8.8, but can ping 1.1.1.1 fine. Also having issues connecting to services running in AWS.
- deleted 6y ago[deleted]
- bikenaga 6y agoComcast seems to be fine (at around 11:50 a.m. EST), so maybe it's just Verizon.
- beachwood23 6y agoI'm in Philly, and I'm having issues with Comcast. Doesn't look like it's isolated to any one provider, or any one site. https://downdetector.com https://downdetector.com is showing many issues across many platforms.
- deleted 6y ago[deleted]
- phenkdo 6y agoboston fios here, having intermittent issues, mostly slow connection.
- xhrpost 6y agoVerizon FiOS. 50% packet loss with many "good" packets in the hundreds of ms when pinging Google. NYC.
- mroche 6y agohttps://twitter.com/VerizonSupport/status/1354109889572982786 https://twitter.com/VerizonSupport/status/135410988957298278... There is a fiber cut in Brooklyn. We have no ETR, as of yet. You can use the MY Fios app for updates. *EAG
- heatmiser 6y agoDoes a cut cable in BK account for outages up and down the east coast?
- mroche 6y agoI'm not knowledgeable enough to answer that question. However, this is the only Twitter comment that mentions this on their page, and I don't have access to a Verizon account to verify their outage status memo. We had a flurry of employees saying their connections were going haywire to the business and this is the only thing I've been able to gather outside of downdetector.
- jasonjayr 6y agoThis is speculation, of course, but: there are several large Internet Exchanges in NYC, and US/EU peering points. It's probable that a fiber cut could have cut several other links, and the failover links are now completely overwhelmed.
- jonknee 6y agoIf it's a fiber that goes into major data centers it certainly could.
- Spooky23 6y agoVerizon had (and apparently still has) a few vulnerable links in the NYC metro area that have wide ranging impacts.
- WarOnPrivacy 6y agohttps://www.datacenterdynamics.com/en/news/brooklyn-fiber-cut-appears-cause-east-coast-verizon-fios-outage/ https://www.datacenterdynamics.com/en/news/brooklyn-fiber-cu...
- asmr 6y agoIssues with FiOS in the Boston area. Traceroute seems to point to an issue originating in NY but that's just speculation. Apparently a lot of other services are down. Might be a Tier 1 or 2 ISP.
- vrtx0 6y agoSame here. Seeing heavy packet loss to most (but not all) sites that route through verizondigitalmedia.com.customer.alter.net in NYC.
- gen220 6y agoMy corporate laptop is affected. My roommate’s corporate laptop is totally unaffected. We’re on the same router/gateway. Any theories out there to explain this phenomenon? Fascinating. (My theory is a network partition, where my roommate’s IP addresses happen to be in regions that are on “our side” of the partition, whereas my IP Addresses are on the “other”, unreachable side).
- hpkuarg 6y agoIs your roommate on a VPN? It looks like Verizon is having trouble only with traffic to certain destinations (perhaps Google and AWS).
- gen220 6y agoThey’re not. I’m on a split VPN, for what it’s worth. Disconnecting even from the split VPN does not help my laptop.
- dx87 6y agoAre you using different DNS servers? My work laptop was using 8.8.8.8 and is getting name resolution errors, but my personal computer is using 1.1.1.1 and working fine.
- gen220 6y agoInterestingly enough, they’re both deferring DNS to our router, and the router is configured to use DNS servers dynamically provided by the ISP.
- Here_For_LOLs 6y agoits most likely your companies RDP infrastructure (or local environment) operates on AWS. Which is down because of this "cut"
- deleted 6y ago[deleted]
- Moogs 6y agoApparently, Verizon's outage page doesn't actually say if there's an outage. You have to log in to see if there's an outage. Too bad logins are failing after ~10 minutes for me. https://www.verizon.com/support/residential/service-outage https://www.verizon.com/support/residential/service-outage
- NovemberWhiskey 6y agoYeah, I'm getting a 504 from their status site even after managing to get through the authn process. Although my packet loss to my IPv6 tunnel provider just went to zero, so that's a good sign here (Manhattan).
- atonse 6y agoHonestly, at this point I've found HN to be the best "Status Page" for this sort of thing. I came here first when I found issues and fiddling around with my custom DNS lookup hierarchy didn't help. FiOS customer in DC Metro.
- xhrpost 6y agoExcept these posts seem to regularly get taken down or flagged. This isn't even on the front page anymore it seems.
- someHnUser 6y agoI'm glad I'm not the only one noticing this. I could usually rely on HN to not be so heavily "filtered" for a lack of a better term but I have noticed the slide into a more controlled/gamed PR outlet.
- dang 6y agoIt set off the flamewar detector. We monitor that list and eventually cancel the penalty for threads that aren't actual flamewars, but it takes a while. If anyone wants to make it faster, emailing hn@ycombinator.com is usually a forcing function. Edit: Also, posts without URLs get a downweight by default. I've turned that off too, and merged the other threads into this one, since it was the first one.
- eMGm4D0zgUAVXc7 6y agoHow does one programmatically detect a flamewar? :)
- dang 6y agoBy proxy, which is why there are false positives.
- unoti 6y ago> How does one programmatically detect a flamewar? :) An NLP/Machine learning model would be a great way! Probably also taking in other features as input such as the number of comments per second, and how deeply nested those comments are. Let's build one!
- dzmien 6y agoI am experiencing connectivity issues on Fios outside of Philadelphia in Pennsylvania. edit: as of 13:20 (GMT-5) it appears to be resolved.
- 813919 6y agoFios outside Baltimore and yeah it's screwed currently.
- crowman 6y agoYep. Second day of classes at George Mason University in Fairfax and no one can connect to Zoom or Blackboard meetings. I'm on FiOS as well.
- couchand 6y agoFor me, mtr reports quick pings (single digit ms mostly), but HTTP traffic is very spotty. Brooklyn FiOS.
- podiki 6y agoIssues here too, I'm on Verizon Fios. Right when starting class, what a mess. (As a colleague joked, Google going down (or Verizon I guess) is like the modern snow day)
- twistedpair 6y agoProps to Slack, it's the only thing working well for me. GMail, GCP, Meet, all dragging and timing out in Boston.
- whatsmyusername 6y agoSlack is spotty for us
- JimiofEden 6y agoPhilly suburbs: Fios has been inconsistent for me for the last hour or so, starting around 11:00
- mikesapienza 6y agoYes. Pittsburgh area is spotty with FiOS right now.
- mikesapienza 6y agoPittsburgh area has spotty FiOS internet service also.
- chockablock 6y agoConnecting to a full-tunnel VPN seems to fix this for me. (FiOS in Brooklyn).
- boleary-gl 6y agoYes! Thank you for this (in hindsight, obvious) idea.
- woohoo7676 6y agoThank you! This also fixed my slowdowns.
- nycben2 6y agoVerizon is having issues in NYC for sure. https://twitter.com/VerizonSupport/status/1354114675093495808 https://twitter.com/VerizonSupport/status/135411467509349580...
- u678u 6y agotypical that link is useless verizon.com/outage. "learn how to reboot your router"
- Fej 6y agoI am on Fios in northeast NJ, issues with some services (Netflix) but not others (Google except significant variance in ping times, Bing, AWS).
- Ansil849 6y agoWhy is this post not showing up on the HN homepage?
- taf2 6y agoGetting reports that it's improving. It's nice that it never actually went down down... just degraded
- willswire 6y agoNoticing degradation in connection quality in Boston, MA. ISP is Verizon FIOS - operating on the 1 Gbps plan.
- sleepydog 6y agoIt seems to have recovered for me. I am using FIOS in Brooklyn.
- gedaxiang 6y agoSame! Also FIOS in BK
- squeaky-clean 6y agoSame, also fios in Brooklyn. Not getting my full speeds but I am getting 100Mbps, which is far better than the 60 Kbps I was getting. Ironically things recovered within 30 seconds me of messaging our systems team to ask if the development VPN was having an outage. That's like a magic rule of internet outages, isn't it? Nothing works until you try to show someone else the issue, then it all works.
- excitednumber 6y agoRestored for me (FIOS Brooklyn, NY)
- excitednumber 6y agoRestored for me (FIOS Brookly NY). Someone get that tech's venmo handle. Human deserves $1.
- excitednumber 6y agoNot to say their time is only worth $1. Just to emphasize my thanks. Anyway. Please do not skewer me. :(
- rrrrrrrrrrrryan 6y agoI'm nitpicking on the title, but "widespread" is one word, not two.
- lacraig2 6y agoI have FiOS and my town is listed here. Internet is still working.
- dave_aiello 6y agoRight now in my part of Bucks County, PA, FiOS' gigabit speed internet is running about normal speed for download and about 3/4 of normal for upload.
- willcipriano 6y agoSmall world, I'm in the same place. I noticed long periods of dropped packets, it looks like existing connections are ok but establishing a new one is a problem. My wife working over a VPN doesn't notice a issue.
- sweetheart 6y agoBucks County represent!
- neogodless 6y agoI was getting 2 Mbps upstream on FiOS just after noon today! But it seems largely back to normal now. (I have 200/200, not Gigabit.)
- HenryNY2020 6y agoGot knock out from Google Meeting a lot recently, even online speedtest show normal.
- carewornalien 6y agoYes, large carriers can't do much when the backhoe strikes.. it happens.. they'll fix it.. and we'll all go on with our lives..
- t0mbstone 6y agoI thought the whole point of the internet and protocols like TCP/IP is that it's supposed to be fault tolerant and traffic is supposed to automatically re-route. Seems kind of weird how a single cable being cut can cause half the country to lose internet...
- tenacious_tuna 6y agoI wonder how much has to do with the cost-optimization pressure on a private carrier. Building robust infrastructure is only valuable to them insofar as it enables them to continue to earn money. Reliable service for the sake of reliable service--especially to residential regions--is a non-starter. Less pessimistically, it's possible that the regions that're affected have natural "choke points" for infrastructure, either moving through the mountains or along highways, etc., which of course leads to a single-point-of-failure, though it may be unlikely.
- centimeter 6y agoTCP doesn't know anything about route failure or rerouting. All it does is handle a few dropped or reordered packets here and there. In principle, you can have systems that very quickly route around damage like this, but in practice it happens so rarely that it's not worth it for consumer ISPs. I think it should exist, but the ISP space is so burdened with red tape that the only way anyone could compete with it was literally to start a rocket ship company and launch a bunch of satellites.
- pbronez 6y agoStarting an ISP isn't all THAT hard. https://arstechnica.com/information-technology/2021/01/jared-mauch-didnt-have-good-broadband-so-he-built-his-own-fiber-isp/ https://arstechnica.com/information-technology/2021/01/jared...
- Gene_Parmesan 6y agoWhen half of my coworkers on a Zoom call (including myself) got kicked within ten seconds of each other, I figured something major had happened. Western PA. Was somewhere around 11:24 EST. Network never went down completely, bitrate just took a nosedive.
- arthurcolle 6y agoI was just about to call FiOS to switch from Xfinity because these data caps are absolutely ridiculous. I am sad now. Aside: Comcast, wtf? 1.2 TB limit per month? Seriously if anyone works at Comcast know that I hate your company with an overwhelming passion.
- garettmd 6y agoTo be fair, I've had FiOS for more than a year now, and this is the first incident I've had with them. I'm pretty comfortable saying they have a much better reputation than Comcast...
- abstractbarista 6y agoWow, that's crazy. I use over 1TB a day. Forever thankful for Google Fiber.
- slivanes 6y agoHow does does one use over 1TB a day?
- arthurcolle 6y agoRunning scrapers can get you there pretty fast ;)
- frakkingcylons 6y agoWow 1TB per day? What is using all that bandwidth? (if you don't mind me asking)
- vondur 6y agoI've had FIOS service since 2007 or so here in SoCal and I can count the outages over this period on one hand.
- tomjakubowski 6y agoI was a long-term Fios customer in SoCal, and they were indeed very reliable up until Verizon sold the local service to Frontier. Glad to hear they're not a problem for everyone at least.
- 2dollars27cents 6y agoI have no technical understanding of fiber networks. The explanation that this was caused by one fiber line in Brooklyn being severed seems unlikely. Curious to see what the experts think.
- kube-system 6y agoI wouldn't doubt it. Single misconfigured routers have caused issues of similar scale.
- im3w1l 6y agoI believe the internet is designed to handle broken links, but is not very robust against misconfigured (or malicious) routers.
- ArchOversight 6y agoIf a 100 Gbps link goes down, and all the other links that could carry that route are only 10 Gbps... it goes downhill quickly. Too much traffic is concentrated at the major carriers which carry a large amount of traffic over relatively little fiber (cause its cheaper) and when that fiber disappears we lose a massive amount of capacity...
- mikem170 6y agoJust to add to that, sometimes the redundancy doesn't work out as planned, maybe another weird set of circumstances, or some routers can't handle spikes in load as the network reconverges, causing other reconverges, confusing bgp routing peers, etc. Fun stuff!
- neogodless 6y agoWhile I'm also not qualified to diagnose the problem, I would imagine it's something like this... The major internet connections are like superhighways, with major central routers acting like exchanges. If a major superhighway had an accident blocking traffic, there would be personnel stationed at exchanges to help re-route the traffic, though that often occurs too late to prevent a bunch of backups. At any rate, people using Google Maps, Waze, etc. would see the accident and would likely be redirected to alternate routes that would normally be much slower - and would also be slowed down by the influx of higher than normal traffic. The fact that the internet still worked albeit much slower than usual leads me to believe that something similar happened. While normally my internet traffic might go through a major hub in NYC and then onward to various carriers and hosts, etc. it had to be re-routed around that hub that is normally sub-optimal, and also ill-equipped to handle the spillover. (And in some cases, bandwidth was saturated, and failed altogether for some users for certain usage.)
- Animats 6y agoToo much is failing from single point outages. The FCC may have to establish standards to prevent over-centralization. In the entire history of the Bell System, no electromechanical central office was ever down for more than 30 minutes for any reason than a natural disaster or, one occasion, a major file. That record has not been maintained in the Internet era.
- dylan604 6y ago>The FCC may have to establish standards to prevent over-centralization. Bwahahahahaha. Sorry, but that's like expecting the FCC to do something that would benefit US citizens vs US corporations. I think we've seen that's just not something they are interested in doing. Maybe a new administration will put as the head of the FCC someone that is "good of citizens" minded and uses a normal size coffee cup, but I'm not holding my breath for anything other than pro-corp rulings from the FCC
- Finnucane 6y agoTo be fair it's slightly more likely with Ajit Pai out.
- Florin_Andrei 6y ago> Maybe a new administration will put as the head of the FCC someone that is "good of citizens" minded and uses a normal size coffee cup Someone who is interested in anything beyond personal profit, status, and influence, would be nice for a change. Also someone who is smart would be very nice.
- tenebrisalietum 6y agoThe Bell System's job was to operate a nationwide telephone system for voice, not high-speed data. Even then, you only had one path to the telephone CO and only one CO it could go to. Outages were rare because voice-grade wire is cheap and all the complexity is handled by the CO. Also, while my Internet may have outages from time to time, I can't recall the last time I couldn't use my cell phone other than being in a bad signal area. For voice, cellular is definitely the way to go as capacity is easily added by adding new towers; not something easy to do with landlines.
- sleepysysadmin 6y ago>There is a fiber cut in Brooklyn. We have no ETR, as of yet. A fiber cut auto heals automatically via bgp. Absolutely no truth in this claim from verizon. Meanwhile google, zoom, aws, and microsoft went down? There isn't any shared datacenters here. There's at least 3 distinct datacenters. So what's the truth? what actually went down?
- jasonjayr 6y agoTraffic is passing, although slowly. I think this revealed that there is insufficient failover capacity.
- mikem170 6y agoCould be that. Also could be too much complexity, as opposed to capacity. Could have been timing didn't help, sometimes you take calculated risks or have to deal with additional complexity during migrations, etc. I've even seen where playing it safe while planning to back out of a manual fix used during a weird outage (circuit went down in one direction or something, just needed to be manually shutdown to re-route everything) brought down the entire network when another outage happened before that evening, i.e. most redundancy plans protect against single failures, not all combination of two failures. They wanted to do all planned changes after business hours, like manually swinging traffic back to the primary hub site. They didn't check the weather on the other side of the country, and the power at their new super-duper data center went down that afternoon during a thunderstorm... I wonder what percentage of fiber cuts don't cause noticeable outages. We'll never know, unless they let us root around in their ticket system. As a customer it is fair to judge them on what you see, though. Some telcos screw up more than others. Best to use two. (Typed over my phone, cuz my broadband is down!)
- ArchOversight 6y agoIf 100 Gbps of capacity suddenly went away, then BGP attempted to route around it, and sent that traffic down 10 Gbps routers instead, which overloaded, or were not ready for the deluge of traffic, then those BGP routes could start flapping, at which point those routes would get pulled, forcing it onto yet other circuits, which may be 1 Gbps and that's how the outage continues to grow. Too much capacity is centralized in certain providers, and if they go down there simply is no backup route that can handle the same traffic patterns.
- Jonnax 6y agoThe FCC said that 3mbps upload speed is all internet users need. They might not be in tune with modern internet usage https://www.theverge.com/2021/1/21/22242635/fcc-ajit-pai-internet-speed-advanced-telecommunications-3mbps https://www.theverge.com/2021/1/21/22242635/fcc-ajit-pai-int...
- cure 6y agoTo be fair, that was just Ajit Pai, who also thought net neutrality was not important. Now that he's gone, I hope he gets subjected to 3mbps upload speeds for the rest of his days.
- birdyrooster 6y agoWell Ajit Pai only has 640K of memory so it’s understandable.
- Brendinooo 6y ago> “We find that the current speed benchmark of 25/3 Mbps remains an appropriate measure by which to assess whether a fixed service is providing advanced telecommunications capability,” the report reads. Not "3mbps upload speed is all internet users need".
- lotsofpulp 6y agoThat’s effectively the same thing, otherwise I don’t see the purpose of that statement.
- richwater 6y agoYou vastly underestimate the cost to provide broadband to rural communities.
- dang 6y agoWe detached this subthread from https://news.ycombinator.com/item?id=25920055 https://news.ycombinator.com/item?id=25920055.
- nwsm 6y agoWe have FiOS in Boston and noticed it. Nothing terrible but for ~15 seconds at a time, recurring sporadically, we lost connection. Happened for a few hours, maybe 10am-1pm EST. Based on other comments we probably just didn't notice the big actual outage that happened all at once.
- adjkant 6y agoCompletely unable to load anything today from 11:30sh to 12:30ish EST in Brooklyn, all seems to be smooth now. First ever issue with FiOS in two years, would still fully recommend them to anyone when comparing to most options, especially Comcast.
- jzer0cool 6y agoWhen houses are created there are building codes to adhere to. For example, when there is a water pipe behind the walls a plumber (a good one) would also put a plate in front of it. In this way, once the walls are in place, drilling away into the wall, or nailing, would hit the metal plate, thus provides some safety mechanism. As another example, underground, in public areas are markings (or at least procedures) to identify such areas gas pipe areas before excavating. How are the fiber optics being cut? Are there no procedures for this or is there some challenges how the infra is setup today?
- XorNot 6y agoThe vast majority of fiber optic cuts are someone digging through the cable underground. You can't realistically prevent this with additional armoring, because it's quite a different failure to someone drilling in the wrong place - the forces are a lot higher, or heavy machinery is involved etc. (see also: the rate at which workers go through water mains in footpaths).
- valbaca 6y agoScale: wrapping all fiber optic cables in a protective metal sheath would be cost prohibitive. Whereas adding a metal sheet to prevent a house from flooding or someone from getting electrocuted makes sense. One idea, borrowed from some chainsaw safety pants, would be to wrap the cable in a fiber web sheath that tangles in whatever cutting equipment https://www.youtube.com/watch?v=r5VSivQe760 https://www.youtube.com/watch?v=r5VSivQe760
- jzer0cool 6y agoLiterally took off the pants testing that! I'm unsure how much is worth getting paid to run a self-test. This product looks pretty neat. Half joking, anyone ever, or witnessed getting eletrocuted trying to get the internet / computer working again? This spun threads reminds me of nature. Spiders, silk works spinning web. Wonder if that's where the inspiration came from.
- gknoy 6y agoTIL that Aramid fibers are not just something made up in bideo games. (: Thanks for sharing that.
- egberts 6y agoI find it weirder that Zoom Verizon got BGP routed through AS42861 via 0.ae15.gw14.iad8.alter.net and that AS42861 isn’t clearly defined under Roblex
- teejmya 6y agoVery interesting. HE's BGP tool sees it https://bgp.he.net/AS42861 https://bgp.he.net/AS42861 edit: so does https://www.peeringdb.com/net/2258 https://www.peeringdb.com/net/2258
- 310260 6y agoNo problems in Jersey City all day. Bleeping Computer article cites someone as saying it's a specific AS that's affected. JC routes through an AS in Newark, NJ and not NYC so, for me at least, there hasn't been an outage.
- betterunix2 6y agoI saw some disruptions in services that people in my household rely on, but I think that is because of disruptions in DCs located in NYC and not FiOS itself.
- kevstev 6y agoHey neighbor, I disagree, though switching my VPN gateway to our DR one in NJ vs our primary in Chicago cleared things up. For about an hour and a half though, remote desktop was struggling, even though my speed tests were fine. I'm on FIOS.
- palencharizard 6y agoEtsy appeared hard down for me (though not for friends a mile over). Homepage kept returning 429
- kevwil 6y agoImagine a world where all our enemies DIDN'T know how fragile our infrastructure is. False security, surely, but still would be somewhat comforting. :)
- nine_zeros 6y agoFirstly, America should strive to not have enemies. We fail at this very badly. Secondly, ZERO countries in the world have non-fragile infrastructure. America's infrastructure is better than most countries and definitely above average even among developed countries, except public transportation.
- eternalsculptor 6y ago> Firstly, America should strive to not have enemies. We fail at this very badly. This comment is not based in reality. Does Australia have many enemies? Does South Korea? Does Japan? Taiwan?
- throw0101a 6y ago> Firstly, America should strive to not have enemies. We fail at this very badly. This will never be possible, even if America does nothing 'wrong' against anyone else. As long as America (flaws and all) is a better place to live than other countries, and can be used as an example of how things can be better (though not perfect), and so will be a target of powers that don't want their own people thinking about improving things. * https://en.wikipedia.org/wiki/The_Road_to_Unfreedom https://en.wikipedia.org/wiki/The_Road_to_Unfreedom * https://www.theguardian.com/books/2018/apr/15/the-road-to-unfreedom-russia-europe-america-timothy-snyder-review-tim-adams https://www.theguardian.com/books/2018/apr/15/the-road-to-un... There are also plenty of bullies out there, who are only checked by American power, and wouldn't mind the US being more isolationist. There are all sorts of problems of how America interacts with the world, but there are many other countries who would do worse things IMHO. America simply existing generates enemies.
- nine_zeros 6y agoAmerica will always have adversaries. But enemies, who what to destroy the country? That's entirely our own creation. We should try to make enemies only when attacked. Not make enemies out of adversaries.
- jp42 6y agoI'm on west coast and my entire community lost internet right now.
- DiggyJohnson 6y agoIs this still ongoing? I'm having issues that I can confirm that as a result of this, and now three days later things are still spotty. Could be conflating multiple problems, of course.