9 ms·
Ask HN: Is there something wrong with internet right now?
Down Detector (http://downdetector.com/) is showing a huge spike in reports over the past 30 minutes. Many of our users are having trouble reaching our LB in Northern Virginia.
Anyone else having this issue or any insight as to what's going on?
- PerfectElement 9y agoComcast says it's been resolved for most customers now: https://twitter.com/comcastcares/status/927636388517838848 https://twitter.com/comcastcares/status/927636388517838848
- Giorgi 9y agoThis is how WW3 will start
- undersuit 9y agoLast night I had trouble with Google, Netflix, Reddit redirect links, and some other sites. I was trying to think of places an explosion could happen that would cause my issues. This morning I've narrowed the explosion's epicenter down to the Wifi card in my PC. :/
- deleted 9y ago[deleted]
- muzani 9y agoMalaysian here. I'm surprising a Comcast issue would reach all the way here. I noticed a lot of sites were working really badly, thought it was some problem with AWS or some damaged internet artery cable.
- exikyut 9y agoThere should be a service sysadmins can run that sends controlled pings, download/upload requests, etc, to specific hosts. The idea being that the results are shared publicly. -- Copypasting https://www.wired.com/story/how-a-tiny-error-shut-off-the-internet-for-parts-of-the-us/ https://www.wired.com/story/how-a-tiny-error-shut-off-the-in... (dated today): -------------- A YEAR AGO, a DDoS attack caused internet outages around the US by targeting the internet-infrastructure company Dyn, which provides Domain Name System services to look up web servers. Monday saw a nationwide series of outages as well, but with a more pedestrian cause: a misconfiguration at Level 3, an internet backbone company—and enterprise ISP—that underpins other big networks. Network analysts say that the misconfiguration was a routing issue that created a ripple effect, causing problems for companies like Comcast, Spectrum, Verizon, Cox, and RCN across the country. Level 3, whose acquisition by CenturyLink closed recently, said in a statement to WIRED that it resolved the issue in about 90 minutes. "Our network experienced a service disruption affecting some customers with IP-based services," the company said. "The disruption was caused by a configuration error." Comcast users started reporting internet outages around the time of the Level 3 outages on Monday, but the company said that it was monitoring "an external network issue" and not a problem with its own infrastructure. RCN confirmed that it had some network problems on Monday because of Level 3. The company said it had restored RCN service by rerouting traffic to a different backbone. The misconfiguration was a "route leak," according to Roland Dobbins, a principal engineer at the DDoS and network-security firm Arbor Networks, which monitors global internet operations. ISPs use "Autonomous Systems," also known as ASes, to keep track of what IP addresses are on which networks, and route packets of data between them. They use the Border Gateway Protocol (BGP) to establish and communicate routes. For example, packets can route between networks A and B, but network A can also route packets to network C through network B, and so on. This is how internet service providers interoperate to let you browse the whole internet, not just the IP addresses on their own networks. In a "route leak," an AS, or multiple ASes, issue incorrect information about the IP addresses on their network, which causes inefficient routing and failures for both the originating ISP and other ISPs trying to route traffic through. Think of it like a series of street signs that help keep traffic flowing in the right directions. If some of them are mislabeled or point the wrong way, assorted chaos can ensue. Route leaks can be malicious, sometimes called "route hijacks" or "BGP hijacks," but Monday's incident seems to have been caused by a simple mistake that ballooned to have national impact. Large outages caused by accidental route leaks have cropped up before. "Folks are looking to tweak routing policies, and make mistakes," Arbor Networks' Dobbins says. The problem could have come as CenturyLink works to integrate the Level 3 network, or could have stemmed from typical traffic engineering and efficiency work. Internet outages of all sizes caused by route leaks have occurred occasionally, but consistently, for decades. ISPs attempt to minimize them using "route filters" that check the IP routes their peers and customers intend to use to send and receive packets, and attempt to catch any problematic plans. But these filters are difficult to maintain on the scale of the modern internet, and can have their own mistakes. Monday's outages reinforce how precarious connectivity really is, and how certain aspects of the internet's architecture—offering flexibility and ease-of-use—can introduce instability into what has become a vital service.
- matthew349hall 9y agoSnapchat went down mid afternoon Eastern Time. My high school was a war zone without Snapchat.
- tomberek 9y agoYes, 4.2.2.1 DNS is down. Probably your problem.
- leesalminen 9y agoThanks for the tip. I'll pass this along to our users.
- lima 9y agoMight as well be a symptom.
- kordlessagain 9y agoSymptom of it being a primary cause would be no loading, then all of a sudden loading of site. However, I've had intermittent connection with HN, so not sure that is necessarily the primary. Edit: clearly spell check is down.
- lexnay 9y agoI thought it was just me. I’m in Colorado. Bandwidth seems to be reporting as normal, however I’m having issues loading Facebook, imgur, some advertisements on reddit, etc.
- leesalminen 9y agoI'm in Colorado as well! Glad to know it's not just me. Are you on CenturyLink or Comcast?
- lexnay 9y agoComcast
- simmons 9y agoIt may have something to do with a large, prolonged power outage in downtown Denver. I've seen at least one ISP tweet about possible disruption to services due to this, and it may be impacting others. http://www.9news.com/news/local/power-outage-in-downtown-denver-affecting-nearly-700-customers/489484960 http://www.9news.com/news/local/power-outage-in-downtown-den...
- EvilMonkeyMat 9y agoI'm in the province of Quebec (Eastern Canada), and we're having huge problems with most websites. Strangely enough, LTE network is working on the three providers I've tried.
- hi5eyes 9y agoSouthern Ontario, everything is slow as all hell
- johndubchak 9y agoOh no, the Internet Monopolies are instituting "Rolling Blackouts" ahead of the FCC decision on Net Neutrality!!
- gpm 9y agoAlso Southern Ontario (Toronto), I've noticed nothing. Been or reddit, HN, many random programming websites, starcraft, etc. I've checked all the urls people said they had problems with so far (spotify, cloudflare, downdetector) and haven't had any.
- 9y ago
- pgroves 9y agodowndetector.com now seems to be down. never a good sign.
- hi5eyes 9y agofacebook, discord, reddit; all unusable what the hell?
- oregontechninja 9y agoRelated thread: https://news.ycombinator.com/item?id=15637412 https://news.ycombinator.com/item?id=15637412 There are I think three threads talking about this right now. This one won the front page and topical title fight.
- preben 9y agoSweden here. Facebook and other select sites down.
- deadcast 9y agoDang I can't even access downdetector. I'm in California in the central valley. Everything is very sluggish and the ol' router/modem reboot did nothing. :/
- abarringer 9y agoyes, lots of sites offline for me. I'm in Atlanta GA.
- kordlessagain 9y agohttps://cloudflare.com https://cloudflare.com is down for me. Comcast in California. HN is really slow.
- EvilMonkeyMat 9y agoSame here. Slack is working great though.
- stormcode 9y agoAnd giphy!
- WalterSobchak 9y agoPhew!
- jmcgough 9y agoI thought it was just me - was about to head to a cafe. On comcast here in Berkeley FWIW.
- daxorid 9y agoNone of the sites mentioned in this or other threads are down from the vantagepoint of Cox in Arizona. Sounds like an ISP issue.
- kordlessagain 9y agoMore likely BGP or similar border gateway routing things. In some cases, gateways can become overloaded because of bad destination routes that other gateways are routing correctly. Not that I'm an expert at this stuff. I'm not.
- JonathonW 9y agoComcast Business in Nashville is terrible right now for anything that's not local (Facebook, Reddit, etc.). Some are worse than others (HN is just slow; Reddit's very slow but loads eventually; Down Detector won't load at all on our work connection now). AT&T (in Nashville) doesn't seem to be impacted; I'm RDP'd into one of my home machines with Uverse and everything's working great there.
- nijaru 9y agoI’m in the SF area and Comcast is between slow and not working. Cellular data is working fine though
- dageshi 9y agoNo problems reaching any site from the UK
- kurtdac 9y agoSpotify has been down for me since ~1:00 EST in Toronto.
- Animats 9y agoSecond Life is reporting major problems. [1] [1] https://status.secondlifegrid.net/ https://status.secondlifegrid.net/
- Twisol 9y agoI'm having significant issues as well -- it happened quite suddenly an hour ago (9:50 Pacific Time, northern California). Thought it was a local issue at first and spent the last hour debugging my network. I think my apartment is on Comcast, although I'm not 100% certain (one of my roommates is in charge of it). Spamming `date` at an affected SSH connection suggests it's entirely a download problem, since all of the timestamps are very close, but they're received multiple seconds apart. Like some other commenters here, my mobile connectivity is just fine. EDIT: Certain sites (including Hacker News) are now qualitatively more accessible, and I'm now hitting expected speeds at http://testmy.net/download http://testmy.net/download. SSH to my DigitalOcean is also better. Here's hoping everything else smoothes out soon!
- Stratoscope 9y agoSame here on Comcast in Menlo Park (CA). T-Mobile is zippy, though, so I guess I'll switch to mobile hotspot for a while.
- Fwirt 9y agoSalem, Oregon here. Everything is zippy. Seems like it's location dependent? Just checked reddit and facebook.
- 24gttghh 9y agoI was just thinking, shouldn't there/couldn't there be a site that aggregates actual statistics (as opposed to the crowd-sourced DownDetector) regarding internet connectivity/health globally to provide an accurate snapshot of what might actually be happening? Edit: Sorta like this[0] I suppose. [0]https://www.akamai.com/us/en/solutions/intelligent-platform/visualizing-akamai/ https://www.akamai.com/us/en/solutions/intelligent-platform/...
- oasisbob 9y agoFor many years there was a site called the Internet Health Report which would show packet loss and problems between multiple carriers. It was pretty useful until Dynatrace deep-sixed it this year. I haven't found a suitable replacement, would be interested to hear any suggestions.
- leesalminen 9y agoSame here! I seriously miss that site.
- mattbeckman 9y agoYeah, I just headed there for the first time in a while. DISAPPOINTED.
- bproven 9y agoIssues here in San Jose area - it appears to be Comcast from what I can tell. Same as others, cellular via ATT is fine.
- blang 9y agosome in Seattle are complaining: https://www.geekwire.com/2017/internet-issues-plague-users-across-u-s-monday-morning/ https://www.geekwire.com/2017/internet-issues-plague-users-a...
- oasisbob 9y agoLots of similar reports on the outages mailing list: https://puck.nether.net/pipermail/outages/2017-November/010891.html https://puck.nether.net/pipermail/outages/2017-November/0108... "Anyone having packet loss, please send traceroutes to the list. There is clearly something happening, but not much useful data yet. Some traceroutes might help narrow down the cause of the problem."
- zelon88 9y agoIT worker here... Everything is on fire and my phone is ringing like crazy.
- monistowl 9y agoDoes that differ in any significant respect from a normal day in IT?
- chipuni 9y agoI'm a Comcast user in Hayward (Bay Area), CA, USA. Some sites are easily available: Google, Yahoo. Some sites are partially available: Twitter, Hacker News. Many sites are completely unavailable: Down Detector.
- manyxcxi 9y agoI was having maddening issues with Comcast Business in Portland, OR this morning. Switched over to my VPN and haven’t had any problems since. My initial pokes at the issues I was seeing were DNS related, but even switching the DNS hosts to Google or OpenDNS didn’t completely fix it. Maybe they’ve got some deep packet inspection issues?
- enoch_r 9y agoAlso Comcast (in Portland, OR) also having major issues, resolved by tethering.
- ruffrey 9y agoI've read anecdotal reports of an Level3 outage that makes Comcast have poor connectivity outside its network. I have websites that are up right now according to monitoring tools, but they are inaccessible to me from home.
- Mz 9y agoMaybe. I am getting a weird issue when trying to open a page for research I am doing for paid work. I can't get to the page and it offers to open as a PDF or something and then fails to open anything. Just more error messages follow.
- janlukacs 9y agoEverything loading blazing fast (Romania).
- collinmanderson 9y agoIt appears to be a Comcast issue: https://news.ycombinator.com/item?id=15637412 https://news.ycombinator.com/item?id=15637412
- chipuni 9y agohttps://twitter.com/ReportOutage/status/927600102918811648 https://twitter.com/ReportOutage/status/927600102918811648 shows a problem with Comcast. That tweet links to http://outage.report/comcast http://outage.report/comcast .
- uncled1023 9y agoI'm in Northern California and also having issues. One thing I noticed is that it is only Downloads having issues. My uploads are as fast as ever. I then connected to my VPN, and everything is working great. I think it is their routing that is having the issues, not the connection itself.
- collinmanderson 9y agoI contacted Comcast Network Operations Center. They confirmed they're aware of the issue and working on fixing it, but didn't provide any more details.
- afpx 9y agoI can’t connect to VPN (on AWS), but AWS CLI is working fine. Weird. But, I’ll just do something else for a while ...
- politician 9y agoThe situation here started to improve within the last two minutes.
- shampto3 9y agoGizmodo is reporting that it is a DDoS attack. https://gizmodo.com/this-is-probably-why-half-the-internet-shut-down-today-1788062835/amp https://gizmodo.com/this-is-probably-why-half-the-internet-s... [edit] Dang, I thought I had checked the date. Sorry everyone!
- kjsahakian 9y agoThat's from the outage on 10/21/16.
- leesalminen 9y agoThat’s from 2016?
- steveadoo 9y agoThat article is from 2016.
- LyndsySimon 9y agoThat's a year old article, FYI.
- pathseeker 9y agoGoes to show you how incredible slow this attack has made the Internet!