12 ms·
T-Mobile, Verizon and AT&T phone calls are failing across the US
- jdofaz 6y agoI called my t-mobile number and got a busy signal, haven't heard that in a while.
- ankurkwv 6y agohttps://downdetector.com/ https://downdetector.com/ looks pretty wild right now.
- dleslie 6y agoDefinitely screenshot worthy for posterity.
- parsimo2010 6y agoThe live outage maps are basically just this: https://xkcd.com/1138/ https://xkcd.com/1138/
- marksc 6y ago5 of the top 10 states by population aren't on the cellular outage maps: Georgia, North Carolina, Ohio, Michigan, and Pennsylvania.
- Shank 6y agoThis is a good case of causation and correlation being conflated. Of course T-Mobile users will report problems with YouTube even if it’s just their carrier. Any localized ISP problem will show up in outage reports for top websites because that’s what everyone uses.
- sjg007 6y agoI mean, the thing is.. they can all get to downdetector.. so :)
- mennis16 6y agoThat is true but I will also note that Facebook Messenger died for me across devices (even on the browser regular Facebook was fine but my messages would not load), and the email discussion I've seen largely says something similar- only Messenger failing for a lot of people around the same time a couple hours ago.
- glitcher 6y agoCan confirm in AZ. My wife and I are on T-Mobile and are unable to call each other, with both of us at home.
- deleted 6y ago[deleted]
- joezydeco 6y agoTMO down in IL for voice but data is working.
- mech422 6y agoWierd...I'm in mesa and just got a call from San Tan on my Cricket phone.
- cbsks 6y agoCricket uses the AT&T network so they shouldn't be affected by this.
- binarymax 6y agoSame issue in NY. Can't get in touch with my wife, but she's out. Voice and Text not working. Can't get through to her at all.
- atlgator 6y agoI know this is the digital age but you could just try speaking to her directly. /s
- krustyburger 6y agoA friend’s phone has been non-functional for hours now. With the size of the T-Mobile user base, this outage almost certainly has already had real human costs in the form of an inability to summon emergency services.
- zamadatix 6y agoDepending what is causing the outage 911 may still function. Or it may not. Just spreading to not assume 911 is unavailable because normal calls don't work/your plan expired/whatever else is going on as 911 gets a few special technical perks that make it available more often than normal calls.
- jaywalk 6y agoExactly right. 911 calls are handled differently at almost every step of the process, starting with the handset and carrying all the way through to the 911 call center.
- dtmmax33 6y agohttps://www.clickorlando.com/news/local/2020/06/15/cell-carrier-outages-impacting-911-system-in-several-areas-in-central-florida/ https://www.clickorlando.com/news/local/2020/06/15/cell-carr... 911 in Central Florida is having issues.
- e79 6y agoPriority access services like 911 are handled very differently. For example, I can’t place any normal outbound calls with my T-Mobile sim, but calling 311 in NYC works fine.
- zarmin 6y agoWhatever is the source, I really hope it's not malicious. The US is vulnerable.
- partiallypro 6y agoBased on how it has unfolded, I'm betting it's a cascading failure. Data seems to be unaffected, it is only voice.
- lallysingh 6y agoMy guess is it's some technical issue related to the Sprint merger. I have no data to back this up.
- dogecoinbase 6y agoThat is the current suggestion: https://puck.nether.net/pipermail/outages/2020-June/013124.html https://puck.nether.net/pipermail/outages/2020-June/013124.h... This is because of an update in the systems from the Sprint side to help merge with T Mobile. That happened from 12am-6am EST and Sprint's systems clashed with T Mobile's. I'm glad I have off today.
- Someone1234 6y agoConsidering it is impacting other non-celluar services (like Facebook), I'm guessing it is an internet backbone issue (e.g. bad BGP routing was broadcast?). PS - This doesn't mean it is unrelated to the Spring/T-Mobile merger, they themselves could be the ones who screwed up routing.
- DebtDeflation 6y agoUh oh. https://www.digitalattackmap.com/ https://www.digitalattackmap.com/ US and Brazil both under a massive DDOS attack right now.
- t3rabytes 6y agoFWIW, that isn't showing live data -- it says June 14 at the top.
- newman8r 6y agoPerhaps incidents like these will increase the popularity of amateur radio. I'm curious if there's any studies on the potential impacts of the US carriers begin down for a day or a week or more.
- reaperducer 6y agoNo need to study anything more than a newspaper archive. Look at the local papers every time a hurricane hits the United States, or there is another major natural disaster over the last 80 years. Ham operators are always a communication lifeline in times of crisis.
- Multicomp 6y agoMy hope is for someone to complete a smartphone-based ad hoc mesh network that can use bluetooth, wifi to pass messages along, up to 6 hops or similar. Currently, the closest thing I'm aware of for that is cwtch which is a ricochet.im rewrite
- WrtCdEvrydy 6y agoThere was one out there for Android that worked great. I wanted to build my own for emergency services and we figured out how to do some of the mesh routing but it didn't go far... WiFi Direct unfortunately only allows you to pair to one phone at a time (while bluetooth allows endless pairing outside of BLE)
- jbj 6y agoI heard of a few, but havent tried them: disaster.radio serval mesh meshtastic a different and cool project worth mentioning, more for distibuting teaching materials as I understand: othernet.is
- pixelface 6y agothis already exists in some forms [1] on top of the fact that amateur radio in general is quite capable of moving data whether by packet radio [2] (including packet radio BBS [3], things like hinternet [4], bbs over amateur radio, or plain old teletype (RTTY) as a means to operate access a console on a remote machine. there's a ton of interesting creative work being done in the space even though it isn't particularly mainstream-popular. [1] http://www.broadband-hamnet.org/ http://www.broadband-hamnet.org/ [2] http://www.choisser.com/packet/ http://www.choisser.com/packet/ [3] https://www.scc-ares-races.org/data/packet/index.html https://www.scc-ares-races.org/data/packet/index.html [4] https://www.kqed.org/quest/8440/ham-radio-helping-to-build-a-fast-and-free-internet https://www.kqed.org/quest/8440/ham-radio-helping-to-build-a...
- jeffbee 6y agoCan't call from landline to t-mobile iphone, just dead air, not even busy/fast-busy/three-tone call fail.
- Multicomp 6y agoSame. AT&T to T-Mobile line. Dead air, insta hang up. Signal messages appear to be unaffected so far, so one more anecdote that data works, but IDK if they were delivered to my group warning them about this via cellular data or wifi.
- sbierwagen 6y agoCalled from ringcentral voip to t-mobile phone on UMA. Rang once, then dead air. Disabled wifi to shut off UMA for a second try, rang a few times then went to busy tone.
- ohhhwell 6y agoGoogle Fi from iPhone to Android worked after a long delay. Failed the other direction :|
- jaden 6y agoSeveral other networks are showing up on downdetector.com too, including Sprint, U.S. Cellular and Consumer Cellular.
- DebtDeflation 6y agoI'm on T-Mobile and can confirm that I have been unable to make or receive calls for the past hour. Also, my home Comcast internet seems to be operating at dial-up speed and with insane latency. Something big is going on.
- novaRom 6y agoAre people still working from home in US?
- DebtDeflation 6y agoYes.
- monadic2 6y agoDepends who you ask.
- ngngngng 6y agoHow about texts? My wife and I couldn't send or receive texts for a while earlier today.
- edoceo 6y agoThree hour delay on texts from Twilio to T-Mobile. No voice connect from Twilio to Verizon, delayed and too.
- DebtDeflation 6y agoA whole bunch of texts people apparently sent me ~4 hours ago just came through.
- burnte 6y agoSome were delivered immediately, some delayed hours, I know some texts were sent to me by my wife still haven't arrived at all.
- monadic2 6y ago
- mlacks 6y agoLooks to be a tech issue related to the Sprint merger. That is the current suggestion: (reposted from elsewhere in this thread: https://puck.nether.net/pipermail/outages/2020-June/013124.html https://puck.nether.net/pipermail/outages/2020-June/013124.h... This is because of an update in the systems from the Sprint side to help merge with T Mobile. That happened from 12am-6am EST and Sprint's systems clashed with T Mobile's. I'm glad I have off today.
- A4ET8a8uTh0 6y agoIf true that would be interesting. Is the merger approval signed, sealed and irreversible?
- bingdig 6y agoNot irreversible (but the DOJ rarely breaks up already merged companies due to the difficulty of "unscrambling the eggs")! This event could create a reason to block the merger ex post, especially given last week's news that DISH may not buy Boost Mobile from T-Mobile, which was a key provision of the merger to reduce its anti-competitive effects.
- Andrex 6y ago"So... you're telling me there's a chance [the merger won't happen]?" -Lloyd, Dumb and Dumber
- SllX 6y agoWell, hopefully it doesn't cause reason to block the merger ex post. Disclosure upfront, I'm nothing more than a satisfied T-Mobile customer that switched away a couple of years ago from AT&T. No other interests in this race. Sprint was a shambling zombie, and any public policy measure that could be undertaken to make Sprint something other than a shambling zombie, and in particular, into a viable competitor would have rewarded Sprint for well, not being competitive. Out of the four major carriers we used to have, we had 2.5 well capitalized competitors with large customer bases and Sprint, a shambling zombie. T-Mobile put together an offer, tendered it to Sprint, and it was accepted, and out of the many ways that Sprint could have been gobbled up by one of its larger and better competitors, T-Mobile serving as daddy to Sprint's mommy was about the best outcome you could have gotten. This puts their customer base, assuming the Sprint customers stay, at relatively equal footing with Verizon and AT&T, and more customers = increased cash flow, and increased cash flow keeps them operating in the black and hopefully with room to invest. I don't care if we have three or four or five major wireless carriers right now so long as I can be satisfied with one of them where I live and work. T-Mobile's "Uncarrier"-like lack of billing shenanigans was one of the things that put them over the edge when I decided to bite the bullet and switch carriers. Also, with StarLink and maybe other systems like it possibly coming online and into service in the next couple of years, I'm not at all convinced it's going to matter that we only have three major "cell carriers" going into the next decade. Let T-Mobile keep all its customers, all of the spectrum they acquired. They paid for it, and I expect them to put it to use on behalf of their customers.
- thebean11 6y agoOh interesting, I couldn't get a verification code from Chase earlier today. Just tried sending a text to myself; I can send but not receive texts. That's pretty wild, it's been like that for >4 hours for me it seems.
- tzs 6y agoI tried logging in to a couple financial sites that should have sent me text codes, but they did not even try. They just accepted my name/password and went to my account page. I tried from a Firefox private window, my work Firefox profile (which has never went to those sites), a Chrome incognito window, and a Chrome instance being run under Selenium, so make sure I had no cookies from a previous visit to allow bypassing the text code. So now I'm a bit confused. I hope the sites aren't doing something stupid, like associating successful text code verification with the IP address, and caching that on their end. Maybe they've temporarily turned off requiring text codes to avoid locking people out during this outage?
- SSLy 6y agoFacebook is experiencing issues regarding chat (including on Instagram) in Europe too. Might be related?
- jedberg 6y agoIt's a good thing there are only three wireless providers so this was easy to track down. /s
- dehrmann 6y agoThere were going to be three one way or another; Sprint wasn't a viable carrier on its own.
- loughnane 6y agoMy wife and I have T-Mobile and so couldn’t call each other... but we could over signal. Not surprising, but still useful. EDIT: if all the other comments are correct and this is _really_ widespread then I am a little bit surprised signal is working well
- supernova87a 6y agoJust to be clear, this is only happening to POTS (?) dialing phone numbers, is that right? It shouldn't (or doesn't) affect Skype/Whatsapp/iMessage/Facetime/etc types of data calls does it? Or is it T-Mobile, Verizon, AT&T subscribers and their phones, cellular phones + data connections?
- jabedude 6y ago> It shouldn't (or doesn't) affect Skype/Whatsapp/iMessage/Facetime/etc types of data calls does it? Anecdotally, this is correct. I was unable to place a normal call today, but was able to Facetime the other device.
- guerrilla 6y agoIt seems to be effecting other things? https://downdetector.com/ https://downdetector.com/
- PascLeRasc 6y agoOh, so that's why no one at my work can get on our VPN now. We need to do away with SMS 2FA immediately.
- vmception 6y agoWhy are you using SMS 2FA?
- PascLeRasc 6y agoI use 1Password 2FA everywhere I can and choose services specifically for that reason. I just switched cell providers today because Verizon didn’t offer it. I don’t have influence over my employers IT though.
- goldenManatee 6y agoAdvocate switching over to something Microsoft Authenticator or Yubico. SMS is unreliable and not transmitted in an encrypted fashion.
- AznHisoka 6y agoDDOS is trending on Twitter now. Could it be an attack?
- vsareto 6y agoThat doesn't seem credible yet https://twitter.com/MalwareTechBlog/status/1272647109833940992 https://twitter.com/MalwareTechBlog/status/12726471098339409...
- adrianpike 6y agoUnlikely that an attack would manifest itself in this way. I think it's an SS7 screwup since it seems that signaling is mostly what's gone pear shaped.
- LinuxBender 6y agoSS7 would not affect website uptime. All the carrier websites are having issues. [edit] As others have pointed out, if b-number routing is failing, then their support sites would be flooded with people checking their account. So maybe the subscribers are in effect overloading the sites. That would make sense.
- Rebelgecko 6y agoCouldn't that just be the result of tens of millions of people simultaneously trying to contact support?
- LinuxBender 6y agoCould be. With SS7 outages, first things you would see is b-number analysis failing (call routing) and SMS text messages failing.
- londons_explore 6y agoAs soon as there is a service outage, all the users go to the carrier website to try to complain or log in to check if they've paid their bill, etc. A website designed for most users to visit once per year when their contract expires likely won't scale to everyone visiting in an hour...
- blhack 6y agoWhat could cause all major cell phone providers to experience outages that began at the same time in the same day? A solar flare seems unlikely, since WiFi does not seem to be effected (for me at least). Routing issue maybe? One possibility: it’s all just T-Mobile, but people are bad at reporting. For instance: if a non technical person tries to call a t-mobile number from their Verizon phone, and experiences an “outage”, they may report this as a problem with their own phone rather than understanding that it was on the receiving end (and knowing to try to call other carriers). This could also explain problems with Facebook messenger (if the T-Mobile outage is effecting data). It’s not a problem with messenger etc. it’s just that messenger is the service that some T-Mobile customers are trying to use and having it not work. So they report it as a problem with the black box they know as their phone. That or we are getting hacked. And if that’s the case then buckle in and charge up your ham radios.
- adrianpike 6y agoEasy: someone fat-fingered SS7 the same way someone fat-fingered BGP. https://en.wikipedia.org/wiki/Signalling_System_No._7 https://en.wikipedia.org/wiki/Signalling_System_No._7 https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-today/ https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-...
- deleted 6y ago[deleted]
- bogomipz 6y agoNo it's not the same at all. A carrier's SS7 configuration does not get propagated out to other carriers.
- jcrawfordor 6y agoQuite possible though is a significant disruption of one of the many services like LNP which are implemented via SS7 and shared by all carriers - there is indeed a complex set of inter-dependencies between carriers that are all somewhat generically referred to as part of the SS7 system, several of which are required to successfully set up a call or deliver a text message.
- coronadisaster 6y agoThe NSA implemented a new system?
- acruns 6y agoit would more likely be that one of their systems being down would cause this ;)
- coronadisaster 6y agoeither way, it is very suspicious
- JCBird1012 6y agoThere were early reports that Level 3 was having some routing issues earlier today, but it's been confirmed that they're not having any problems whatsoever (for anyone who may have been suspecting that) https://twitter.com/zackwhittaker/status/1272648878639255553?s=20 https://twitter.com/zackwhittaker/status/1272648878639255553...
- hnrodey 6y agoT-Mobile user here, had to restart my device (iPhone) in order to make calls. The data network seems stable as I was able to FaceTime another person via the T-Mobile data network.
- tzs 6y agoSame here. Just read about this outage, confirmed I could not place calls. Restarted my iPhone, and now I can place calls.
- deleted 6y ago[deleted]
- koolba 6y agoFun fact: If your 2FA is based on SMS then you can’t log in until this is resolved.
- acruns 6y agotypically you just disable the 2fa requirement until the issue is resolved.
- alanbernstein 6y agoIf you can disable 2FA, without using 2FA to authenticate first... is it 2FA?
- adrianpike 6y agoWell I'm sure you'd have to enter your password to disable 2FA. /s, obviously.
- deadwing0 6y agoMany sites give you a recovery code of some sort that you can use to disable 2fa in the event you were to lose access to your authenticator app or texts.
- mrgzg1 6y agoLiterally wasted my whole day on this :(
- partiallypro 6y agoOff topic, but I had 2FA setup on my phone for various things, and had it stolen overseas. I was able to lock the phone and erase it...but I was stuck overseas without being able to access anything or reset it. There has to be a better way.
- 0xdeadb00f 6y agoTOTP with password protected key backups in the cloud? (Bonus: recovery keys you keep offline, in your wallet or something)
- LeoPanthera 6y agoTwitter moved really fast on this one, and not in a good way. 4chan created a meme map declaring that the USA was under a DDOS cyberattack, and all of Twitter apparently immediately decided that this was the truth - including some US senators. (It does not appear to be a DDOS attack.)
- drunkcatsdgaf 6y agoyoure doing the same thing. 4chan didnt create a "meme", the map youre seeing is from digitalattackmap.
- wybiral 6y agoHow do people get digitalattack map to show today? I can only see yesterday on there and some people are posting images that say the 15th. Either way, it's being taken out of context. There are always DDoS attacks like that. The past couple of months has had constant traffic on their timeline.
- drunkcatsdgaf 6y agosome other maps are showing the same stuff, but DDOS maps are inconsistent due to how the data is gathered. None of them are truely a panopticon. I have my own theory that these sites are actually the ones astroturfing this stuff when services go down.
- adamfeldman 6y agoUpdate #2 from T-Mobile's President of Technology (3:02 PM PT): "Teams continue to work as quickly as possible to fix the voice & messaging problems some are seeing. Data services are now available & some calls are completing. Alternate services like WhatsApp, Signal, iMessage, Facetime etc. are available. Thanks for your patience." https://twitter.com/NevilleRay/status/1272650750665953280 https://twitter.com/NevilleRay/status/1272650750665953280
- ourcat 6y agoDDOS Attack apparently. https://twitter.com/OutageAlert2020/status/1272638788011995141 https://twitter.com/OutageAlert2020/status/12726387880119951...
- ourcat 6y agoConfused why this got -4 on it? Is linking to graphs about the issue frowned upon here? When submitted, it looked quite relevant to the discussion within the minutes beforehand.
- Pwntastic 6y agoLikely the same reason that a few other comments were downvoted: there wasn't an actual DDOS attack. Additionally, your linked graph doesn't actually contain any context or resources that could be used to verify it. It is in fact an uncredited photo from [0] which was refuted in the children of a few other comments[1,2] that were available at the time of your post [0]: https://www.digitalattackmap.com/ https://www.digitalattackmap.com/ [1]: https://news.ycombinator.com/item?id=23533054 https://news.ycombinator.com/item?id=23533054 [2]: https://news.ycombinator.com/item?id=23533510 https://news.ycombinator.com/item?id=23533510 ️
- kossTKR 6y agoWhy are so many people in this thread sure that this is "just Sprint merger mishap" when i and many others have been unable to use Facebook Messenger and Instagram chat in Europe? What does those services have to do with US cellular networks?
- mynameisvlad 6y agoExactly, what do those services have to do with US cellular networks? Nothing. Hence why nobody is talking about them in this thread.
- kossTKR 6y agoI came here from the DDOS thread on the HN frontpage because Messenger was down (and instagram DM's), which i haven't tried before here in Scandinavia in my 10 years of FB. Same for my colleagues. I see at least 3 other people mentioning it in this thread, and lots on Reddit and on Twitter in connection to these articles. That for me was at least was at little interesting when the overall theme of this new "incident" has been "something large scale going on".
- hindsightbias 6y agoThe test before the storm
- serf 6y agoWhat's up with the networks that went down a few hours staggered from the rest? Frontier Networks, Playstation Network, and Spectrum all went down early this morning; and I mean actually down. I had a Fingbox at my parents house tattle on Frontier Networks as soon as the connection got less spotty, and then the network dropped again. The ISP/data failures are more interesting to me, to be honest, than the cell networks. I wonder if they're related. The cell networks all seem to share a lot of cascading failures, historically. It's intriguing that something is happening to affect the rest of the world. The cynic in me says that someone just re-routed the internet through whatever new DPI-esque system is now in place, but I hope the reason for this mess is more innocent.
- mke 6y agoThe worst part: we have to wait 5 years to hear about this on Darknet Diaries!
- blakeg 6y agoMajor ddos attack
- Good_Karma 6y agoHow about too many IMSI catchers being deployed at the same time ? Like basically ~ all of them at the same time right ? Couldn't that cause some unforeseen disturbances ?
- peter_d_sherman 6y agoWhat might be needed is something like voice-packet across voice-network tracing/logging... sort of like, given a voice connection ID, tag packets going through it in a certain way, when routers or forwarding equipment en route see this tag, then send back an "I received the packet at <timestamp>" message, or something like that. If something like that already exists, then T-Mobile should know exactly where the network is failing, and why. If not, then maybe it, or something like it would be good to have... I have learned from many years of software engineering that a working system is NOT proof that the system will continue to work into the future, that good logging/tracing of all system components is a necessity not a nicety, and that if you can't tell exactly when/where/why/how your system failed within 5 minutes of looking at log files -- you need to revamp your logging system so that that becomes possible. If it takes T-Mobile more than 5 minutes to accurately diagnose the root cause of this by looking at log files, then that is proof (to me, as an Engineer) that their logging system ranges from suboptimal to non-existent, and that fixing that so that this never happens again (via fixing their logging system) -- should be their absolute first corporate priority -- after whatever is causing this is fixed. Just one man's opinion. I will now entertain disagreements and flames...
- soheil 6y agoCome to think of it seems like telecomm companies have been elevated to the national security level of importance. I'm not sure if it has always been true, but with more and more reliance on tech that depends on these companies the danger is escalating. Does anyone know if these companies are vetted and inspected on any regular basis by government agencies the same way airlines for example are? A trivial mistake by one of these carriers can have dire ramifications for the entire nation as it seems to have been the cause of the current outage.
- _sbrk 6y agoIjit Pai is on the case. Everything will be better soon. The current FCC leadership is about as inept as the Fed, except that they're not ashamed to admit they're selling out to carriers.