7 ms·

“Massive DDoS attack” just T-Mobile error?

127 points by sethbannon 6y ago

alfiedotwtf 6y ago
"It starts with T-Mobile. They were making some changes to their network configurations today. Unfortunately, it went badly." KATE: Wait a minute, the fourteenth, that's the same day the worm ends its run. I mean... Da Vinci virus, didn't Phreak say that's what he was being charged with? Look, they blame hackers!
momothereal 6y ago
Out of curiosity, how does a T-Mobile error affect services like Messenger and Facebook? I live in Canada and they were unreachable for about an hour. My ISP has no relation to T-Mobile. Is the FB network solely dependent on a mobile carrier?
- boomboomsubban 6y ago
  Are you sure the problem wasn't on your end? The Twitter thread explains why some sites were listed as down, and shows Facebook up for the day.
  - xkyscore 6y ago
    Facebook itself was up but messaging was unavailable, same with Instagram messaging.
  - Rebelgecko 6y ago
    it sounds like there may have been a separate issue with facebook? https://developers.facebook.com/status/issues/2510474695911231/ https://developers.facebook.com/status/issues/25104746959112...
- erikerikson 6y ago
  Certainly if you are connecting through T-Mobile hardware either as a customer or as a customer of a provider who contracted with T-Mobile on your behalf. Even if you weren't connecting through T-Mobile managed hardware, if the T-Mobile customer base found or was provisioned with alternative connection mechanisms then they could have been competing with you for bandwidth. The direct answer is no, network outages don't effect services but if you can't access them the experience is though they were out. Consider a friend you want to call. They may be alive and well (or even besides you) but if your phone can't reach theirs you still can't talk to them on your phones.
  - joe_the_user 6y ago
    I use T-mobile for both voice and data. I only had the problem of people not being to make phone calls to me. No problem calling and no problem connecting to Facebook. Of course, T-mobile is huge so paradoxical effects make sense. Edit: Facebook messaging from the web interface was up also
- cbhl 6y ago
  Some ideas: - Facebook sends SMS 2FA. Users on T-mo would be unable to receive codes. What behavior does the 2FA-sending code have in this condition? Does the process just sit there and time out (possibly exhausting some finite server resource, like file descriptors)? Are there automated retries in a client? Are humans retrying aggressively because they are frustrated by being unable to log in? - SWEs and SREs on on-call rotations may have configured pages to go to a cell phone on T-mo. Usually the three fallbacks are "phone call", "sms", and "pubsub/push notification", but both voice and data were broken today. May have taken longer to notice/escalate/resolve an outage. - Messenger may have seen increased load because people were using it to replace voice and texts that they would have normally sent over T-mo.
deleted 6y ago
[deleted]
deleted 6y ago
[deleted]
graton 6y ago
I think this is what caused the problems with the MVNO (Mobile Virtual Network Operator) I use. I noticed today that calling and texting weren't working on our phones. Our MVNO uses T-Mobile as its network, so this kind of explains everything. Luckily I use Google Voice as my primary # and it rang on my computer but I didn't understand why it wasn't ringing on my cell phone. Now I know :)
aritraghosh007 6y ago
Would love to see if T-Mobile publishes an RCA later. There seems to be so many conflicting reports speculating about a DDoS attack such as this https://www.forbes.com/sites/jessedamiani/2020/06/15/t-mobile-fortnite-instagram-comcast-and-chase-bank-have-all-experienced-outages-some-believe-the-us-has-been-hit-by-large-scale-ddos-attack-others-are-skeptical/#1cd4058e4b2a https://www.forbes.com/sites/jessedamiani/2020/06/15/t-mobil...
- boomboomsubban 6y ago
  From the post, the "attackmap" always looks like that and downdetector was showing that those sites were down for T-Mobile users. Other than that, the only report listed was an anonymous Twitter account.
- downerending 6y ago
  Having dealt with T-Mobile at length regarding local tower problems, I doubt we'll see anything but a blanket denial. And perhaps a map showing that we all have "excellent" coverage.
dang 6y ago
Related previous thread: https://news.ycombinator.com/item?id=23532560 https://news.ycombinator.com/item?id=23532560
dekhn 6y ago
I thought it was funny when somebody linked to an "attack map" which was really just a heatmap of the US population (https://xkcd.com/1138/ https://xkcd.com/1138/) and then went on to speculate it was the US gov't messing with protestors.

127 points by sethbannon 6y ago

"It starts with T-Mobile. They were making some changes to their network configurations today. Unfortunately, it went badly." KATE: Wait a minute, the fourteenth, that's the same day the worm ends its run. I mean... Da Vinci virus, didn't Phreak say that's what he was being charged with? Look, they blame hackers!

momothereal 6y ago

Out of curiosity, how does a T-Mobile error affect services like Messenger and Facebook? I live in Canada and they were unreachable for about an hour. My ISP has no relation to T-Mobile. Is the FB network solely dependent on a mobile carrier?

boomboomsubban 6y ago
Are you sure the problem wasn't on your end? The Twitter thread explains why some sites were listed as down, and shows Facebook up for the day.
- xkyscore 6y ago
  Facebook itself was up but messaging was unavailable, same with Instagram messaging.
- Rebelgecko 6y ago
  it sounds like there may have been a separate issue with facebook? https://developers.facebook.com/status/issues/2510474695911231/ https://developers.facebook.com/status/issues/25104746959112...
erikerikson 6y ago
Certainly if you are connecting through T-Mobile hardware either as a customer or as a customer of a provider who contracted with T-Mobile on your behalf. Even if you weren't connecting through T-Mobile managed hardware, if the T-Mobile customer base found or was provisioned with alternative connection mechanisms then they could have been competing with you for bandwidth. The direct answer is no, network outages don't effect services but if you can't access them the experience is though they were out. Consider a friend you want to call. They may be alive and well (or even besides you) but if your phone can't reach theirs you still can't talk to them on your phones.
- joe_the_user 6y ago
  I use T-mobile for both voice and data. I only had the problem of people not being to make phone calls to me. No problem calling and no problem connecting to Facebook. Of course, T-mobile is huge so paradoxical effects make sense. Edit: Facebook messaging from the web interface was up also
cbhl 6y ago
Some ideas: - Facebook sends SMS 2FA. Users on T-mo would be unable to receive codes. What behavior does the 2FA-sending code have in this condition? Does the process just sit there and time out (possibly exhausting some finite server resource, like file descriptors)? Are there automated retries in a client? Are humans retrying aggressively because they are frustrated by being unable to log in? - SWEs and SREs on on-call rotations may have configured pages to go to a cell phone on T-mo. Usually the three fallbacks are "phone call", "sms", and "pubsub/push notification", but both voice and data were broken today. May have taken longer to notice/escalate/resolve an outage. - Messenger may have seen increased load because people were using it to replace voice and texts that they would have normally sent over T-mo.

deleted 6y ago

[deleted]

deleted 6y ago

[deleted]

graton 6y ago

I think this is what caused the problems with the MVNO (Mobile Virtual Network Operator) I use. I noticed today that calling and texting weren't working on our phones. Our MVNO uses T-Mobile as its network, so this kind of explains everything. Luckily I use Google Voice as my primary # and it rang on my computer but I didn't understand why it wasn't ringing on my cell phone. Now I know :)

aritraghosh007 6y ago

Would love to see if T-Mobile publishes an RCA later. There seems to be so many conflicting reports speculating about a DDoS attack such as this https://www.forbes.com/sites/jessedamiani/2020/06/15/t-mobile-fortnite-instagram-comcast-and-chase-bank-have-all-experienced-outages-some-believe-the-us-has-been-hit-by-large-scale-ddos-attack-others-are-skeptical/#1cd4058e4b2a https://www.forbes.com/sites/jessedamiani/2020/06/15/t-mobil...

boomboomsubban 6y ago
From the post, the "attackmap" always looks like that and downdetector was showing that those sites were down for T-Mobile users. Other than that, the only report listed was an anonymous Twitter account.
downerending 6y ago
Having dealt with T-Mobile at length regarding local tower problems, I doubt we'll see anything but a blanket denial. And perhaps a map showing that we all have "excellent" coverage.

dang 6y ago

Related previous thread: https://news.ycombinator.com/item?id=23532560 https://news.ycombinator.com/item?id=23532560

dekhn 6y ago

I thought it was funny when somebody linked to an "attack map" which was really just a heatmap of the US population (https://xkcd.com/1138/ https://xkcd.com/1138/) and then went on to speculate it was the US gov't messing with protestors.