5 ms·
I see multiple posts here speculating on cyberattack—as opposed to "we pushed a bad configuration update which messed everything up irreparably"—you know, like
by schmuckonwheels 9mo ago
I see multiple posts here speculating on cyberattack—as opposed to "we pushed a bad configuration update which messed everything up irreparably"—you know, like it has been every other time before this.
E.g., Cloudflare, Meta (who in doing so also locked themselves out of the building), and didn't some bumbling major Canadian telco knock themselves offline for like a week not too long ago?
- embedding-shape 9mo agoThat's not true though, sometimes it's inexplicitly DNS too.
- schmuckonwheels 9mo agoMy money is on expired certificate. If so, successful test.
- Xylakant 9mo agoThe great thing is that with all of the encrypted/signed DNS thingys, you now have not one, but multiple options to combine “invalid certificate” and DNS in one outage. You no longer need to choose!
- snorbleck 9mo agoit's always DNS
- jandrese 9mo agoOne of these times they will be right and you will never hear the end of the time they were first to recognize the start of a cyberattack.
- Bratmon 9mo agoYou know what they say about a stopped clock: it's wrong 1,438 times a day.
- jermaustin1 9mo agoIf it is a quartz watch, it might be 47,120,384 times per day.
- calvinmorrison 9mo agobetter odds than the lottery!
- eth0up 9mo agoNot if you're playing Florida Pick 4, which is supposed to be 1:10000, but for me seems 1:U+221E
- btreesOfSpring 9mo agoYour post reflects another online observation. With the rise of online sports books, this sort of predictive doomerism has flooded almost every team's online comment section. It no longer feel like fandom or community in the same way. Just lots of voices that will be glad to say, "I told you so," in the loss and crickets with the W. Wish there was some accountability mechanism for all the negative noise broadcasted into the channel.
- deleted 9mo ago[deleted]
- yifanl 9mo agoIn general, whats the expected cost of being loudly, obviously, publicly and obnoxiously wrong? Keep in mind who the President is.
- ocdtrekkie 9mo agoVerizon had issues routing calls to a provider I'm aware of yesterday, and had to make some sort of change today to fix it. I'm definitely thinking bad configuration update.
- CGMthrowaway 9mo agoIt's affecting every mobile carrier (ATT, TMO), not just Verizon
- daveguy 9mo agoNo, it's affecting Verizon and people on ATT & TMO trying to get in touch with the Verizon customers who are affected. Note the tiny fraction of people reporting for ATT & TMO compared to Verizon on DownDetector: https://downdetector.com/ https://downdetector.com/ (You have to click the links to see actual magnitude because the graphs are scaled to show relative outage within a given service -- order of 150k for Verizon vs 1.5k for the others.)
- shimman 9mo agoI worked at a major ISP and we had a similar situation where the North East went down and the RC was a fiber cut at a major node in Philly. These network topologies are incredibly complex and edges you think wouldn't exist have ways of suddenly appearing when things go awry.
- Nextgrid 9mo agoCyberattacks are a good scapegoat for any large incompetent non-tech company that is unable to admit a mistake. (tech companies are more open to admitting actual mistakes - and reluctant to disclose cyberattacks even if there actually was one - where as non-tech ones would rather allude to an attack than admit a mistake) Cyberattack scenarios pretty much never make sense in case of complete outages; if you have the access required to cause such an outage it’s always more profitable to keep this access and use it for covert spying/targeted attacks or save it for later than to burn it by causing a massive, visible problem.
- venturecruelty 9mo agoIn a dead empire, sufficiently advanced rot is indistinguishable from malice.
- Meekro 9mo agoInfrastructure in general seems worse than 20 years ago. Our talent for black-bagging dictators has never been stronger, though!
- devilbunny 9mo ago20 years ago much less of the infrastructure of everyday life depended on an always-on network connection. Smartphones in particular were a relatively niche product. I didn’t even have a cell phone (and not because I was too young), much less expect it to work all the time.
- cheeseprocedure 9mo agoYeah, the Canadian telco was Rogers. Total recovery took multiple days. From the Wikipedia writeup: > In a letter to the CRTC, Rogers stated that the deletion of a routing filter on its distribution routers caused all possible routes to the internet to pass through the routers, exceeding the capacity of the routers on its core network. https://en.wikipedia.org/wiki/2022_Rogers_Communications_outage https://en.wikipedia.org/wiki/2022_Rogers_Communications_out...
- mikepurvis 9mo agoThe major Canadian outage was Rogers in 2022: https://en.wikipedia.org/wiki/2022_Rogers_Communications_outage https://en.wikipedia.org/wiki/2022_Rogers_Communications_out... I took my kids swimming that day and the pool couldn't take our money since the payment terminal was on the Rogers cellular network, so it was a free family swim.
- deleted 9mo ago[deleted]