6 ms·
Curious as to what could cause a global outage like that. The system consists of birds in the air and individual ground stations. There's no big choke point tha
by Rooster61 1y ago
Curious as to what could cause a global outage like that. The system consists of birds in the air and individual ground stations. There's no big choke point that I'm aware of that could cause the whole shebang to go dark. Am I missing something here?
- rhcom2 1y agoSoftware?
- 0xffff2 1y agoI'm hardly an expert, but it seems like usually stuff like this happens to major backbones because of a bad configuration being rolled out too fast. Maybe the pushed a bad DNS/BGP/<insert networking acronym here> update to every satellite and/or ground station near simultaneously.
- chrisstanchak 1y agoIt just takes one
- sekh60 1y ago/s not-/s BGP?
- erulastiel 1y agoA bad configuration pushed to network
- chasd00 1y agothat's going to be a lot of crew dragon trips to go hold the power button down for 5 seconds on all those satellites to reset to factory default.
- AdamJacobMuller 1y agoI volunteer.
- x2tyfi 1y agoAlmost always. Most people look at global networks like black magic and don’t realize this. For most large businesses, 90%+ of major network events are caused by internally-driven network config changes. Depending on how far along the business is towards automation, a proportion of the 90% can be attributed to a human going “off script” - I.e.: making a change that had not been reviewed.
- markasoftware 1y agoSame things have been said of eg the massive Facebook years ago. It was DNS, which affected service discovery across the whole org. The massive Roblox outage was also caused by service discovery failure (Consul). If your machines don't know which other machines to talk to, shit goes bad fast.
- supportengineer 1y agoThis happens enough that it seems like someone would have created a better alternative.
- airspresso 1y agoEverything has failure modes, the only difference is how hard they are to trigger
- ben_w 1y agoThere is a non-zero probability that Musk personally tried to use Grok to make and roll out a software update.
- deleted 1y ago[deleted]
- reputablenixer 1y ago[dead]
- JumpCrisscross 1y agoUnless Musk is outside the country, the outage started in the late morning [1]. If this happened at 3AM, your hypothesis would be more likely. [1] https://www.pcmag.com/news/spacexs-starlink-hit-by-major-outage https://www.pcmag.com/news/spacexs-starlink-hit-by-major-out...
- cyanydeez 1y agoTook ketamine, went blackout at 3Am, told hself future me can handle yhis, scheduled deployment in morning. Future self woke up this afternoon completely unaware.
- ben_w 1y agoHmm. It may be that I myself am too tired right now, but I'm afraid I don't follow the logic on that one? But thanks for the link anyway.
- dave_universetf 1y agoBGP is the canonical way to take your entire global network offline within seconds. Glancing at BGP looking glasses, Starlink's prefixes seem to still be announced, but there could still be an accidental blackhole or routing loop within their AS, or something broken in one of their transit providers. No idea if that's what's going on, but routing protocols are one of a few effectively global control planes that can go wrong very quickly like this.
- indigodaddy 1y agoIf it were BGP/routing, you would think we'd be able to still get a signal and the modem would think it's healthy (although maybe not if the issue prevented us from obtaining our public IP), we just wouldn't be able to route to any dst. In the current case we don't have a signal (orange light on the modem)
- FollowingTheDao 1y agoThat sounds bad…
- ta1243 1y agoYes, before the drop out my traffic was coming from a downlink station in Bulgaria, on an IP on AS14593 Traceroute from "the internet" back to that IP reaches AS14593 just fine, and my endpoint doesn't get beyond the first hop of the local starlink router. Whatever it is, it doesn't look like a peering problem
- indigodaddy 1y agohttps://mtr.ping.pe https://mtr.ping.pe ftw for MTR from "the internet" ? :)
- ta1243 1y agoFrom various monitoring points I have on multiple internet connections. One of the promises of starlink was it would stay in space as long as possible before being downlinked, giving far lower latency, alas that hasn't happened yet, and traffic will run thousands of miles in the wrong direction before being downlinked. For example from one location to another I have 360ms via Starlink but just 200ms rtt via local provision (5g p2p wireless then optical). On another it used to downlink in Lagos, but now it downlinks in Nairobi, meaning traffic to Lagos routes Nairobi -> Marseille -> Lagos, taking far longer than it used to. A shame really.
- kevinqi 1y agoI mean, why do systems go down at all? a lot of big outages are simple misconfiguration or cascading failure from what seemed like small changes. It's rarely due to the physical constraints of the world
- modeless 1y agoCyberattack. Given the usage in active wars, it's a big juicy target. There's precedent too; Russia hacked Viasat at the start of the Ukraine war. They actually bricked user terminals in that attack.
- oblio 1y agoFYI, Vodafone and other major European mobile networks were down during the last few days. This is hybrid warfare, not a coincidence. Russia needs to learn some manners, the hard way.
- cluckindan 1y ago[flagged]
- LunaSea 1y agoLuxembourg too yesterday : https://paperjam.lu/article/panne-de-post-lombre-des-hackers-prorusses https://paperjam.lu/article/panne-de-post-lombre-des-hackers...
- reaperducer 1y agoThe article you linked to states 2,500 reports on DD. Does that count as "major" now?
- cluckindan 1y agoWhy wouldn’t it count if multiple operators are down for multiple thousands of people for several hours? Not even emergency numbers were working.
- reaperducer 1y ago2,500 people out of 70,000,000 isn't "major." And your own link states that 999 service was restored in 61 minutes. Emergency service lines go down somewhere in the world almost every day. Often for longer than that. HN didn't used to be a place for people who like to gin up conspiracies, and engage in speculative hyperbole.
- mixdup 1y agoThe fleet of satellites is still managed centrally, even if the physical infrastructure is distributed globally to be able to see satellites in different hemispheres, at some point they are very likely managing these from a single control plane
- AdamJacobMuller 1y agoI wonder what they do for out of band on satellites. It's not like you can have a tech go out and reboot it or reinstall windows.
- cyanydeez 1y agoTheir space infra is all disposable. Its GUGD.
- lxgr 1y agoEach individual satellite, yes. But killing all of them at the same time, e.g. due to a bad software or configuration update, might be the end of the company.
- cyanydeez 1y agoyeah, that would be an interesting dillema. Lucky for us, they technically arn't designed to be anything but disposable, so they'll fall from the heavens eventually.
- lxgr 1y agoI bet they have some pretty robust hardware watchdogs that allow rebooting, booting from firmware OTA etc. in case of local data corruption. Here's a great talk on roughly this topic: https://media.ccc.de/v/38c3-hacking-yourself-a-satellite-recovering-beesat-1 https://media.ccc.de/v/38c3-hacking-yourself-a-satellite-rec...
- nothacking_ 1y agoI'd bet on a bad update or configuration change. (likely one that prevents the affected systems from reaching the internet and being automatically rolled back)
- system2 1y agoIt is always the DNS.
- Whatarethese 1y agoshutdown now. Forgot the /r
- x______________ 1y agoMy money is on a misconfigured switch or hub that allowed a broadcast storm that shut down all of their edge devices! Additional points for a user to cause the loop by plugging an rj-45 where they weren't supposed to, despite no warning or label telling them not to!
- cyanydeez 1y ago...in space?
- _verandaguy 1y agoCat 9 cables are rated for low crosstalk links in LEO, though I'm not sure how Starlink solved the problem where the cable would tend to wind around the earth because of the speed difference between LEO and the Earth's rotation.
- saltcured 1y agoWell, obviously the sats have to juggle cables. Only connect to one for a little bit and then toss it to the next in the train...
- zamadatix 1y agoI think the network putting out warning labels to prevent loops is probably worse off than the place that just forgot to leave some form of loop prevention enabled.
- heraldgeezer 1y agoAre you saying they are one big L2 domain? I love when devs talk about networking.
- esseph 1y agoThey said "hub" Heh.
- nikanj 1y agoThe usual dark triad: DNS, BGP and IPv6
- lxgr 1y agoThey still have tons of ground-based infrastructure that has all the regular failure modes. I suspect the satellites also route IP traffic statically, and they only do so to a fixed set of ground gateways all operated by Starlink (compared to e.g. Iridium, which makes in-space switching decisions and can at least continue ongoing phone calls between two devices without actually needing a reachable ground gateway). National telcos manage to take themselves out countrywide from time to time too, so I'd assume the same usual suspects apply: Routing problems, botched global configuration changes etc. If anything, due to centralization, ubiquitous and effectively free connectivity, and centralization/automation of configuration changes, the blast radius of any given error or malicious action has probably been steadily increasing over the years.
- FollowingTheDao 1y agoI was browsing Reddit and X and a lot of people saying that their base stations couldn’t even find a satellite. I find that pretty odd and can’t make sense out of it if it was simply a network problem. You would expect to be able to at least connect to a satellite unless the satellite has no Internet connection so it does not make itself available to be connected to you. Anyone have any deeper knowledge about this for the curious?
- scoofy 1y agoKessler syndrome?
- beefnugs 1y agoToo many HEILs near the rack power buttons