11 ms·
EU's GPS satellites have been down for four days in mysterious outage
- Wellshit 7y agoMy best guess is a GDPR popup klutzing the system
- saagarjha 7y ago> In a statement published after this article's publication, the GSA blamed the Galileo outage on "a technical incident related to its ground infrastructure." I wonder what this could mean?
- rando444 7y agoSince there is nothing available but speculation, I'm going to put my money on a network issue.
- dcomp 7y agoMy 2p: I think an atomic clock has failed
- solarkraft 7y agoThis sounds like it'd fit the accuracy continually lowering before the system was turned off.
- galangalalgol 7y agothat was my first thought as well but surely they can replace it faster than this? also, the search and rescue signal still works. search and rescue signal are in a different band, so maybe antenna issues, or it has its own time reference. or maybe it's being jammed like the gps signals in the middle East, they are the same band.
- _Codemonkeyism 7y agoIt looks more like they took the service down intentionally due to problems - it's still in beta. And kept the SAR signal intentionally up.
- tialaramex 7y agoA COSPAS SARSAT MEOSAR is essentially a separate platform that just happens to be traveling on the same vehicle. As deployed today the two needn't have any electronic connection although they do. Eventually a MEOSAR is intended to use the accompanying navigation transmitter to send replies from ground SAR. Initially an acknowledgement ("We see your distress beacon" and by implication "somebody will try to help") and perhaps later more. But that requires proof of concept and then newer beacons as today's beacons, even the ones with GPS (many rely on COSPAS SARSAT to triangulate their position itself) do not know such an acknowledgment might arrive or how to interpret it.
- wolrah 7y agoInsert "it's always DNS" meme here...
- benj111 7y agoIsnt it always lupus?.... Or sarcoidosis.
- blaser-waffle 7y agoA haiku shamelessly stolen from elsewhere: It's not DNS There's no way it's DNS It was DNS
- mtmail 7y agocopying from https://news.ycombinator.com/item?id=20427544 https://news.ycombinator.com/item?id=20427544 Guenter Hein, Professor Emeritus of Excellence at the University FAF Munich told us, “As far as I know, it is a problem of the PTF [Precise Timing Facility] in Italy – time has an impact on the whole constellation!”
- saagarjha 7y agoBut the satellites should be carrying their own clocks?
- mikeash 7y agoThey do, but they’re less accurate than the ground station’s clocks. If not updated regularly, they’ll drift, with a gradual loss of positional accuracy.
- brootstrap 7y agowe had this issues on our EC2 servers getting out of sync with other systems. Had to schedule some crontabs to ntp update yo. Maybe them EU dudes forgot to sync their clocks?
- a012 7y agoRun chronyd on every Linux box is in my checklist (ansible playbook). It's not much but more accurate than a cronjob.
- tzs 7y agoSounds like a bug we hit at AWS a while back. The clocks on several of our instances started running at incorrect speeds. One was gaining something like 10 seconds every minute or something ridiculous like that. The rate was steady, just very far off from one minute per minute. The Linux timekeeping system can normally deal with a clock that is steady but not the right rate by learning what correction to apply, but these were far enough off to be well beyond the maximum correction. This only affected particular instance types, so changing to a non-affect type could fix it, but all the working types were more expensive than the types we were using which was annoying. Sometimes you could change to a working type then back to the buggy type and end up with one that did not have the problem. There were a few people asking about this in the support forums, but apparently it was only hitting cheapskates like us who had not paid extra for support. Finally, someone paid for incident support and opened a ticket and Amazon quickly found and fixed the problem.
- pgt 7y agoNot trying to be funny here, but I would not be surprised if an automated Windows update borked something. Recently heard about an Antarctic base that lost heating for 36 hears after a Windows Update triggered a several GB download over the base's modem downlink.
- wrkronmiller 7y agoCan you provide a link to that Antarctic incident? I'm curious how a network outage would affect a system as critical as heating...
- eightysixfour 7y agoThe heater required an active internet connection and available bandwidth?
- privong 7y ago> The heater required an active internet connection and available bandwidth? My guess is that the heater was being run by the machine that required the update and the update download process meant the machine was not available for usual operation. But I don't really know.
- eightysixfour 7y agoWindows downloads the updates separately from installing them. Good guess though.
- huhtenberg 7y ago... unless it needs to do a quick follow-up update after applying one of the pre-downloaded ones. And then not three reboots later you get your machine back.
- martin_a 7y agoSounds like a pretty normal device to me in the times of IoT...
- stupidcar 7y agoGiven that the project is funded by €10 billion of EU taxes, I hope we'll get transparency sooner rather than later on exactly what's gone wrong, regardless of who it might embarrass.
- rtkwe 7y agoLike so? [0] Seems like they're still fixing it and it was in a pilot phase anyways. [0] https://www.gsa.europa.eu/newsroom/news/update-availability-some-galileo-initial-services https://www.gsa.europa.eu/newsroom/news/update-availability-...
- lallysingh 7y ago"Galileo provides ‘initial services’ since December 2016. During this initial "pilot" phase preceding the ‘full operational services’ phase, Galileo signals are used in combination with other satellite navigation systems, which allows for the detection of technical issues before the system becomes fully operational."
- aw3c2 7y agoNo, that gives almost zero information as to the known cause: > A technical incident in the Galileo ground infrastructure is affecting the functioning of the Galileo system, as a result of which there is a temporary interruption of some of the Galileo initial services. > The cause of the technical incident is identified and recovery actions are implemented to ensure that the nominal service is resumed as soon as possible while safeguarding quality of the services.
- reneberlin 7y agoTo me that reads like: ransomware has hit our control center. Windows (tm)
- the_mitsuhiko 7y agoConsidering they are running this as an early preview I'm not sure if we are going to get much in terms of public response. Maybe now that it's a long outage journalists will be poking enough. Pretty sure universities and other institutions would be able to inquire though.
- t3rabytes 7y agoIt's a system/service that's still in beta, I'd expect some occasional issues until it's deemed "production-ready".
- jhayward 7y agoEverything on orbit and on the ground is production product. When you are several billion euros in, have 24 satellites on orbit, and have been broadcasting with the 'Valid' bit set for 2 years you aren't really in 'beta' as we use it. They have been in production mode in all but name. Given the life/safety implications, there are some rather rigid qualification checks that have to be done to be 'fully certified'. That's what's been going on.
- benj111 7y agoSo what would break if GPS went down for 4 days? Sat nav obviously, Amazon package tracking, Grindr? (I assume that falls back to cell tower triangulation???). I guess theres some more serious things, any examples?
- systemtest 7y agoShutting down GPS would be a good way to avert the upcoming attack on Area 51.
- benj111 7y agoI'm guessing this is a reference that's going over my head?
- msavelyev 7y agohttps://www.facebook.com/events/area-51/storm-area-51-they-cant-stop-all-of-us/448435052621047/ https://www.facebook.com/events/area-51/storm-area-51-they-c...
- JamesCoyne 7y agoYes. https://old.reddit.com/r/OutOfTheLoop/comments/cc3qnt/whats_up_with_all_these_area_51_memes/ https://old.reddit.com/r/OutOfTheLoop/comments/cc3qnt/whats_...
- crysin 7y agohttps://www.independent.co.uk/news/world/americas/storm-area-51-latest-aliens-sighting-us-military-ready-nevada-a9004086.html https://www.independent.co.uk/news/world/americas/storm-area... https://www.cnn.com/2019/07/13/us/area-51-raid-memes-trnd/index.html https://www.cnn.com/2019/07/13/us/area-51-raid-memes-trnd/in... https://deadline.com/2019/07/area-51-storming-may-be-countered-with-force-says-us-air-force-1202646103/ https://deadline.com/2019/07/area-51-storming-may-be-counter... https://www.washingtonpost.com/national-security/2019/07/13/half-million-people-signed-up-storm-area-what-happens-if-they-actually-show-up/?utm_term=.2b6da3a97b39 https://www.washingtonpost.com/national-security/2019/07/13/...
- ppod 7y agoThey put the Precise Timing Facility in Italy?
- herpes 7y agoWhat's wrong with Italy?
- outside1234 7y agoNothing runs on time in Italy. Someone on this thread is going to say that is racism, but its backed up by a lot of data.
- MichaelApproved 7y agoCan you share that data with the rest of us?
- rbrtdrmpc- 7y agoNothing runs on time at Consorzio Torino Time (CTT), the guys quoted previously, also "The PTF in Italy been coordinated by the Consorzio Torino Time (CTT), with the partnership and support of SpT and T4Science in Switzerland." so i'm pretty sure as an Italian who enjoys to stab Caesar and switch sides during war, THE BLAME IS ON THE SWISS
- rbrtdrmpc- 7y agoA little rain today, despite that things are pretty good.
- toyg 7y agoCasual racism, because fuck yeah. Italy has a pretty big space-related industrial sector, and excellent academic standards in some fields. It might not produce geniuses with the same frequency of the past (Galvani, Volta, Marconi, Fermi and so on) but it's not some random backwater.
- martin_a 7y ago
- supermatt 7y agoI had someone come and measure boundaries for some land on Friday, supposedly with centimetre precision. Does this mean that the markers they installed are likely in the wrong place?
- tinco 7y agoNo, those markers work by listening to the GPS signals for an extended period of time and maybe PPK or perhaps an alternative network that's not satellite based.
- supermatt 7y agoSo it would be regular GPS assisted with some basestation at a known location to boost accuracy, as opposed to GNSS (Galileo)?
- tinco 7y agoYes, I'm not in land surveying myself, but my company builds georeferenced 3D models of buildings at high resolution and sub cm precision, and we never rely solely on GPS information. There's too many things that can go wrong with GPS to reliably fly a drone around a building for example.
- AnssiH 7y ago"GNSS" just means satellite positioning systems in general, and e.g. "GPS" and "Galileo" are examples of GNSS systems. At least regular receivers (e.g. in phones) utilize signals from multiple satellite constellations simultaneously, not sure if this is the case here as well.
- jhayward 7y agoYes, exactly. They use the signal as received at a nearby marker previously surveyed with high accuracy to calculate a local 'correction' to the GPS/GNSS signal, and give that correction to the roving units.
- module0000 7y ago
- michaelt 7y agoI'm more familiar with the older US GPS system than I am with Galileo, but I can make an informed guess. GPS satellites broadcast "ephemerides" letting receivers work out where the satellite is - and some other details like clock error and predicted effects of the ionosphere. You can make a rough prediction of these weeks in advance, but not precisely enough to meet GPS's precision requirements. Up-to-date short term predictions are needed to make things precise enough, so a ground-based 'ground segment' or 'control segment' broadcasts updates to these every 30 minutes or so. The 'ground segment' is comprised of ~20 monitoring stations (at known locations, measuring the errors on signals from satellites), one 'master control station' that aggregates this data and decides what update to send next, 4 ground antennas that can send that update to the satellite no matter what side of the globe it happens to be on, and a backup MCS. This is what is meant by "ground infrastructure". The design of GPS is informed by its cold-war-era-military origins, so there's a backup master control station, and the satellites have an 'Autonav mode' letting them operate for 60 days in even if the entire ground segment has been destroyed. Assuming the Galileo system has a broadly similar design, you can imagine the criteria for a four-day outage: * Failure of both the master control station and the backup, in a way that takes more than 4 days to recover from. Maybe their backup control station wasn't fully commissioned yet, wasn't fully redundant, or some IT problem hit both sites at once? * Discovery of some fault severe enough they felt they had to shut down until it was fixed, and the fix couldn't be rushed out. * Either satellites have no autonav mode, or it somehow failed (such as due to bad data from the ground segment), or it's imprecise enough they decided pushing people onto GPS was a more responsible option.
- makomk 7y agoThe rumour I saw was that the entire ground segment is dependent on a single Precise Timing Facility in Italy, which went down and took everything with it. Not sure what the holdover capability is like without the ground stations, but I'd venture a guess it's not great,
- michaelt 7y agoAccording to a report from 2009 [1] "The Galileo ground segment has two GCCs, each with a Precise Timing Facility (PTF) providing a very stable reference GST to all elements in the GCC. The two PTFs are developed by separate companies; this diversification is applied to reduce the possibility of common failures in this important element." I suppose it's possible the second GCC hadn't been fully commissioned? Or the switchover failed in some very hard to recover way? [1] https://reports.nlr.nl/xmlui/bitstream/handle/10921/210/TP-2009-082.pdf?sequence=1&isAllowed=y https://reports.nlr.nl/xmlui/bitstream/handle/10921/210/TP-2...
- deleted 7y ago[deleted]
- dfeojm-zlib 7y agoThis is a good reminder to buy and use satnav modules supporting multiple positioning systems and technologies: most of GLONASS, GPS, BDS, Galileo, IRNSS, SkyHook, etc. Disclaimer: Trimble Ltd. alum
- droduduwaherbal 7y agoAM HERBALIST DR ODUDUWA, I Cure Different Types Of Human Infections And Diseases Using Natural Roots And Herbs. I’m also a professional spell caster and my major areas of concentration are Sexually transmitted infections (STI), also referred to as sexually transmitted diseases (STD) e.g; herpes 1&2, HPV, diabetes 1&2, Hepatitis B, HIV and venereal diseases (VD)” contact me on my email; dr.oduduwa111@gmail.com or website oduduwatraditionalherbalremedies.wordpress.com