7 ms·

GPS error caused '12 hours of problems' for companies

96 points by majke 11y ago

fudged71 11y ago
Funny, I just saw this post yesterday by DARPA about preventing GPS issues: http://gazette.com/defense-department-researchers-push-plan-aimed-at-avoiding-worldwide-gps-meltdown/article/1569000 http://gazette.com/defense-department-researchers-push-plan-...
skykooler 11y ago
Interesting. Would the 13-microsecond discrepancy mean that GPS positions were over two miles off during those 12 hours?
- bjcubsfan 11y ago
  No, the discrepancy was only on a timing message for determining UTC. This is not used for determining position.
  - cnvogel 11y ago
    Specifically, one or few of the sattelites sent a wrong value "A0" in its "Page 18 of subframe 4" message. Page 37, §2.4.5.5 in http://www.gps.gov/technical/ps/1995-SPS-signal-specification.pdf http://www.gps.gov/technical/ps/1995-SPS-signal-specificatio... https://www.febo.com/pipermail/time-nuts/2016-January/095692.html https://www.febo.com/pipermail/time-nuts/2016-January/095692...
- ars 11y ago
  Probably not if they were all off by the same amount. Position is based on relative difference between time signals received from different satellites, not absolute.
  - bjcubsfan 11y ago
    This error only occurred on a certain revision of GPS satellites, not all of them. It would have been a problem, except luckily the error was only on a message for determining UTC. The error did not affect the message or clock that are used in determining position.
spencerhakim 11y ago
It's fascinating that an amount of time small enough to be completely meaningless to most of us can have such a big impact.
madaxe_again 11y ago
I'm curious as to why they're using atomic clocks in space for a high resolution time source, rather than an atomic clock on earth... Latency?
- fennecfoxen 11y ago
  At a guess, they need a very small number of clocks (for uniformity) and they use the existing clocks in space because that signal distribution was more effective than new terrestrial infrastructure.
- jewel 11y ago
  I think it's a matter of price. GPS receivers are cheap and widely supported by timekeeping software. The cheapest atomic clock I can find is $1,500, but it's a hard thing to search for since the term is used to refer to radio-synchronized clocks. If you did have an atomic clock in every datacenter, you'd still need to make sure that they didn't develop a problem that caused them to drift apart, and also you'd need to make sure that they were in sync with each other in the first place.
  - ploxiln 11y ago
    it would be a real problem if they drifted at all ... that's kinda the point of atomic clocks, no?
    - detaro 11y ago
      Of course atomic clocks drift as well, and as soon as you run multiple you have to keep that in mind. GPS (or rather a GPS-controlled quartz) is probably better quality than cheap atomic clocks. GPS is a convenient way to get well-synced precise time nearly everywhere you can see the sky.
      - ploxiln 11y ago
        GPS satellites themselves use atomic clocks (and to find your position you use their also-broadcast orbital positions and math for the effects of relativity etc)
pjc50 11y ago
In particular, DAB depends on exact time synchronisation to avoid interference. Every transmitter in the country has to broadcast the same OFDM symbol at the same time. https://en.wikipedia.org/wiki/Single-frequency_network https://en.wikipedia.org/wiki/Single-frequency_network The kit is supposed to accomodate loss of GPS, but evidently doesn't handle error so well: http://www.microsemi.com/products/timing-synchronization-systems/time-frequency-distribution/gps-instruments/4370a-dvb-sync-source http://www.microsemi.com/products/timing-synchronization-sys...
- elfchief 11y ago
  It's not that surprising that it has a problem with error, because the reality is, there's no correct solution for dealing with this kind of error. The first question, of course, is "how do you determine when there's an error?" Half the GPS constellation was broadcasting the data, and for almost all purposes it was 'correct' data, other than being wrong. Perhaps ground stations could determine that they're getting different data from different satellites (if they have some good ones in view), or maybe they could just decide that the new parameters differed enough from the old ones to be implausible, but neither of these is a cut and dry "I definitely have an error" ... but let's say you have a foolproof way to determine that A0/A1 are incorrect. Great. THEN what do you do? You could just decide to use the previous value for that parameter... which is great for a single clock, but what about clocks elsewhere that have a different value (the one being broadcast by the rest of the satellites) or an out of date value (say, a clock that was powered off while traveling in a broadcast van)? Barring external communication, you could have an arbitrary number of different interpretations of the current time -- a different one for each ground station! You could also just accept that the new parameters are correct, but how do you then get your clock to the new correct time, without having a second (or other time period) that's significantly longer or shorter than a second? There's a lot of gear that expects a second to be a specific length, which is not gonna be happy if the length changes. (Think of playing back an audio file that's exactly 30 minutes long, and needs to start and stop at specific times, down to the bit. You then configure your system to send one bit every (1/bitrate) seconds exactly. How does that system recover when 13µs of bits go missing? How do you coordinate that behavior over multiple disconnected systems around the world which may have gotten the same information at different times (or not at all)? From reading the article, it sounds like what happened was that the units didn't fail because of the GPS issue, but that they start throwing alarms (probably while going into a mode where they ran off only their internal (disciplined) oscillator. This seems like absolutely the right thing to do -- "uh, boss, things look really not-right here, and I can't fix it without breaking something, you really need to come deal with this yourself." (Note the comments about 'thousands of system warnings' but none about 'cellphones broke.') This is, from what I can see, really the only reasonable behavior these devices could have. (disclaimer: I don't have any direct data on what broke vs. what complained loudly and woke people up. I'd love to have some solid, not-parsed-by-the-tech-media data on this topic, if anyone has any.)
basicplus2 11y ago
more like.. what secret military mission was taking place during this time...
- chippy 11y ago
  or space craft, satellite launch from an "unfriendly" state...
- jrockway 11y ago
  This would have no effect on military uses. It was a static piece of data that was misconfigured on one satellite, not the actual time code on any satellite. Disrupting GPS is less useful for military goals these days, since there are independent positioning constellations, like GLONASS.
jevinskie 11y ago
This is the press release that I received: https://ghostbin.com/paste/hdj3j https://ghostbin.com/paste/hdj3j
Johnny_Brahms 11y ago
"After the initial jolt, one of the first effects you'd notice would be that your GPS would stop working. The satellites would stay in roughly the same orbits, but the delicate timing that the GPS system is based on would be completely ruined within hours" https://what-if.xkcd.com/67/ https://what-if.xkcd.com/67/

96 points by majke 11y ago

Funny, I just saw this post yesterday by DARPA about preventing GPS issues: http://gazette.com/defense-department-researchers-push-plan-aimed-at-avoiding-worldwide-gps-meltdown/article/1569000 http://gazette.com/defense-department-researchers-push-plan-...

skykooler 11y ago

Interesting. Would the 13-microsecond discrepancy mean that GPS positions were over two miles off during those 12 hours?

bjcubsfan 11y ago
No, the discrepancy was only on a timing message for determining UTC. This is not used for determining position.
- cnvogel 11y ago
  Specifically, one or few of the sattelites sent a wrong value "A0" in its "Page 18 of subframe 4" message. Page 37, §2.4.5.5 in http://www.gps.gov/technical/ps/1995-SPS-signal-specification.pdf http://www.gps.gov/technical/ps/1995-SPS-signal-specificatio... https://www.febo.com/pipermail/time-nuts/2016-January/095692.html https://www.febo.com/pipermail/time-nuts/2016-January/095692...
ars 11y ago
Probably not if they were all off by the same amount. Position is based on relative difference between time signals received from different satellites, not absolute.
- bjcubsfan 11y ago
  This error only occurred on a certain revision of GPS satellites, not all of them. It would have been a problem, except luckily the error was only on a message for determining UTC. The error did not affect the message or clock that are used in determining position.

spencerhakim 11y ago

It's fascinating that an amount of time small enough to be completely meaningless to most of us can have such a big impact.

madaxe_again 11y ago

I'm curious as to why they're using atomic clocks in space for a high resolution time source, rather than an atomic clock on earth... Latency?

fennecfoxen 11y ago
At a guess, they need a very small number of clocks (for uniformity) and they use the existing clocks in space because that signal distribution was more effective than new terrestrial infrastructure.
jewel 11y ago
I think it's a matter of price. GPS receivers are cheap and widely supported by timekeeping software. The cheapest atomic clock I can find is $1,500, but it's a hard thing to search for since the term is used to refer to radio-synchronized clocks. If you did have an atomic clock in every datacenter, you'd still need to make sure that they didn't develop a problem that caused them to drift apart, and also you'd need to make sure that they were in sync with each other in the first place.
- ploxiln 11y ago
  it would be a real problem if they drifted at all ... that's kinda the point of atomic clocks, no?
  - detaro 11y ago
    Of course atomic clocks drift as well, and as soon as you run multiple you have to keep that in mind. GPS (or rather a GPS-controlled quartz) is probably better quality than cheap atomic clocks. GPS is a convenient way to get well-synced precise time nearly everywhere you can see the sky.
    - ploxiln 11y ago
      GPS satellites themselves use atomic clocks (and to find your position you use their also-broadcast orbital positions and math for the effects of relativity etc)

pjc50 11y ago

In particular, DAB depends on exact time synchronisation to avoid interference. Every transmitter in the country has to broadcast the same OFDM symbol at the same time. https://en.wikipedia.org/wiki/Single-frequency_network https://en.wikipedia.org/wiki/Single-frequency_network The kit is supposed to accomodate loss of GPS, but evidently doesn't handle error so well: http://www.microsemi.com/products/timing-synchronization-systems/time-frequency-distribution/gps-instruments/4370a-dvb-sync-source http://www.microsemi.com/products/timing-synchronization-sys...

elfchief 11y ago
It's not that surprising that it has a problem with error, because the reality is, there's no correct solution for dealing with this kind of error. The first question, of course, is "how do you determine when there's an error?" Half the GPS constellation was broadcasting the data, and for almost all purposes it was 'correct' data, other than being wrong. Perhaps ground stations could determine that they're getting different data from different satellites (if they have some good ones in view), or maybe they could just decide that the new parameters differed enough from the old ones to be implausible, but neither of these is a cut and dry "I definitely have an error" ... but let's say you have a foolproof way to determine that A0/A1 are incorrect. Great. THEN what do you do? You could just decide to use the previous value for that parameter... which is great for a single clock, but what about clocks elsewhere that have a different value (the one being broadcast by the rest of the satellites) or an out of date value (say, a clock that was powered off while traveling in a broadcast van)? Barring external communication, you could have an arbitrary number of different interpretations of the current time -- a different one for each ground station! You could also just accept that the new parameters are correct, but how do you then get your clock to the new correct time, without having a second (or other time period) that's significantly longer or shorter than a second? There's a lot of gear that expects a second to be a specific length, which is not gonna be happy if the length changes. (Think of playing back an audio file that's exactly 30 minutes long, and needs to start and stop at specific times, down to the bit. You then configure your system to send one bit every (1/bitrate) seconds exactly. How does that system recover when 13µs of bits go missing? How do you coordinate that behavior over multiple disconnected systems around the world which may have gotten the same information at different times (or not at all)? From reading the article, it sounds like what happened was that the units didn't fail because of the GPS issue, but that they start throwing alarms (probably while going into a mode where they ran off only their internal (disciplined) oscillator. This seems like absolutely the right thing to do -- "uh, boss, things look really not-right here, and I can't fix it without breaking something, you really need to come deal with this yourself." (Note the comments about 'thousands of system warnings' but none about 'cellphones broke.') This is, from what I can see, really the only reasonable behavior these devices could have. (disclaimer: I don't have any direct data on what broke vs. what complained loudly and woke people up. I'd love to have some solid, not-parsed-by-the-tech-media data on this topic, if anyone has any.)

basicplus2 11y ago

more like.. what secret military mission was taking place during this time...

chippy 11y ago
or space craft, satellite launch from an "unfriendly" state...
jrockway 11y ago
This would have no effect on military uses. It was a static piece of data that was misconfigured on one satellite, not the actual time code on any satellite. Disrupting GPS is less useful for military goals these days, since there are independent positioning constellations, like GLONASS.

jevinskie 11y ago

This is the press release that I received: https://ghostbin.com/paste/hdj3j https://ghostbin.com/paste/hdj3j

Johnny_Brahms 11y ago

"After the initial jolt, one of the first effects you'd notice would be that your GPS would stop working. The satellites would stay in roughly the same orbits, but the delicate timing that the GPS system is based on would be completely ruined within hours" https://what-if.xkcd.com/67/ https://what-if.xkcd.com/67/