8 ms·

Multiple Boeing 787s in China experienced GPS 20 years rollover issue

126 points by jcfrei 8y ago

deleted 8y ago
[deleted]
PaulHoule 8y ago
Funny I got an old Tom Tom for $10 a few weeks ago, too old to get software updates, and I was pleased to see that it worked just fine after the rollover.
- 420codebro 8y ago
  It isn't really the Nav/Positioning that has issue- but the time. Check the time on your Tom Tom unit. Is it 2019?
  - bookofjoe 8y ago
    Sort of related: my Sony TV, circa 2013, ALWAYS has this opening message (even if there's been no power outage that I've noticed): "Power outage. Time needs to be reset." I've been ignoring it since 2014 because it takes about 5 minutes and a half-dozen menus correctly sequenced (but only after multiple FAILs) to reset correctly.
- labster 8y ago
  At the [dead] commenter: Timekeeping is an integral part of how the Global Positioning System works, to the extent that relativistic time dilation is a factor. You simply cannot use a GPS receiver with a broken time implementation, at least not without significant accuracy issues.
  - deadCommenter 8y ago
    Funny how we have to pretend people don’t exist, in order to preserve the decorum and subculture around here.
  - myself248 8y ago
    But internal time is not the same as display time. Internally, the week counter just rolled over. That's fine; all the almanac data is handled in that same format internally. Some receivers may need a reset when this counter rolls, and may need to relearn the almanac/ephemeris from the sky, but once that's done, Sunday of week 0 is just the same as any other Sunday of any other week 0. Navigation works just fine because it's all internally consistent. But display time, ahh, now that's where things get interesting. Sunday of week 0 could be displayed as 2019 April 7, or it could be displayed as 1999 August 22, or it could be displayed as 1980 January 6. Knowing which epoch you're in is actually considerably trickier than working consistently internally, since there's no information in the satellite signal to tell you this, by design. It comes from applying an external offset. Receivers have lots of ways to do this, and quite a few simply have it baked in and cannot handle a rollover, so as far as they're concerned, week 0 is in 1999, period. They'll navigate just fine but they'll display the date wrong, because they're applying the wrong offset.
    - PaulHoule 8y ago
      Funny enough my Tom Tom made some noise and seemed to boot with a different sequence than usual, in particular it insisted that I remove the USB cable that I use to power it from the car. I did that, and it found the signal in 50 seconds and worked normally, correct date and everything.
    - labster 8y ago
      Cool, see, I knew this was worth talking about. While I've learned about GPS as a way to gather meteorological data[0] via radio occultation, I never had to deal with the raw data stream myself. Thanks for your expertise. [0]: https://en.wikipedia.org/wiki/Constellation_Observing_System_for_Meteorology,_Ionosphere,_and_Climate https://en.wikipedia.org/wiki/Constellation_Observing_System...
    - joelhaasnoot 8y ago
      The easiest way of doing this is to assume time will never travel and store the latest computed date right? If you've seen 2001, it's never going to be 1999 again, so it must be 2019. I understand that will require either flash or battery backed RAM, etc to keep this stored, but seems relatively trivial in most applications... (side effect would be that if the backup battery is dead or all batteries are disconnected, you would revert to 1999)
- dsfyu404ed 8y ago
  Any tips for someone looking for similarly priced antiquated GPS's to shove in the glove-box of all the cars in their fleet?
  - deleted 8y ago
    [deleted]
  - PaulHoule 8y ago
    There were four of them here: https://ithacareuse.org/ https://ithacareuse.org/ there also was a $5 pair of Bose headphones that I was stupid to not buy the instant I saw them. The Tom Tom I have is too old to get updated maps, but it's not a problem driving around in upstate NY. In New Hampshire it might be another thing, but where I live they build new buildings but not new roads.
gwbas1c 8y ago
Dumb question: What is the rollover issue?
- usermac 8y ago
  I had to look it up as well. I found this link: https://www.orolia.com/resources/blog/lisa-perdue/2018/gps-2019-week-rollover-what-you-need-know https://www.orolia.com/resources/blog/lisa-perdue/2018/gps-2... From the text comes: "However, the field that contains the week number is a 10-bit binary number. This limits the range of the week number to 0 – 1023, or 1024 total weeks."
- Someone1234 8y ago
  Not a dumb question at all. GPS used ten bits for datetime. With a week counter and seconds into that week since 6th of January, 1980. After 1024 weeks it resets back to 0. This has happened once already on 21st of August, 1999. This is the second rollover/reset of the week counter on 6th of April, 2019. Newer versions of GPS messages use thirteen digits instead of ten digits allowing for a larger week counter (granting 8,192 weeks), causing the first rollover not to occur until the year 2137. https://en.wikipedia.org/wiki/Global_Positioning_System#Timekeeping https://en.wikipedia.org/wiki/Global_Positioning_System#Time...
  - kevhito 8y ago
    Instead of adding 3 bits, maybe they should have dropped 3 bits. A rollover every ~20 years is just asking for latent buts to blow up. A rollover every ~2.5 years is just business as usual. Go big or go small. Don't design things to break just after you retire but before your coworkers do.
    - Rebelgecko 8y ago
      OTOH, with the current setup a receiver will know the correct date as long as it is turned on at least once every 1024 weeks, since it can detect the rollover. More frequent rollovers might cause issues with timing on devices that are infrequently turned on. It's sort of a moot point for most of us once there's greater adoption of the L2 and L5 CNAV signals— those use a week counter that won't roll over until most if not all of us are dead. Then it can be someone else's problem :)
petee 8y ago
I don't get how some planes were affected, but not all; Every 787 in China should be running the same software version, let alone the world, no? Are there any reasons a fleet plane might be using a different version? I'm coming up short on reasons, other than language, or maybe special hardware modifications?
- j16sdiz 8y ago
  They have regular software updates. Just like Windows update, nobody do that regularly
  - petee 8y ago
    Haha, that's surprising though, since these are important systems, if there is an update, doesn't it mean something really needs to fixed, i.e, gps rollover? Any bug left unpatched seems unacceptable Maybe I have more faith in the 'system' than I should...
    - kijin 8y ago
      We're talking about the same manufacturer that's responsible for the 737 MAX fiasco. I'm not sure how much trust I would want to place in their software.
      - icebraining 8y ago
        You're putting trust in their software whether you upgrade it or not.
        sslayer 8y ago
        You forget that Boeing doesn't own these planes. These aren't Tesla's. We are currently in an era were it seems to be expected that the manufacturer automatically update their software without the end user intervention, ie Tesla, Windows, Apple, etc... Thats not how it was done in the past, particularly when the bandwidth didn't exist.
js2 8y ago
Related discussion which explains the rollover issue: https://news.ycombinator.com/item?id=19593227 https://news.ycombinator.com/item?id=19593227
mschuetz 8y ago
What's the reasoning for the current GPS time format and it's resulting issues? Can't we just use a 64bit integer timestamp? That would give us roughly 580 years in nanosecond resolution?
- crooked-v 8y ago
  > What's the reasoning for the current GPS time format and it's resulting issues? Somebody wanted to save money when building the original satellites in the 1970s.
- bmurray7jhu 8y ago
  The GPS signal format was was designed in the 1970s. Any change to the signal format must be backwards compatible with legacy receivers. There have been some efforts to modernize the GPS signal format. In 2014, select GPS satellites began broadcasting an additional message (CNAV) with a 13 bit week number, allowing a ~157 epoch.
- jhayward 8y ago
  > What's the reasoning for the current GPS time format and it's resulting issues? GPS satellites and receivers last on the order of 20 years so it takes a long time to age out of legacy support. The latest generation of satellites, GPS-III, have a new week number format that allows for 13-bit week numbers (120 year rollover). The reason for the low number bits in week number is simple, it's broadcast on a very low bit rate (50 bits per second) and every frame has the week number in it. So you get faster fixes if you don't pad out with useless zeros.
  - throw0101a 8y ago
    > The reason for the low number bits in week number is simple, it's broadcast on a very low bit rate (50 bits per second) and every frame has the week number in it. So you get faster fixes if you don't pad out with useless zeros. It should be noted that the coders could have used data windowing and pivot dates to make certain assumptions: > When ntpd(8) receives an unresolved timestamp from an upstream server, that timestamp could be based in any era prior to the current wall-clock date - but by definition we may not know the wall-clock date with confidence yet (not having achieved sync). > To resolve this ambiguity, NTP also uses an internal pivot date. It chooses the era that would minimize the absolute value of the time difference between the resolved timestamp and its internal pivot date. > This resolution technique is part of the NTP specification. It will normally resolve to the current era, but may in cases where the pivot is within a half-cycle of a rollover time resolve to either the previous or the next era. > An ntpd(8) instance’s pivot date will be the date it was compiled and built. > It is fairly easy to see that this guarantees correct resolution if the pivot dates of the communicating ntpds are within a half-cycle (88 years) of each other, even if there is a rollover between the pivot dates. * https://docs.ntpsec.org/latest/rollover.html#ntp_pivots https://docs.ntpsec.org/latest/rollover.html#ntp_pivots * https://en.wikipedia.org/wiki/Date_windowing https://en.wikipedia.org/wiki/Date_windowing
userbinator 8y ago
The 787 didn't even exist when the GPS time last rolled over, so it's quite unusual to see it have this problem --- unless Boeing reused the software from a previous version, which is not surprising given the regulations of the industry, but that would imply other older models also have the problem.
- acqq 8y ago
  > The 787 didn't even exist when the GPS time last rolled over, so it's quite unusual to see it have this problem It's not as unusual as you'd think: https://blog.fosketts.net/2019/04/06/gps-time-rollover-failures-keep-happening-but-theyre-almost-done/ https://blog.fosketts.net/2019/04/06/gps-time-rollover-failu... There you see a nice summary of the problem and an example a Mercedes car from, if I understood, 2008 experiencing the same type of problems. Also discussed on HN: https://news.ycombinator.com/item?id=19593227 https://news.ycombinator.com/item?id=19593227 It's like having a Y2K bug software and saying "but that software didn't exist before 1900." It doesn't matter when the "epochs" are, it's just about what the design covered or didn't. I've personally fixed the trading software which failed to correctly implement the correct rollover of the decades (i.e. the transition from year 9 to year 0 in the names), not to mention Y2K. It was simply tested before only "in the same decade." It's definitely a failure in the development process, for the software that is supposed to work in some bigger time frame, and not only shortly.
  - syockit 8y ago
    I'm rather interested in what kind of software developed after Y2K still exhibiting Y2K bug, and how it affects the usage. Is it made with the assumption that it's going to record only post-Y2K dates, and somehow user manages to input pre-Y2K date?
    - avar 8y ago
      A lot of software was never fixed for the Y2K problem. People just went from e.g. entering a date of 99 to 00 as the new millennium rolled around, and dealt with the pain of any time offset calculations being incorrect and needing to be manually fixed for a few months. Some of it's doubtless still in use and people are entering "19" as the year today, and if it's still in use in 2099 we'll have the exact same problem all over again with the exact same software. That may seem crazy today, but people in the 1950s would have thought you were crazy if you said B-52s were still going to be flying in 2020. There's doubtless going to be people still running Windows 95 on some emulator in 2100 to use some legacy application that keeps chugging along. Some software we use today might be in use for thousands of years or more, unlike airframes it isn't subject to any natural decay or wear.
Thermolabile 8y ago
Can someone give me a source of it?
AnssiH 8y ago
Someone on reddit said yesterday a KLM flight (apparently B777) was grounded because of incorrect date (in some Honeywell equipment?): https://www.reddit.com/r/sysadmin/comments/babyvv/anyone_hit_by_the_gps_rollover_event_today/ekb3ev3/ https://www.reddit.com/r/sysadmin/comments/babyvv/anyone_hit...
- Aromasin 8y ago
  We had some Honeywell engineers at my workplace for a system integration meeting. Over lunch I got the opportunity to talk with them. This came up in conversation. Apparently it's very much all-hands-on-deck at the moment - almost every system they've currently got on-board Boeing aircraft is being assessed, and they're in all hours of the day to get new revisions through. I do not envy them right now.
  - mehrdadn 8y ago
    Is it because they didn't see the GPS rollover coming up (it was in the news though)? Or did they not think their products would be affected?
    - Aromasin 8y ago
      Partly that (I think there were some legacy systems that were missed), partly that every minor clerical and drawing error that got passed through because of how insignificant it was (nothing that would affect the integrity of the aircraft I should add) has been flagged by Boeing. They want to look proactive to their customers and shareholders I suppose. That means lots of tiny, insignificant changes to hundreds of drawings. It's nothing technically challenging for the most part - just lots and lots of paperwork.

126 points by jcfrei 8y ago

deleted 8y ago

[deleted]

Funny I got an old Tom Tom for $10 a few weeks ago, too old to get software updates, and I was pleased to see that it worked just fine after the rollover.

420codebro 8y ago
It isn't really the Nav/Positioning that has issue- but the time. Check the time on your Tom Tom unit. Is it 2019?
- bookofjoe 8y ago
  Sort of related: my Sony TV, circa 2013, ALWAYS has this opening message (even if there's been no power outage that I've noticed): "Power outage. Time needs to be reset." I've been ignoring it since 2014 because it takes about 5 minutes and a half-dozen menus correctly sequenced (but only after multiple FAILs) to reset correctly.
labster 8y ago
At the [dead] commenter: Timekeeping is an integral part of how the Global Positioning System works, to the extent that relativistic time dilation is a factor. You simply cannot use a GPS receiver with a broken time implementation, at least not without significant accuracy issues.
- deadCommenter 8y ago
  Funny how we have to pretend people don’t exist, in order to preserve the decorum and subculture around here.
- myself248 8y ago
  But internal time is not the same as display time. Internally, the week counter just rolled over. That's fine; all the almanac data is handled in that same format internally. Some receivers may need a reset when this counter rolls, and may need to relearn the almanac/ephemeris from the sky, but once that's done, Sunday of week 0 is just the same as any other Sunday of any other week 0. Navigation works just fine because it's all internally consistent. But display time, ahh, now that's where things get interesting. Sunday of week 0 could be displayed as 2019 April 7, or it could be displayed as 1999 August 22, or it could be displayed as 1980 January 6. Knowing which epoch you're in is actually considerably trickier than working consistently internally, since there's no information in the satellite signal to tell you this, by design. It comes from applying an external offset. Receivers have lots of ways to do this, and quite a few simply have it baked in and cannot handle a rollover, so as far as they're concerned, week 0 is in 1999, period. They'll navigate just fine but they'll display the date wrong, because they're applying the wrong offset.
  - PaulHoule 8y ago
    Funny enough my Tom Tom made some noise and seemed to boot with a different sequence than usual, in particular it insisted that I remove the USB cable that I use to power it from the car. I did that, and it found the signal in 50 seconds and worked normally, correct date and everything.
  - labster 8y ago
    Cool, see, I knew this was worth talking about. While I've learned about GPS as a way to gather meteorological data[0] via radio occultation, I never had to deal with the raw data stream myself. Thanks for your expertise. [0]: https://en.wikipedia.org/wiki/Constellation_Observing_System_for_Meteorology,_Ionosphere,_and_Climate https://en.wikipedia.org/wiki/Constellation_Observing_System...
  - joelhaasnoot 8y ago
    The easiest way of doing this is to assume time will never travel and store the latest computed date right? If you've seen 2001, it's never going to be 1999 again, so it must be 2019. I understand that will require either flash or battery backed RAM, etc to keep this stored, but seems relatively trivial in most applications... (side effect would be that if the backup battery is dead or all batteries are disconnected, you would revert to 1999)
dsfyu404ed 8y ago
Any tips for someone looking for similarly priced antiquated GPS's to shove in the glove-box of all the cars in their fleet?
- deleted 8y ago
  [deleted]
- PaulHoule 8y ago
  There were four of them here: https://ithacareuse.org/ https://ithacareuse.org/ there also was a $5 pair of Bose headphones that I was stupid to not buy the instant I saw them. The Tom Tom I have is too old to get updated maps, but it's not a problem driving around in upstate NY. In New Hampshire it might be another thing, but where I live they build new buildings but not new roads.

gwbas1c 8y ago

Dumb question: What is the rollover issue?

usermac 8y ago
I had to look it up as well. I found this link: https://www.orolia.com/resources/blog/lisa-perdue/2018/gps-2019-week-rollover-what-you-need-know https://www.orolia.com/resources/blog/lisa-perdue/2018/gps-2... From the text comes: "However, the field that contains the week number is a 10-bit binary number. This limits the range of the week number to 0 – 1023, or 1024 total weeks."
Someone1234 8y ago
Not a dumb question at all. GPS used ten bits for datetime. With a week counter and seconds into that week since 6th of January, 1980. After 1024 weeks it resets back to 0. This has happened once already on 21st of August, 1999. This is the second rollover/reset of the week counter on 6th of April, 2019. Newer versions of GPS messages use thirteen digits instead of ten digits allowing for a larger week counter (granting 8,192 weeks), causing the first rollover not to occur until the year 2137. https://en.wikipedia.org/wiki/Global_Positioning_System#Timekeeping https://en.wikipedia.org/wiki/Global_Positioning_System#Time...
- kevhito 8y ago
  Instead of adding 3 bits, maybe they should have dropped 3 bits. A rollover every ~20 years is just asking for latent buts to blow up. A rollover every ~2.5 years is just business as usual. Go big or go small. Don't design things to break just after you retire but before your coworkers do.
  - Rebelgecko 8y ago
    OTOH, with the current setup a receiver will know the correct date as long as it is turned on at least once every 1024 weeks, since it can detect the rollover. More frequent rollovers might cause issues with timing on devices that are infrequently turned on. It's sort of a moot point for most of us once there's greater adoption of the L2 and L5 CNAV signals— those use a week counter that won't roll over until most if not all of us are dead. Then it can be someone else's problem :)

petee 8y ago

I don't get how some planes were affected, but not all; Every 787 in China should be running the same software version, let alone the world, no? Are there any reasons a fleet plane might be using a different version? I'm coming up short on reasons, other than language, or maybe special hardware modifications?

j16sdiz 8y ago
They have regular software updates. Just like Windows update, nobody do that regularly
- petee 8y ago
  Haha, that's surprising though, since these are important systems, if there is an update, doesn't it mean something really needs to fixed, i.e, gps rollover? Any bug left unpatched seems unacceptable Maybe I have more faith in the 'system' than I should...
  - kijin 8y ago
    We're talking about the same manufacturer that's responsible for the 737 MAX fiasco. I'm not sure how much trust I would want to place in their software.
    - icebraining 8y ago
      You're putting trust in their software whether you upgrade it or not.
      - sslayer 8y ago
        You forget that Boeing doesn't own these planes. These aren't Tesla's. We are currently in an era were it seems to be expected that the manufacturer automatically update their software without the end user intervention, ie Tesla, Windows, Apple, etc... Thats not how it was done in the past, particularly when the bandwidth didn't exist.

js2 8y ago

Related discussion which explains the rollover issue: https://news.ycombinator.com/item?id=19593227 https://news.ycombinator.com/item?id=19593227

mschuetz 8y ago

What's the reasoning for the current GPS time format and it's resulting issues? Can't we just use a 64bit integer timestamp? That would give us roughly 580 years in nanosecond resolution?

crooked-v 8y ago
> What's the reasoning for the current GPS time format and it's resulting issues? Somebody wanted to save money when building the original satellites in the 1970s.
bmurray7jhu 8y ago
The GPS signal format was was designed in the 1970s. Any change to the signal format must be backwards compatible with legacy receivers. There have been some efforts to modernize the GPS signal format. In 2014, select GPS satellites began broadcasting an additional message (CNAV) with a 13 bit week number, allowing a ~157 epoch.
jhayward 8y ago
> What's the reasoning for the current GPS time format and it's resulting issues? GPS satellites and receivers last on the order of 20 years so it takes a long time to age out of legacy support. The latest generation of satellites, GPS-III, have a new week number format that allows for 13-bit week numbers (120 year rollover). The reason for the low number bits in week number is simple, it's broadcast on a very low bit rate (50 bits per second) and every frame has the week number in it. So you get faster fixes if you don't pad out with useless zeros.
- throw0101a 8y ago
  > The reason for the low number bits in week number is simple, it's broadcast on a very low bit rate (50 bits per second) and every frame has the week number in it. So you get faster fixes if you don't pad out with useless zeros. It should be noted that the coders could have used data windowing and pivot dates to make certain assumptions: > When ntpd(8) receives an unresolved timestamp from an upstream server, that timestamp could be based in any era prior to the current wall-clock date - but by definition we may not know the wall-clock date with confidence yet (not having achieved sync). > To resolve this ambiguity, NTP also uses an internal pivot date. It chooses the era that would minimize the absolute value of the time difference between the resolved timestamp and its internal pivot date. > This resolution technique is part of the NTP specification. It will normally resolve to the current era, but may in cases where the pivot is within a half-cycle of a rollover time resolve to either the previous or the next era. > An ntpd(8) instance’s pivot date will be the date it was compiled and built. > It is fairly easy to see that this guarantees correct resolution if the pivot dates of the communicating ntpds are within a half-cycle (88 years) of each other, even if there is a rollover between the pivot dates. * https://docs.ntpsec.org/latest/rollover.html#ntp_pivots https://docs.ntpsec.org/latest/rollover.html#ntp_pivots * https://en.wikipedia.org/wiki/Date_windowing https://en.wikipedia.org/wiki/Date_windowing

userbinator 8y ago

The 787 didn't even exist when the GPS time last rolled over, so it's quite unusual to see it have this problem --- unless Boeing reused the software from a previous version, which is not surprising given the regulations of the industry, but that would imply other older models also have the problem.

acqq 8y ago
> The 787 didn't even exist when the GPS time last rolled over, so it's quite unusual to see it have this problem It's not as unusual as you'd think: https://blog.fosketts.net/2019/04/06/gps-time-rollover-failures-keep-happening-but-theyre-almost-done/ https://blog.fosketts.net/2019/04/06/gps-time-rollover-failu... There you see a nice summary of the problem and an example a Mercedes car from, if I understood, 2008 experiencing the same type of problems. Also discussed on HN: https://news.ycombinator.com/item?id=19593227 https://news.ycombinator.com/item?id=19593227 It's like having a Y2K bug software and saying "but that software didn't exist before 1900." It doesn't matter when the "epochs" are, it's just about what the design covered or didn't. I've personally fixed the trading software which failed to correctly implement the correct rollover of the decades (i.e. the transition from year 9 to year 0 in the names), not to mention Y2K. It was simply tested before only "in the same decade." It's definitely a failure in the development process, for the software that is supposed to work in some bigger time frame, and not only shortly.
- syockit 8y ago
  I'm rather interested in what kind of software developed after Y2K still exhibiting Y2K bug, and how it affects the usage. Is it made with the assumption that it's going to record only post-Y2K dates, and somehow user manages to input pre-Y2K date?
  - avar 8y ago
    A lot of software was never fixed for the Y2K problem. People just went from e.g. entering a date of 99 to 00 as the new millennium rolled around, and dealt with the pain of any time offset calculations being incorrect and needing to be manually fixed for a few months. Some of it's doubtless still in use and people are entering "19" as the year today, and if it's still in use in 2099 we'll have the exact same problem all over again with the exact same software. That may seem crazy today, but people in the 1950s would have thought you were crazy if you said B-52s were still going to be flying in 2020. There's doubtless going to be people still running Windows 95 on some emulator in 2100 to use some legacy application that keeps chugging along. Some software we use today might be in use for thousands of years or more, unlike airframes it isn't subject to any natural decay or wear.

Thermolabile 8y ago

Can someone give me a source of it?

AnssiH 8y ago

Someone on reddit said yesterday a KLM flight (apparently B777) was grounded because of incorrect date (in some Honeywell equipment?): https://www.reddit.com/r/sysadmin/comments/babyvv/anyone_hit_by_the_gps_rollover_event_today/ekb3ev3/ https://www.reddit.com/r/sysadmin/comments/babyvv/anyone_hit...

Aromasin 8y ago
We had some Honeywell engineers at my workplace for a system integration meeting. Over lunch I got the opportunity to talk with them. This came up in conversation. Apparently it's very much all-hands-on-deck at the moment - almost every system they've currently got on-board Boeing aircraft is being assessed, and they're in all hours of the day to get new revisions through. I do not envy them right now.
- mehrdadn 8y ago
  Is it because they didn't see the GPS rollover coming up (it was in the news though)? Or did they not think their products would be affected?
  - Aromasin 8y ago
    Partly that (I think there were some legacy systems that were missed), partly that every minor clerical and drawing error that got passed through because of how insignificant it was (nothing that would affect the integrity of the aircraft I should add) has been flagged by Boeing. They want to look proactive to their customers and shareholders I suppose. That means lots of tiny, insignificant changes to hundreds of drawings. It's nothing technically challenging for the most part - just lots and lots of paperwork.