10 ms·
Gpsd bug may create a 1024 week time warp on October 23
- colejohnson66 5y agoHow do we not have a way to predict leap seconds?
- kelnos 5y agoI believe we used to, but only recently have we began to understand that the Earth's rotation is not slowing down in a linear/predictable fashion. A comment on the bug says this has something to do with global warming, but I don't really have much context here.
- btilly 5y agoThe Earth's rotation is complicated. See https://www.smithsonianmag.com/smart-news/global-warming-changing-how-fast-earth-spins-180957550/ https://www.smithsonianmag.com/smart-news/global-warming-cha... for how global warming is theoretically speeding up the Earth's rotation (melting ice causes land to rise at the poles, and moving rock turns out to be more important than water migrating from pole to equator) and https://phys.org/news/2015-12-scientists-reveal-rotation-earth-core.html https://phys.org/news/2015-12-scientists-reveal-rotation-ear... for how interactions of the core, mantle and crust are currently slowing the day down.
- crote 5y agoLeap seconds correct for the difference between the time as measured by atomic clocks and the time determined by solar observations. Turns out the rotation speed of Earth varies. Things like tides, earthquakes, and climate change can affect it. There is no formula for that, the only thing you can do is measure and issue a leap second when required.
- oceanghost 5y agoWe're floating through space and that space doesn't have a uniform shape and then there's the n-body-problem. Interestingly enough we're finding out that a lot of our solar cycles are probably due to massive things in very long orbits in our solar system.
- nonfamous 5y agoThe number of leap seconds required is determined by the Earth's rotation speed, which isn't constant. In the same way that an ice skater extending his arms slows down, shifts in mass on the Earth can alter its rotation speed. Earthquakes, icemelt, atmospheric warming, and even the filling of the Three Gorges Dam can have an effect at the scale required for GPS synchronization.
- pithon 5y agoThis isn't even really a GPS thing, it's UTC time that bounces around because it wants to stay synchronized to the day/night cycle. GPS time is continuous.
- packetslave 5y agoQuoting from that thread: > I don't think gpsd has any reason to be predicting when a future leap-second is going to occu Until last year, leap seconds had been very predicable. The effect of global warming on earth rotational speeds was only very recently seen, or even predicted. But, yes, going forward, that needs to change. > And the code in question is clearly expecting only positive leap-seconds. Yes, because until 2020, the thought of a negative leap second was unthinkable. I would welcome you testing that and seeing what falls out.
- swman 5y agoIt'd be cool if we could just invent a new standard time system that's independent from things like that which add variation or unexpected randomness. I mean sure it would be annoying but its a one time change and our generation has to endure the pain of upgrading our systems, but it would be worth it no?
- wmf 5y agoThere's not really anything to invent or upgrade. Just never add any more leap seconds (the past ones can't be safely removed). We need to lobby the IERS harder.
- sp332 5y agoWe already have one of those. https://en.m.wikipedia.org/wiki/International_Atomic_Time https://en.m.wikipedia.org/wiki/International_Atomic_Time TAI is just UTC before the leap second adjustments.
- wmf 5y agoSwitching everyone from UTC to TAI is too much work and if you try to use TAI while everyone else is on UTC you'll run into off-by-37 bugs everywhere. It's better to keep using UTC but add no more leap seconds to it.
- sp332 5y agoIf people didn't need leap seconds, they would already not be using them. There's absolutely no use case for adding leap seconds for a few decades and then stopping. Either put them in or don't.
- wmf 5y agoLeap seconds are the right thing to do but people didn't realize all the bugs and costs they would trigger. Now that we understand these costs we can and should change our minds.
- tialaramex 5y agoWe decided to have leap seconds to match UTC which otherwise needn't care and could be based on TAI, against UT, which depends on the gentle spinning of the large rock we all live on. The IERS https://www.iers.org/ https://www.iers.org/ is in charge of monitoring the spinning of the Earth. On the basis of their assessment a decision is made every six months whether to inject (or indeed remove) a leap second. If we decided not to match UTC to UT and thus we did not care precisely how quickly or slowly the Earth is spinning, we could abolish leap seconds. If you meant, "Why can't we precisely predict the motions of a vast rock floating in space years into the future" then I don't know what to tell you. We're not God?
- remexre 5y ago> If you meant, "Why can't we precisely predict the motions of a vast rock floating in space years into the future" then I don't know what to tell you. We're not God? IMO this feels like the more interesting one to explore: are leap seconds wholly from things that are within measurement error in existing rotation, or is the Earth's rotational rate actually changing? If we had leap minutes as the smallest increment, could we predict them out centuries in advance? etc
- jhayward 5y agoNo, they are not measurement error. The moment of inertia of the earth changes over time in irregular ways (e.g, melting glaciers, etc). Even a major earthquake can show up in the earth’s rotation. Leap seconds account for the accumulated difference in the rotational period from time to time.
- wumpus 5y agoWhy not just go back to the Julian calendar? You'd avoid a lot of software bugs.
- gerikson 5y agoYou jest, but I like to use Julian days (fun fact: different Julian!) to calculate day ranges. It's built in to SQLite so it's really convenient.
- foepys 5y agoIn the same vain there is no algorithm to precisely predict moon phases. A complete cycle takes proximately 27 days but not really. To find out you have to look into the sky which can also be a bit subjective.
- callesgg 5y agoI don’t understand if this is an actual bug. I have heard that the timing on gps is somehow delivered as weeks and that the bitsize of the variable keeping track of the weeks is to small. So every now and then the weeks reset and this is managed through overrides in the clients. Is this bug not just referencing that thing, the override of the week rollover?
- aidenn0 5y agoYes and no. GPS has 10 bits for the week number (so 1024 weeks) This code is using the number of leap seconds that have happened to sanity group of 1024 weeks we are in. The assumption is that by December of 2022, we would have another leap second, so if we had fewer than 19 total leap seconds, then something has gone wrong. However due to incorrect arithmetic, this sanity check is looking at October 2021. Further comments point out that the sanity check need not be in production code at all, but should be moved to test code.
- fanf2 5y agoGPS also has a field indicating when the previous or next leap second was or will be; this field is 8 bits or about 5 years. The last leap second was in 2016 and no future leap second has been announced. So GPS needs to mark a non-leap to keep the week offset in bounds. This happened before in 2003 during the previous long gap in leap seconds (1998-2005).
- dheera 5y agoWhy do people use gpsd instead of just reading $GPGLL or $GPRMC from /dev/ttyACM0 or /dev/ttyUSB0 or whatever, which always seemed far more reliable to me?
- gsich 5y agoAllows multiple applications to use a GPS source.
- dheera 5y agoOkay, then why not just take the /dev/ttyACM0 output and redirect it to a standard TCP port with like 5 lines of code? And can't multiple applications read a /dev/tty device read-only with just e.g. "cat /dev/ttyACM0"?
- BenjiWiebe 5y agoNo, multiple applications cannot read the same data from a tty device.
- rcxdude 5y agoTechnically you can (in that you won't usually get an error), but the result is not what you would want: each byte only gets delivered to one of the readers, in a somewhat unpredictable manner, either mangling the data or starving one of the readers
- AlotOfReading 5y agoThe faq answers this [1]. The issue is that GNSS vendors and standards need to clean up their act before you can do this reliably across different receivers. [1] https://gpsd.gitlab.io/gpsd/faq.html#why_not_parse_nmea https://gpsd.gitlab.io/gpsd/faq.html#why_not_parse_nmea
- dheera 5y agoI suppose. But I've had so many more problems with gpsd (especially when e.g. USB enumerates devices randomly) that on outdoor robots I've switched to parsing NMEA strings and binary data over serial directly, specifically for reliability reasons. Also I've had gpsd think some other non-GPS serial device was GPS, took up the port, and I got frustrated at its incompetency and apt-get uninstalled it.
- craigds 5y agoBeing unfamiliar with GPSD, what devices/services would this be likely to affect?
- wmf 5y agoNTP servers.
- offmycloud 5y agoAndroid phones and tablets. "In addition, the Android smartphone operating system (from version 4.0 onwards and possibly earlier; we don't know for sure when the change happened) uses GPSD to monitor the phone's on-board GPS, so every location-aware Android app is indirectly a GPSD client."
- offmycloud 5y agoSource: https://gpsd.io/ https://gpsd.io/
- axus 5y agoWe just replaced some old GPS time-servers that have a similar bug.
- toomuchtodo 5y agoWhat did you replace them with?
- benbojangles 5y agooh cool, because that's my birthday
- fisherjeff 5y agoSomewhat unrelated: Can someone explain the rationale for writing comparisons in the ordering they're using (e.g., 2180 < week)? I've seen similar before and always thought it seemed error-prone to not write them the way they'd be spoken aloud, but happy to entertain other explanations.
- gabagool 5y agoEven more unrelated, but equally pedantic, I've always thought it's weird when people write "null != val" instead of "val != null" for the same reason. When said out loud, "null is not val" just feels wrong.
- new_guy 5y agoIsn't that just Yoda notation? https://en.wikipedia.org/wiki/Yoda_conditions https://en.wikipedia.org/wiki/Yoda_conditions
- hobs 5y agoIn PowerShell you'll get a linter warning because things like (look at the -): if (@() -eq $null) { 'true' }else { 'false' } # false if (@() -ne $null) { 'true' }else { 'false' } # false
- mgdlbp 5y agoAnd there also exist expressions for the left-hand side that make both expressions truthy, like [0]: $v = $null, $null, 1 ($v -eq $null) -and ($v -ne $null) # True In both these cases, this is caused by the language having built-in binary operators for when the left-hand side expression is a collection type that perform the operation elementwise and return an array. Interestingly, it seems like PowerShell's operator overload resolution in general depends entirely on the type of the LHS. I say 'seems' because I couldn't find any sort of language specification when I looked into it a while ago like what C# and VB.NET have, and testing seemed to confirm that this was the case. Now, searching the PowerShell Core source, it seems from [1] that this is indeed the implementation. This contrasts with C# [2] and VB.NET [3], where binary operator overload resolution is treated as if the candidate operator implementations were a two-parameter method group, making the resolution process 'commutative' (though not always commutative in practice as the operators themselves can still have different LHS and RHS types {Edit: example from the CLR [4]: +(Point, Size) but not +(Size, Point)}). [0] https://blog.iisreset.me/schrodingers-argumentlist/ https://blog.iisreset.me/schrodingers-argumentlist/ [1] https://github.com/PowerShell/PowerShell/blob/master/src/System.Management.Automation/engine/interpreter/LightCompiler.cs#L805 https://github.com/PowerShell/PowerShell/blob/master/src/Sys... [2] https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/language-specification/expressions#binary-operator-overload-resolution https://docs.microsoft.com/en-us/dotnet/csharp/language-refe... [3] https://docs.microsoft.com/en-us/dotnet/visual-basic/reference/language-specification/expressions#operator-resolution https://docs.microsoft.com/en-us/dotnet/visual-basic/referen... [4] https://source.dot.net/#System.Drawing.Primitives/System/Drawing/Point.cs,89 https://source.dot.net/#System.Drawing.Primitives/System/Dra...
- mithusingh32 5y agoThis brings back so many nightmares at my old job. These kind of things would creep up on us all that time and we'd spend a week or so scratching our heads in how the hell did our systems would travel time.
- Someone 5y agoFrom the gpsd homepage (https://gpsd.gitlab.io/gpsd/index.html https://gpsd.gitlab.io/gpsd/index.html): “GPSD is everywhere in mobile embedded systems. It underlies the map service on Android phones. It's ubiquitous in drones, robot submarines, and driverless cars. It's increasingly common in recent generations of manned aircraft, marine navigation systems, and military vehicles.” FTA: “Will it be cherry-picked back to the 3.20, 3.21, and 3.22 branches? gpsd does not have enough volunteers to maintain "branches". Some distros try to cherry pick, but usually make things worse. This bug was announced on gpsd-dev and gpsd-users email lists. So the packagers for several distros already saw it. What they do is what they do.” So, it seems gpsd is like the tz database, a few volunteers maintaining an essential part of our software infrastructure.
- TeMPOraL 5y ago> So, it seems gpsd is like the tz database, a few volunteers maintaining an essential part of our software infrastructure. More than that. Software is now running the economy, controlling safety and security of the physical world. gpsd, like Tz database, cURL, SQLite and the Linux Kernel, should be seen as critical planetary infrastructure, period. Safety of our economy and our physical well-being increasingly depends on them being operational. And yes, it's worrying that we ended up in a situation, where the building blocks of our technological society are maintained by underpaid volunteers.
- gsich 5y ago>FTA: “Will it be cherry-picked back to the 3.20, 3.21, and 3.22 branches? Just update.
- darylteo 5y agoAs a business, on a scale of 1-10 how worried should I be?
- nineteen999 5y agoDepends, how much does your business depend on GPS time?
- darylteo 5y agoWell, some other comments indicate that this might affect NTP servers? Would that indicate some kind of follow on effect on database timestamps? Timezone localisation/conversions? Replication setups?
- nineteen999 5y agoMost people using NTP servers do not use GPS units directly, but sync them to public NTP servers - for example https://www.ntppool.org/en/ https://www.ntppool.org/en/. Redhat for example ships these as the default upstream NTP servers. if the ntppool.org servers use GPS and gpsd, they are likely to have patched the issue well in advance. I work in a business where we do use serial and network connected GPS devices as stratum 0 timesources for NTP, and yes we have concerns about the implications of this bug on some of our remote devices. If the gpsd starts sending incorrect time/date to the local ntpd it will probably be marked as a false ticker. We have multiple GPS based NTP servers in our datacenters as fallbacks, however we will probably need to check for a firmware update from them for this issue from the vendor.
- darylteo 5y agoGreat, thanks! I'll continue to keep an eye on the situation at least. :)
- gsich 5y agoIt most likely does not. You can get accurate (down to ~10ms) time from other NTP sources. What you want from a GPS based NTP server is the PPS output, which is accurate to a few ns.
- AnonHP 5y agoI saw in another comment here by offmycloud [1] that this affects: > Android phones and tablets. "In addition, the Android smartphone operating system (from version 4.0 onwards and possibly earlier; we don't know for sure when the change happened) uses GPSD to monitor the phone's on-board GPS, so every location-aware Android app is indirectly a GPSD client." Can someone explain how the patch for this will reach all Android devices (especially the large number of devices running older versions of the OS and not getting any updates at all)? What exactly are the consequences for these users? [1]: https://news.ycombinator.com/item?id=28045191 https://news.ycombinator.com/item?id=28045191
- vimda 5y agoIt seems like the bug was introduced in 2019, so it's presumably phones released since then. Still a problem, but not "all Android phones" bad.
- fomine3 5y agoIt is surprising that this bug found in just three month ago from the day.
- beatthatflight 5y agoNote the heading is an error. In the comments of the bug : "Ooops, 16 Oct, 2021, was supposed to b 31 Dec 2022. My calendar error. That needs to be fixed."
- terom 5y agoMy understanding of the comments is that the week 2180 code was supposed to be 2022-12-31, but the code is actually 2021-10-24 (end of week 2180). In other words, the heading is correct.
- beatthatflight 5y agohah, yes, came back to correct my statement. Thanks!
- sydthrowaway 5y agoIs global warming really affecting the Earth’s rotation?
- datenwolf 5y agoYes. It's the classic ice skater effect. Climate gets warmer => Ice (on mountains) melts. Molten ice = water flows down into the ocean, moment of inertia of the Earth is reduced => Earth's rotation speeds up. Just to put things into perspective: The elevation of the land surface at the Earth's south pole is over 2800m above sea level. The highest point in greenland is over 3600m above sea level.
- dolmen 5y agoGlobal warming is breaking our software. From https://gitlab.com/gpsd/gpsd/-/issues/144#note_633612324 https://gitlab.com/gpsd/gpsd/-/issues/144#note_633612324 > Until last year, leap seconds had been very predicable. The effect of global warming on earth rotational speeds was only very recently seen, or even predicted. But, yes, going forward, that needs to change.
- riobard 5y agoAnd _negative_ leap seconds! My mind is blown.