9 ms·
Ntpd won't save you from one particular rogue bit
- simcop2387 9y agoI wonder if any of the other ntp implementations (chrony et al) suffer from this same issue?
- jlgaddis 9y agoYeah, it's an issue with NTP the "protocol" not NTP the "program".
- grogers 9y agoTo expand on this, NTP uses timestamps with 32 bit seconds (plus fractions of a second), so if you manually step your clock some multiple of ~136 years, the protocol inputs will be the exact same so you'd have no way of knowing you were off from the server. However, you could imagine an NTP implementation which hardcodes the approximate starting time to get the right era. You'd only have to recompile every lifetime or so to keep it up to date.
- jlgaddis 9y ago> However, you could imagine an NTP implementation which hardcodes the approximate starting time to get the right era. You'd only have to recompile every lifetime or so to keep it up to date. Yep, see "NTP pivot dates" [0]: > When ntpd(8) receives a unresolved timestamp from an upstream server that timestamp could be based in any era ... To resolve this ambiguity, NTP also uses an internal pivot date ... An ntpd(8) instance’s pivot date will be the date it was compiled and built. [0]: https://docs.ntpsec.org/latest/rollover.html#ntp_pivots https://docs.ntpsec.org/latest/rollover.html#ntp_pivots
- jlgaddis 9y agoIt won't save you from this particular case but you're safe so long as your clock is off by no more than ~68 years.
- cgsmith 9y agoThe other use case is manufactured hardware with their time set for the unix epoch and depend on it for updating.
- whouweling 9y agoInteresting edge case! This is why it may be sensible to do some extra checks in things like product expiration batch jobs, to check if the previous run was not to far in the future or past and refuse to run in that case. Scary to think what might happen if some database purge process is running after this bit flip!
- viraptor 9y agoThis is going to give me actual nightmares. Not only database cleanups, but there are quite a few automated backup cycling scripts which would happily throw out terabytes of data this way...
- nolok 9y agoAs parent said, it's usually a very good idea to check for time delay and ensure it "makes sense" before doing anything destructive. Doesn't cost much, can save a lot, and adding a command switch to bypass it allows to skip the "haven't booted in a couple week and now everything refuses to run" problem. A much more common case of when such a check help is "last time run is in the future", that you face every time there is a clock issue. Some scripts... Don't react too well about that.
- deleted 9y ago[deleted]
- poizan42 9y ago> If the resulting time that ntpd sees is more than a few milliseconds off, it'll step the clock, and that will clear out the future time. It shouldn't do that, otherwise it's going to break in 2036. (The "future time" it's going to clear off will set the time back to 1900 once we are past February 7, 2036) Edit: I don't know if I didn't explain myself well enough. Because the protocol only has 32 bits of seconds it cannot tell apart September 27, 2017 and November 4, 2153. This means that ntpd absolutely must trust that it is in the correct 68-year span. But according to the author ntpd "clears out the future time" if it is more than a few milliseconds off. This violates the spec and is also inconsistent as it happily keeps up with the future date as long as it doesn't have to step the clock.
- jlgaddis 9y agoIt won't, this is accounted for in the spec; cf. "pivot dates" (IIRC).
- poizan42 9y agoI know that it shouldn't, but the author said that it did in the blog post.
- nolok 9y agoI often see this type of comment on many subject and I find it super weird in a way. I mean, if it took you less than a minute after reading it to spot the issue despite not being familiar with the details before, it's usually safe to assume whoever made the specs and spent days on it also did.
- petecooper 9y agoI get numerous requests to my (macOS) computer for ntpd to connect to shady subnets when I'm connected to a particular commercial VPN: https://twitter.com/petecooper/status/911946604759977984 https://twitter.com/petecooper/status/911946604759977984 https://pbs.twimg.com/media/DKficrvW4AA1Hxm.jpg:large https://pbs.twimg.com/media/DKficrvW4AA1Hxm.jpg:large Numerous hosts across numerous networks, perhaps two or three an hour. I've wondered what exactly would be gained by resetting a clock to a different time – this is a useful article.
- gog 9y agoWhich VPN provider are you talking about? Are you sure that it is not just NTP servers being served via DHCP once the connection is established and your computer trying to use those provided by your VPN?
- petecooper 9y agoIt's PureVPN. I'm not 100% sure about any of the ntpd connection requests, they're not predictable in their appearance. Some sessions are very quiet (zero requests), others I get a bunch of incoming connection requests for smbd, and other odd things. I really should start taking notes rather than just deny the connections.
- KekDemaga 9y agoLeakage from other clients perhaps?
- petecooper 9y agoThat's my gut feeling, yes.
- gog 9y agoIt looks like your VPN provider does not isolate traffic between clients connected to the same server. That is pretty bad, security vise.
- userbinator 9y agoThis means you can't actually persist the bad time to your RTC. I wonder where that limit comes from, since the RTC in a standard PC stores both the year(00-99) and century(also 00-99) in BCD, so a date in 2153 should be representable. This reminds me, one interesting thing I've noticed over the years is that early PC's RTCs were pretty accurate, but the most recent ones, as in the past few years, are horrible (as much as +/- several seconds per day), and the ones in smartphones even worse. Maybe because they're assuming NTP, so have cut costs by using less accurate crystals? I've had systems in storage for many years, and the RTC was within a few seconds of the current time when turned on again.
- lucaspiller 9y agoI have a heating thermostat that has optional ‘smart’ features that allow you to control it via your phone. I tried it when I first got it, but it seemed a bit pointless, so I turned off the feature and the wifi connection. That was a month ago, and the onboard clock has now drifted by around 15 minutes...
- dspillett 9y agoI can't say I've ever had a PC with a particularly accurate RTC, though in recent years I've run NTP on everything I can so not noticed any effects. Anecdotally clocks in small mobile devices like phones do seem less accurate though, at least when they don't have access to the network to update themselves. I've not looked into it but assumed temperature variances, which a device like a phone will experience to a larger extent, were part of the problem. Of course phones syncing with the network relies on their clocks agreeing with the rest of the world - for some months my provider seemed to be a minute or so behind which meant I had to adjust a little when gauging if I was just going to make train I was rushing towards or if I'd likely missed it leaving already...
- jofer 9y agoI've noticed the same thing, but I'm always surprised by it. Particularly with smartphones. A GPS is basically just a very, very precise clock and a radio receiver. Why don't smartphones use the GPS clock? I'm sure there's a good reason (and some probably do), but it sill surprises me. (Presumably it's battery related -- the GPS clock can't be used unless GPS is fully enabled or something similar.)
- tonyg 9y agoWe will continue to suffer problems like this so long as we continue to use languages which offer machine words in place of general integers.
- wruza 9y ago>Time is hard. I learned that once tried to implement "universal" datetime library for fun and education. There is wall clock, which leaps for political decisions; universal time coordinated which leaps on schedule to ease astronomical differences; atomic time of few sorts; numerous computer clocks; time zones which can go back and forth for north and south, land and sea; non-gregorian calendars and the fact that there is no 0 AD and no 0 BC; that dates were offset for many days few times in gregorian history; special and general relativity errors; and of course integer overflow issues. I may have missed few points, but still, time is hard.
- moviuro 9y agoSee this[0] article which does a good tour of numerous issues with time. [0]http://FalsehoodsAboutTime.com http://FalsehoodsAboutTime.com
- Dylan16807 9y ago> universal time coordinated which leaps on schedule to ease astronomical differences "Schedule" is a bit generous there. Oh and there's TAI for just counting seconds but you're discouraged from using it that way because of some slightly-suspicous reasoning about retrospective calibration.
- deepsun 9y agoThat's interesting, could you elaborate, or give a hyperlink on why TAI is discouraged?
- scottlamb 9y agoI think it boils down to compatibility - POSIX and SUS specify that CLOCK_REALTIME is UTC, and platform libc time conversion routines expect this. So everyone uses UTC. On Linux, there's now a CLOCK_TAI, but I'm not sure how usable it is. Does it have the correct offset on boot? when ntpd starts? and you have to convert back and forth to interact with the rest of the world, and there's no call to get the time and the conversion factor atomically, which is what you'd probably want if you're using the same timestamp internally and externally. (I think there's some non-privileged call to get the conversion; you could call that, get the time, and then call it again; retry if it's changed. but that's annoying.) And of course this is non-portable. edit: additionally, if you have a stored, bare TAI value, what can you do with it other than see how many seconds older it is than the current time? nothing on a standard system stores when previous leap seconds happened iirc so you can't convert to UTC (and thus can't convert to civil time) unless you roll your own table (and conversion routines) or always store the offset with every TAI (increasing storage overhead).
- mnarayan01 9y agoUnless you're running it with additional configuration (e.g. the -g option), I don't think ntpd will save you from any bit flips other than the 10 least significant: You'll be outside the 1000s panic threshold.
- Fnoord 9y agoIts not recommended to use ntpd with -g argument in production. An attacker can MITM NTP protocol. The 1000s threshold severely limits this attack. The attack can be used e.g. in defeating TOTP. I'm not sure if this rogue bit can be used to attack TOTP. Can anyone clarify?
- gerdesj 9y agoWell that was fun! I tried out the timeshift program and my PC duly went a bit mad. ntpq -p showed all being well. Chromium threw a fit because news.ycombinator.com's SSL cert had expired and offered to reset the clock which it could not do, given that I'm not in the habit of running apps as root. My Kerberos tickets all expired so Evolution lost contact and so did quite a few other things. MariaDB dumped core. systemd's timers went berserk so things like my LetsEncrypt cert tried renew itself. I'm still hunting through some odd looking log files but overall things seem to be back to normal when I reran timeshift and returned to current time.
- winkywooster 9y agoSimilarly with my Mac, within moments it became unusable. Even trying to reboot from the command line and I got a message that it was waiting to acquire a lock on /. Hard reset and everything is back to normal.
- netsharc 9y agoiOS devices had a bug that hard locks it if you set the time to 1/1/1970 (on US timezones, that sets it to negative territory), I can't believe they don't do any tests for their OSes to survive this...
- ReverseCold 9y agoIs there a problem with locking current time to the release date of the product?
- ifoundthetao 9y agoHm, I wasn't able to replicate it. I'm using a Kali Linux VM. I'll try some other OSes and see if it works on there. Was anyone else successful with the PoC code?
- sp332 9y agoIs your VM host resetting the time in the guest?
- ifoundthetao 9y agoThat's possible. I'll try it on a physical system in a bit. See what I can do.
- gerdesj 9y agoIt works flawlessly on a physical system. There was rather a lot of red text in journalctl -r 8)
- wyldfire 9y agoOne source of RTC weirdness is the fact that inb/outb to/from the RTC and other legacy parts of the ISA aren't protected by mutex on linux. So if you're unlucky enough to collide with some other program doing something innocuous with the RTC you can accidentally set bits that will never be set by the RTC. Many RTCs decide not to tick anymore when that happens. If all your programs use /dev/rtc and ioctl()s you are probably a little safer because there will be coarse locks around the RTC itself, those will serialize the activity. But IIRC the inb/outb stuff can be done from user space (as superuser) and even if you're only reading you have to write to the address register which could break a write-in-progress by sending its output to the wrong RTC field.
- exikyut 9y ago> Many RTCs decide not to tick anymore when that happens. Of course I laughed when I read that, but then I realized I was interpreting that line to mean "it declares shenanigans and stops reporting time so you realize something broke." Just wanted to clarify - do you mean the above, or "of course software cannot kill hardware!!1" "won't tick anymore"?
- wyldfire 9y agoYeah, IIRC it really stopped ticking (until you write a valid value at which point it would resume). Note that it doesn't stop reporting the time in this condition it will just give you back all the garbage fields that you wrote before. So an interesting thing happens when the system tries to transform that into a UTC wall clock basis and it usually ends up with a really wild interpretation of the date (decades/centuries off, similar to the problem described in TFA).
- exikyut 9y ago> Yeah, IIRC it really stopped ticking (until you write a valid value at which point it would resume). Ah, okay then. Good to know I can't accidentally thousands of dedicated servers :P (ie, via NTP MITM, writing to hw RTC...) > Note that it doesn't stop reporting the time in this condition it will just give you back all the garbage fields that you wrote before. I don't know why I didn't remember this last night: Linux uses the RTC as a poor-man's NVRAM that will persist across a reboot. Provides <24 bits of data to work with (yay! ...not). https://wiki.ubuntu.com/DebuggingKernelSuspend https://wiki.ubuntu.com/DebuggingKernelSuspend, useful info in https://github.com/torvalds/linux/blob/e34bac726d27056081d0250c0e173e4b155aa340/drivers/base/power/trace.c https://github.com/torvalds/linux/blob/e34bac726d27056081d02... > So an interesting thing happens when the system tries to transform that into a UTC wall clock basis and it usually ends up with a really wild interpretation of the date (decades/centuries off, similar to the problem described in TFA). Right.
- gumby 9y agoThis is a nice bug! And not really a clock issue -- many programs/protocols could have such a bug.