15 ms·
Windows feature that resets system clocks based on random data is wreaking havoc
- binkHN 3y agoThe article doesn’t talk about a fix—scary. I know OpenBSD uses a similar SSL-based method for time keeping on startup, but I’ve never experienced an issue like this.
- tedunangst 3y agoOpenbsd does an https (with expiry checks disabled) GET to google.com (or another server) and uses the Date header in the response.
- JohnFen 3y agoIf it's assuming that it can make an internet connection, why does it do that rather than NTP?
- toast0 3y agoNTP is trivially MITM. Secure NTP is basically not deployed anywhere. Someone with a google.com cert that's valid other than expiration is probably google and probably has a working clock.
- anthk 3y agoYou can always use a gpsd(4) compatible USB dongle as the time source.
- JohnFen 3y agoOr run their own NTP server. That's what I do in my home network.
- toast0 3y agoSure, if you have such a dongle, and it has sufficient view of the sky. Adding a 'free' https based sanity check on top of NTP seems like a good balance of cost vs reward. gps (and friends) is a nice way to get access to a very accurate timesource, but it's not always worth the cost.
- labawi 3y agoGPS doesn't send full time. It wraps every 1024 weeks, or about 20 years, hence you need a (very) rough basis in case the system is over 20 years old.
- somat 3y agoIt is a sanity check to protect against malicious ntp actors. http://man.openbsd.org/ntpd.conf#CONSTRAINTS http://man.openbsd.org/ntpd.conf#CONSTRAINTS
- simendsjo 3y agoIt's easy to turn off the feature, and its described in their original blog post. The article doesn't mention it directly though.
- newman314 3y agoThe fix is to disable. Here's the PowerShell cmd to do so: Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\W32Time\Config" -Name "UtilizeSslTimeData" -Type DWord -Value 0
- toast0 3y agoTL:DR, the w32time service will sometimes try to bootstrap the clock with data from TLS handshakes. TLS 1.0-1.2 use a random value in Client Hello/Server Hello which was originaly specified as a uint32 gmt_unix_time plus 28 random bytes. It's better [1] to fill the whole structure with 32 bytes of random data (and TLS 1.3 specifies it without reference to time). If you interpret the random data as a unix time, sometimes you're going to get weird results. It's not clear to me what servers w32time probes to get these values, either. I've seen other services that use https to bootstrap time in case other clocks are unavailable or suspect (or use a limited expiration certificate to authenticate!), it's a bit difficult because you have to ignore or postpone checking certificate expiration when validating the x.509 certificates, parse the http date header, and then presumably check that the date provided is within the time the certificates involved are valid. [1] https://datatracker.ietf.org/doc/html/draft-mathewson-no-gmtunixtime-00#section-2 https://datatracker.ietf.org/doc/html/draft-mathewson-no-gmt...
- simendsjo 3y agoI wrote more details in a ServerFault answer before contacting Ars Technica: https://serverfault.com/a/1132383/45588 https://serverfault.com/a/1132383/45588
- nicolaslem 3y agoI sometimes experience a similar issue with my linux laptop where time jumps to the year 2077 when waking up from sleep. My guess is that it is a hardware glitch as it doesn't happen often, but when it does it is quite impactful. One of the annoying consequences is that some parts of the system decide to clean up "old" data. Surely data that has been stale for 50 years can be deleted, right? I cannot imagine the impact of something similar happening on a production server.
- lalo2302 3y agoI had that with old mac os versions. Fun to finally know why it happened
- matja 3y agoCoincidence? $ TZ=UTC date --date=@$[$(date +%s)*2] Wed 31 Mar 15:33:50 UTC 2077
- jstanley 3y agoNice spot. Perhaps something that is meant to add a delta to a timestamp accidentally adds 2 timestamps together.
- almostnormal 3y agoHow is the hardware clock connected? It it is something serial it could be an offset by 1 bit.
- rep_lodsb 3y agoIf it's an x86 laptop, it will be compatible with the MC146818 chip found in the original IBM AT, and return the time in BCD format as yyyy/mm/dd hh/mm/ss.
- tenebrisalietum 3y agoOn modern systems the simulated MC146818 is part of the chipset.
- drcongo 3y agoBlows my mind that anyone would use Windows on a server.
- netmare 3y agoEven worse, some use Linux as their desktop OS. Imagine that...
- kibwen 3y agoIndeed, and then consider that some poor souls even use Windows or Mac on the desktop.
- lamp987 3y agoIt just works in 2023.
- justapassenger 3y agoIt does. As long as you carefully select HW.
- blibble 3y agolet me present you the Windows 11 supported CPU list: https://learn.microsoft.com/en-us/windows-hardware/design/minimum/supported/windows-11-supported-intel-processors https://learn.microsoft.com/en-us/windows-hardware/design/mi... anything earlier than 8th gen core? fuck you
- lamp987 3y agoI said 2023. Not 2013.
- justapassenger 3y agoAnd? Off the top - most powerful and energy efficient laptop for last few years (M* macbooks) have either no or very little support. I’ve been running Linux since 1990s. It’s best it ever was with the HW support, but saying it’s not an issue is just a straight up lie.
- yetanotherloss 3y agoI can't imagine the sequence of horrible decisions that led to doing this. Like, why would the time service ever want to depend on all this insanity when if it has a network and everything else is bizarro world, just like scrape the time and date text from weather.gov. Or just accept that absent NTP, w32time maybe just shouldn't try to set the clock to whatever a circus clown tells it? This sort of reminds me how there is a buried registry setting to tell w32time to not place the firmware clock at very high stratum in the time sources list which also is just a giant WTF in how those two behaviors were decided on.
- Varriount 3y agoThis is what puzzles me - while time drift does occur, it tends to be in the form of minor errors which build up over time; large jumps are relatively rare. You'd think the mechanism described in the article would have a failsafe ensuring that only minor time recalibrations are performed.
- toast0 3y agoIt kind of depends. There's a lot of poor clocks at boot up time, but a continuously running host usually doesn't get too off, too quickly; I've seen some things, but even at 10% fast/slow, you don't have to jump days unless you're checking infrequently. On the other hand, with virtualization, who knows how long it's really been between clock ticks. Let's say someone suspends a VM for a couple months and then unsuspends it; hopefully the clock is synchronized, but maybe it isn't, and you do need to jump ahead. I thought I saw in the article that there are failsafes, but the allowed range is still pretty broad. But reporting is going to tend to be dominated by larger values, because smalller jumps are less likely to be noticed or more likely to be considered an accidental change.
- Wowfunhappy 3y agoEvery VM software I've ever used automatically syncs the guest OS's clock to the host OS's clock. I've never used VMs in a server context, but do servers not do this too? It seems like the logical solution here.
- przemeq 3y agoWow, we encountered a similar problem on one of our servers. We couldn't determine the cause. We set up an alarm to notify us when such a change occurs, and we've developed a procedure to handle the consequences. Great discovery!
- stefan_ 3y agoFunny, they made a bad copy of tlsdate and sold it as a super new innovation.
- jmuguy 3y agoWindows Time bullshit was one of the most annoying things I dealt with during my years as an IT guy. Registering and unregistering w32time, trying different NTP servers. Trying to figure out why domain systems werent getting their time from the DC. It always felt so... stupid. Surely having the correct time on a device isnt that complicated. Turns out, its not, unless you're on Windows. Somewhat ironic that these days the only Windows system I have to deal with is my gaming PC. It refuses to sync with time.windows.com.
- yetanotherloss 3y agoIt's probably still the case that Windows will consider the firmware clock more reliable than anything but a stratum 1 (direct gps/glonas/atomic) or maybe 2 time source which can result in truly bizarre behavior if the firmwar/hypervisor drifts too far from dozens of stratum 2 or 3 active directory servers. It logs no messages about this logic fork. I don't recall the exact conditions that determine this but troubleshooting it is an exercise in thinking you're having a stroke.
- deleted 3y ago[deleted]
- mschuster91 3y agoWindows isn't the only thing with issues. Both ntpd and systemd-timesyncd can be very annoying if upstream servers send a bad stratum value, and the maintainence team of upstream is a bunch of morons taking weeks to fix it. You can't say "forcibly use IP w.x.y.z no matter what" to either of them.
- Symbiote 3y agoI had this recently when one of the time servers I use was accidentally reset (somehow) and ended up on the previous GPS era, i.e. 20 years wrong. /etc/systemd/timesyncd.conf NTP=w.x.y.z Or /etc/ntp.conf server w.x.y.z iburst with unaffected servers fixed it.
- 3y ago
- mlichvar 3y agoA better solution to secure bootstrapping of time would be NTP+NTS (RFC 8915) using self-signed certificates with unlimited time validity, which can be preloaded with the OS and updated via normal OS updates if the server key is compromised. They would probably need to run their own servers. There are some public NTS servers (e.g. Cloudflare and Netnod), but I have not seen any using long-term certificates specifically for this use case.
- zokier 3y agoWhile this Windows feature does sound quite bonkers as described, it is also baffling to me that the timekeeping on computers is such a mess; would it really be that difficult to have my multithousand dollar computer keep time at least as well as a dollar-store quartz watch? Have time already set in factory, and be correct to within few hours at least; enough to do networking and more accurate time syncing safely.
- HPsquared 3y agoComputers have a lot more temperature variations than a typical quartz watch. In the absence of temperature correction, they'll be less accurate.
- RetroTechie 3y agoEnter the temperature compensated crystal oscillator (TXCO). A few $ at most should get you one. This shouldn't be overkill for a server dedicated to timekeeping on a large company network, right? In a datacenter one could even go for a matchbox sized atomic clock. Last I checked (years ago) such a miracle device could be had for ~$1500. Then: how often does atomic clock (or even TXCO) fail, in practice? My guess: only about as often as even the backup power fails. Note that either should provide accurate timekeeping without even going outside a physical location. That is: without using 'random' NTP server on the internet, GPS or whatever.
- deleted 3y ago[deleted]
- SAI_Peregrinus 3y agoI've got a GPS-disciplined Oven Controlled Crystal Oscillator (OCXO) hooked up to a raspberry Pi running Chrony on my home network. It provides a pretty good time source (and more importantly for me a pretty good 10MHz reference). There are rack-mount 1U NTP servers built for the purpose for about what you estimate. Most atomic clocks will have 20+ year lifetimes. OCXOs age out a good bit faster, but they're rather cheap (<$200 for ones good enough to provide holdover for a GPS module). TCXOs don't age as quickly (they're not literally in an oven), I'm not sure how long they can be expected to last.
- xg15 3y agoWhy not use all that STS cleverness to get a "rough" estimate of the current time, then use that estimate to securely connect to an NTP server and get the actual time?
- zoky 3y agoWhy even bother with all of the STS cleverness? The article suggests a perfectly good way to get a rough but trusted time stamp by… scraping it from a known good HTTP server. Anyway, STS as described is an obviously broken protocol. It’s not clear to me that it is even capable of reliably getting a rough time.
- xg15 3y agoMy understanding was that they want to pull the time over a secure (TLS) connection only - but to establish a secure connection, they already have to have the time in advance. That kind of catch-22 would always appear, no matter whether the thing you want to connect to is an NTP or an HTTP server. So far, it all makes sense. It stops making sense when their attempt at breaking the catch-22 seems to do so by in fact ditching the "secure connection" requirement, just doing so in an obfuscated manner. I agree, if you already violate your own rule, you could have saved everyone a lot of hassle and just pull the time from a plaintext HTTP server. But then you might just as well ask a plaintext NTP server and go back to where everything started...
- zoky 3y agoI think the idea is that NTP is only supposed to work if your clock is close enough to correct, in order to prevent wackiness from ensuing due to sudden extreme jumps in the system time, presumably from a misconfiguration, compromised server, or whatever. I suppose you could just rely on an insecure NTP and force the date correction, but given that the default is not to do that and you have to force it, there’s probably some reason why you shouldn’t. So, if your goal is to get the accurate time over a secure connection, getting a “close enough” time over an insecure connection and confirming/adjusting the time over secure NTP doesn’t actually seem like the craziest way to do it. I don’t really know why bootstrapping over HTTP then adjusting via secure NTP is better than bootstrapping via plaintext NTP then confirming with SSL enabled, but I guess it at least means two servers and protocols would have to be compromised.
- JohnFen 3y agoThat feature is simply insane. How did MS think this was a good idea? To address the issue they were trying to address (what happens when a mission critical server's RTC malfunctions or the battery dies?), Windows should treat it no differently than any other hardware malfunction in a mission-critical server: raise an alarm so a system operator can take a look and address the issue.
- pmontra 3y agoFrom the article > “The false assumption is that most SSL implementations return the server time,” Simen said. “This was probably true in a Microsoft-only ecosystem back when they implemented it, but at that time [when STS was introduced], OpenSSL was already sending random data instead.” And they link the reason for sending random data at https://mailarchive.ietf.org/arch/msg/tls/_clS-TIIlZUcid_2S4WPej9iMWk/ https://mailarchive.ietf.org/arch/msg/tls/_clS-TIIlZUcid_2S4... The post starts with > Here's something I wrote about removing a fingerprinting opportunity from the current TLS protocol. So the point is not that it was a bad idea, it is that MS didn't keep up with the world and didn't respond to requests from their customers to fix the problem.
- JohnFen 3y agoYes, I read that. I think that if you're making mission-critical systems rely on nonguaranteed behavior from systems you don't control, then it's a Bad Idea.
- biogene 3y agoThe point is that it was guaranteed behavior before the spec was changed. As an aside, I do find it amusing that the memo was co-authored by a Googler. Google has the best fingerprinting & tracking tech baked into their core products. I guess they don't want competition! :) https://www.ietf.org/archive/id/draft-mathewson-no-gmtunixtime-00.txt https://www.ietf.org/archive/id/draft-mathewson-no-gmtunixti...
- 3y ago
- nneonneo 3y agoI wonder if this feature might be “infectious”: if a handful of servers wind up with the wrong time, and start propagating that wrong time in their server handshakes, it might cause a cascade of other servers to reset their times too. In any case, it seems like a poor solution especially given that random timestamps can occasionally look valid (they’re random!). The problem they’re trying to solve seems to be to obtain a trustworthy timestamp in the presence of a potentially hostile network - one that could be presenting expired, cracked certificates for every connection to arbitrarily tamper with all SSL connections. It’s not a trivial problem, for sure, but resetting perfectly valid clocks using this algorithm seems like a strict downgrade….
- cratermoon 3y agoAt the very least the system should do what NTP does: refuse to change the system time if the delta is too large.
- simendsjo 3y agoFunny thing is that it does refuse to change the time when the delta is large. So it trusts these random bytes in the TLS handshake blindly, but when it gets the correct time from the time server some seconds later, it doesn't dare to change the time as the delta is too large!
- smallstepforman 3y agoI’ve used Linux on an embedded system which gets time/date via proprietary monitoring protocol. Seting Linux time/date isn’t instant, Linux will run an accelerated clock until it reaches the target time. This bizzare behaviour is a hack to fix apps/services which have too high delta time which cause other problems. Its frustrating for us since when we receive a clock update, our clocks can take hours to sync. Meanwhile our packet clocks are inaccurate. This only happens during testing (by gov regulators) and to pass the tests, we had to outsmart the way the system clock works when receiving a large date/time diff. All of these work-arounds for unneeded hacks (like in the article) drives my temper way up. Just do the simple thing in kernel space or the OS level. No suprises.
- Johnny555 3y agoNTP will step the clock instead of slewing it if the offset exceeds a threshold, your proprietary algorithm should do the same if the time to adjust the clock is too long. I don’t think it’s unreasonable for apps to assume that time is continuous and always moving forward, so slewing seems like a good solution to the problem of time synchronization, if you need instant synchronization you can always step the clock yourself. Under ordinary conditions, the clock discipline gradually slews the clock to the correct time, so that the time is effectively continuous and never stepped forward or backward. If, due to extreme network congestion, an offset spike exceeds the step threshold, by default 128 ms, the spike is discarded. However, if offset spikes greater than the step threshold persist for an interval more than the stepout threshold, by default 300 s, the system clock is stepped to the correct time.
- gorkish 3y agoGod knows how windows machines keep time these days, except to say that they basically don't. WSL2 has not ever been able to keep time consistent between the host and VM. You can find the issue threads on gitlab going back years at this point.
- buildbot 3y agoUgh I hate this issue. Literally just ran: sudo ntpdate time.windows.com Because my WSL VM drifted apparently 29000 seconds into the past for...reasons... causing the az cli login to fail. With an ugly traceback, of course.
- londons_explore 3y agoI wonder if you could use this 'feature' to exploit a system? Set up a bunch of servers all over the internet with innocuous web pages. Get all of them to include in their SSL headers the exact identical timestamp of July 5th 1998. Then get the user to connect to all those domains (eg. with a page with a bunch of iframes). The Secure Time service will see that lots of remote servers all agree with high confidence that the date is July 5th 1998. So the system clock gets set to then. Then you use a leaked yet expired cert (or maybe one signed with a broken algorithm) to impersonate an update server or steal valuable cookies/tokens. Expired certs typically fall out the back of revocation systems too - so it really is just the expiry date/time that protects against their use.
- drvdevd 3y agoI was wondering what the newly exposed vector of exploitation was here and I think you nailed it.
- jaegrqualm 3y agoTFA seems to only mention errors where the time is set to the future, which would probably indicate that Microsoft at least thought of this. Their responses seem to indicate that they also don't think that it's a security issue, which means they likely don't know of any _explicit_ way to exploit this. It seems to me that it could still be used to bring down a windows server right around the time that you wanted to, which is still a potentially serious security concern.
- gerdesj 3y agoAt the very least you can can screw with Kerberos which requires a default of something like five mins time sync. That's a denial of service. Keep it up for long enough and the device will fall off AD as well.
- _dain_ 3y agoBitcoin? Heartbleed? SSL? Wow anon, you must have hit your head hard. C'mon, we're gonna be late for the Windows 98 launch party!
- hospitalJail 3y agoThe point that Windows became less reliable than Linux Desktop happened last month. Windows 7 is still more reliable, but with the updates being over, people are forced to upgrade. But Windows 11 is less reliable than popular Linux Desktops. There might be an exception if you have NVIDIA, but I'd say less than a few percent of people have a video card.
- newman314 3y agoPowerShell invocation to disable this: Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\W32Time\Config" -Name "UtilizeSslTimeData" -Type DWord -Value 0 Reload using: W32tm.exe /config /update
- simendsjo 3y agoRemember to tell w32tm to reload the configuration too.
- HankB99 3y agoI always thought that it was a bad decision that Windows tracked time as local time. IMO they got it wrong right from the get-go. It made more sense to me to track time in an unambiguous manner(*) like seconds since midnight January 1. 1970 GMT and then convert to whatever representation is convenient locally. (*) I suppose this does not work as well on Mars. I suspect that the time on the various extra-planetary vehicles is tied to earth time but would not work so well if Mars were ever colonized.
- Terr_ 3y agoHmm, so essentially it's sampling times from different servers it thinks are unlikely to be conspiring, and those times are not as reliable as expected. Since it's fun to imagine (and then find flaws in proposals) here's an idea for y'all: 1. If the device reaches out to a dozen servers at different usually-trustable domains (e.g. microsoft.com, nist.gov) and they _all_ have "expired" certificates, that becomes a sign that My Local Time Is Unreliable. (Or that the machine is being booted up by an attacker who has connected it to a fake micro-internet.) 2. The Windows device contains a cert or public key which does not auto-expire based on clock-time, but is only replaced as a side-effect of OS patches. This cert/key exists for one specific purpose, allowing it to securely get a time from an MS time fallback server. Such a server would not normally be queried, and can have very relaxed performance/accuracy/uptime requirements. It could literally just be a GET call returning a epoch integer. 3. This way the client system can bootstrap up to something close-enough-to-current that the regular NTP process can be completed for a more-precise time. Obviously this can fail if the attacker injects a fake cert/key into the machine... but at that point isn't it Game Over anyway? The same attacker could just put in their own certificate authority for any service. It may not work for computers that are left unplugged and unpatched for a few years, but it sounds like the real issue are live servers, or ones that just suffered some power-supply hiccup.
- b112 3y ago“You may ask—why doesn’t the device ask the nearest time server for the current time over the network?” Microsoft engineers wrote. “Since the device is not in a state to communicate securely over the network, it cannot obtain time securely over the network as well, unless you choose to ignore network security or at least punch some holes into it by making exceptions.” Let me guess, time isn't set, so dnssec is broken, so ntp servers won't reolve, so you can't set the time? So lame, ntp servers shouldn't be using dnssec.
- cryptonector 3y agoWhat on Earth! Look, you can use (with some care) time from a Kerberos KDC for the domain/realm to which the host is joined if you like, but you can't use the time from random application servers! What were they thinking?!
- dullcrisp 3y agoDoes anyone discuss the hypothesis that it’s a statistical bug in the implementation and that there’s some small but non-negligible probability that the servers see a pattern of random timestamps that convinces them of a false time? Because the would be my naive guess as to the cause.
- Spooky23 3y agoIt makes me feel better, at least I’m not alone in getting garbage support from them. Our TAM is like a funeral director. Always expressing his deep sorrow.
- spartanatreyu 3y agoA bit of a tangent, but this time-checking heuristic reminds me of the game "Halo 3: ODST". The story takes place in a future African megacity before, during, and after an alien invasion. Part of the game's story is shown from an AI's perspective who controls the city's operations trying to evacuate the Human survivors. Whenever the story is shown through the AI's perspective, you can see an overlay of the AI's working memory including which suburb's cameras are being monitored, the emotional state of the AI, and the current running timestamp. Some of the aliens end up withdrawing from within the city by making an FTL jump which causes extreme localised time dilation. Immediately afterwards, the timestamp in the AI's working memory is frozen because it's not sure what the exact timestamp is anymore, each system it connects to gives it a slightly different time depending on how close it was to the FTL event. It guesses a likely time and uses that until sometime later it is able to reconnect with other unaffected human networks and resumes the correct timestamps. This time heuristic was shown in the game, but also makes an appearance of sorts in the announcement trailer (seen in the bottom right): https://www.youtube.com/watch?v=WroxHMo6B_k https://www.youtube.com/watch?v=WroxHMo6B_k
- predictabl3 3y agoI am so happy you shared this here, I had no idea. Really lovely, thank you.
- nightowl_games 3y agoHow is a basic web application supposed to treat client time? In some of my systems, the server responds with its timestamp, the client records it's delta between itself and the server, and we use that delta as an offset into stuff like auth token refresh. My main PC has the incorrect timezone in its bios, so it's always 6hr in the future when I first log in..it's interesting to see what breaks. Rocket League for instance doesnt work at all. Starcraft 2 works fine. Windows was unable to sync itself in this situation for months, but has fixed itself lately so it can sync to the right time after a few minutes
- justinsaccount 3y ago> Microsoft recommending STS be turned off when the server receives reliable timekeeping through the Network Time Protocol. Why then are the first lines of the STS code not something like // server receives reliable timekeeping through the Network Time Protocol. if (ntp_enabled() && ntp_working()) return 0;
- fnordpiglet 3y agoI honestly struggle to recall a time when there was a windows feature that wasn’t : * broken on arrival and never really fixed * negligently implemented that causes serious harm * a dark pattern that no one actually wants but improves the windows margins through covert means (data harvesting, embedded adware, etc) I’m sure they exist, but the prevalence of the badness is so striking.
- oskarw85 3y agoIn my opinion writing that Microsoft fucked up where clearly other software puts bogus data in timestamp fields is laughable. Shit goes in - shit goes out. I think MS should improve implementation in two ways: - ask known trusted sources (like their own servers) for SSL timestamp - use that timestamp only internally (as opposed to system-wide) to acquire valid NTP timestamp