9 ms·

Make sure your time servers are configured correctly. My department had several Silicon Graphics workstations. Running IRIX, they machines determine amongst the

Make sure your time servers are configured correctly. My department had several Silicon Graphics workstations. Running IRIX, they machines determine amongst themselves which has the most accurate time and they vote that one as the Timemaster. Any new machines added would take their time from the Timemaster. The oldest machine was Timemaster and 8.5 minutes off. Took me week to figure out why my brand workstation had bad time. Fun times

saltedonion 5y ago
Man. Time based bugs are some of the worst. I spent weeks trying to figure out what was wrong with some of my scripts running under WSL. Apparently wsl Linux kernel had a bug that could cause time to drift by minutes.
- dmos62 5y ago
  My most memorable bug (because of how long debugging took) was inconsistent use of local and UTC timezones. Learned to localize timedates as late as possible (or preferably never) and delocalize as early as possible.
  - znpy 5y ago
    as if it is a rite of passage to become an adult engineer, i'm now facing timezone issues at work. no real problems yet, but the utc vs local timezone has to be handled. Could you please share what did you read on this topic? Thank you!
    - dmos62 5y ago
      I don't think I read anything specific on the topic. I'd simply suggest only using local time in the views. Hopefully that's possible in your use case. Be defensive in your design.
- treeman79 5y ago
  This is why mission critical systems like Boeing airliners require a reboot. At some point it’s just a lot more practical and safer.
  - sokoloff 5y ago
    That was done via Airworthiness Directive (a mandatory prod patch for aviation), not by original engineering design intent. Similar story with the Patriot missile defense battery. * - https://www.federalregister.gov/documents/2020/03/23/2020-06092/airworthiness-directives-the-boeing-company-airplanes https://www.federalregister.gov/documents/2020/03/23/2020-06... * - https://barrgroup.com/sites/default/files/case-study-patriot-missile-defects.pdf https://barrgroup.com/sites/default/files/case-study-patriot...
- donkarma 5y ago
  My Windows 10 clock goes 2-5x faster when I hibernate and come back
  - throwaway2048 5y ago
    There is a funny bug with Windows Vista in KVM/QEMU where the clock ticks about 1000x faster than it should. You can see the hour hand moving in the clock panel, and animations play at warp speed, and media playback is extremely screwy.
- imglorp 5y ago
  Oohh, are we doing time bugs? Sun T4 servers around early 2010s, and I forget which Solaris release this was, had a beaut of a random clock jump. Every now and then, certain apps on the a system would crash around the same time. We'd scrape through the logs and usually see our app cratered, some databases, ntpd, sshd naturally. Logging of timestamps was iffy, obviously. The ntpd was the obvious suspect because, despite keeping a nice low offset to its peers like 1ms or two, for months at a time, out of the blue it would confess something like "time offset is too large, I can't fix that so I'll exit!". After chasing Sun ntpd bug reports[1] for a while, we ruled it out when we saw a pattern in the undamaged logs that looked like 09:59:58.000 ... 09:59:59.123 ... 09:09:01.345 ... 09:09:02.123 ... Yep the system clock really had jumped back almost an hour. That explained everything about the userspace going nuts including ntpd exiting as a symptom and not a culprit. After some Sun support and some sunbugs searching [1 again] we found the T4 in that Solaris rev had a hardware RTC with separate registers for H, M, S etc and a write mutex protecting them, but no read mutex. It was possible to read the RTC while it was being updated, which happened when it was syncing the OS clock to the RTC, or something like that. Fixed in a later release. 1: RIP sunbugs database. It was such a mature relationship where Sun would let everyone see what they were working on and customers could participate or at least know about known issues. I would love to find an archive. Of course Oracle shut that off immediately so you had to open a ticket and ask.
- gregmac 5y ago
  My favorite time bug: Someone on a team I was on spent probably weeks working on a way to process an inbound sensor data stream (which was generated by another bit of software also maintained by us). The data was akin to an odometer, increasing over time based on usage. The trouble was there appeared to be two parallel streams of the same name, which were always offset from each other and while the offset varied they were always roughly close. The algorithm this person came up with eventually sorted them into "thing_1" and "thing_2" streams, at which point it got turned over to me to display. It actually worked pretty well, for what it's worth. I started asking how am I supposed to show this to a user to make use of, what does this even represent?... but never got an adequate answer. So I started looking at the whole chain, and what I found was the piece of software generating the data had a small bug: it used "hh" instead of "HH" in the timestamp, but also no am/pm. The timestamps were supposed to be 24h and looked like it, but 9:02AM and 9:02PM both came in as "09:02". To confirm, we checked the database and, sure enough, every bit of data was between 1:00am and 12:59pm. In the end we fixed the timestamp bug and threw out the processing code. It's simultaneously a hilarious date bug, a face-palming colossal waste of time, and a lesson on how not to run technical teams (the manager had no technical skill, and the "team" of ~7 was very siloed and each operated more like teams of 1-2).
- throwaway2048 5y ago
  A very annoying DNS over HTTPS/TLS circular dependency bug manifests itself if your device doesn't have an RTC, or the battery is dead, or the clock is sufficiently skewed. Clock is fucked, so TLS certs don't verify due to validity times, so DNS is broken, so NTP cant look up domains, so the clock can't be set...
  - exikyut 5y ago
    At the end of the day since TLS depends on correct time to trust certificates, I guess the "everything is fine" solution is to fetch the DoH server's TLS cert, inspect the start and end dates, set the system time to the exact middle, then helicopter over NTP a bit to make sure it came up and changed the time to something hopefully correct. On the one hand there's not very much else you can do since you're pointing at bits of thin air and saying "there's the trust chain" in the first place, but on the other hand plaintext DNS is... not much better? Of course that's when the existential "why even DoH in the first place" starts (with side servings of "this feels so wrong putting it on the security report")... (...Why do I suddenly feel like disabling certificate verification is going to catch on in a big way in embedded ntpds, almost like a standard best practice... aaaaaaaaa)
    - mrtesthah 5y ago
      The best and simplest solution would just be to require an administrator to input the current date manually in that circumstance.
TacticalCoder 5y ago
TOTP issues related to bogus clocks are soooooo common that I've got my own public TOTP "secret" (so not really a secret) which I use to verify that my various devices running TOTP authenticators have the correct time (my phone, wife's phone, an airline/airgapped device running TOTP etc.). It's so bad that Google's own authenticator has a "time synch" functionality or something like that in the very TOTP app (and it helps!). This speaks volume as to how bad and how not-solved-at-all the issue of drifting/wrong clocks is. I much prefer U2F.
- orev 5y ago
  TOTP systems are supposed to implement a sliding window to account for reasonable clock differences, usually within 5 minutes. Devices further off from that really do have a problem, and should not be trusted. I also find it hard to believe that so many devices are so far off, given the ubiquitous access to GPS (atomic clock) signals and NTP on networks. This has been a solved problem for a long time.
  - nfoz 5y ago
    > TOTP systems are supposed to implement a sliding window to account for reasonable clock differences, usually within 5 minutes. The default is 30 seconds, as per the RFC: https://datatracker.ietf.org/doc/html/rfc6238 https://datatracker.ietf.org/doc/html/rfc6238 (not sure that meaningfully changes what you were saying, but just fyi.)
    - orev 5y ago
      30 seconds is the default “time step” (4.2), but in talking about the transmission delay window (5.2), which the RFC recommends “at most 1 time step”, but also says that validation should occur for both the previous and next time step window. However in practical implementations, 5 minutes on both sides is typically used.
  - Avamander 5y ago
    As long as an Android or OSX phone has internet access and it isn't totally messed up, it'll pull time from NTP if the modem's time doesn't exist. I too have doubts that many devices are so far off.

saltedonion 5y ago

Man. Time based bugs are some of the worst. I spent weeks trying to figure out what was wrong with some of my scripts running under WSL. Apparently wsl Linux kernel had a bug that could cause time to drift by minutes.

dmos62 5y ago
My most memorable bug (because of how long debugging took) was inconsistent use of local and UTC timezones. Learned to localize timedates as late as possible (or preferably never) and delocalize as early as possible.
- znpy 5y ago
  as if it is a rite of passage to become an adult engineer, i'm now facing timezone issues at work. no real problems yet, but the utc vs local timezone has to be handled. Could you please share what did you read on this topic? Thank you!
  - dmos62 5y ago
    I don't think I read anything specific on the topic. I'd simply suggest only using local time in the views. Hopefully that's possible in your use case. Be defensive in your design.
treeman79 5y ago
This is why mission critical systems like Boeing airliners require a reboot. At some point it’s just a lot more practical and safer.
- sokoloff 5y ago
  That was done via Airworthiness Directive (a mandatory prod patch for aviation), not by original engineering design intent. Similar story with the Patriot missile defense battery. * - https://www.federalregister.gov/documents/2020/03/23/2020-06092/airworthiness-directives-the-boeing-company-airplanes https://www.federalregister.gov/documents/2020/03/23/2020-06... * - https://barrgroup.com/sites/default/files/case-study-patriot-missile-defects.pdf https://barrgroup.com/sites/default/files/case-study-patriot...
donkarma 5y ago
My Windows 10 clock goes 2-5x faster when I hibernate and come back
- throwaway2048 5y ago
  There is a funny bug with Windows Vista in KVM/QEMU where the clock ticks about 1000x faster than it should. You can see the hour hand moving in the clock panel, and animations play at warp speed, and media playback is extremely screwy.
imglorp 5y ago
Oohh, are we doing time bugs? Sun T4 servers around early 2010s, and I forget which Solaris release this was, had a beaut of a random clock jump. Every now and then, certain apps on the a system would crash around the same time. We'd scrape through the logs and usually see our app cratered, some databases, ntpd, sshd naturally. Logging of timestamps was iffy, obviously. The ntpd was the obvious suspect because, despite keeping a nice low offset to its peers like 1ms or two, for months at a time, out of the blue it would confess something like "time offset is too large, I can't fix that so I'll exit!". After chasing Sun ntpd bug reports[1] for a while, we ruled it out when we saw a pattern in the undamaged logs that looked like 09:59:58.000 ... 09:59:59.123 ... 09:09:01.345 ... 09:09:02.123 ... Yep the system clock really had jumped back almost an hour. That explained everything about the userspace going nuts including ntpd exiting as a symptom and not a culprit. After some Sun support and some sunbugs searching [1 again] we found the T4 in that Solaris rev had a hardware RTC with separate registers for H, M, S etc and a write mutex protecting them, but no read mutex. It was possible to read the RTC while it was being updated, which happened when it was syncing the OS clock to the RTC, or something like that. Fixed in a later release. 1: RIP sunbugs database. It was such a mature relationship where Sun would let everyone see what they were working on and customers could participate or at least know about known issues. I would love to find an archive. Of course Oracle shut that off immediately so you had to open a ticket and ask.
gregmac 5y ago
My favorite time bug: Someone on a team I was on spent probably weeks working on a way to process an inbound sensor data stream (which was generated by another bit of software also maintained by us). The data was akin to an odometer, increasing over time based on usage. The trouble was there appeared to be two parallel streams of the same name, which were always offset from each other and while the offset varied they were always roughly close. The algorithm this person came up with eventually sorted them into "thing_1" and "thing_2" streams, at which point it got turned over to me to display. It actually worked pretty well, for what it's worth. I started asking how am I supposed to show this to a user to make use of, what does this even represent?... but never got an adequate answer. So I started looking at the whole chain, and what I found was the piece of software generating the data had a small bug: it used "hh" instead of "HH" in the timestamp, but also no am/pm. The timestamps were supposed to be 24h and looked like it, but 9:02AM and 9:02PM both came in as "09:02". To confirm, we checked the database and, sure enough, every bit of data was between 1:00am and 12:59pm. In the end we fixed the timestamp bug and threw out the processing code. It's simultaneously a hilarious date bug, a face-palming colossal waste of time, and a lesson on how not to run technical teams (the manager had no technical skill, and the "team" of ~7 was very siloed and each operated more like teams of 1-2).
throwaway2048 5y ago
A very annoying DNS over HTTPS/TLS circular dependency bug manifests itself if your device doesn't have an RTC, or the battery is dead, or the clock is sufficiently skewed. Clock is fucked, so TLS certs don't verify due to validity times, so DNS is broken, so NTP cant look up domains, so the clock can't be set...
- exikyut 5y ago
  At the end of the day since TLS depends on correct time to trust certificates, I guess the "everything is fine" solution is to fetch the DoH server's TLS cert, inspect the start and end dates, set the system time to the exact middle, then helicopter over NTP a bit to make sure it came up and changed the time to something hopefully correct. On the one hand there's not very much else you can do since you're pointing at bits of thin air and saying "there's the trust chain" in the first place, but on the other hand plaintext DNS is... not much better? Of course that's when the existential "why even DoH in the first place" starts (with side servings of "this feels so wrong putting it on the security report")... (...Why do I suddenly feel like disabling certificate verification is going to catch on in a big way in embedded ntpds, almost like a standard best practice... aaaaaaaaa)
  - mrtesthah 5y ago
    The best and simplest solution would just be to require an administrator to input the current date manually in that circumstance.

TacticalCoder 5y ago

TOTP issues related to bogus clocks are soooooo common that I've got my own public TOTP "secret" (so not really a secret) which I use to verify that my various devices running TOTP authenticators have the correct time (my phone, wife's phone, an airline/airgapped device running TOTP etc.). It's so bad that Google's own authenticator has a "time synch" functionality or something like that in the very TOTP app (and it helps!). This speaks volume as to how bad and how not-solved-at-all the issue of drifting/wrong clocks is. I much prefer U2F.

orev 5y ago
TOTP systems are supposed to implement a sliding window to account for reasonable clock differences, usually within 5 minutes. Devices further off from that really do have a problem, and should not be trusted. I also find it hard to believe that so many devices are so far off, given the ubiquitous access to GPS (atomic clock) signals and NTP on networks. This has been a solved problem for a long time.
- nfoz 5y ago
  > TOTP systems are supposed to implement a sliding window to account for reasonable clock differences, usually within 5 minutes. The default is 30 seconds, as per the RFC: https://datatracker.ietf.org/doc/html/rfc6238 https://datatracker.ietf.org/doc/html/rfc6238 (not sure that meaningfully changes what you were saying, but just fyi.)
  - orev 5y ago
    30 seconds is the default “time step” (4.2), but in talking about the transmission delay window (5.2), which the RFC recommends “at most 1 time step”, but also says that validation should occur for both the previous and next time step window. However in practical implementations, 5 minutes on both sides is typically used.
- Avamander 5y ago
  As long as an Android or OSX phone has internet access and it isn't totally messed up, it'll pull time from NTP if the modem's time doesn't exist. I too have doubts that many devices are so far off.