3 ms·

Link is to the transcript of a podcast, intro: Clock synchronization, keeping all of the clocks on your network set to the “correct” time, sounds straightforwar

Link is to the transcript of a podcast, intro: Clock synchronization, keeping all of the clocks on your network set to the “correct” time, sounds straightforward: our smartphones sure don’t seem to have trouble with it. Next, keep them all accurate to within 100 microseconds, and prove that you did -- now things start to get tricky. In this episode, Ron talks with Chris Perl, a systems engineer at Jane Street about the fundamental difficulty of solving this problem at scale and how we solved it.

ithkuil 5y ago
It's not hard to make something that works ok most of the time. It's much harder to make it work all the time or at least to know for sure when it didn't work. It all depends on what is at stakes: if your clocks are wrong, are you just mildly inconvenienced? Do you get a cached result that should have expired? Is it a security problem (are you accepting a certificate that has expired)? Is it a correctness problem (is the causal order of a database update been inverted by mistake)? There is a spectrum of severity of problems caused by clock sync, and whether the solutions are trivially solved or provably impossible (or anything in between) ultimately depends on what you're trying to do.
- gerdesj 5y ago
  Not sure why you were DVd. You have mentioned many valid points related to timesync.
gerdesj 5y ago
"the fundamental difficulty of solving this problem at scale and how we solved it" Time is hard and the funny thing is - it's relative! Anyone who says they have "solved" time is delusional. The best you can do is reduce errors and problems related to timekeeping to an acceptable level. I've "solved" time at work with three Raspberry Pis with GPS boards/hats and aerials. My requirements are that logs be correlatable and Kerberos etc work - so milli-second is enough. My Pis have a spread of around 0.002ms ie 2us according to ntpq -p. It doesn't take much to make it far worse. When I update them, I do one at a time per day. Accuracy drops a bit. I might put in three more at home. ntp needs a lot of sources to converge properly. Ideally five or more. I have quite a decent connection to work. ... reads transcript. Oh well done - you discovered PTP. Anyone who actually knows about this stuff like to comment on ntp vs ptp?
- bradknowles 5y ago
  The standard rule is that you need 2n + 1 upstream time sources to protect against n falsetickers, but this falls down for the case of n=1. If you want to protect yourself against just one falseticker, you need at least four upstream time sources, to handle the case of what you do after you kick the one falseticker off the island. We explain this at http://support.ntp.org/bin/view/Support/SelectingOffsiteNTPServers#Section_5.3.3 http://support.ntp.org/bin/view/Support/SelectingOffsiteNTPS.... PTP is great for high resolution time synchronization on the local LAN, but doesn’t leave the local lan segment without special network support in the switches and hubs. It is not designed for good stable time sync over the long term. If you’re running a High Frequency Trading operation and it’s really important to know precisely which nanosecond or which picosecond a particular transaction occurred, then you want PTP. If you’re concerned about keeping all your servers in good enough time sync that you can debug the order of operations as they might show up in the log files on different servers, then you want NTP. And NTP can be configured to use PTP as a good quality Stratum 0 refclock. If you wish to learn more about PTP, the Network Time Foundation supports the Linux PTP project, see https://www.nwtime.org/projects/linuxptp/ https://www.nwtime.org/projects/linuxptp/ Disclaimer: I have been involved in supporting the Public NTP Project since 2003, and related projects like the NTF, the nwtime.org site, etc….
  - nezgar 5y ago
    ... "local LAN" is a tad redundant. :)
    - vlovich123 5y ago
      I understood it to mean within the same subnet or switch. If that’s what the author intended then I think it’s not redundant.
    - bradknowles 5y ago
      You are correct. I meant the local LAN segment. I think I may have tried to go back and edit that clarification into the message, but failed. And then failed to post a follow up clarification. So, good catch! Thanks!

ithkuil 5y ago

It's not hard to make something that works ok most of the time. It's much harder to make it work all the time or at least to know for sure when it didn't work. It all depends on what is at stakes: if your clocks are wrong, are you just mildly inconvenienced? Do you get a cached result that should have expired? Is it a security problem (are you accepting a certificate that has expired)? Is it a correctness problem (is the causal order of a database update been inverted by mistake)? There is a spectrum of severity of problems caused by clock sync, and whether the solutions are trivially solved or provably impossible (or anything in between) ultimately depends on what you're trying to do.

gerdesj 5y ago
Not sure why you were DVd. You have mentioned many valid points related to timesync.

gerdesj 5y ago

"the fundamental difficulty of solving this problem at scale and how we solved it" Time is hard and the funny thing is - it's relative! Anyone who says they have "solved" time is delusional. The best you can do is reduce errors and problems related to timekeeping to an acceptable level. I've "solved" time at work with three Raspberry Pis with GPS boards/hats and aerials. My requirements are that logs be correlatable and Kerberos etc work - so milli-second is enough. My Pis have a spread of around 0.002ms ie 2us according to ntpq -p. It doesn't take much to make it far worse. When I update them, I do one at a time per day. Accuracy drops a bit. I might put in three more at home. ntp needs a lot of sources to converge properly. Ideally five or more. I have quite a decent connection to work. ... reads transcript. Oh well done - you discovered PTP. Anyone who actually knows about this stuff like to comment on ntp vs ptp?

bradknowles 5y ago
The standard rule is that you need 2n + 1 upstream time sources to protect against n falsetickers, but this falls down for the case of n=1. If you want to protect yourself against just one falseticker, you need at least four upstream time sources, to handle the case of what you do after you kick the one falseticker off the island. We explain this at http://support.ntp.org/bin/view/Support/SelectingOffsiteNTPServers#Section_5.3.3 http://support.ntp.org/bin/view/Support/SelectingOffsiteNTPS.... PTP is great for high resolution time synchronization on the local LAN, but doesn’t leave the local lan segment without special network support in the switches and hubs. It is not designed for good stable time sync over the long term. If you’re running a High Frequency Trading operation and it’s really important to know precisely which nanosecond or which picosecond a particular transaction occurred, then you want PTP. If you’re concerned about keeping all your servers in good enough time sync that you can debug the order of operations as they might show up in the log files on different servers, then you want NTP. And NTP can be configured to use PTP as a good quality Stratum 0 refclock. If you wish to learn more about PTP, the Network Time Foundation supports the Linux PTP project, see https://www.nwtime.org/projects/linuxptp/ https://www.nwtime.org/projects/linuxptp/ Disclaimer: I have been involved in supporting the Public NTP Project since 2003, and related projects like the NTF, the nwtime.org site, etc….
- nezgar 5y ago
  ... "local LAN" is a tad redundant. :)
  - vlovich123 5y ago
    I understood it to mean within the same subnet or switch. If that’s what the author intended then I think it’s not redundant.
  - bradknowles 5y ago
    You are correct. I meant the local LAN segment. I think I may have tried to go back and edit that clarification into the message, but failed. And then failed to post a follow up clarification. So, good catch! Thanks!