5 ms·

> Microprocessor based Network Time Protocol (NTP) servers suffer from a large amount of timestamp jitter, due to the hardware and Operating System (OS) being s

> Microprocessor based Network Time Protocol (NTP) servers suffer from a large amount of timestamp jitter, due to the hardware and Operating System (OS) being shared among other applications. So instead of configuring Linux correctly and dedicating a core for your NTP you have created custom hardware. Congratulations, now you have two problems. I have been working on algorithmic trading and it is not that hard to reliably (like 100% of the time) respond within couple microseconds. You will never notice it, though, because your network will introduce way more variability. This is especially the case with NTP which usually is a single box shared between large number of servers throughout your network.

varjag 6y ago
Yes, doing things fast is the unremarkable part. Doing things fast and synchronously over the network is the trickier one.
gbrown_ 6y ago
> dedicating a core for your NTP Heh as soon as I read this I thought "algo trader?". If you don't mind me asking outside of trading how often do you see systems seriously manage jitter (specifically I'm thinking about interrupt management). I'm in HPC and whilst some such management is present it seems seldom thought about.
- lmilcin 6y ago
  Heh, I already thought somebody will point it out. See, they decided to create custom hardware just to do NTP. How is instead dedicating a single core a worse solution? Setting dedicated core is basically configuration detail. There are no gains from custom hardware as network variability would mask them. If you want really good time guarantee because having right time is critical for your application, you put atomic clocks in your servers as one well known company does. Also, it is customary to have dedicated machines for various functions. You would not normally want to mix some different types of loads that conflict with each other, for example high security with low security, high throughput with real time, etc. You could just configure a single machine dedicated to NTP (which is customary), but instead of sharing cores between OS and NTP (and introducing jitter) you can have separate core for the OS and separate for NTP so that NTP can work undisturbed. So if you maintain an NTP instance and have problem with jitter and you red this article my hint is: don't set up custom FPGA to fix your problem, just dedicate single core on a machine you already have and you will get as good results as you can.
  - gbrown_ 6y ago
    Oh I wasn't meaning specifically in the context of NTP, just generally.
    - lmilcin 6y ago
      > Oh I wasn't meaning specifically in the context of NTP, just generally. So you decided to just answer with a generalization without regard for the particular case of NTP? Are you frequently building custom hardware when default OS settings do not suit your needs? Linux has a lot of configuration potential. It is wise to explore and understand various knobs available before you decide to complicate your life and build something very complex that could easily be replaced with a one line shell script.
      - gbrown_ 6y ago
        Have you perhaps mixed up my comment with this one https://news.ycombinator.com/item?id=26152783 https://news.ycombinator.com/item?id=26152783? I was just asking about your thoughts of how outside of trading jitter is seldom considered. I appreciate it's off-topic from the actual HN post but I was curious given your experience in trading.
  - Unklejoe 6y ago
    > How is instead dedicating a single core a worse solution? It is pretty much guaranteed to perform worse, if that's what you mean. Is it worth the effort and extra complexity? Probably not. The timestamping jitter with an FPGA would probably be on the order of tens of nanoseconds while we're talking microseconds with a software solution. > There are no gains from custom hardware as network variability would mask them. I _think_ multiple cascaded jitters in a system add with the square root of the sum of squares, so there would still be an improvement. Consider a system where you had 10 microseconds of inherent jitter, then you added another 10 microseconds on top of that. The total is not 10 microseconds - it would be 14.
    - lmilcin 6y ago
- hansor 6y ago
  NTP is hilariously insufficient for HFT, we have far better protocol around: PTP. https://en.wikipedia.org/wiki/Precision_Time_Protocol https://en.wikipedia.org/wiki/Precision_Time_Protocol
  - gbrown_ 6y ago
    I'm aware, I was talking about jitter in general.
  - Bootvis 6y ago
    Jane Street has an interesting podcast on this very subject: https://signalsandthreads.com/clock-synchronization/ https://signalsandthreads.com/clock-synchronization/
Unklejoe 6y ago
> I have been working on algorithmic trading and it is not that hard to reliably (like 100% of the time) respond within couple microseconds. As someone who works in the flight test instrumentation industry (that primarily uses PTP rather than NTP), my first thought was "is a couple of microseconds supposed to be good?". With PTP, I usually achieve RMS offsets of < 50 nanoseconds over gigabit Ethernet. I sort of agree with your point in that it's probably pointless implementing NTP inside of an FPGA, but I'd like to extend it further and say that if you really care about accurate timing, you should be using PTP rather than NTP anyway. In that case, an FPGA solution makes much more sense. However, in the systems I've worked with, the FPGA only implements a few pieces of core functionality - like the frame time stamping and the numerically-frequency-controlled clock. The actual protocol is usually implemented in software, except in the case of transparent switches. That said, most modern NICs have PTP hardware, but you still usually need some external logic to actually use these synchronized clocks.
- lmilcin 6y ago
  On a "normal" box the time is usually combination of various sources neither of which was really designed to provide accurate, sub microsecond, absolute time. (That except possibly for this "PTP hardware" in NICs that I don't know anything about) So even if you somehow devise a piece of hardware to pass the accurate time to the operating system, there would be no way to track it accurately. Now, I think that really accurate absolute timing on machines isn't really that critical. For example, for consistency protocols usually it is enough to provide guarantee of 1s around true time. For algorithmic trading we were mostly interested in responding as quickly as possible to the event that came from the stock exchange. The packets always came with a known delay and the timing wasn't all that important except for debugging. Everything was done on a single box (so no need to synchronize with anything else) and even on that box the events were processed such that there was no need to coordinate with other threads, each thread just consumed, processed and published results.
  - mlyle 6y ago
    Most Intel 10 gigabit adapters support PTP. It's just about inserting the exact time the frame will be sent into a sent frame, and noting the exact time the frame is received. All that's left is a small amount of clock jitter and cable delays. High end switches support PTP themselves so you don't need to worry about queueing delays, but you don't need a switch that supports PTP to get high resolution, low jitter, low uncertainty time-- at least on reasonably sized networks that are usually not saturated. There are many algorithmic trading applications that rely upon high quality time derived from a single coherent source, along with telecommunications, control systems, instrumentation, etc. And there are other distributed systems approaches where true ordering is nice: yes, you can provide ordering within a system of dependent events using a Lamport clock, vector clock, etc, but without high quality time you can also correctly sequence causally related events originating outside your tightly coupled system in many use cases. That said: PTP relies on a tightly coupled master and realistically is intended for a tightly coupled, hierarchal system. NTP is a better "internet" time protocol-- lots of logic for clock precedence, averaging of multiple sources, slower control system, etc.
1_person 6y ago
The uncontrollable and unobservable SMM interrupts of most modern CPUs add sufficient jitter to the reference clock sampling that there is no correct configuration. You generally must use an FPGA or appropriate DSP/microcontroller to achieve precision beyond ~10s of nanoseconds, which is entirely achievable even with NTP between hosts that have low contention 10+Gbps per transceiver lane interfaces / properly configured DCB/QoS so the time synchronization packets always egress without delay. PTP can, of course, achieve even better precision. Round trip time / transmission time ("respond within couple microseconds") are irrelevant, it's a rising or falling edge feature of the packet burst which timestamps the sample and the interval between these samples that's used to train the clocks. This can be accurate to within femtoseconds at the extreme.
- lmilcin 6y ago
  Let's say you get FPGA and then what? Where are you going to use that signal? On a server that has "uncontrollable and unobservable SMM interrupts"?
  - 1_person 6y ago
    Sure, why not? It's going to offer an incremental improvement in accuracy for the server's derived clock when the reference clock is not affected by that and other sources of jitter. Beyond the accuracy, the hardware is significantly cheaper and more efficient than a server with a general purpose x86 CPU. For a dedicated network reference clock, an FPGA or ASIC solution is simply better in every measurable way. It is more complex, to be sure, but the complexity needn't be your concern.

varjag 6y ago

Yes, doing things fast is the unremarkable part. Doing things fast and synchronously over the network is the trickier one.

gbrown_ 6y ago

> dedicating a core for your NTP Heh as soon as I read this I thought "algo trader?". If you don't mind me asking outside of trading how often do you see systems seriously manage jitter (specifically I'm thinking about interrupt management). I'm in HPC and whilst some such management is present it seems seldom thought about.

lmilcin 6y ago
Heh, I already thought somebody will point it out. See, they decided to create custom hardware just to do NTP. How is instead dedicating a single core a worse solution? Setting dedicated core is basically configuration detail. There are no gains from custom hardware as network variability would mask them. If you want really good time guarantee because having right time is critical for your application, you put atomic clocks in your servers as one well known company does. Also, it is customary to have dedicated machines for various functions. You would not normally want to mix some different types of loads that conflict with each other, for example high security with low security, high throughput with real time, etc. You could just configure a single machine dedicated to NTP (which is customary), but instead of sharing cores between OS and NTP (and introducing jitter) you can have separate core for the OS and separate for NTP so that NTP can work undisturbed. So if you maintain an NTP instance and have problem with jitter and you red this article my hint is: don't set up custom FPGA to fix your problem, just dedicate single core on a machine you already have and you will get as good results as you can.
- gbrown_ 6y ago
  Oh I wasn't meaning specifically in the context of NTP, just generally.
  - lmilcin 6y ago
    > Oh I wasn't meaning specifically in the context of NTP, just generally. So you decided to just answer with a generalization without regard for the particular case of NTP? Are you frequently building custom hardware when default OS settings do not suit your needs? Linux has a lot of configuration potential. It is wise to explore and understand various knobs available before you decide to complicate your life and build something very complex that could easily be replaced with a one line shell script.
    - gbrown_ 6y ago
      Have you perhaps mixed up my comment with this one https://news.ycombinator.com/item?id=26152783 https://news.ycombinator.com/item?id=26152783? I was just asking about your thoughts of how outside of trading jitter is seldom considered. I appreciate it's off-topic from the actual HN post but I was curious given your experience in trading.
- Unklejoe 6y ago
  > How is instead dedicating a single core a worse solution? It is pretty much guaranteed to perform worse, if that's what you mean. Is it worth the effort and extra complexity? Probably not. The timestamping jitter with an FPGA would probably be on the order of tens of nanoseconds while we're talking microseconds with a software solution. > There are no gains from custom hardware as network variability would mask them. I _think_ multiple cascaded jitters in a system add with the square root of the sum of squares, so there would still be an improvement. Consider a system where you had 10 microseconds of inherent jitter, then you added another 10 microseconds on top of that. The total is not 10 microseconds - it would be 14.
  - lmilcin 6y ago
hansor 6y ago
NTP is hilariously insufficient for HFT, we have far better protocol around: PTP. https://en.wikipedia.org/wiki/Precision_Time_Protocol https://en.wikipedia.org/wiki/Precision_Time_Protocol
- gbrown_ 6y ago
  I'm aware, I was talking about jitter in general.
- Bootvis 6y ago
  Jane Street has an interesting podcast on this very subject: https://signalsandthreads.com/clock-synchronization/ https://signalsandthreads.com/clock-synchronization/

Unklejoe 6y ago

> I have been working on algorithmic trading and it is not that hard to reliably (like 100% of the time) respond within couple microseconds. As someone who works in the flight test instrumentation industry (that primarily uses PTP rather than NTP), my first thought was "is a couple of microseconds supposed to be good?". With PTP, I usually achieve RMS offsets of < 50 nanoseconds over gigabit Ethernet. I sort of agree with your point in that it's probably pointless implementing NTP inside of an FPGA, but I'd like to extend it further and say that if you really care about accurate timing, you should be using PTP rather than NTP anyway. In that case, an FPGA solution makes much more sense. However, in the systems I've worked with, the FPGA only implements a few pieces of core functionality - like the frame time stamping and the numerically-frequency-controlled clock. The actual protocol is usually implemented in software, except in the case of transparent switches. That said, most modern NICs have PTP hardware, but you still usually need some external logic to actually use these synchronized clocks.

lmilcin 6y ago
On a "normal" box the time is usually combination of various sources neither of which was really designed to provide accurate, sub microsecond, absolute time. (That except possibly for this "PTP hardware" in NICs that I don't know anything about) So even if you somehow devise a piece of hardware to pass the accurate time to the operating system, there would be no way to track it accurately. Now, I think that really accurate absolute timing on machines isn't really that critical. For example, for consistency protocols usually it is enough to provide guarantee of 1s around true time. For algorithmic trading we were mostly interested in responding as quickly as possible to the event that came from the stock exchange. The packets always came with a known delay and the timing wasn't all that important except for debugging. Everything was done on a single box (so no need to synchronize with anything else) and even on that box the events were processed such that there was no need to coordinate with other threads, each thread just consumed, processed and published results.
- mlyle 6y ago
  Most Intel 10 gigabit adapters support PTP. It's just about inserting the exact time the frame will be sent into a sent frame, and noting the exact time the frame is received. All that's left is a small amount of clock jitter and cable delays. High end switches support PTP themselves so you don't need to worry about queueing delays, but you don't need a switch that supports PTP to get high resolution, low jitter, low uncertainty time-- at least on reasonably sized networks that are usually not saturated. There are many algorithmic trading applications that rely upon high quality time derived from a single coherent source, along with telecommunications, control systems, instrumentation, etc. And there are other distributed systems approaches where true ordering is nice: yes, you can provide ordering within a system of dependent events using a Lamport clock, vector clock, etc, but without high quality time you can also correctly sequence causally related events originating outside your tightly coupled system in many use cases. That said: PTP relies on a tightly coupled master and realistically is intended for a tightly coupled, hierarchal system. NTP is a better "internet" time protocol-- lots of logic for clock precedence, averaging of multiple sources, slower control system, etc.

1_person 6y ago

The uncontrollable and unobservable SMM interrupts of most modern CPUs add sufficient jitter to the reference clock sampling that there is no correct configuration. You generally must use an FPGA or appropriate DSP/microcontroller to achieve precision beyond ~10s of nanoseconds, which is entirely achievable even with NTP between hosts that have low contention 10+Gbps per transceiver lane interfaces / properly configured DCB/QoS so the time synchronization packets always egress without delay. PTP can, of course, achieve even better precision. Round trip time / transmission time ("respond within couple microseconds") are irrelevant, it's a rising or falling edge feature of the packet burst which timestamps the sample and the interval between these samples that's used to train the clocks. This can be accurate to within femtoseconds at the extreme.

lmilcin 6y ago
Let's say you get FPGA and then what? Where are you going to use that signal? On a server that has "uncontrollable and unobservable SMM interrupts"?
- 1_person 6y ago
  Sure, why not? It's going to offer an incremental improvement in accuracy for the server's derived clock when the reference clock is not affected by that and other sources of jitter. Beyond the accuracy, the hardware is significantly cheaper and more efficient than a server with a general purpose x86 CPU. For a dedicated network reference clock, an FPGA or ASIC solution is simply better in every measurable way. It is more complex, to be sure, but the complexity needn't be your concern.