5 ms·

> dedicating a core for your NTP Heh as soon as I read this I thought "algo trader?". If you don't mind me asking outside of trading how often do you see syste

> dedicating a core for your NTP Heh as soon as I read this I thought "algo trader?". If you don't mind me asking outside of trading how often do you see systems seriously manage jitter (specifically I'm thinking about interrupt management). I'm in HPC and whilst some such management is present it seems seldom thought about.

lmilcin 6y ago
Heh, I already thought somebody will point it out. See, they decided to create custom hardware just to do NTP. How is instead dedicating a single core a worse solution? Setting dedicated core is basically configuration detail. There are no gains from custom hardware as network variability would mask them. If you want really good time guarantee because having right time is critical for your application, you put atomic clocks in your servers as one well known company does. Also, it is customary to have dedicated machines for various functions. You would not normally want to mix some different types of loads that conflict with each other, for example high security with low security, high throughput with real time, etc. You could just configure a single machine dedicated to NTP (which is customary), but instead of sharing cores between OS and NTP (and introducing jitter) you can have separate core for the OS and separate for NTP so that NTP can work undisturbed. So if you maintain an NTP instance and have problem with jitter and you red this article my hint is: don't set up custom FPGA to fix your problem, just dedicate single core on a machine you already have and you will get as good results as you can.
- gbrown_ 6y ago
  Oh I wasn't meaning specifically in the context of NTP, just generally.
  - lmilcin 6y ago
    > Oh I wasn't meaning specifically in the context of NTP, just generally. So you decided to just answer with a generalization without regard for the particular case of NTP? Are you frequently building custom hardware when default OS settings do not suit your needs? Linux has a lot of configuration potential. It is wise to explore and understand various knobs available before you decide to complicate your life and build something very complex that could easily be replaced with a one line shell script.
    - gbrown_ 6y ago
      Have you perhaps mixed up my comment with this one https://news.ycombinator.com/item?id=26152783 https://news.ycombinator.com/item?id=26152783? I was just asking about your thoughts of how outside of trading jitter is seldom considered. I appreciate it's off-topic from the actual HN post but I was curious given your experience in trading.
- Unklejoe 6y ago
  > How is instead dedicating a single core a worse solution? It is pretty much guaranteed to perform worse, if that's what you mean. Is it worth the effort and extra complexity? Probably not. The timestamping jitter with an FPGA would probably be on the order of tens of nanoseconds while we're talking microseconds with a software solution. > There are no gains from custom hardware as network variability would mask them. I _think_ multiple cascaded jitters in a system add with the square root of the sum of squares, so there would still be an improvement. Consider a system where you had 10 microseconds of inherent jitter, then you added another 10 microseconds on top of that. The total is not 10 microseconds - it would be 14.
  - lmilcin 6y ago
    > Is it worth the effort and extra complexity? Probably not. That's pretty much definition of a worse solution.
    - Unklejoe 6y ago
      Not necessarily when it comes to something like HFT, where they do things like run point to point microwave links to reduce latency by microseconds.
  - 1_person 6y ago
    The precision achievable with FPGAs is well beyond 10s of nanoseconds. The high rate transceivers are used with capacitative dividers or internal signal propagation delays defining the reference intervals for time to digital conversion in extreme accuracy applications, these and similar approaches yield precision to picosecond scale and beyond.
    - Unklejoe 6y ago
      Right, but Ethernet timestamping usually occurs on a symbol or byte level inside the MAC, so for GigE, you're looking at 8 nanoseconds per symbol. This could be improved by oversampling the symbols, but at least with GigE, there's (I think) 4 cycles of uncertainty each time the link comes up due to how the clocking agreement between master/slave is established. It's also somewhat difficult to run large counters faster than a few hundred MHz inside of an FPGA. I've worked with systems that use two different clocks - one small counter for Ethernet timestamping, and another main clock, with a periodic latching between the two to correlate them, but it's uncommon. There's also the fact that most PHYs operate by having a FIFO with data being clocked in and clocked out on different clock domains. The way I understand it, the PHY FIFO fills up half way, and only then does it start draining into the MAC. This is to allow for oscillator frequency errors between each device. This is also why there's a maximum frame size and max oscillator tolerance specified for Ethernet - so that this FIFO doesn't under or overrun before the entire frame is received. I think sync-e can improve this quite a bit. Of course, FPGAs themselves are capable of much tighter timings, but in practice, it's usually implemented this way.
hansor 6y ago
NTP is hilariously insufficient for HFT, we have far better protocol around: PTP. https://en.wikipedia.org/wiki/Precision_Time_Protocol https://en.wikipedia.org/wiki/Precision_Time_Protocol
- gbrown_ 6y ago
  I'm aware, I was talking about jitter in general.
- Bootvis 6y ago
  Jane Street has an interesting podcast on this very subject: https://signalsandthreads.com/clock-synchronization/ https://signalsandthreads.com/clock-synchronization/

lmilcin 6y ago

Heh, I already thought somebody will point it out. See, they decided to create custom hardware just to do NTP. How is instead dedicating a single core a worse solution? Setting dedicated core is basically configuration detail. There are no gains from custom hardware as network variability would mask them. If you want really good time guarantee because having right time is critical for your application, you put atomic clocks in your servers as one well known company does. Also, it is customary to have dedicated machines for various functions. You would not normally want to mix some different types of loads that conflict with each other, for example high security with low security, high throughput with real time, etc. You could just configure a single machine dedicated to NTP (which is customary), but instead of sharing cores between OS and NTP (and introducing jitter) you can have separate core for the OS and separate for NTP so that NTP can work undisturbed. So if you maintain an NTP instance and have problem with jitter and you red this article my hint is: don't set up custom FPGA to fix your problem, just dedicate single core on a machine you already have and you will get as good results as you can.

gbrown_ 6y ago
Oh I wasn't meaning specifically in the context of NTP, just generally.
- lmilcin 6y ago
  > Oh I wasn't meaning specifically in the context of NTP, just generally. So you decided to just answer with a generalization without regard for the particular case of NTP? Are you frequently building custom hardware when default OS settings do not suit your needs? Linux has a lot of configuration potential. It is wise to explore and understand various knobs available before you decide to complicate your life and build something very complex that could easily be replaced with a one line shell script.
  - gbrown_ 6y ago
    Have you perhaps mixed up my comment with this one https://news.ycombinator.com/item?id=26152783 https://news.ycombinator.com/item?id=26152783? I was just asking about your thoughts of how outside of trading jitter is seldom considered. I appreciate it's off-topic from the actual HN post but I was curious given your experience in trading.
Unklejoe 6y ago
> How is instead dedicating a single core a worse solution? It is pretty much guaranteed to perform worse, if that's what you mean. Is it worth the effort and extra complexity? Probably not. The timestamping jitter with an FPGA would probably be on the order of tens of nanoseconds while we're talking microseconds with a software solution. > There are no gains from custom hardware as network variability would mask them. I _think_ multiple cascaded jitters in a system add with the square root of the sum of squares, so there would still be an improvement. Consider a system where you had 10 microseconds of inherent jitter, then you added another 10 microseconds on top of that. The total is not 10 microseconds - it would be 14.
- lmilcin 6y ago
  > Is it worth the effort and extra complexity? Probably not. That's pretty much definition of a worse solution.
  - Unklejoe 6y ago
    Not necessarily when it comes to something like HFT, where they do things like run point to point microwave links to reduce latency by microseconds.
- 1_person 6y ago
  The precision achievable with FPGAs is well beyond 10s of nanoseconds. The high rate transceivers are used with capacitative dividers or internal signal propagation delays defining the reference intervals for time to digital conversion in extreme accuracy applications, these and similar approaches yield precision to picosecond scale and beyond.
  - Unklejoe 6y ago
    Right, but Ethernet timestamping usually occurs on a symbol or byte level inside the MAC, so for GigE, you're looking at 8 nanoseconds per symbol. This could be improved by oversampling the symbols, but at least with GigE, there's (I think) 4 cycles of uncertainty each time the link comes up due to how the clocking agreement between master/slave is established. It's also somewhat difficult to run large counters faster than a few hundred MHz inside of an FPGA. I've worked with systems that use two different clocks - one small counter for Ethernet timestamping, and another main clock, with a periodic latching between the two to correlate them, but it's uncommon. There's also the fact that most PHYs operate by having a FIFO with data being clocked in and clocked out on different clock domains. The way I understand it, the PHY FIFO fills up half way, and only then does it start draining into the MAC. This is to allow for oscillator frequency errors between each device. This is also why there's a maximum frame size and max oscillator tolerance specified for Ethernet - so that this FIFO doesn't under or overrun before the entire frame is received. I think sync-e can improve this quite a bit. Of course, FPGAs themselves are capable of much tighter timings, but in practice, it's usually implemented this way.

hansor 6y ago

NTP is hilariously insufficient for HFT, we have far better protocol around: PTP. https://en.wikipedia.org/wiki/Precision_Time_Protocol https://en.wikipedia.org/wiki/Precision_Time_Protocol

gbrown_ 6y ago
I'm aware, I was talking about jitter in general.
Bootvis 6y ago
Jane Street has an interesting podcast on this very subject: https://signalsandthreads.com/clock-synchronization/ https://signalsandthreads.com/clock-synchronization/