6 ms·

The biggest danger in NTP isn't memory safety (though good on this project for tackling it), it's (a) the inherent risks in implementing a protocol based on tr

The biggest danger in NTP isn't memory safety (though good on this project for tackling it), it's (a) the inherent risks in implementing a protocol based on trivially spoofable UDP that can be used to do amplification and reflection and (b) emergent resonant behavior from your implementation that will inadvertently DDOS critical infrastructure when all 100m installed copies of your daemon decide to send a packet to NIST in the same microsecond. I'm happy to see more ntpd implementations but always a little worried.

timmytokyo 2y ago
I really wish more internet infrastructure would switch to using NTS. It addresses these kinds of issues.
- jaas 2y ago
  ntpd-rs support NTS, I agree it would be great if more people used it!
- 1over137 2y ago
  Never heard of it. Shockingly little on wikipedia for example.
  - codetrotter 2y ago
    Yeah. Seems it doesn’t even have its own article there. Only a short mention in the main article about NTP itself: > Network Time Security (NTS) is a secure version of NTPv4 with TLS and AEAD. The main improvement over previous attempts is that a separate "key establishment" server handles the heavy asymmetric cryptography, which needs to be done only once. If the server goes down, previous users would still be able to fetch time without fear of MITM. NTS is currently supported by several time servers, including Cloudflare. It is supported by NTPSec and chrony.
    - westurner 2y ago
      "RFC 8915: Network Time Security for the Network Time Protocol" (2020) https://www.rfc-editor.org/rfc/rfc8915.html https://www.rfc-editor.org/rfc/rfc8915.html "NTS RFC Published: New Standard to Ensure Secure Time on the Internet" (2020) https://www.internetsociety.org/blog/2020/10/nts-rfc-published-new-standard-to-ensure-secure-time-on-the-internet/ https://www.internetsociety.org/blog/2020/10/nts-rfc-publish... : > NTS is basically two loosely coupled sub-protocols that together add security to NTP. NTS Key Exchange (NTS-KE) is based on TLS 1.3 and performs the initial authentication of the server and exchanges security tokens with the client. The NTP client then uses these tokens in NTP extension fields for authentication and integrity checking of the NTP protocol messages that exchange time information. From "Simple Precision Time Protocol at Meta" https://news.ycombinator.com/item?id=39306209 https://news.ycombinator.com/item?id=39306209 : > How does SPTP compare to CERN's WhiteRabbit, which is built on PTP [and NTP NTS]? White Rabbit Project: https://en.wikipedia.org/wiki/White_Rabbit_Project https://en.wikipedia.org/wiki/White_Rabbit_Project
  - rnijveld 2y ago
    I'm afraid this is a pretty common sentiment. NTS has been out for several years already and is implemented in several implementations (including our ntpd-rs implementation, and others like chrony and ntpsec). Yet its usage is low and meanwhile the fully unsecured and easily spoofable NTP remains the default, in effect allowing anyone to manipulate your clock almost trivially (see our blog post about this: https://tweedegolf.nl/en/blog/121/hacking-time https://tweedegolf.nl/en/blog/121/hacking-time). Hopefully we can get NTS to the masses more quickly in the coming years and slowly start to decrease our dependency on unsigned NTP traffic, just as we did with unencrypted HTTP traffic.
  - denton-scratch 2y ago
    I hadn't heard of NTS until a Debian upgrade quietly installed ntpsec. It seems to now be the Debian default.
    - rlaager 2y ago
      ntp has been replaced by ntpsec in Debian. (I am the Debian ntpsec package maintainer.) By default, NTPsec on Debian uses the NTP Pool, so no NTS. But NTPsec does support NTS if you are running your own server and supports it opt-in on the client side. As far as I know, the Debian “default” is systemd-timesyncd. That is what you get out of the box. (Though, honestly, I automate most of my Linux installs, so I don’t interact with a stock install very often.) AFAIK, systemd-timesyncd does not support NTS at all. Doing NTS on a pool would be quite complicated. The easy way is to share the same key across the pool. That is obviously not workable when pool servers are run by different people. The other way would be to have an another out-of-band protocol where the pool NTP servers share their key with the centralized pool NTS-KE servers. Nobody has built that, and it’s non-trivial.
      - tialaramex 2y ago
        Ah not quite, I think pooling would be rather easier than you've thought, there are Let's Encrypt people here, but let me explain what you'd do to have N unrelated machines which are all able to successfully claim they are some-shared-name.example Each such machine mints (as often as it wants, but at least once) a document called a Certificate Signing Request. This is a signed (thus cannot be forged) document but it's public (so it needn't be confidential) and it basically says "Here's my public key, I claim I am some-shared-name.example, and I've signed this document with my private key so you can tell it was me who made it". The centralized service collects these public documents for legitimate members of the pool and it asks a CA to issue certificates for them. The CA wants a CSR, that's literally what it asks for -- Let's Encrypt clients actually just make one for you automatically, they still need one. Then the certificates are likewise public documents and can be just provided to anybody who wants them (including the NTP pool servers they're actually for which can collect a current certificate periodically). So you're only moving two public, signed, documents, which isn't hard to get right, you should indeed probably do this out-of-band but you aren't sharing the valuable private key anywhere, that's a terrible idea as well as being hard to do correctly it's just unnecessary.
rnijveld 2y ago
I agree that amplification and reflection definitely are worries, which is why we are working towards NTS becoming a default on the internet. NTS would prevent responses by a server from a spoofed packet and at the same time would make sure that NTP clients can finally start trusting their time instead of hoping that there are no malicious actors anywhere near them. You can read about it on our blog as well: https://tweedegolf.nl/en/blog/122/a-safe-internet-requires-secure-time https://tweedegolf.nl/en/blog/122/a-safe-internet-requires-s... One thing to note about amplification: amplification has always been something that NTP developers have been especially sensitive to. I would say though that protocols like QUIC and DNS have far greater amplification risks. Meanwhile, our server implementation forces that responses can never be bigger than the requests that initiated them, meaning that no amplification is possible at all. Even if we would have allowed bigger responses, I cannot imagine NTP responses being much bigger than two or three times their related request. Meanwhile I've seen numbers for DNS all the way up to 180 times the request payload. As for your worries: I think being a little cautious keeps you alert and can prevent mistakes, but I also feel that we've gone out of our way to not do anything crazy and hopefully we will be a net positive in the end. I hope you do give us a try and let us know if you find anything suspicious. If you have any feedback we'd love to hear it!
- dfc 2y ago
  > I cannot imagine NTP responses being much bigger than two or three times their related request. I think you must be limiting your imagination to ntp requests related to setting the time. There are a lot of other commands in the protocol used for management and metrics. The `monlist` command was good for 200x amplification. https://blog.cloudflare.com/understanding-and-mitigating-ntp-based-ddos-attacks https://blog.cloudflare.com/understanding-and-mitigating-ntp...
  - rnijveld 2y ago
    Ah right! I always forget about that since we don’t implement the management protocol in ntpd-rs. I think it’s insane that stuff should go over the same socket as the normal time messages. Something I don’t ever see us implementing.
- 2y ago

timmytokyo 2y ago

I really wish more internet infrastructure would switch to using NTS. It addresses these kinds of issues.

jaas 2y ago
ntpd-rs support NTS, I agree it would be great if more people used it!
1over137 2y ago
Never heard of it. Shockingly little on wikipedia for example.
- codetrotter 2y ago
  Yeah. Seems it doesn’t even have its own article there. Only a short mention in the main article about NTP itself: > Network Time Security (NTS) is a secure version of NTPv4 with TLS and AEAD. The main improvement over previous attempts is that a separate "key establishment" server handles the heavy asymmetric cryptography, which needs to be done only once. If the server goes down, previous users would still be able to fetch time without fear of MITM. NTS is currently supported by several time servers, including Cloudflare. It is supported by NTPSec and chrony.
  - westurner 2y ago
    "RFC 8915: Network Time Security for the Network Time Protocol" (2020) https://www.rfc-editor.org/rfc/rfc8915.html https://www.rfc-editor.org/rfc/rfc8915.html "NTS RFC Published: New Standard to Ensure Secure Time on the Internet" (2020) https://www.internetsociety.org/blog/2020/10/nts-rfc-published-new-standard-to-ensure-secure-time-on-the-internet/ https://www.internetsociety.org/blog/2020/10/nts-rfc-publish... : > NTS is basically two loosely coupled sub-protocols that together add security to NTP. NTS Key Exchange (NTS-KE) is based on TLS 1.3 and performs the initial authentication of the server and exchanges security tokens with the client. The NTP client then uses these tokens in NTP extension fields for authentication and integrity checking of the NTP protocol messages that exchange time information. From "Simple Precision Time Protocol at Meta" https://news.ycombinator.com/item?id=39306209 https://news.ycombinator.com/item?id=39306209 : > How does SPTP compare to CERN's WhiteRabbit, which is built on PTP [and NTP NTS]? White Rabbit Project: https://en.wikipedia.org/wiki/White_Rabbit_Project https://en.wikipedia.org/wiki/White_Rabbit_Project
- rnijveld 2y ago
  I'm afraid this is a pretty common sentiment. NTS has been out for several years already and is implemented in several implementations (including our ntpd-rs implementation, and others like chrony and ntpsec). Yet its usage is low and meanwhile the fully unsecured and easily spoofable NTP remains the default, in effect allowing anyone to manipulate your clock almost trivially (see our blog post about this: https://tweedegolf.nl/en/blog/121/hacking-time https://tweedegolf.nl/en/blog/121/hacking-time). Hopefully we can get NTS to the masses more quickly in the coming years and slowly start to decrease our dependency on unsigned NTP traffic, just as we did with unencrypted HTTP traffic.
- denton-scratch 2y ago
  I hadn't heard of NTS until a Debian upgrade quietly installed ntpsec. It seems to now be the Debian default.
  - rlaager 2y ago
    ntp has been replaced by ntpsec in Debian. (I am the Debian ntpsec package maintainer.) By default, NTPsec on Debian uses the NTP Pool, so no NTS. But NTPsec does support NTS if you are running your own server and supports it opt-in on the client side. As far as I know, the Debian “default” is systemd-timesyncd. That is what you get out of the box. (Though, honestly, I automate most of my Linux installs, so I don’t interact with a stock install very often.) AFAIK, systemd-timesyncd does not support NTS at all. Doing NTS on a pool would be quite complicated. The easy way is to share the same key across the pool. That is obviously not workable when pool servers are run by different people. The other way would be to have an another out-of-band protocol where the pool NTP servers share their key with the centralized pool NTS-KE servers. Nobody has built that, and it’s non-trivial.
    - tialaramex 2y ago
      Ah not quite, I think pooling would be rather easier than you've thought, there are Let's Encrypt people here, but let me explain what you'd do to have N unrelated machines which are all able to successfully claim they are some-shared-name.example Each such machine mints (as often as it wants, but at least once) a document called a Certificate Signing Request. This is a signed (thus cannot be forged) document but it's public (so it needn't be confidential) and it basically says "Here's my public key, I claim I am some-shared-name.example, and I've signed this document with my private key so you can tell it was me who made it". The centralized service collects these public documents for legitimate members of the pool and it asks a CA to issue certificates for them. The CA wants a CSR, that's literally what it asks for -- Let's Encrypt clients actually just make one for you automatically, they still need one. Then the certificates are likewise public documents and can be just provided to anybody who wants them (including the NTP pool servers they're actually for which can collect a current certificate periodically). So you're only moving two public, signed, documents, which isn't hard to get right, you should indeed probably do this out-of-band but you aren't sharing the valuable private key anywhere, that's a terrible idea as well as being hard to do correctly it's just unnecessary.

rnijveld 2y ago

I agree that amplification and reflection definitely are worries, which is why we are working towards NTS becoming a default on the internet. NTS would prevent responses by a server from a spoofed packet and at the same time would make sure that NTP clients can finally start trusting their time instead of hoping that there are no malicious actors anywhere near them. You can read about it on our blog as well: https://tweedegolf.nl/en/blog/122/a-safe-internet-requires-secure-time https://tweedegolf.nl/en/blog/122/a-safe-internet-requires-s... One thing to note about amplification: amplification has always been something that NTP developers have been especially sensitive to. I would say though that protocols like QUIC and DNS have far greater amplification risks. Meanwhile, our server implementation forces that responses can never be bigger than the requests that initiated them, meaning that no amplification is possible at all. Even if we would have allowed bigger responses, I cannot imagine NTP responses being much bigger than two or three times their related request. Meanwhile I've seen numbers for DNS all the way up to 180 times the request payload. As for your worries: I think being a little cautious keeps you alert and can prevent mistakes, but I also feel that we've gone out of our way to not do anything crazy and hopefully we will be a net positive in the end. I hope you do give us a try and let us know if you find anything suspicious. If you have any feedback we'd love to hear it!

dfc 2y ago
> I cannot imagine NTP responses being much bigger than two or three times their related request. I think you must be limiting your imagination to ntp requests related to setting the time. There are a lot of other commands in the protocol used for management and metrics. The `monlist` command was good for 200x amplification. https://blog.cloudflare.com/understanding-and-mitigating-ntp-based-ddos-attacks https://blog.cloudflare.com/understanding-and-mitigating-ntp...
- rnijveld 2y ago
  Ah right! I always forget about that since we don’t implement the management protocol in ntpd-rs. I think it’s insane that stuff should go over the same socket as the normal time messages. Something I don’t ever see us implementing.
2y ago