8 ms·

This seems like a weird place to be touting memory safety. It's ntpd, it doesn't seem like a place for any sort of attack vector and it's been running on many

This seems like a weird place to be touting memory safety. It's ntpd, it doesn't seem like a place for any sort of attack vector and it's been running on many VMs without exploding memory for a while now. I'd think there are far more critical components to rewrite in a memory safe language than the clock synchronizer.

luma 2y ago
It's present on loads of systems, it's a very common service to offer, it's a reasonably well-constrained use case, and the fact that nobody thinks about it might be a good reason to think about it. They can't boil the ocean but one service at a time is a reasonable approach. I'll flip the question around, why not start at ntpd?
- cogman10 2y ago
  > I'll flip the question around, why not start at ntpd? Easy, because there are loads of critical infrastructure written in C++ that is commonly executed on pretty much every VM and exposed in such a way that vulnerabilities are disasterous. For example, JEMalloc is used by nearly every app compiled in *nix. Perhaps systemd which is just about everywhere running everything. Maybe sshd, heaven knows it's been the root of many attacks.
  - hedora 2y ago
    There’s a nice pure-rust ssh client/server already. Systemd should just be scrapped. This week’s wtf “systemd-tmpfile —-purge” intentionally changed its behavior to “rm -rf /home”. Confirmed not-a-bug. There are dozens of other comparable screwups in that stack, even ignoring the long list of CVEs (including dns and ntp, I think). Rust can’t fix that. I haven’t heard of any issues with jemalloc, though that seems reasonable (assuming calling rust from C doesn’t break compiler inlining, etc).
    - forbiddenlake 2y ago
      > Confirmed not-a-bug. Initially closed not a bug, but then Poettering overruled that decision and implemented a fix which is already released. https://github.com/systemd/systemd/commit/e76015738942246db70f444b3567afd1b132f824 https://github.com/systemd/systemd/commit/e76015738942246db7...
    - yjftsjthsd-h 2y ago
      > There’s a nice pure-rust ssh client/server already. Prod ready, audited, non-buggy in actual use? And if so, do you have a link so I can start test-deploying it?
  - kelnos 2y ago
    > For example, JEMalloc is used by nearly every app compiled in nix.* JEmalloc is used by very very very few apps compiled for *nix. That's a conscious decision that an app developer needs to make (and few would bother without specialized needs) or a distro packager needs to shoehorn into their package builds (which most/all would not do).
oconnor663 2y ago
> it's been running on many VMs without exploding memory for a while now Most of the security bugs we hear about don't cause random crashes on otherwise healthy machines, because that tends to get them noticed and fixed. It's the ones that require complicated steps to trigger that are really scary. When I look at NTP, I see a service that: - runs as root - talks to the network - doesn't usually authenticate its traffic - uses a bespoke binary packet format - almost all network security depends on (for checking cert expiration) That looks to me like an excellent candidate for a memory-safe reimplementation.
- cogman10 2y ago
  > runs as root ntpd can (and should) run as a user > talks to the network Makes outbound requests to the network. For it to be compromised, the network itself or a downstream server needs to be compromised. That's very different from something like hosting an http server. > doesn't usually authenticate its traffic Yes it does. ntp uses TLS to communicate with it's well known locations. > uses a bespoke binary packet format Not great but also see above where it's talking to well known locations authenticated and running as a user. It's a service that to be compromised requires state level interference.
  - franga2000 2y ago
    > It's a service that to be compromised requires state level interference For servers it's definitely harder, although definitely not only state-level, but another big issue could be client-side. NTP servers can be set by DHCP, so the admin of any network you connect to could exploit such a bug against you. And once you have code execution on a desktop OS, all bets are off, even if you're not under the primary UID. It's not the most important threat vector, but it also doesn't seem as difficult as some of the other system services to rewrite, so I'd say it was a good first step for the memory-safe-everything project.
  - woodruffw 2y ago
    I don’t think NTP uses TLS by default. The closest equivalent I can find online is NTS, which was only standardized in 2020 (and for which I can’t find clear adoption numbers). (But regardless: “the transport layer is secure” is not a good reason to leave memory unsafe code on the network boundary. Conventional wisdom in these settings is to do the “defense in depth” thing and assume that transport security can be circumvented.)
jaas 2y ago
I'm the person driving this. NTP is worth moving to a memory safe language but of course it's not the single most critical thing in our entire stack to make memory safe. I don't think anyone is claiming that. It's simply the first component that got to production status, a good place to start. NTP is a component worth moving to a memory safe language because it's a widely used critical service on a network boundary. A quick Google for NTP vulnerabilities will show you that there are plenty of memory safety vulnerabilities lurking in C NTP implementations: https://www.cvedetails.com/vulnerability-list/vendor_id-2153/NTP.html https://www.cvedetails.com/vulnerability-list/vendor_id-2153... Some of these are severe, some aren't. It's only a matter of time though until another severe one pops up. I don't think any critical service on a network boundary should be written in C/C++, we know too much at this point to think that's a good idea. It will take a while to change that across the board though. If I had to pick the most important thing in the context of Let's Encrypt to move to a memory safe language it would be DNS. We have been investing heavily in Hickory DNS but it's not ready for production at Let's Encrypt yet (our usage of DNS is a bit more complex than the average use case). https://github.com/hickory-dns/hickory-dns https://github.com/hickory-dns/hickory-dns Work is proceeding at a rapid pace and I expect Hickory DNS to be deployed at Let's Encrypt in 2025.
- landmarker10 2y ago
  [flagged]
  - kaql 2y ago
    [flagged]
    - devwastaken 2y ago
      It's not censorship to be downvoted for low quality posts. Please reference "intellectual honesty" and research how to have legitimate conversations. Reddit/Twitter/4chan esque communication is not appropriate for serious spaces.
      - 2y ago
lambdaone 2y ago
NTP is a ubiquitous network service that runs directly exposed to the Internet, and that seems to me like a good thing to harden. Making NTP more secure does not stop anyone else from working on any other project.
rnijveld 2y ago
I do think that memory safety is important for any network service. The probability of something going horribly wrong when a network packet is parsed in a wrong way is just too high. NTP typically does have more access to the host OS than other daemons, with it needing to adjust the system clock. Of course, there are many other services that could be made memory safe, and maybe there is some sort of right or smart order in which we should make our core network infrastructure memory safe. But everyone has their own priorities here, and I feel like this could end up being an endless debate of whatabout-ism. There is no right place to start, other than to just start. Aside from memory safety though, I feel like our implementation has a strong focus on security in general. We try and make choices that make our implementation more robust than what was out there previously. Aside from that, I think the NTP space has had an under supply of implementations, with there only being a few major open source implementations (like ntpd, ntpsec and chrony). Meanwhile, NTP is one of those pieces of technology at the core of many of the things we do on the modern internet. Knowing the current time is one of these things you just need in order to trust many of the things we take for granted (without knowledge of the current time, your TLS connection could never be trusted). I think NTP definitely deserves this attention and could use a bunch more attention.
astrobe_ 2y ago
Agreed. It is a "good old" binary protocol, so the many gotchas of text protocols are not there.

luma 2y ago

It's present on loads of systems, it's a very common service to offer, it's a reasonably well-constrained use case, and the fact that nobody thinks about it might be a good reason to think about it. They can't boil the ocean but one service at a time is a reasonable approach. I'll flip the question around, why not start at ntpd?

cogman10 2y ago
> I'll flip the question around, why not start at ntpd? Easy, because there are loads of critical infrastructure written in C++ that is commonly executed on pretty much every VM and exposed in such a way that vulnerabilities are disasterous. For example, JEMalloc is used by nearly every app compiled in *nix. Perhaps systemd which is just about everywhere running everything. Maybe sshd, heaven knows it's been the root of many attacks.
- hedora 2y ago
  There’s a nice pure-rust ssh client/server already. Systemd should just be scrapped. This week’s wtf “systemd-tmpfile —-purge” intentionally changed its behavior to “rm -rf /home”. Confirmed not-a-bug. There are dozens of other comparable screwups in that stack, even ignoring the long list of CVEs (including dns and ntp, I think). Rust can’t fix that. I haven’t heard of any issues with jemalloc, though that seems reasonable (assuming calling rust from C doesn’t break compiler inlining, etc).
  - forbiddenlake 2y ago
    > Confirmed not-a-bug. Initially closed not a bug, but then Poettering overruled that decision and implemented a fix which is already released. https://github.com/systemd/systemd/commit/e76015738942246db70f444b3567afd1b132f824 https://github.com/systemd/systemd/commit/e76015738942246db7...
  - yjftsjthsd-h 2y ago
    > There’s a nice pure-rust ssh client/server already. Prod ready, audited, non-buggy in actual use? And if so, do you have a link so I can start test-deploying it?
- kelnos 2y ago
  > For example, JEMalloc is used by nearly every app compiled in nix.* JEmalloc is used by very very very few apps compiled for *nix. That's a conscious decision that an app developer needs to make (and few would bother without specialized needs) or a distro packager needs to shoehorn into their package builds (which most/all would not do).

oconnor663 2y ago

> it's been running on many VMs without exploding memory for a while now Most of the security bugs we hear about don't cause random crashes on otherwise healthy machines, because that tends to get them noticed and fixed. It's the ones that require complicated steps to trigger that are really scary. When I look at NTP, I see a service that: - runs as root - talks to the network - doesn't usually authenticate its traffic - uses a bespoke binary packet format - almost all network security depends on (for checking cert expiration) That looks to me like an excellent candidate for a memory-safe reimplementation.

cogman10 2y ago
> runs as root ntpd can (and should) run as a user > talks to the network Makes outbound requests to the network. For it to be compromised, the network itself or a downstream server needs to be compromised. That's very different from something like hosting an http server. > doesn't usually authenticate its traffic Yes it does. ntp uses TLS to communicate with it's well known locations. > uses a bespoke binary packet format Not great but also see above where it's talking to well known locations authenticated and running as a user. It's a service that to be compromised requires state level interference.
- franga2000 2y ago
  > It's a service that to be compromised requires state level interference For servers it's definitely harder, although definitely not only state-level, but another big issue could be client-side. NTP servers can be set by DHCP, so the admin of any network you connect to could exploit such a bug against you. And once you have code execution on a desktop OS, all bets are off, even if you're not under the primary UID. It's not the most important threat vector, but it also doesn't seem as difficult as some of the other system services to rewrite, so I'd say it was a good first step for the memory-safe-everything project.
- woodruffw 2y ago
  I don’t think NTP uses TLS by default. The closest equivalent I can find online is NTS, which was only standardized in 2020 (and for which I can’t find clear adoption numbers). (But regardless: “the transport layer is secure” is not a good reason to leave memory unsafe code on the network boundary. Conventional wisdom in these settings is to do the “defense in depth” thing and assume that transport security can be circumvented.)

jaas 2y ago

I'm the person driving this. NTP is worth moving to a memory safe language but of course it's not the single most critical thing in our entire stack to make memory safe. I don't think anyone is claiming that. It's simply the first component that got to production status, a good place to start. NTP is a component worth moving to a memory safe language because it's a widely used critical service on a network boundary. A quick Google for NTP vulnerabilities will show you that there are plenty of memory safety vulnerabilities lurking in C NTP implementations: https://www.cvedetails.com/vulnerability-list/vendor_id-2153/NTP.html https://www.cvedetails.com/vulnerability-list/vendor_id-2153... Some of these are severe, some aren't. It's only a matter of time though until another severe one pops up. I don't think any critical service on a network boundary should be written in C/C++, we know too much at this point to think that's a good idea. It will take a while to change that across the board though. If I had to pick the most important thing in the context of Let's Encrypt to move to a memory safe language it would be DNS. We have been investing heavily in Hickory DNS but it's not ready for production at Let's Encrypt yet (our usage of DNS is a bit more complex than the average use case). https://github.com/hickory-dns/hickory-dns https://github.com/hickory-dns/hickory-dns Work is proceeding at a rapid pace and I expect Hickory DNS to be deployed at Let's Encrypt in 2025.

landmarker10 2y ago
[flagged]
- kaql 2y ago
  [flagged]
  - devwastaken 2y ago
    It's not censorship to be downvoted for low quality posts. Please reference "intellectual honesty" and research how to have legitimate conversations. Reddit/Twitter/4chan esque communication is not appropriate for serious spaces.
    - 2y ago

lambdaone 2y ago

NTP is a ubiquitous network service that runs directly exposed to the Internet, and that seems to me like a good thing to harden. Making NTP more secure does not stop anyone else from working on any other project.

rnijveld 2y ago

I do think that memory safety is important for any network service. The probability of something going horribly wrong when a network packet is parsed in a wrong way is just too high. NTP typically does have more access to the host OS than other daemons, with it needing to adjust the system clock. Of course, there are many other services that could be made memory safe, and maybe there is some sort of right or smart order in which we should make our core network infrastructure memory safe. But everyone has their own priorities here, and I feel like this could end up being an endless debate of whatabout-ism. There is no right place to start, other than to just start. Aside from memory safety though, I feel like our implementation has a strong focus on security in general. We try and make choices that make our implementation more robust than what was out there previously. Aside from that, I think the NTP space has had an under supply of implementations, with there only being a few major open source implementations (like ntpd, ntpsec and chrony). Meanwhile, NTP is one of those pieces of technology at the core of many of the things we do on the modern internet. Knowing the current time is one of these things you just need in order to trust many of the things we take for granted (without knowledge of the current time, your TLS connection could never be trusted). I think NTP definitely deserves this attention and could use a bunch more attention.

astrobe_ 2y ago

Agreed. It is a "good old" binary protocol, so the many gotchas of text protocols are not there.