8 ms·

A Memory Safe Implementation of the Network Time Protocol

101 points by SGran 4y ago

Thaxll 4y ago
- jeffbee 4y ago
  There's no reason to believe there isn't some vulnerability lurking in chrony, which is a highly privileged process. These are exactly the starting points I would hope for. On the other hand, the statement that they did not study chrony because they couldn't understand it does not exactly fill me with confidence.
  - kelnos 4y ago
    To be fair, chrony, at least on my system, is not running as root. I'm not sure what mechanism it uses to set the system time, whether it's capabilities, or a setuid helper program, but I think it's safe to assume that, if compromised, the only malicious thing that it could do would be to set my system time to something incorrect. Which isn't nothing, but also isn't much, either.
- bogota 4y ago
  They are justifying the rewrite by saying that it is memory safe. I don’t think its a bad goal but if they don’t do the work to maintain and package these for distros it wont make it very far. However im always surprised by how many people are fanatics of rust and they do contribute a lot to packaging at least for Gentoo which is my daily driver. I could see a lot of these taking off if it does show a reduction in attack surface and it works seamlessly as a replacement for existing solutions hopefully not forcing a new config on everyone.
  - akira2501 4y ago
    Doesn't Rust just panic and abort on runtime memory errors? So, we maybe get memory security, but we seemingly do nothing for DoS. Isn't that a large attack vector for NTP, given it's primary utility in other protocols?
    - steveklabnik 4y ago
      Rust the language doesn’t know anything about heap allocation, so in a strict sense, no. The standard library provides a number of APIs that abort on allocation failures, yes. There are currently nightly-only APIs for some of them to return Result instead, and they’ll hit stable eventually. You could also not use them if you don’t want that behavior.
    - black_puppydog 4y ago
      No, rust avoids most memory errors by proving at compile time that they don't exist. That's why it forces you to write code with lifetime annotations and such, so that thst proof becomes feasible. I say "most" because it also allows you to get out the footguns, meaning you can sidestep this proof mechanism. But you do that by declaring a block/function as "unsafe", so if you do find a memory bug, you know exactly where to start your search.
    - Veliladon 4y ago
      > Doesn't Rust just panic and abort on runtime memory errors? Technically yes if you directly access an index that's OOB but stuff like Vecs and Arrays have the get and get_mut methods which allows you to try to retrieve an element (or slice). If the element is in bounds you get a Some<T> with a reference to the element (or slice) and if it fails it'll return an Error type which can be handled rather than panicking out. It's not even slower to use the .get method rather than direct indexes because attempts to access indexes directly are implemented as .get.unwrap()
    - staticassertion 4y ago
      That's right, DoS in Rust is still a thing you can have. But it's no worse than in memory unsafe languages, since memory unsafety can also lead to DoS (and in a much worse way - there are methods for managing panics, managing segfaults is much harder).
- wyager 4y ago
  People find vulnerabilities in old software all the time.
  - mro_name 4y ago
    expat, sigh. openssl, sigh again.
- staticassertion 4y ago
  It's a networked, privileged process. If my goal were "try to ensure that more of my OS is memory safe" I'd probably start somewhere similar. A lot of this post is specifically addressing the justification for this work so idk, I'd suggest responding to that directly.
- bsder 4y ago
  NTP takes in untrusted input from the net, does complex processing on it, and requires the ability to do something administrative on your system (adjust the time). This gives NTP a lot of attack surface and makes it a likely vector for compromise. Rewriting NTP in any language safer than C/C++ is likely to be a good idea.
  - yencabulator 4y ago
    That drives home the point that NTP would likely benefit from privilege separation too. One component talks to the network, another component adjusts time. Second component would enforce gradual skewing only. (Or a time jump once in early boot, per policy. And it could enforce the max size of that jump too, implement a policy where the machine boots into repair mode if it was more than 5 minutes off the desired time, etc.) A very simple one-way data channel between the components would make attacking the second component near impossible. With that, and an attacker that completely pwns component 1: At worst they could start a very gradual skew, which could then be picked up by monitoring before it causes much of an issue.
- kelnos 4y ago
  > a rewrite for the sake of safety is not sufficient imo I would disagree; I think a rewrite for the sake of safety, especially when we are talking about a piece of server software that deals with untrusted clients, is often worth it. Certainly it isn't always: rewriting huge, complex pieces of software for the sake of anything (including safety) may just not be justifiable. Like, sure, maybe the Linux kernel would benefit from a rewrite in Rust, but I don't think that would be a good idea. NTP is a fairly small protocol, and at least the server portion -- where I'd be the most worried about memory safety issues -- can be implemented in Rust without too much difficulty. As I understand it, NTP clients can be a bit more complicated. But I still expect they're much less complicated than many other types of clients. > Are they going to maintain that forever and push Linux distro and the industry to move to that new solution? Why not? That's how open source Linux software works. Someone builds it, and if there's enough interest, it gets adopted, and attracts new contributors and maintainers over time. If there isn't enough interest, it withers away. That may not be your idea of the best use of your time (and I might agree), but who are we to tell others what to do with their time?
- tptacek 4y ago
  Please don't comment about the voting on comments. It never does any good, and it makes boring reading. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
manfre 4y ago
> Another benefit of Rust is that we can use its standard library and package ecosystem, so our NTP implementation is much smaller (hence easier to validate) than the alternatives It might be easier to validate the code in their repo, but I feel like they are ignoring the effort that would be needed to validate all of the very large number of dependencies.
- staticassertion 4y ago
  If their major concern is memory unsafety it's a lot easier. Most dependencies don't use any unsafe, and instead there's usually just a few libraries pulled in across them that do. One of the best parts of auditing rust (for memory unsafety) is that you can just "grep for unsafe" and know exactly where to start.
- strangemonad 4y ago
  There is a bunch of well funded work to tackle validating various aspects of rust, the std library, and ecosystem. For example rust-belt led by Derek Dryer https://plv.mpi-sws.org/rustbelt/ https://plv.mpi-sws.org/rustbelt/
- raggi 4y ago
  What is the most fascinating thing you learned when you read ntpd's configure script? What is the most interesting thing you learned reading glibc? Were you at all concerned when you discovered that the sources come from http-only servers and only have un-signed md5's for checksums? Did you find the support for HP-UX distracting?
  - deleted 4y ago
    [deleted]
- brundolf 4y ago
  Most of the dependencies I see listed in this project are upstanding, household-name crates. Personally I'd feel more confident using those (which have many other eyes on them) than maintaining custom in-house implementations of complex (but standard) building-blocks
nathas 4y ago
I'm really looking forward to the client implementation. I worked on a Rust NTP server where I used to work. It was truly faster than ntpd or chrony, which is a meaningful benefit when you're talking about something that sends out data about clocks and time. Unfortunately the server is relatively easy to build. The client, however, is where a LOT of the intelligence and difficulty lies.
- jandrese 4y ago
  The tricky part isn't the protocol, it's all of the interfaces with weirdo hardware clocks. Even just parsing GPGGA messages from a serial port can be tricky when you're trying to keep the timing tight.
  - tptacek 4y ago
    There is immense value in replacing simple NTP deployments, which don't interface with weirdo hardware clocks, with a memory-safe alternative; those simple deployments dwarf the weird ones. It is fine (good, even) for there to be multiple viable implementations of NTP, fit for different purposes.
- toast0 4y ago
  Assuming a pure network client, there's not that much to build if you make simplifying assumptions, and build off the work of others. You need something to get a list of servers, and maybe update the list overtime (if you're using a dns name). For each server, you need to poll to accumulate a list of delays and offset. If you want to make the delay precise, you can try to get the NIC to timestamp the NTP packets. Once you've got sufficient samples from a server, you can use a regression to approximate the time offset and frequency offset for that server. If you've got more than one server, you can choose one to follow (ala the ntp reference implementation) or merge the data from some selection. This is kind of tricky, but there are many examples to choose from. Some implementations use metrics from multiple servers to try to estimate how much of the delay is asymmetric and use that to enhance the clock precision; but the reference implementation didn't do that and most people didn't mind. Once you've figured out which time and frequency offsets to do, send it to the OS via settimeofday and ntp_adjtime. This is a bit trickier if you don't want to step the clock, but you can calculate a frequency offset to slew the clock in the direction you want over the time you find acceptable. Reference ntp drops old measurements after an adjustment, but you can also scale them by your adjustments and keep using them. You might want to have some feedback loop to adjust your polling rates, but either way. None of this code needs to be particularly fast either. Ideally, very little in between timestamp and send, and receive and timestamp (NIC timestamping helps tremendously here, if that's possible); and you also want to have a minimum of delay during offset operations too. But the protocol parsing, and offset calculations can take as long as you like. I've got a one-shot Erlang ntp client in 108 lines of Erlang, and 50 lines of nif (which is mostly unpacking the arguments) for an unpublished hobby os. It's not perfect, but it does pretty ok. If it ran continuously, it would probably meet or beat reference ntpd. Reference ntpd does a whole lot more cool stuff, of course.
infamouscow 4y ago
It's unclear why Rust was chosen over OCaml or Haskell.
- mtlmtlmtlmtl 4y ago
  It's pretty obvious actually. They want memory safety and predictable, good performance. Rust is the goto language for that in sysdev today. Go is a much better candidate than the ones you mention, but there are legitimate reasons to want to avoid GC. Additionally, the intersection of people who are interested in working on low level system daemons and people who prefer Haskell/OCaml must be pretty small compared to Rust.
  - infamouscow 4y ago
    - mtlmtlmtlmtl 4y ago
      Performance being important in system development is pretty much a given, and doesn't necessarily even need to be mentioned. You could argue that systems software is well defined as software where security, performance, and stability are more important than all other concerns. I think the fact that your comment didn't elaborate is why you're being downvoted, not for asking questions. It can be interpreted as a question, but it can also easily be interpreted as a shallow dismissal of their choice of Rust, which is a fairy common thing on HN. If you had laid out some details on why the choice puzzled you, your comment would probably have been interpreted more charitably.
      - thwarted 4y ago
        Performance is especially important when dealing with time syncing.
        tedunangst 4y ago
        I think consistency is more important than actual execution time. Python will certainly have more latency responding to time queries, but it will be consistent and likely indistinguishable from a faster program.
yuuta 4y ago
I don't see the whole point. It's like creating yet-another-ntp-implementation while other well-known implementations are known to be working good and safely on billions of devices. It is easier to report a security issue to ntpd or chrony instead of creating a new one.
- tptacek 4y ago
  The overwhelming majority of NTP deployments (by device count) don't benefit from any of the complexity or flexibility of chrony and ntpd, but do suffer from the memory-unsafety of those programs, and pass that unsafety on to the rest of us. The case for a memory-safe 80%-use-case NTP server is very strong.
  - ok_dad 4y ago
    > the memory-unsafety of those programs How do you know they are unsafe? Have they been audited and memory un-safety been found? I don't understand why the Rust community automatically assumes there are always memory safety issues with C, for example. I get that it's possible to write unsafe programs in C, but not all programs are automatically unsafe just because they are written in C. I need to do more research, because I don't actually trust that just writing a program in Rust will result in total memory safety, so these are actually questions I want answers to, not just me trying to attack Rust or anything. Thanks!
    - tptacek 4y ago
      It's not just the Rust community. I don't especially like Rust, but I fully buy into the argument that code written in memory-unsafe languages is materially less safe than code that is. There are plenty of memory-safe options, and rewriting software to be memory safe --- especially when there's a clear, simple common case to seize on --- is a positive step for Internet safety.
      - ok_dad 4y ago
        I totally get that, but my stance on the matter is that some of the software we're talking about is just as memory safe as a new Rust rewrite because it doesn't do anything unsafe, but the rewrite could introduce other bugs and differences that could break things. I would say I don't stand on the side of "rewrite nothing", but I'm more of a realist here, in that we absolutely cannot "rewrite everything" perfectly in a memory safe language, and we should first determine if a particular tool should be rewritten in a memory safe language by doing some analysis and testing on that tool. Certainly, even though I know no Rust and am not an expert in memory safety, I would say that in the future we should try not to write totally new software in memory unsafe languages, but I'm not everyone so I can't make that rule and ensure it sticks.
bitwize 4y ago
Wellp, so much for the need for NTPsec.

101 points by SGran 4y ago

jeffbee 4y ago
There's no reason to believe there isn't some vulnerability lurking in chrony, which is a highly privileged process. These are exactly the starting points I would hope for. On the other hand, the statement that they did not study chrony because they couldn't understand it does not exactly fill me with confidence.
- kelnos 4y ago
  To be fair, chrony, at least on my system, is not running as root. I'm not sure what mechanism it uses to set the system time, whether it's capabilities, or a setuid helper program, but I think it's safe to assume that, if compromised, the only malicious thing that it could do would be to set my system time to something incorrect. Which isn't nothing, but also isn't much, either.
bogota 4y ago
They are justifying the rewrite by saying that it is memory safe. I don’t think its a bad goal but if they don’t do the work to maintain and package these for distros it wont make it very far. However im always surprised by how many people are fanatics of rust and they do contribute a lot to packaging at least for Gentoo which is my daily driver. I could see a lot of these taking off if it does show a reduction in attack surface and it works seamlessly as a replacement for existing solutions hopefully not forcing a new config on everyone.
- akira2501 4y ago
  Doesn't Rust just panic and abort on runtime memory errors? So, we maybe get memory security, but we seemingly do nothing for DoS. Isn't that a large attack vector for NTP, given it's primary utility in other protocols?
  - steveklabnik 4y ago
    Rust the language doesn’t know anything about heap allocation, so in a strict sense, no. The standard library provides a number of APIs that abort on allocation failures, yes. There are currently nightly-only APIs for some of them to return Result instead, and they’ll hit stable eventually. You could also not use them if you don’t want that behavior.
  - black_puppydog 4y ago
    No, rust avoids most memory errors by proving at compile time that they don't exist. That's why it forces you to write code with lifetime annotations and such, so that thst proof becomes feasible. I say "most" because it also allows you to get out the footguns, meaning you can sidestep this proof mechanism. But you do that by declaring a block/function as "unsafe", so if you do find a memory bug, you know exactly where to start your search.
  - Veliladon 4y ago
    > Doesn't Rust just panic and abort on runtime memory errors? Technically yes if you directly access an index that's OOB but stuff like Vecs and Arrays have the get and get_mut methods which allows you to try to retrieve an element (or slice). If the element is in bounds you get a Some<T> with a reference to the element (or slice) and if it fails it'll return an Error type which can be handled rather than panicking out. It's not even slower to use the .get method rather than direct indexes because attempts to access indexes directly are implemented as .get.unwrap()
  - staticassertion 4y ago
    That's right, DoS in Rust is still a thing you can have. But it's no worse than in memory unsafe languages, since memory unsafety can also lead to DoS (and in a much worse way - there are methods for managing panics, managing segfaults is much harder).
wyager 4y ago
People find vulnerabilities in old software all the time.
- mro_name 4y ago
  expat, sigh. openssl, sigh again.
staticassertion 4y ago
It's a networked, privileged process. If my goal were "try to ensure that more of my OS is memory safe" I'd probably start somewhere similar. A lot of this post is specifically addressing the justification for this work so idk, I'd suggest responding to that directly.
bsder 4y ago
NTP takes in untrusted input from the net, does complex processing on it, and requires the ability to do something administrative on your system (adjust the time). This gives NTP a lot of attack surface and makes it a likely vector for compromise. Rewriting NTP in any language safer than C/C++ is likely to be a good idea.
- yencabulator 4y ago
  That drives home the point that NTP would likely benefit from privilege separation too. One component talks to the network, another component adjusts time. Second component would enforce gradual skewing only. (Or a time jump once in early boot, per policy. And it could enforce the max size of that jump too, implement a policy where the machine boots into repair mode if it was more than 5 minutes off the desired time, etc.) A very simple one-way data channel between the components would make attacking the second component near impossible. With that, and an attacker that completely pwns component 1: At worst they could start a very gradual skew, which could then be picked up by monitoring before it causes much of an issue.
kelnos 4y ago
> a rewrite for the sake of safety is not sufficient imo I would disagree; I think a rewrite for the sake of safety, especially when we are talking about a piece of server software that deals with untrusted clients, is often worth it. Certainly it isn't always: rewriting huge, complex pieces of software for the sake of anything (including safety) may just not be justifiable. Like, sure, maybe the Linux kernel would benefit from a rewrite in Rust, but I don't think that would be a good idea. NTP is a fairly small protocol, and at least the server portion -- where I'd be the most worried about memory safety issues -- can be implemented in Rust without too much difficulty. As I understand it, NTP clients can be a bit more complicated. But I still expect they're much less complicated than many other types of clients. > Are they going to maintain that forever and push Linux distro and the industry to move to that new solution? Why not? That's how open source Linux software works. Someone builds it, and if there's enough interest, it gets adopted, and attracts new contributors and maintainers over time. If there isn't enough interest, it withers away. That may not be your idea of the best use of your time (and I might agree), but who are we to tell others what to do with their time?
tptacek 4y ago
Please don't comment about the voting on comments. It never does any good, and it makes boring reading. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

> Another benefit of Rust is that we can use its standard library and package ecosystem, so our NTP implementation is much smaller (hence easier to validate) than the alternatives It might be easier to validate the code in their repo, but I feel like they are ignoring the effort that would be needed to validate all of the very large number of dependencies.

staticassertion 4y ago
If their major concern is memory unsafety it's a lot easier. Most dependencies don't use any unsafe, and instead there's usually just a few libraries pulled in across them that do. One of the best parts of auditing rust (for memory unsafety) is that you can just "grep for unsafe" and know exactly where to start.
strangemonad 4y ago
There is a bunch of well funded work to tackle validating various aspects of rust, the std library, and ecosystem. For example rust-belt led by Derek Dryer https://plv.mpi-sws.org/rustbelt/ https://plv.mpi-sws.org/rustbelt/
raggi 4y ago
What is the most fascinating thing you learned when you read ntpd's configure script? What is the most interesting thing you learned reading glibc? Were you at all concerned when you discovered that the sources come from http-only servers and only have un-signed md5's for checksums? Did you find the support for HP-UX distracting?
- deleted 4y ago
  [deleted]
brundolf 4y ago
Most of the dependencies I see listed in this project are upstanding, household-name crates. Personally I'd feel more confident using those (which have many other eyes on them) than maintaining custom in-house implementations of complex (but standard) building-blocks

nathas 4y ago

I'm really looking forward to the client implementation. I worked on a Rust NTP server where I used to work. It was truly faster than ntpd or chrony, which is a meaningful benefit when you're talking about something that sends out data about clocks and time. Unfortunately the server is relatively easy to build. The client, however, is where a LOT of the intelligence and difficulty lies.

jandrese 4y ago
The tricky part isn't the protocol, it's all of the interfaces with weirdo hardware clocks. Even just parsing GPGGA messages from a serial port can be tricky when you're trying to keep the timing tight.
- tptacek 4y ago
  There is immense value in replacing simple NTP deployments, which don't interface with weirdo hardware clocks, with a memory-safe alternative; those simple deployments dwarf the weird ones. It is fine (good, even) for there to be multiple viable implementations of NTP, fit for different purposes.
toast0 4y ago
Assuming a pure network client, there's not that much to build if you make simplifying assumptions, and build off the work of others. You need something to get a list of servers, and maybe update the list overtime (if you're using a dns name). For each server, you need to poll to accumulate a list of delays and offset. If you want to make the delay precise, you can try to get the NIC to timestamp the NTP packets. Once you've got sufficient samples from a server, you can use a regression to approximate the time offset and frequency offset for that server. If you've got more than one server, you can choose one to follow (ala the ntp reference implementation) or merge the data from some selection. This is kind of tricky, but there are many examples to choose from. Some implementations use metrics from multiple servers to try to estimate how much of the delay is asymmetric and use that to enhance the clock precision; but the reference implementation didn't do that and most people didn't mind. Once you've figured out which time and frequency offsets to do, send it to the OS via settimeofday and ntp_adjtime. This is a bit trickier if you don't want to step the clock, but you can calculate a frequency offset to slew the clock in the direction you want over the time you find acceptable. Reference ntp drops old measurements after an adjustment, but you can also scale them by your adjustments and keep using them. You might want to have some feedback loop to adjust your polling rates, but either way. None of this code needs to be particularly fast either. Ideally, very little in between timestamp and send, and receive and timestamp (NIC timestamping helps tremendously here, if that's possible); and you also want to have a minimum of delay during offset operations too. But the protocol parsing, and offset calculations can take as long as you like. I've got a one-shot Erlang ntp client in 108 lines of Erlang, and 50 lines of nif (which is mostly unpacking the arguments) for an unpublished hobby os. It's not perfect, but it does pretty ok. If it ran continuously, it would probably meet or beat reference ntpd. Reference ntpd does a whole lot more cool stuff, of course.

infamouscow 4y ago

It's unclear why Rust was chosen over OCaml or Haskell.

mtlmtlmtlmtl 4y ago
It's pretty obvious actually. They want memory safety and predictable, good performance. Rust is the goto language for that in sysdev today. Go is a much better candidate than the ones you mention, but there are legitimate reasons to want to avoid GC. Additionally, the intersection of people who are interested in working on low level system daemons and people who prefer Haskell/OCaml must be pretty small compared to Rust.
- infamouscow 4y ago
  - mtlmtlmtlmtl 4y ago
    Performance being important in system development is pretty much a given, and doesn't necessarily even need to be mentioned. You could argue that systems software is well defined as software where security, performance, and stability are more important than all other concerns. I think the fact that your comment didn't elaborate is why you're being downvoted, not for asking questions. It can be interpreted as a question, but it can also easily be interpreted as a shallow dismissal of their choice of Rust, which is a fairy common thing on HN. If you had laid out some details on why the choice puzzled you, your comment would probably have been interpreted more charitably.
    - thwarted 4y ago
      Performance is especially important when dealing with time syncing.
      - tedunangst 4y ago
        I think consistency is more important than actual execution time. Python will certainly have more latency responding to time queries, but it will be consistent and likely indistinguishable from a faster program.

yuuta 4y ago

I don't see the whole point. It's like creating yet-another-ntp-implementation while other well-known implementations are known to be working good and safely on billions of devices. It is easier to report a security issue to ntpd or chrony instead of creating a new one.

tptacek 4y ago
The overwhelming majority of NTP deployments (by device count) don't benefit from any of the complexity or flexibility of chrony and ntpd, but do suffer from the memory-unsafety of those programs, and pass that unsafety on to the rest of us. The case for a memory-safe 80%-use-case NTP server is very strong.
- ok_dad 4y ago
  > the memory-unsafety of those programs How do you know they are unsafe? Have they been audited and memory un-safety been found? I don't understand why the Rust community automatically assumes there are always memory safety issues with C, for example. I get that it's possible to write unsafe programs in C, but not all programs are automatically unsafe just because they are written in C. I need to do more research, because I don't actually trust that just writing a program in Rust will result in total memory safety, so these are actually questions I want answers to, not just me trying to attack Rust or anything. Thanks!
  - tptacek 4y ago
    It's not just the Rust community. I don't especially like Rust, but I fully buy into the argument that code written in memory-unsafe languages is materially less safe than code that is. There are plenty of memory-safe options, and rewriting software to be memory safe --- especially when there's a clear, simple common case to seize on --- is a positive step for Internet safety.
    - ok_dad 4y ago
      I totally get that, but my stance on the matter is that some of the software we're talking about is just as memory safe as a new Rust rewrite because it doesn't do anything unsafe, but the rewrite could introduce other bugs and differences that could break things. I would say I don't stand on the side of "rewrite nothing", but I'm more of a realist here, in that we absolutely cannot "rewrite everything" perfectly in a memory safe language, and we should first determine if a particular tool should be rewritten in a memory safe language by doing some analysis and testing on that tool. Certainly, even though I know no Rust and am not an expert in memory safety, I would say that in the future we should try not to write totally new software in memory unsafe languages, but I'm not everyone so I can't make that rule and ensure it sticks.

bitwize 4y ago

Wellp, so much for the need for NTPsec.