15 ms·

More Memory Safety for Let's Encrypt: Deploying ntpd-rs

348 points by Dunedan 2y ago

skilled 2y ago
[flagged]
- mianosm 2y ago
  The Jonestown massacre was actually grape flavor-aid: https://www.vox.com/2015/5/23/8647095/kool-aid-jonestown-flavor-aid https://www.vox.com/2015/5/23/8647095/kool-aid-jonestown-fla... They really do appear to be all in on avoiding memory leaks from C/CPP: > Over the next few years we plan to continue replacing C or C++ software with memory safe alternatives in the Let’s Encrypt infrastructure: OpenSSL and its derivatives with Rustls, our DNS software with Hickory, Nginx with River, and sudo with sudo-rs. Memory safety is just part of the overall security equation, but it’s an important part and we’re glad to be able to make these improvements. It seems like a really challenging endeavor, but I appreciate their desire to maintain uptime and a public service like they do.
- _flux 2y ago
  Is it completely unwarranted, though? It seems most of the issues listed here are indeed memory safety bugs that are more difficult to pull off in memory-safe languages such as Rust: https://www.cvedetails.com/vulnerability-list/vendor_id-2153/NTP.html https://www.cvedetails.com/vulnerability-list/vendor_id-2153...
- pjmlp 2y ago
  Because since the Morris Worm in 1988, there are still plenty of networking facing services that keep being written in C and C++, and without the necessary sanitary precautions. True, there are plenty of alternatives for many of those networking services, not necessarily Rust.
  - danudey 2y ago
    I hate writing code in golang, because I have to pepper every single function call with `if err != nil`, but then I think about how much C code I've seen that doesn't do that and I wonder how much of it should.
    - SAI_Peregrinus 2y ago
      Lots! Also lots of C code that doesn't properly check `errno` for things like `strtol` to check for errors that aren't returned directly from a function.
    - hot_gril 2y ago
      Golang's error handling is safer than C's, but it's more cumbersome than it needs to be. In high-level code, nearly every func you write can return an error, and 99% of the time you're just going to pass the error up. Webserver will catch all and send 4xx or 5xx, for example. Exceptions are a lot more convenient and encourage solid error handling. Rust chose a good in-between (the "?" unwrapping syntax). In mid or low level code, anything can still fail, but I suspect the performance impact of supporting exceptions or similar for basic operations (integer overflow, div by 0, etc) wouldn't be worthwhile vs just crashing the program or doing something else. Interestingly, Rust doesn't crash in this case: https://doc.rust-lang.org/book/ch03-02-data-types.html https://doc.rust-lang.org/book/ch03-02-data-types.html
      - pjmlp 2y ago
        Every now and then I think about forking Go and adding something like ?, and Pascal/Modula-2 like enumerations. However then I realise, why bother, and go back into using C#, Java, D instead.
        hot_gril 2y ago
        The only thing I seriously plan to use Golang for is implementing a scripting language in my spare time. Its threading model (greenthreads flexibly mapped to OS threads) is attractive for that.
- tialaramex 2y ago
  Correctness matters, in their particular game that's especially true although I'm doubtful of common insistence that it's better for this or that software to be fast than correct. Rust is really good for correctness. Take "Hello, World", the obvious toy program. Someone tried giving it various error states instead of (as would be usual) a normal happy terminal environment. In C or C++ the canonical "Hello, World" program terminates successfully despite any amount of errors, it just doesn't care about correctness. The default Rust Hello World, the one you get out of the box when you make a new project, or you'd show people on a "My First Rust Program" course, will complain about the errors when they happen. Because doing so is correct. It's the New Jersey style. The priority for these languages was simplicity of implementation. It's more important that you can cobble together a C compiler easily than that the results are useful or worthwhile. This contributed to C's survival, but we pay the price until we give it up.
  - akaletF 2y ago
    If println panics if the write fails that is kind of cheating, isn't it. Yes, toy C programs do not check the return value of printf. So what.
    - dralley 2y ago
      I would venture to say that most C programs don't check the return value of printf, including ones that ought not to be "toys"
      - akaletF 2y ago
        I don't know. Most serious programs will use write() and then check the return value. In locations where it does not matter (say a test suite that is guaranteed to signal an error but an fprintf() error message could fail in theory) not checking is fine I think. You will not see the message if you get a panic either ...
        sans-seraph 2y ago
        > Most serious programs will use write() and then check the return value. Similarly in Rust, serious programs will use writeln rather than println, and will receive the standard compiler warning if the Result produced by writeln is ignored.
- itishappy 2y ago
  > I struggle to understand why they would say that as the opening statement in such a matter-of-fact manner. TFA's second sentence explains the facts of the matter along with the flavor of Kool-Aid they stock: > The CA software itself is written in memory safe Golang, but from our server operating systems to our network equipment, lack of memory safety routinely leads to vulnerabilities that need patching.
- bakugo 2y ago
  [flagged]
- hot_gril 2y ago
  What's incorrect about that statement?
- kelnos 2y ago
  > I struggle to understand why they would say that as the opening statement in such a matter-of-fact manner. Because it is a fact. An obvious, obvious fact to anyone who has been working in this ecosystem for any amount of time. > Drinking the Rust kool-aid by the sound of it. Would you say the same thing if they'd instead decided to use golang, zig, nim, ocaml, etc.? If not, maybe consider that your emotional stance around Rust is coloring your judgment. Our modern systems are built on a house of cards where security is concerned. I agree that the "RiiR" meme is tiresome and dumb, but I'm tired of seeing report after report of new vulnerabilities found in foundational libraries and programs. The majority of those vulnerabilities are of the type that Rust won't even let you compile. Languages like C and C++ have their place, but for most applications, there are safer alternatives that don't require harsh compromises or significant trade offs.
akira2501 2y ago
Why does your ntpd have a json dependency?
- danudey 2y ago
  This is a good question to ask, especially in the age of everything pulling in every possible dependency just to get one library function or an `isNumeric()` convenience function. The answer is that there is observability functionality which provides its results as JSON output via a UNIX socket[0]. As far as I can see, there's no other JSON functionality anywhere else in the code, so this is just to allow for easily querying (and parsing) the daemon's internal state. (I'm not convinced that JSON is the way to go here, but that's the answer to the question) [0] https://docs.ntpd-rs.pendulum-project.org/development/code-structure/#observability-task https://docs.ntpd-rs.pendulum-project.org/development/code-s...
  - motrm 2y ago
    If the pieces of state are all well known at build time - and trusted in terms of their content - it may be feasible to print out JSON 'manually' as it were, instead of needing to use a JSON library, print "{" print "\"some_state\": \""; print GlobalState.Something.to_text(); print "\", "; print "\"count_of_frobs\": "; print GlobalState.FrobsCounter; print "}"; Whether it's worth doing this just to rid yourself of a dependency... who knows.
    - syncsynchalt 2y ago
      Even better to just use TSV. Hand-rolling XML or JSON is always a smell to me, even if it's visibly safe.
      - hackernudes 2y ago
        Do you mean TLV (tag-length-value)? I can't figure out what TSV is.
        2y ago
akaletF 2y ago
[flagged]
NelsonMinar 2y ago
I like the idea of NTPD in Rust. Is there anything to read about how well ntpd-rs performs? Would love a new column for chrony's comparison: https://chrony-project.org/comparison.html https://chrony-project.org/comparison.html Particularly interested in the performance stats, how well the daemon keeps time in the face of various network problems. Chrony is very good at this. Some of the other NTP implementations (not on that chart) are so bad they shouldn't be used in production.
- rnijveld 2y ago
  In our internal testing we are very close to Chrony with our synchronization performance, some of our testing data and an explanation of our algorithm is published in our repository: https://github.com/pendulum-project/ntpd-rs/tree/main/docs/algorithm https://github.com/pendulum-project/ntpd-rs/tree/main/docs/a... Given the amount of testing we (and other parties) have done, and given the strong theoretical foundation of our algorithm I’m pretty confident we’d do well in many production environments. If you do find any performance issues though, we’d love to hear about them!
  - hcfman 2y ago
    Ah okay. Nice. Love to know whether you have plans to also provide gps time synchronisation? I guess if the point was to deal with security issues surrounding network communications I’d guess the answer is no. But if the goals was also be a dominant player in time synchronisation then it might be a nice to have. BTW. Letsencrypt certificates are the best. I install them with pretty much every installation of my other software. Thanks guys.
ComputerGuru 2y ago
Unlike say, coreutils, ntp is something very far from being a solved problem and the memory safety of the solution is unfortunately going to play second fiddle to its efficacy. For example, we only use chrony because it’s so much better than whatever came with your system (especially on virtual machines). ntpd-rs would have to come at least within spitting distance of chrony’s time keeping abilities to even be up for consideration. (And I say this as a massive rust aficionado using it for both work and pleasure.)
- syncsynchalt 2y ago
  The biggest danger in NTP isn't memory safety (though good on this project for tackling it), it's (a) the inherent risks in implementing a protocol based on trivially spoofable UDP that can be used to do amplification and reflection and (b) emergent resonant behavior from your implementation that will inadvertently DDOS critical infrastructure when all 100m installed copies of your daemon decide to send a packet to NIST in the same microsecond. I'm happy to see more ntpd implementations but always a little worried.
  - timmytokyo 2y ago
    I really wish more internet infrastructure would switch to using NTS. It addresses these kinds of issues.
    - jaas 2y ago
      ntpd-rs support NTS, I agree it would be great if more people used it!
    - 1over137 2y ago
      Never heard of it. Shockingly little on wikipedia for example.
      - codetrotter 2y ago
        Yeah. Seems it doesn’t even have its own article there. Only a short mention in the main article about NTP itself: > Network Time Security (NTS) is a secure version of NTPv4 with TLS and AEAD. The main improvement over previous attempts is that a separate "key establishment" server handles the heavy asymmetric cryptography, which needs to be done only once. If the server goes down, previous users would still be able to fetch time without fear of MITM. NTS is currently supported by several time servers, including Cloudflare. It is supported by NTPSec and chrony.
cogman10 2y ago
This seems like a weird place to be touting memory safety. It's ntpd, it doesn't seem like a place for any sort of attack vector and it's been running on many VMs without exploding memory for a while now. I'd think there are far more critical components to rewrite in a memory safe language than the clock synchronizer.
- luma 2y ago
  It's present on loads of systems, it's a very common service to offer, it's a reasonably well-constrained use case, and the fact that nobody thinks about it might be a good reason to think about it. They can't boil the ocean but one service at a time is a reasonable approach. I'll flip the question around, why not start at ntpd?
  - cogman10 2y ago
    > I'll flip the question around, why not start at ntpd? Easy, because there are loads of critical infrastructure written in C++ that is commonly executed on pretty much every VM and exposed in such a way that vulnerabilities are disasterous. For example, JEMalloc is used by nearly every app compiled in *nix. Perhaps systemd which is just about everywhere running everything. Maybe sshd, heaven knows it's been the root of many attacks.
    - hedora 2y ago
      There’s a nice pure-rust ssh client/server already. Systemd should just be scrapped. This week’s wtf “systemd-tmpfile —-purge” intentionally changed its behavior to “rm -rf /home”. Confirmed not-a-bug. There are dozens of other comparable screwups in that stack, even ignoring the long list of CVEs (including dns and ntp, I think). Rust can’t fix that. I haven’t heard of any issues with jemalloc, though that seems reasonable (assuming calling rust from C doesn’t break compiler inlining, etc).
      - forbiddenlake 2y ago
        > Confirmed not-a-bug. Initially closed not a bug, but then Poettering overruled that decision and implemented a fix which is already released. https://github.com/systemd/systemd/commit/e76015738942246db70f444b3567afd1b132f824 https://github.com/systemd/systemd/commit/e76015738942246db7...
_joel 2y ago
Reading this reminded me of ntpsec, anyone actually use that?
- move-on-by 2y ago
  Yes, Debian transitioned to NTPSec with bookworm. The NTP package is just a dummy transitional package to that installs NTPsec. https://packages.debian.org/bookworm/net/ntp https://packages.debian.org/bookworm/net/ntp
  - _joel 2y ago
    Interesting, thanks
nubinetwork 2y ago
The problem with ntp isn't the client, it's the servers having to deal with forged UDP packets. Will ntpd ever become TCP-only? Sadly I'm not holding my breath. I stopped running a public stratum 3 server ~10 years ago.
- Faaak 2y ago
  On the contrary, I'm hosting a stratum 1 and 2 stratum 2s (at my previous company we offered 3 stratum 1s) on the ntp pool. It's useful, used, and still needed :-)
- brohee 2y ago
  When one can make a stratum 1 server for $100, there is very little reason for the continuous existence of public NTP servers. ISP can offer the service to their customers, and any company with a semblance of IT dept can have its own stratum 1.
  - ssl-3 2y ago
    One can build a GPS-backed stratum 1 server for a lot less than $100 in hardware (and I have done so in the past). It was a fun little project for me, but it involved a confluence of skillsets that many [especially smaller] companies may not collectively possess. And even then, it needs to be documented so someone else can work on it, and that maintenance has to actually-happen, and it needs a view of the sky in order for it to chooch, and it also needs redundancy. This takes time. (And if this IT department doesn't work for free, then the cost is very quickly a lot more than $100.) And going full-assed with one or more actually-outside antennas can also be problematic, since it's a Bad Day when (eg) grounding is done improperly and lightning comes by to fuck their shit up. And ISPs can (and certainly should!) provide decent NTP servers. That's the logical place for shared servers, network-wise. But the one's choice of ISP is often limited by geography, and the most-viable ISP here stopped publishing what their NTP servers are -- if they still exist in a form that customers can use, they don't publish this information. (It wasn't always this way and I remember using them years ago when they were more forthcoming. They were shitty NTP servers with high jitter. Good enough for some things, I suppose, but not as good as the members of the public NTP pool tend to be.) I mean: Many ISPs can't even manage to handle DNS queries quickly. When public servers like 8.8.8.8 and 1.1.1.1 (or the semi-public ones like 4.2.2.1) are faster than the ISP's own servers, then that's a problem. And it's a stupid problem, and it should not happen. But it does happen. So thus, public NTP servers are useful for many -- including those who have a tinkered-together NTP server with a GPS antenna in a window somewhere, where public servers can be used as backup. It's good to have options, and it's nice that some organizations provide some options for the greater network.
johnklos 2y ago
[flagged]
- dequan 2y ago
  I agree that it would be great if the ecosystem was a bit slower to use every new version and it does seem like things are beginning to tend in that direction as many foundational crates have begun declaring MSRVs of !LATEST. However I don't think the pace of updates really changes anything in terms of tool chain security. If Rust decided to go to a 36 week release cycle, each release would just have 6x as much stuff in it. If you can't keep up reviewing N changes in a 6 week release cycle, moving to a 6*X release cycle will not help you review N*X changes.
  - chaoskitty 2y ago
    Serious question: why does it need to change so much and so often? I know nothing about rust development, so I'm curious about why it's worlds different from the development of other toolchains.
    - sans-seraph 2y ago
      Let us be clear that the notion of "change" being referred to here is forward compatibility, not backward compatibility. The user is commenting on the fact that Rust library authors make use of new features as they become available, and as a result in order to compile Rust code you will often need a recent version of the compiler, or otherwise you will need to find an older version of the library in question. In addition Rust was born from Mozilla and imitates Firefox's rapid release schedule of one release per six weeks. This does not mean that Rust releases are substantial or that they are traumatizing, only that they are frequent. The contract with users is that Rust releases must be painless so as to not fatigue users and discourage them from upgrading. The success of this painless upgrade strategy is proved by the fact that library authors are so quick to upgrade, as mentioned. This is in contrast to other languages where historically a new version of their compiler might only be released as infrequently as once per three years. It seems that these languages have begun taking queues from Rust as even Java now releases once per six months.
xvilka 2y ago
BGP probably should be the next.
hoseja 2y ago
Free pair of knee-high socks with every cert.
mre 2y ago
I spoke with Folkert, one of the developers on this project, on the 'Rust in Production' podcast. Some of you might find it interesting: https://corrode.dev/podcast/s01e05-tweede-golf/ https://corrode.dev/podcast/s01e05-tweede-golf/
hcfman 2y ago
If you want to setup a chrony time server that maintains accuracy to within a microsecond and doesn’t do this with a network connection then you could try my sbts-aru project and just not use the audio recorder parts of it. https://github.com/hcfman/sbts-aru https://github.com/hcfman/sbts-aru It installs with a single command on all Raspberry Pi versions and takes care of all the dependencies, configuration and startup order details to install and start working with one command. It’s a sound localizing audio recorder platform and that’s why it also sets up accurate time. It’s using GPS to get its time from.

348 points by Dunedan 2y ago

skilled 2y ago

[flagged]

mianosm 2y ago
The Jonestown massacre was actually grape flavor-aid: https://www.vox.com/2015/5/23/8647095/kool-aid-jonestown-flavor-aid https://www.vox.com/2015/5/23/8647095/kool-aid-jonestown-fla... They really do appear to be all in on avoiding memory leaks from C/CPP: > Over the next few years we plan to continue replacing C or C++ software with memory safe alternatives in the Let’s Encrypt infrastructure: OpenSSL and its derivatives with Rustls, our DNS software with Hickory, Nginx with River, and sudo with sudo-rs. Memory safety is just part of the overall security equation, but it’s an important part and we’re glad to be able to make these improvements. It seems like a really challenging endeavor, but I appreciate their desire to maintain uptime and a public service like they do.
_flux 2y ago
Is it completely unwarranted, though? It seems most of the issues listed here are indeed memory safety bugs that are more difficult to pull off in memory-safe languages such as Rust: https://www.cvedetails.com/vulnerability-list/vendor_id-2153/NTP.html https://www.cvedetails.com/vulnerability-list/vendor_id-2153...
pjmlp 2y ago
Because since the Morris Worm in 1988, there are still plenty of networking facing services that keep being written in C and C++, and without the necessary sanitary precautions. True, there are plenty of alternatives for many of those networking services, not necessarily Rust.
- danudey 2y ago
  I hate writing code in golang, because I have to pepper every single function call with `if err != nil`, but then I think about how much C code I've seen that doesn't do that and I wonder how much of it should.
  - SAI_Peregrinus 2y ago
    Lots! Also lots of C code that doesn't properly check `errno` for things like `strtol` to check for errors that aren't returned directly from a function.
  - hot_gril 2y ago
    Golang's error handling is safer than C's, but it's more cumbersome than it needs to be. In high-level code, nearly every func you write can return an error, and 99% of the time you're just going to pass the error up. Webserver will catch all and send 4xx or 5xx, for example. Exceptions are a lot more convenient and encourage solid error handling. Rust chose a good in-between (the "?" unwrapping syntax). In mid or low level code, anything can still fail, but I suspect the performance impact of supporting exceptions or similar for basic operations (integer overflow, div by 0, etc) wouldn't be worthwhile vs just crashing the program or doing something else. Interestingly, Rust doesn't crash in this case: https://doc.rust-lang.org/book/ch03-02-data-types.html https://doc.rust-lang.org/book/ch03-02-data-types.html
    - pjmlp 2y ago
      Every now and then I think about forking Go and adding something like ?, and Pascal/Modula-2 like enumerations. However then I realise, why bother, and go back into using C#, Java, D instead.
      - hot_gril 2y ago
        The only thing I seriously plan to use Golang for is implementing a scripting language in my spare time. Its threading model (greenthreads flexibly mapped to OS threads) is attractive for that.
tialaramex 2y ago
Correctness matters, in their particular game that's especially true although I'm doubtful of common insistence that it's better for this or that software to be fast than correct. Rust is really good for correctness. Take "Hello, World", the obvious toy program. Someone tried giving it various error states instead of (as would be usual) a normal happy terminal environment. In C or C++ the canonical "Hello, World" program terminates successfully despite any amount of errors, it just doesn't care about correctness. The default Rust Hello World, the one you get out of the box when you make a new project, or you'd show people on a "My First Rust Program" course, will complain about the errors when they happen. Because doing so is correct. It's the New Jersey style. The priority for these languages was simplicity of implementation. It's more important that you can cobble together a C compiler easily than that the results are useful or worthwhile. This contributed to C's survival, but we pay the price until we give it up.
- akaletF 2y ago
  If println panics if the write fails that is kind of cheating, isn't it. Yes, toy C programs do not check the return value of printf. So what.
  - dralley 2y ago
    I would venture to say that most C programs don't check the return value of printf, including ones that ought not to be "toys"
    - akaletF 2y ago
      I don't know. Most serious programs will use write() and then check the return value. In locations where it does not matter (say a test suite that is guaranteed to signal an error but an fprintf() error message could fail in theory) not checking is fine I think. You will not see the message if you get a panic either ...
      - sans-seraph 2y ago
        > Most serious programs will use write() and then check the return value. Similarly in Rust, serious programs will use writeln rather than println, and will receive the standard compiler warning if the Result produced by writeln is ignored.
itishappy 2y ago
> I struggle to understand why they would say that as the opening statement in such a matter-of-fact manner. TFA's second sentence explains the facts of the matter along with the flavor of Kool-Aid they stock: > The CA software itself is written in memory safe Golang, but from our server operating systems to our network equipment, lack of memory safety routinely leads to vulnerabilities that need patching.
bakugo 2y ago
[flagged]
hot_gril 2y ago
What's incorrect about that statement?
kelnos 2y ago
> I struggle to understand why they would say that as the opening statement in such a matter-of-fact manner. Because it is a fact. An obvious, obvious fact to anyone who has been working in this ecosystem for any amount of time. > Drinking the Rust kool-aid by the sound of it. Would you say the same thing if they'd instead decided to use golang, zig, nim, ocaml, etc.? If not, maybe consider that your emotional stance around Rust is coloring your judgment. Our modern systems are built on a house of cards where security is concerned. I agree that the "RiiR" meme is tiresome and dumb, but I'm tired of seeing report after report of new vulnerabilities found in foundational libraries and programs. The majority of those vulnerabilities are of the type that Rust won't even let you compile. Languages like C and C++ have their place, but for most applications, there are safer alternatives that don't require harsh compromises or significant trade offs.

akira2501 2y ago

Why does your ntpd have a json dependency?

danudey 2y ago
This is a good question to ask, especially in the age of everything pulling in every possible dependency just to get one library function or an `isNumeric()` convenience function. The answer is that there is observability functionality which provides its results as JSON output via a UNIX socket[0]. As far as I can see, there's no other JSON functionality anywhere else in the code, so this is just to allow for easily querying (and parsing) the daemon's internal state. (I'm not convinced that JSON is the way to go here, but that's the answer to the question) [0] https://docs.ntpd-rs.pendulum-project.org/development/code-structure/#observability-task https://docs.ntpd-rs.pendulum-project.org/development/code-s...
- motrm 2y ago
  If the pieces of state are all well known at build time - and trusted in terms of their content - it may be feasible to print out JSON 'manually' as it were, instead of needing to use a JSON library, print "{" print "\"some_state\": \""; print GlobalState.Something.to_text(); print "\", "; print "\"count_of_frobs\": "; print GlobalState.FrobsCounter; print "}"; Whether it's worth doing this just to rid yourself of a dependency... who knows.
  - syncsynchalt 2y ago
    Even better to just use TSV. Hand-rolling XML or JSON is always a smell to me, even if it's visibly safe.
    - hackernudes 2y ago
      Do you mean TLV (tag-length-value)? I can't figure out what TSV is.
      - 2y ago

akaletF 2y ago

[flagged]

NelsonMinar 2y ago

I like the idea of NTPD in Rust. Is there anything to read about how well ntpd-rs performs? Would love a new column for chrony's comparison: https://chrony-project.org/comparison.html https://chrony-project.org/comparison.html Particularly interested in the performance stats, how well the daemon keeps time in the face of various network problems. Chrony is very good at this. Some of the other NTP implementations (not on that chart) are so bad they shouldn't be used in production.

rnijveld 2y ago
In our internal testing we are very close to Chrony with our synchronization performance, some of our testing data and an explanation of our algorithm is published in our repository: https://github.com/pendulum-project/ntpd-rs/tree/main/docs/algorithm https://github.com/pendulum-project/ntpd-rs/tree/main/docs/a... Given the amount of testing we (and other parties) have done, and given the strong theoretical foundation of our algorithm I’m pretty confident we’d do well in many production environments. If you do find any performance issues though, we’d love to hear about them!
- hcfman 2y ago
  Ah okay. Nice. Love to know whether you have plans to also provide gps time synchronisation? I guess if the point was to deal with security issues surrounding network communications I’d guess the answer is no. But if the goals was also be a dominant player in time synchronisation then it might be a nice to have. BTW. Letsencrypt certificates are the best. I install them with pretty much every installation of my other software. Thanks guys.

ComputerGuru 2y ago

Unlike say, coreutils, ntp is something very far from being a solved problem and the memory safety of the solution is unfortunately going to play second fiddle to its efficacy. For example, we only use chrony because it’s so much better than whatever came with your system (especially on virtual machines). ntpd-rs would have to come at least within spitting distance of chrony’s time keeping abilities to even be up for consideration. (And I say this as a massive rust aficionado using it for both work and pleasure.)

syncsynchalt 2y ago
The biggest danger in NTP isn't memory safety (though good on this project for tackling it), it's (a) the inherent risks in implementing a protocol based on trivially spoofable UDP that can be used to do amplification and reflection and (b) emergent resonant behavior from your implementation that will inadvertently DDOS critical infrastructure when all 100m installed copies of your daemon decide to send a packet to NIST in the same microsecond. I'm happy to see more ntpd implementations but always a little worried.
- timmytokyo 2y ago
  I really wish more internet infrastructure would switch to using NTS. It addresses these kinds of issues.
  - jaas 2y ago
    ntpd-rs support NTS, I agree it would be great if more people used it!
  - 1over137 2y ago
    Never heard of it. Shockingly little on wikipedia for example.
    - codetrotter 2y ago
      Yeah. Seems it doesn’t even have its own article there. Only a short mention in the main article about NTP itself: > Network Time Security (NTS) is a secure version of NTPv4 with TLS and AEAD. The main improvement over previous attempts is that a separate "key establishment" server handles the heavy asymmetric cryptography, which needs to be done only once. If the server goes down, previous users would still be able to fetch time without fear of MITM. NTS is currently supported by several time servers, including Cloudflare. It is supported by NTPSec and chrony.

cogman10 2y ago

This seems like a weird place to be touting memory safety. It's ntpd, it doesn't seem like a place for any sort of attack vector and it's been running on many VMs without exploding memory for a while now. I'd think there are far more critical components to rewrite in a memory safe language than the clock synchronizer.

luma 2y ago
It's present on loads of systems, it's a very common service to offer, it's a reasonably well-constrained use case, and the fact that nobody thinks about it might be a good reason to think about it. They can't boil the ocean but one service at a time is a reasonable approach. I'll flip the question around, why not start at ntpd?
- cogman10 2y ago
  > I'll flip the question around, why not start at ntpd? Easy, because there are loads of critical infrastructure written in C++ that is commonly executed on pretty much every VM and exposed in such a way that vulnerabilities are disasterous. For example, JEMalloc is used by nearly every app compiled in *nix. Perhaps systemd which is just about everywhere running everything. Maybe sshd, heaven knows it's been the root of many attacks.
  - hedora 2y ago
    There’s a nice pure-rust ssh client/server already. Systemd should just be scrapped. This week’s wtf “systemd-tmpfile —-purge” intentionally changed its behavior to “rm -rf /home”. Confirmed not-a-bug. There are dozens of other comparable screwups in that stack, even ignoring the long list of CVEs (including dns and ntp, I think). Rust can’t fix that. I haven’t heard of any issues with jemalloc, though that seems reasonable (assuming calling rust from C doesn’t break compiler inlining, etc).
    - forbiddenlake 2y ago
      > Confirmed not-a-bug. Initially closed not a bug, but then Poettering overruled that decision and implemented a fix which is already released. https://github.com/systemd/systemd/commit/e76015738942246db70f444b3567afd1b132f824 https://github.com/systemd/systemd/commit/e76015738942246db7...

_joel 2y ago

Reading this reminded me of ntpsec, anyone actually use that?

move-on-by 2y ago
Yes, Debian transitioned to NTPSec with bookworm. The NTP package is just a dummy transitional package to that installs NTPsec. https://packages.debian.org/bookworm/net/ntp https://packages.debian.org/bookworm/net/ntp
- _joel 2y ago
  Interesting, thanks

nubinetwork 2y ago

The problem with ntp isn't the client, it's the servers having to deal with forged UDP packets. Will ntpd ever become TCP-only? Sadly I'm not holding my breath. I stopped running a public stratum 3 server ~10 years ago.

Faaak 2y ago
On the contrary, I'm hosting a stratum 1 and 2 stratum 2s (at my previous company we offered 3 stratum 1s) on the ntp pool. It's useful, used, and still needed :-)
brohee 2y ago
When one can make a stratum 1 server for $100, there is very little reason for the continuous existence of public NTP servers. ISP can offer the service to their customers, and any company with a semblance of IT dept can have its own stratum 1.
- ssl-3 2y ago
  One can build a GPS-backed stratum 1 server for a lot less than $100 in hardware (and I have done so in the past). It was a fun little project for me, but it involved a confluence of skillsets that many [especially smaller] companies may not collectively possess. And even then, it needs to be documented so someone else can work on it, and that maintenance has to actually-happen, and it needs a view of the sky in order for it to chooch, and it also needs redundancy. This takes time. (And if this IT department doesn't work for free, then the cost is very quickly a lot more than $100.) And going full-assed with one or more actually-outside antennas can also be problematic, since it's a Bad Day when (eg) grounding is done improperly and lightning comes by to fuck their shit up. And ISPs can (and certainly should!) provide decent NTP servers. That's the logical place for shared servers, network-wise. But the one's choice of ISP is often limited by geography, and the most-viable ISP here stopped publishing what their NTP servers are -- if they still exist in a form that customers can use, they don't publish this information. (It wasn't always this way and I remember using them years ago when they were more forthcoming. They were shitty NTP servers with high jitter. Good enough for some things, I suppose, but not as good as the members of the public NTP pool tend to be.) I mean: Many ISPs can't even manage to handle DNS queries quickly. When public servers like 8.8.8.8 and 1.1.1.1 (or the semi-public ones like 4.2.2.1) are faster than the ISP's own servers, then that's a problem. And it's a stupid problem, and it should not happen. But it does happen. So thus, public NTP servers are useful for many -- including those who have a tinkered-together NTP server with a GPS antenna in a window somewhere, where public servers can be used as backup. It's good to have options, and it's nice that some organizations provide some options for the greater network.

johnklos 2y ago

[flagged]

dequan 2y ago
I agree that it would be great if the ecosystem was a bit slower to use every new version and it does seem like things are beginning to tend in that direction as many foundational crates have begun declaring MSRVs of !LATEST. However I don't think the pace of updates really changes anything in terms of tool chain security. If Rust decided to go to a 36 week release cycle, each release would just have 6x as much stuff in it. If you can't keep up reviewing N changes in a 6 week release cycle, moving to a 6*X release cycle will not help you review N*X changes.
- chaoskitty 2y ago
  Serious question: why does it need to change so much and so often? I know nothing about rust development, so I'm curious about why it's worlds different from the development of other toolchains.
  - sans-seraph 2y ago
    Let us be clear that the notion of "change" being referred to here is forward compatibility, not backward compatibility. The user is commenting on the fact that Rust library authors make use of new features as they become available, and as a result in order to compile Rust code you will often need a recent version of the compiler, or otherwise you will need to find an older version of the library in question. In addition Rust was born from Mozilla and imitates Firefox's rapid release schedule of one release per six weeks. This does not mean that Rust releases are substantial or that they are traumatizing, only that they are frequent. The contract with users is that Rust releases must be painless so as to not fatigue users and discourage them from upgrading. The success of this painless upgrade strategy is proved by the fact that library authors are so quick to upgrade, as mentioned. This is in contrast to other languages where historically a new version of their compiler might only be released as infrequently as once per three years. It seems that these languages have begun taking queues from Rust as even Java now releases once per six months.

xvilka 2y ago

BGP probably should be the next.

hoseja 2y ago

Free pair of knee-high socks with every cert.

mre 2y ago

I spoke with Folkert, one of the developers on this project, on the 'Rust in Production' podcast. Some of you might find it interesting: https://corrode.dev/podcast/s01e05-tweede-golf/ https://corrode.dev/podcast/s01e05-tweede-golf/

hcfman 2y ago

If you want to setup a chrony time server that maintains accuracy to within a microsecond and doesn’t do this with a network connection then you could try my sbts-aru project and just not use the audio recorder parts of it. https://github.com/hcfman/sbts-aru https://github.com/hcfman/sbts-aru It installs with a single command on all Raspberry Pi versions and takes care of all the dependencies, configuration and startup order details to install and start working with one command. It’s a sound localizing audio recorder platform and that’s why it also sets up accurate time. It’s using GPS to get its time from.