8 ms·

Encrypted DNS and NTP = Deadlock

139 points by cyounkins 4y ago

LinuxBender 4y ago
There may be a clunky work around, depending on what else is on the embedded devices. If it has curl, one could create a local host entry for a known site yes, bad practice I know and then curl --head over plain http to get the date header and use hwclock to bootstrap the system time prior to starting up NTP. Some NTP daemons also have a way to do something like this. Another method would be to bootstrap NTP with a few known dedicated public NTP servers that are in /etc/hosts then switch to the pool. These are all clunky methods but I have seen them used out of desperation for a myriad of "but you shouldn't do that" reasons. A cron job could check a management site for the latest configuration from a json or plain text file so that the device does not fall too far out of sync.
- RedShift1 4y ago
  There are multiple "shouldn't do that" and "bad practice" things that saved my ass out in the wild. The "don't do that" people are usually not the ones out in the field having to keep things working...
jedisct1 4y ago
You can add IP addresses of the NTP servers being used in the captive-portals.txt file, which serves records even before the operating system considers a network interface as active. Or add cert_ignore_timestamp=true to the main configuration file. Initially, cert expiration won't be checked, but as soon as a DNS server will be reachable, this feature will automatically disable itself. This is for dnscrypt-proxy. Alternative clients may have something similar.
CaliforniaKarl 4y ago
Assuming devices like this get their IP via DHCP, there is a solution that does not involve hard-coding IPs into software. DHCP option 42 (defined in RFC 2132) can be used to specify multiple NTP server IPv4 addresses. (There’s also DHCP option 4, but that’s used to specify the IP for the older RFC 868 time protocol.) DHCPv6 has option 31 for SNTP (via deprecated RFC 4075), and option 56 for NTP (via RFC 5908). So, that would probably be the best option: Get an NTP address from DHCP or DHCPv6, use that to set your clock, bring up DNS over TLS/HTTPS, re-configure NTP with your preferred source, re-sync your clock, and then continue booting!
- nine_k 4y ago
  Potentially this has a hole: if a wrong time is set initially, then wrong certificates may be trusted, so spoofed TLS sessions with compromised (and expired) certificates would work, potentially giving a way to install something malicious. It requires some massive spoofing on the outside though. If the initial NTP addresses are within a datacenter LAN (say, 10.x.x.x, etc), they are likely harder to spoof. Should be fine their case (servers).
  - cyounkins 4y ago
    Yeah I guess in general if the security of TLS depends on correct timekeeping (eg a compromised key enables an attacker to use an old cert), then in theory we should secure the time sync protocol. The NIST servers page [1] describes an authenticated+encrypted NTP for VIPs, but I don't know a solution for the layperson. [1] https://tf.nist.gov/tf-cgi/servers.cgi https://tf.nist.gov/tf-cgi/servers.cgi
    - nine_k 4y ago
      For TLS, you need a roughly correct time; being many minutes off is usually acceptable. No need for GPS clocks and other such stuff. Ideally your machine should have a functioning battety-backed RTC. The vast majority of larger machines do. In a data center, DHCP or well-known local addresses should offer hard-to-spoof poiners to local NTP servers for bootstrapping. I don't see a large problem here; a reasonable startup sequence that makes sure a correct time is set before attempting TLS connections should just work. DNS requiring TLS and thus a correct system time is slightly novel, so approaches ignoring it expectedly fail.
      - RedShift1 4y ago
        What do you do with embedded devices where the RTC battery has died? Replace the battery but you have no way to set the clock and the device won't connect to anything anymore because its clock is off, so just throw it away then? Doesn't sound environmentally friendly... And yes CR2032 batteries can last a long time but they can also fail, just a a month ago I had to replace one that was only a year old.
exabrial 4y ago
DNS over TCP/TLS is a stupid idea for a lot of reasons. First off, DNSSec already takes care of integrity protection. No need to re-implement the wheel by adding TLS on top of it… and let’s not forget we’re doing in 150 packets which one we previously did in about two packets. DNS needs to be connectionless; it’s building block protocol for TCP. DnsCurve is much closer to what we actually need.
- bcrl 4y ago
  DNS over TCP is fine, and I'd far prefer it to be the default over UDP, as any packet loss with DNS over UDP results in rather lengthy timeouts (typically >1 second). DNS over TCP fixes that (or at least reduces the delay to RTT). It's DNS over TLS that's the problem. Adding HTTPS into the mix is just sheer madness.
- cyounkins 4y ago
  I've written a bit about DNSSEC. Adoption is only around 1-5%, deployment is a footgun [1], and validation currently imposes significant performance penalties [2], although that could be improved. I've also written about performance of DNS over TLS [3] and found it to be negligible. The TLS setup is only done infrequently. [1] https://ianix.com/pub/dnssec-outages.html https://ianix.com/pub/dnssec-outages.html [2] https://cyounkins.medium.com/costs-and-benefits-of-local-dnssec-validation-53c72ca9059b https://cyounkins.medium.com/costs-and-benefits-of-local-dns... [3] https://cyounkins.medium.com/performance-of-dns-over-tls-4f4b8f938bc https://cyounkins.medium.com/performance-of-dns-over-tls-4f4...
  - gsich 4y ago
    Negligible only if you have enough DNS queries to keep the connection alive. Last I checked Quad9 and Cloudflare will close the connection quickly, regardless of EDNS keepalive setting.
- zinekeller 4y ago
  > DNS needs to be connectionless; it’s building block protocol for TCP. This doesn't make any sense as you could use DNS and TCP separately (for example finding a hostname using DNS to connect your video streaming ingestion server running via UDP and hardcoded addresses to bootstrap installation files via TCP respectively).
rsimmons 4y ago
Can we change the title to Deadclock
miyuru 4y ago
> It would be great to see Google or Cloudflare use their infrastructure to provide anycasted NTP IP addresses. Google, Cloudflare and Facebook has vanity IPv6 address, pretty sure they are all static anycast IPs. time.google.com - 2001:4860:4806:: time.cloudflare.com - 2606:4700:f1::123 time.facebook.com - 2a03:2880:ff0c::123
- WirelessGigabit 4y ago
  Mind you that Google smeared the leap second. Not sure if the others do, but I think is important to recognize when selecting an NTP.
  - zokier 4y ago
    As long as they do not start smearing leap days I don't think it matters much for bootstrapping. You only need time accurate enough for validating certs, after that you can use whatever pool you want.
  - plantain 4y ago
    Smearing is probably what most people actually want...
    - oittaa 4y ago
      Yeah, this comes up every now and then in these discussions. Both Amazon and Google have explained quite well why smearing is probably the best way to handle leap seconds.
- mike_d 4y ago
  I also run a pair of worldwide anycasted NTP instances if you don't want to deal with smearing or depending on the same company for DNS and time. 45.127.112.2 45.127.113.2
  - cyounkins 4y ago
    Cool! I searched but couldn't find any docs - looks like it might have been for ntpjs? What was the process like to get an ASN to do this?
tristor 4y ago
> It would be great to see Google or Cloudflare use their infrastructure to provide anycasted NTP IP addresses. They do. Cloudflare does anyway. time.cloudflare.com is backed by a set of anycasted IPs covering around 275 POPs.
- cyounkins 4y ago
  That's great! But I'd want to see it documented - I searched but couldn't find IP addresses, only the domain. For me time.cloudflare.com = 162.159.200.123
  - tristor 4y ago
    The great thing about anycast is the IPs are the same for everyone, it’s just which POP routing converges on that is different. DNS and anycast are combined primarily for load balancing and failover. Anyone can get and use the IPs directly with dig.
    - cyounkins 4y ago
      Right, but unless it's documented how do you know that the DNS entry won't change? They could change infra so it's DNS load-balanced instead of anycast and still at time.cloudflare.com It wasn't clear but I provided the IP so others could validate it's the same for them and (in the future) that it hasn't changed.
dblitt 4y ago
A similar issue happens when a laptop (with a bad battery) loses its time and can't connect to 802.1x WPA2 enterprise wifi. At my high school, we had laptop carts that were notorious for losing their time, and nobody could log in because they were bound to AD over wifi. The system was offline because it would reject the RADIUS certificate of the wifi network due to the time being wrong. We had to manually log in as local admin to change the time or plug them all into ethernet until they could connect to NTP.
CGamesPlay 4y ago
I think there should be an option in TLS clients to ignore expiration times, explicitly for cases like this. All other validations are performed, just the not-before and not-after times are waived. Of all the validations, this one seems like the one that causes problems in edge cases most frequently. I'm definitely not saying that expiry times should be ignored by default; just that client should have the option to do it. This is obviously a problem for embedded devices, but it even goes to the browser level: when a certificate expired yesterday but is otherwise valid, I as a user want to be able to ignore that and only that error, particularly for a pinned certificate, since it's more likely to be incompetent sysadmin than malicious attacker.
- infotogivenm 4y ago
  This would be the solution I’d go with. For the first DoH call made by NTP after startup, just ignore expiry timestamps.
raggi 4y ago
https://datatracker.ietf.org/doc/html/draft-ietf-ntp-roughtime https://datatracker.ietf.org/doc/html/draft-ietf-ntp-roughti... roughtime has been in development since agl proposed it in 2016. It'd be nice to see it get over the line - until then tlsdate is easy to apply for approximate bootstrapping.
thayne 4y ago
Another potential way to work around this is to use unencrypted DNS for the NTP lookup only. In most cases you probably don't care about your lookup for an ntp server being confidential.
gonzo 4y ago
Back in the day, Unix wrote the time of day in the superblock for the root fs before unmounting it and rebooting.
- phh 4y ago
  It still does: sudo LC_ALL=C /sbin/tune2fs -l /dev/nvme0n1 |grep 2022 tune2fs 1.46.6-rc1 (12-Sep-2022) Last mount time: Thu Dec 29 20:05:22 2022 Last write time: Fri Dec 30 11:14:35 2022 Last checked: Wed Aug 31 09:37:57 2022 (It looks like "Last write time" is actually "Last umount time", because it is not refreshed during usage of the FS)
- cyounkins 4y ago
  Interesting! Unfortunately due to write endurance many networking devices mount their filesystem read-only. This means they also sometimes lack a log file describing why the device shut down!
- Jenda_ 4y ago
  This is handled for example by fake-hwclock package in Debian (also RaspberryPi OS, installed by default) - it saves it in file, and even updates it every hour (so you won't teleport more than a hour back in time after unclean reboot). However, it of course does not work when you mount read-only because you don't want your microSD card to fail.
deleted 4y ago
[deleted]
Joel_Mckay 4y ago
In general, a time-travel sanity check at bootup is wise to handle when the RTC clock battery fails or is just installed (i.e. if the hardware jumps decades back into the past, than the local NTP/DNS/DHCP service is briefly paused while an abnormally large leap-forwards is forced from the time configuration commands. During this process, both client and server side SSL will usually be dropped due to the abnormality.) Rookie mistake, like not using UTC time on the servers. =)
- cyounkins 4y ago
  If your clock was just reset, how do you know what time it is? How do you trust that your own clock is accurate? How do you know your clock wasn't incorrect before? Usually the answer to all these is NTP.
  - Joel_Mckay 4y ago
    The epoch "zero time" on most hardware RTC is decades in the past. Thus, it is generally a safe assumption to check if the system release date is in the future. One often doesn't want to trip this fix every boot as it can have collateral consequences in poorly written software. We have a no time-travelers policy in most situations, and recommend others try to also constrain transactional-validity temporal-windows. Some hardware fakes an RTC by writing the current time to a cache file on disk on power cycles, but it would take a hour to explain why this is unwise (even if using GPS stratum time). =)
mvip 4y ago
We've seen this (and similar TLS related issues) a fair bit at Screenly when working with Raspberry Pis. The best workaround that we've found is to use the date in a HTTP Header to set the initial time (if we detect this condition): $ curl -sI http://api.screenlyapp.com http://api.screenlyapp.com | grep Date Date: Fri, 30 Dec 2022 08:43:56 GMT With this set, you should be able to trigger the initial NTP service to start and set the date. There's a Rust library here that can parse these dates for you: https://docs.rs/httpdate/0.3.2/httpdate/ https://docs.rs/httpdate/0.3.2/httpdate/
- ianai 4y ago
  Do you just pipe that into a date set command?
  - mvip 4y ago
    No, we parse it and set it over DBUS. If you want to do something more turnkey, take a look at htpdate (https://linux.die.net/man/8/htpdate https://linux.die.net/man/8/htpdate).
- ThePowerOfFuet 4y ago
  Why does Screenly have their API available on port 80 at all? Great way to leak cookies or tokens...
  - mvip 4y ago
    We obviously don't use HTTP for serving our API. The HTTP end-point is just a 301, but it still contains the Date field: $ curl -I http://api.screenlyapp.com http://api.screenlyapp.com HTTP/1.1 301 Moved Permanently Date: Fri, 30 Dec 2022 09:01:44 GMT [...]
    - ThePowerOfFuet 4y ago
      Regardless of it being a 301, the cookie or token is already sent, and thus already burned, by the time the response code is returned. Your API should not even be listening on port 80, period.
eniac111 4y ago
Happens to me several times a year. My PiHole is running on LXD :)
KirillPanov 4y ago
Same problem with Wireguard and NTP. You can't (usefully) tunnel NTP inside of wireguard, because if your clock is wrong your peers won't talk to you anymore. This is my personal pet peeve. IMHO the wireguard handshake needs to be extended to allow one peer (the one that didn't reboot) to reply to a packet with a non-monotonically-increasing nonce with some signal saying "hey, here is the last nonce I got from you". Obviously this reply would be encrypted. Then hazmat-free hardware could use these replies to reset its nonce (for that particular peer only) if we haven't had a successful handshake with that peer since the last reboot and/or the system clock is implausible. Obviously this behavior would be off-by-default. I would enable it for my batteryless routers.
- ilyt 4y ago
  I think the solution here is to get on finally securing NTP, not try to hack around it
  - KirillPanov 4y ago
    That's a fairly glib comment; have you thought it through? Most of the "secure this" wrappers like TLS+X.509 assume a clock.
remram 4y ago
I thought the title read "encrypted (DNS and NTP)", e.g. "encrypted DNS and encrypted NTP".
bobdvb 4y ago
I think hard coding IPs is generally a bad idea, it might work for one or two users, but if this became standard practice then it would cause issues. I think it would be saner to say "if you don't have valid time (e.g. less than system/kernel build date) then don't use encrypted DNS.". Then NTP domains can be looked up, the answer will be correct enough to set a clock. Alternatively it would be good to use an anycast IP for NTP. This is normally a bad idea because it makes calculating skew hard/unreliable, but that really should just mean a poorly sync'ed clock. So set the Anycast clock to be an intentionally high/poor Stratum score, list this along with a DNS based address so it's used until the encrypted DNS can be resolved with a better Stratum score. So, Dear Akamai/Cloudflare/MANGA/etc. please provide a high stratum, Anycast address for basic, approximate NTP.
notwokeno 4y ago
DNS is a public database. Encrypting it is a bit silly anyway IMO. Certainly between the recursive and authoritative resolvers.

139 points by cyounkins 4y ago

There may be a clunky work around, depending on what else is on the embedded devices. If it has curl, one could create a local host entry for a known site yes, bad practice I know and then curl --head over plain http to get the date header and use hwclock to bootstrap the system time prior to starting up NTP. Some NTP daemons also have a way to do something like this. Another method would be to bootstrap NTP with a few known dedicated public NTP servers that are in /etc/hosts then switch to the pool. These are all clunky methods but I have seen them used out of desperation for a myriad of "but you shouldn't do that" reasons. A cron job could check a management site for the latest configuration from a json or plain text file so that the device does not fall too far out of sync.

RedShift1 4y ago
There are multiple "shouldn't do that" and "bad practice" things that saved my ass out in the wild. The "don't do that" people are usually not the ones out in the field having to keep things working...

jedisct1 4y ago

You can add IP addresses of the NTP servers being used in the captive-portals.txt file, which serves records even before the operating system considers a network interface as active. Or add cert_ignore_timestamp=true to the main configuration file. Initially, cert expiration won't be checked, but as soon as a DNS server will be reachable, this feature will automatically disable itself. This is for dnscrypt-proxy. Alternative clients may have something similar.

CaliforniaKarl 4y ago

Assuming devices like this get their IP via DHCP, there is a solution that does not involve hard-coding IPs into software. DHCP option 42 (defined in RFC 2132) can be used to specify multiple NTP server IPv4 addresses. (There’s also DHCP option 4, but that’s used to specify the IP for the older RFC 868 time protocol.) DHCPv6 has option 31 for SNTP (via deprecated RFC 4075), and option 56 for NTP (via RFC 5908). So, that would probably be the best option: Get an NTP address from DHCP or DHCPv6, use that to set your clock, bring up DNS over TLS/HTTPS, re-configure NTP with your preferred source, re-sync your clock, and then continue booting!

nine_k 4y ago
Potentially this has a hole: if a wrong time is set initially, then wrong certificates may be trusted, so spoofed TLS sessions with compromised (and expired) certificates would work, potentially giving a way to install something malicious. It requires some massive spoofing on the outside though. If the initial NTP addresses are within a datacenter LAN (say, 10.x.x.x, etc), they are likely harder to spoof. Should be fine their case (servers).
- cyounkins 4y ago
  Yeah I guess in general if the security of TLS depends on correct timekeeping (eg a compromised key enables an attacker to use an old cert), then in theory we should secure the time sync protocol. The NIST servers page [1] describes an authenticated+encrypted NTP for VIPs, but I don't know a solution for the layperson. [1] https://tf.nist.gov/tf-cgi/servers.cgi https://tf.nist.gov/tf-cgi/servers.cgi
  - nine_k 4y ago
    For TLS, you need a roughly correct time; being many minutes off is usually acceptable. No need for GPS clocks and other such stuff. Ideally your machine should have a functioning battety-backed RTC. The vast majority of larger machines do. In a data center, DHCP or well-known local addresses should offer hard-to-spoof poiners to local NTP servers for bootstrapping. I don't see a large problem here; a reasonable startup sequence that makes sure a correct time is set before attempting TLS connections should just work. DNS requiring TLS and thus a correct system time is slightly novel, so approaches ignoring it expectedly fail.
    - RedShift1 4y ago
      What do you do with embedded devices where the RTC battery has died? Replace the battery but you have no way to set the clock and the device won't connect to anything anymore because its clock is off, so just throw it away then? Doesn't sound environmentally friendly... And yes CR2032 batteries can last a long time but they can also fail, just a a month ago I had to replace one that was only a year old.

exabrial 4y ago

DNS over TCP/TLS is a stupid idea for a lot of reasons. First off, DNSSec already takes care of integrity protection. No need to re-implement the wheel by adding TLS on top of it… and let’s not forget we’re doing in 150 packets which one we previously did in about two packets. DNS needs to be connectionless; it’s building block protocol for TCP. DnsCurve is much closer to what we actually need.

bcrl 4y ago
DNS over TCP is fine, and I'd far prefer it to be the default over UDP, as any packet loss with DNS over UDP results in rather lengthy timeouts (typically >1 second). DNS over TCP fixes that (or at least reduces the delay to RTT). It's DNS over TLS that's the problem. Adding HTTPS into the mix is just sheer madness.
cyounkins 4y ago
I've written a bit about DNSSEC. Adoption is only around 1-5%, deployment is a footgun [1], and validation currently imposes significant performance penalties [2], although that could be improved. I've also written about performance of DNS over TLS [3] and found it to be negligible. The TLS setup is only done infrequently. [1] https://ianix.com/pub/dnssec-outages.html https://ianix.com/pub/dnssec-outages.html [2] https://cyounkins.medium.com/costs-and-benefits-of-local-dnssec-validation-53c72ca9059b https://cyounkins.medium.com/costs-and-benefits-of-local-dns... [3] https://cyounkins.medium.com/performance-of-dns-over-tls-4f4b8f938bc https://cyounkins.medium.com/performance-of-dns-over-tls-4f4...
- gsich 4y ago
  Negligible only if you have enough DNS queries to keep the connection alive. Last I checked Quad9 and Cloudflare will close the connection quickly, regardless of EDNS keepalive setting.
zinekeller 4y ago
> DNS needs to be connectionless; it’s building block protocol for TCP. This doesn't make any sense as you could use DNS and TCP separately (for example finding a hostname using DNS to connect your video streaming ingestion server running via UDP and hardcoded addresses to bootstrap installation files via TCP respectively).

rsimmons 4y ago

Can we change the title to Deadclock

miyuru 4y ago

> It would be great to see Google or Cloudflare use their infrastructure to provide anycasted NTP IP addresses. Google, Cloudflare and Facebook has vanity IPv6 address, pretty sure they are all static anycast IPs. time.google.com - 2001:4860:4806:: time.cloudflare.com - 2606:4700:f1::123 time.facebook.com - 2a03:2880:ff0c::123

WirelessGigabit 4y ago
Mind you that Google smeared the leap second. Not sure if the others do, but I think is important to recognize when selecting an NTP.
- zokier 4y ago
  As long as they do not start smearing leap days I don't think it matters much for bootstrapping. You only need time accurate enough for validating certs, after that you can use whatever pool you want.
- plantain 4y ago
  Smearing is probably what most people actually want...
  - oittaa 4y ago
    Yeah, this comes up every now and then in these discussions. Both Amazon and Google have explained quite well why smearing is probably the best way to handle leap seconds.
mike_d 4y ago
I also run a pair of worldwide anycasted NTP instances if you don't want to deal with smearing or depending on the same company for DNS and time. 45.127.112.2 45.127.113.2
- cyounkins 4y ago
  Cool! I searched but couldn't find any docs - looks like it might have been for ntpjs? What was the process like to get an ASN to do this?

tristor 4y ago

> It would be great to see Google or Cloudflare use their infrastructure to provide anycasted NTP IP addresses. They do. Cloudflare does anyway. time.cloudflare.com is backed by a set of anycasted IPs covering around 275 POPs.

cyounkins 4y ago
That's great! But I'd want to see it documented - I searched but couldn't find IP addresses, only the domain. For me time.cloudflare.com = 162.159.200.123
- tristor 4y ago
  The great thing about anycast is the IPs are the same for everyone, it’s just which POP routing converges on that is different. DNS and anycast are combined primarily for load balancing and failover. Anyone can get and use the IPs directly with dig.
  - cyounkins 4y ago
    Right, but unless it's documented how do you know that the DNS entry won't change? They could change infra so it's DNS load-balanced instead of anycast and still at time.cloudflare.com It wasn't clear but I provided the IP so others could validate it's the same for them and (in the future) that it hasn't changed.

dblitt 4y ago

A similar issue happens when a laptop (with a bad battery) loses its time and can't connect to 802.1x WPA2 enterprise wifi. At my high school, we had laptop carts that were notorious for losing their time, and nobody could log in because they were bound to AD over wifi. The system was offline because it would reject the RADIUS certificate of the wifi network due to the time being wrong. We had to manually log in as local admin to change the time or plug them all into ethernet until they could connect to NTP.

CGamesPlay 4y ago

I think there should be an option in TLS clients to ignore expiration times, explicitly for cases like this. All other validations are performed, just the not-before and not-after times are waived. Of all the validations, this one seems like the one that causes problems in edge cases most frequently. I'm definitely not saying that expiry times should be ignored by default; just that client should have the option to do it. This is obviously a problem for embedded devices, but it even goes to the browser level: when a certificate expired yesterday but is otherwise valid, I as a user want to be able to ignore that and only that error, particularly for a pinned certificate, since it's more likely to be incompetent sysadmin than malicious attacker.

infotogivenm 4y ago
This would be the solution I’d go with. For the first DoH call made by NTP after startup, just ignore expiry timestamps.

raggi 4y ago

https://datatracker.ietf.org/doc/html/draft-ietf-ntp-roughtime https://datatracker.ietf.org/doc/html/draft-ietf-ntp-roughti... roughtime has been in development since agl proposed it in 2016. It'd be nice to see it get over the line - until then tlsdate is easy to apply for approximate bootstrapping.

thayne 4y ago

Another potential way to work around this is to use unencrypted DNS for the NTP lookup only. In most cases you probably don't care about your lookup for an ntp server being confidential.

gonzo 4y ago

Back in the day, Unix wrote the time of day in the superblock for the root fs before unmounting it and rebooting.

phh 4y ago
It still does: sudo LC_ALL=C /sbin/tune2fs -l /dev/nvme0n1 |grep 2022 tune2fs 1.46.6-rc1 (12-Sep-2022) Last mount time: Thu Dec 29 20:05:22 2022 Last write time: Fri Dec 30 11:14:35 2022 Last checked: Wed Aug 31 09:37:57 2022 (It looks like "Last write time" is actually "Last umount time", because it is not refreshed during usage of the FS)
cyounkins 4y ago
Interesting! Unfortunately due to write endurance many networking devices mount their filesystem read-only. This means they also sometimes lack a log file describing why the device shut down!
Jenda_ 4y ago
This is handled for example by fake-hwclock package in Debian (also RaspberryPi OS, installed by default) - it saves it in file, and even updates it every hour (so you won't teleport more than a hour back in time after unclean reboot). However, it of course does not work when you mount read-only because you don't want your microSD card to fail.

deleted 4y ago

[deleted]

Joel_Mckay 4y ago

In general, a time-travel sanity check at bootup is wise to handle when the RTC clock battery fails or is just installed (i.e. if the hardware jumps decades back into the past, than the local NTP/DNS/DHCP service is briefly paused while an abnormally large leap-forwards is forced from the time configuration commands. During this process, both client and server side SSL will usually be dropped due to the abnormality.) Rookie mistake, like not using UTC time on the servers. =)

cyounkins 4y ago
If your clock was just reset, how do you know what time it is? How do you trust that your own clock is accurate? How do you know your clock wasn't incorrect before? Usually the answer to all these is NTP.
- Joel_Mckay 4y ago
  The epoch "zero time" on most hardware RTC is decades in the past. Thus, it is generally a safe assumption to check if the system release date is in the future. One often doesn't want to trip this fix every boot as it can have collateral consequences in poorly written software. We have a no time-travelers policy in most situations, and recommend others try to also constrain transactional-validity temporal-windows. Some hardware fakes an RTC by writing the current time to a cache file on disk on power cycles, but it would take a hour to explain why this is unwise (even if using GPS stratum time). =)

mvip 4y ago

We've seen this (and similar TLS related issues) a fair bit at Screenly when working with Raspberry Pis. The best workaround that we've found is to use the date in a HTTP Header to set the initial time (if we detect this condition): $ curl -sI http://api.screenlyapp.com http://api.screenlyapp.com | grep Date Date: Fri, 30 Dec 2022 08:43:56 GMT With this set, you should be able to trigger the initial NTP service to start and set the date. There's a Rust library here that can parse these dates for you: https://docs.rs/httpdate/0.3.2/httpdate/ https://docs.rs/httpdate/0.3.2/httpdate/

ianai 4y ago
Do you just pipe that into a date set command?
- mvip 4y ago
  No, we parse it and set it over DBUS. If you want to do something more turnkey, take a look at htpdate (https://linux.die.net/man/8/htpdate https://linux.die.net/man/8/htpdate).
ThePowerOfFuet 4y ago
Why does Screenly have their API available on port 80 at all? Great way to leak cookies or tokens...
- mvip 4y ago
  We obviously don't use HTTP for serving our API. The HTTP end-point is just a 301, but it still contains the Date field: $ curl -I http://api.screenlyapp.com http://api.screenlyapp.com HTTP/1.1 301 Moved Permanently Date: Fri, 30 Dec 2022 09:01:44 GMT [...]
  - ThePowerOfFuet 4y ago
    Regardless of it being a 301, the cookie or token is already sent, and thus already burned, by the time the response code is returned. Your API should not even be listening on port 80, period.

eniac111 4y ago

Happens to me several times a year. My PiHole is running on LXD :)

KirillPanov 4y ago

Same problem with Wireguard and NTP. You can't (usefully) tunnel NTP inside of wireguard, because if your clock is wrong your peers won't talk to you anymore. This is my personal pet peeve. IMHO the wireguard handshake needs to be extended to allow one peer (the one that didn't reboot) to reply to a packet with a non-monotonically-increasing nonce with some signal saying "hey, here is the last nonce I got from you". Obviously this reply would be encrypted. Then hazmat-free hardware could use these replies to reset its nonce (for that particular peer only) if we haven't had a successful handshake with that peer since the last reboot and/or the system clock is implausible. Obviously this behavior would be off-by-default. I would enable it for my batteryless routers.

ilyt 4y ago
I think the solution here is to get on finally securing NTP, not try to hack around it
- KirillPanov 4y ago
  That's a fairly glib comment; have you thought it through? Most of the "secure this" wrappers like TLS+X.509 assume a clock.

remram 4y ago

I thought the title read "encrypted (DNS and NTP)", e.g. "encrypted DNS and encrypted NTP".

bobdvb 4y ago

I think hard coding IPs is generally a bad idea, it might work for one or two users, but if this became standard practice then it would cause issues. I think it would be saner to say "if you don't have valid time (e.g. less than system/kernel build date) then don't use encrypted DNS.". Then NTP domains can be looked up, the answer will be correct enough to set a clock. Alternatively it would be good to use an anycast IP for NTP. This is normally a bad idea because it makes calculating skew hard/unreliable, but that really should just mean a poorly sync'ed clock. So set the Anycast clock to be an intentionally high/poor Stratum score, list this along with a DNS based address so it's used until the encrypted DNS can be resolved with a better Stratum score. So, Dear Akamai/Cloudflare/MANGA/etc. please provide a high stratum, Anycast address for basic, approximate NTP.

notwokeno 4y ago

DNS is a public database. Encrypting it is a bit silly anyway IMO. Certainly between the recursive and authoritative resolvers.