15 ms·

Going in circles without a real-time clock

230 points by X-Cubed 2y ago

koala_man 2y ago
tl;dr: machine is unable synchronize its clock because it won't trust the NTP server's DNSSEC certificate because its clock isn't synchronized. Oof.
- iforgotpassword 2y ago
  Yeah in hindsight it might not be the best idea to enable this for ntp.org, especially when the ntp protocol itself isn't secure.
- kakkoko 2y ago
  https://github.com/systemd/systemd/commit/abf4e5c1d3ad767bc0ed67883e8e4d916af095ec https://github.com/systemd/systemd/commit/abf4e5c1d3ad767bc0... While systemd-timesyncd contains a workaround for this, chrony does not seem to.
- jandrese 2y ago
  This is the perennial problem with trying to secure NTP. I've also seen people suggest that we just use HTTPS for time sync, not realizing that if your clocks are too far off you can't make a HTTPS connection.
  - toast0 2y ago
    It takes more creativity, but you can start to make a narrowing range of times if you're careful. You'll have to ignore DNSSEC time based validation while you start up, of course... Start with the last time you were properly synced: it's most likely less than a month behind then. Then take a sampling of certificates offered from well known sites. If the certificates validate, other than time, no reasonable CA that you trust issues not-before dates in the future, so it's not before the most recent not-before date in a certificate. It's probably not much after the not-after dates either, well know sites tend to replace their certificates (but maybe you're getting MITMed with a compromised, old certificate). Maybe ask for OSCP stapling, which should get you a more narrow range of times. I think OSCP responses are valid for a week? But if you get several from different https servers, chances are good you'll narrow the range. In the case as suggested that the time was only off by about a day, IMHO, something is seriously wrong if it can't get to sync from there, but I guess I'm not really surprised either.
  - creeble 2y ago
    You can use plain HTTP for time sync. Almost all HTTP servers respond with a time header.
    - jandrese 2y ago
      The whole point of the exercise was to make it secure though. If you don't care about MITM attackers then NTP works great.
      - numpad0 2y ago
        I don't get the supposed security aspect of getting false time. Hacker gives you 1980-01-01 or 4096-13-32 and mess up CRL and ruin your day...how. Years ago I've tried privilege escalation exploit to play with a phone and it involved rolling back date to unexpire signature, so I know there is exploit potential, but it... it just feels like RTC bootstrap problem should be something solvable.
      - creeble 2y ago
        Isn't that a provably pointless exercise though? Security protocols (at least the ones in common use) require certificates or keys that eventually expire, because of the risk of a permanent key being compromised. If they expire, the protocol needs time. QED.
- deleted 2y ago
  [deleted]
lifthrasiir 2y ago
> By "everything else", I also mean WireGuard. Did you know that if your machine gets far enough out of sync, that'll stop working, too? I had no idea that it apparently includes time in its crypto stuff, but what other explanation is there? WireGuard only requires a monotonic clock, because it periodically rotates keys to provide the forward secrecy. Peer clocks are otherwise not required to be synchronized [1]. I guess the clock had a higher rate than usual and it couldn't be corrected due to the lack of RTC? [1] https://www.wireguard.com/papers/wireguard.pdf#page=7 https://www.wireguard.com/papers/wireguard.pdf#page=7 "In fact, it does not even have to be an accurate timestamp; it simply must be a per-peer monotonically increasing 96-bit number."
- josephg 2y ago
  Why does it use the time then? Why not just increment its own 96 bit number whenever you use it?
  - lifthrasiir 2y ago
    Because it is required to be monotonic per peer. WireGuard has no intrinsic states, so multiple machines with the same peer key will be seen as a single peer and that is actually a legitimate use of WireGuard. These machines would have to be synchronized to each other (but not necessarily to the external clock), and using a time is a straightforward and reasonable way to ensure this.
    - apexalpha 2y ago
      So could you share a single set of keys on multiple machines and Wireguard will work, as long as all machines use the same ntp server?
      - lifthrasiir 2y ago
        Technically yes, but I don't think (but haven't exactly confirmed) that you can initiate a new session to itself.
wodenokoto 2y ago
What is she referring to by “THE ONE”?
- semireg 2y ago
  The know it all. This “wouldn’t have happened to them because they are” THE ONE. Also known as, “that guy” as in, “don’t be THAT GUY.”
  - k1t 2y ago
    As sibling comment somewhat points out, she is literally referencing The One (2001) https://m.imdb.com/title/tt0267804/ https://m.imdb.com/title/tt0267804/
- bjackman 2y ago
  It's pre-empting the sibling comment saying "RTC module is $1"
- dmvdoug 2y ago
  See, e.g., https://rachelbythebay.com/w/2018/04/28/meta/ https://rachelbythebay.com/w/2018/04/28/meta/
  - Joker_vD 2y ago
    Amazing. So when you read about some weird (and mostly self-inflicted) problem you've never had in your life, you are not allowed to express your surprise because by doing that you're implying you're better than other people, and that would be horrible (the implying part is what is horrible, I think? Or maybe actually being better is also wrong? Can you even be better―yeah, I guess it's possible to be better than someone else at something).
    - Ensorceled 2y ago
      You're totally allowed to express your surprise, you're totally allowed to be a condescending jerk while doing it, you're totally allowed to call it "self-inflicted". The author is allowed to ask you to shut up and go away if you do.
M95D 2y ago
RTC module for Raspberry is ~ $1.
- icehawk 2y ago
  and you certaintly get what you pay for, DS3231s fry almost as easily as MAX3232s, (And you can do the same thing were you write the time to the RTC before you have the correct time)
  - Joel_Mckay 2y ago
    I have used these in product designs before with zero failures. Where exactly did you buy your samples? We found they drift less than 28s a year in standalone mode. Also, were repeatable within +-18ms when using a NTP client. Microchip RTCs are harder to setup, but also seem to work just as well (ignoring the goofy epoch). Chip shortage caused people to do unspeakable things.... ;-)
    - hcfman 2y ago
      You could buy an rtc and put up with that kind of clock drift, or.... for the same money or less you could be a GPS and have none of that drift.
      - r2_pilot 2y ago
        (a bit more money for the GPS to be fair) not even mentioning the fact that you should have clear-ish access to the sky, the time taken for you to receive the time data, and the possibility of jamming. I could go on but the two methods have differing modes of operation and failure despite ultimately being capable of fulfilling the need for a reasonably synchronized clock
  - M95D 2y ago
    I have a DS1307 installed ever since I bought a RPi3B+. It didn't fail and it's accurate enough until NTP updates the time. Are you sure you connected it correctly?
jlubawy 2y ago
My raspberry pi is also currently dead, something it didn't like about updating the OS caused it to be bricked. All updates were done within the OS, so didn't expect anything bad to happen. I could reflash the OS, but I also "don't have the time". I'll just use my laptop until I do have the time someday to resurrect it.
- hcfman 2y ago
  Quite likely installing the sbts-aru as a base project would have prevented this as well. Loads of SD cards get corrupted historically without apparent good reason. Using a memory-based overlayFS file system solves so many of these problems. sbts-aru, as well as setting up a stratum #1 time server sets up a resilient memory overlayFS as a base.
riobard 2y ago
Speaking of RTC battery, I've recently come to the realization that I have to make sure that BIOS battery is not absolutely dead in always-on PC boxes. Background: I use an x86 box as home router. I've changed the configuration in the BIOS that it should automatically boot up on power. However if the BIOS battery is dead, the config will be lost and it will revert to default settings, which is not to boot on power. But there is no way to know how much juice is left in the BIOS battery, thus there is no way to issue warnings that the battery should be replaced soon. If there is power loss AND the battery is dead, when the power comes back up again, the router will just stay off indefinitely until it's manually turned on. That's when I realized why most consumer routers do not feature RTC nor do they require batteries.
- qiqitori 2y ago
  Some unsolicited solutions, maybe not for you but somebody in a similar situation: maybe replace it with a supercapacitor (needs some wiring changes, otherwise it won't ever be charged, should last long enough for most power outages), or use a stack of coin cells in parallel (difficult due to physical dimensions, and they'll still go dead at some point). You can also short two wires on the ATX supply to automatically turn on the computer when power is supplied (which may not be good enough if you have to press a key during boot to dismiss the low battery warning). Or hack the BIOS :)
  - riobard 2y ago
    Somehow I feel reliability might take a hit… :|
  - mkesper 2y ago
    If using supercapacitors, use some known not for leaking else you're back to square one (do a search for capacitor recap).
    - duskwuff 2y ago
      That issue doesn't apply to supercapacitors (nor to any capacitor manufactured in the last 15-20 years, really). It affected some aluminum electrolytic capacitors manufactured with a defective electrolyte in the early 2000s.
Joel_Mckay 2y ago
If you are running Bind or SSL/x.509 certs, than an RTC is pretty much a requirement on the raspberry pi. Network time leaps over NTP even when working (first big leap was allowed etc.) can still bork a lot of services. Local stratum 1 GPS time (even with PPS pin) servers based on Pi's can suffer similar issues when they lose satellite lock (weather etc.) Have a great day =)
- hcfman 2y ago
  Having said that, I've never personally seen my stratum #1 Pi loose sync due to weather and I run about 4 of em here. I also use a cheap 7 euro ublox 7m. So long as you position it or an antennae with a reasonable view of the sky that shouldn't happen I reckon. Why all the complex solutions when it's so easy and cheap now to install a stratum #1 time server, even a Pi 3 will suffice. It works on a Pi Zero, but then you only have a wifi link. The sbts-au project make's it incredibly easy to setup. git lone and then install. One command, it's that simple.
  - teruakohatu 2y ago
    Any advice for a cheap USB GPS for a stratum #1 time server? EDIT: Any any advice on how essential PPS is?
    - Joel_Mckay 2y ago
      The PPS can be good (pi kernel support is stable), but one needs to be careful as some modules PPS signal has a great deal of instability. Some U-Blox modules support the additional ground based navigation signal and cellular beacons. This is recommended in addition to the 3 satellite networks classes, but note some modules will not work with some antennas (get the unified hardware with built in antenna/lna if unsure about RF notes). Best regards, =)
      - hcfman 2y ago
        If you have good sky visibility, then I don't need an antenna. Though this often tends to mean putting a Pi outside. I'm also using just the first Antenna I find actually.
mplaisier 2y ago
I ran into a similar problem on a Raspberry Pi, where my program would ignore data because it was deemed too old. It turns out a Pi uses fake-hwclock. This tool stores a timestamp regularly in a file. On boot the Pi reads from that file to set the time. When the Pi has been turned off for longer then the initial time can be hours or days out of sync. The Pi then takes a couple of seconds to obtain a valid time via NTP, but my program ran quickly enough to work with the old timestamp. After a couple of minutes it would typically work again.
zokier 2y ago
It's unfortunate that using DHCP to get rough time isn't more popular, especially for devices without RTC. Sure DHCP can be spoofed and should be considered more of a last resort, but it sure would be better than nothing and ending up in this kind of a situation where nothing works anymore. Previous discussion on similar thing: "Encrypted DNS + NTP = Deadlock" https://news.ycombinator.com/item?id=34177331 https://news.ycombinator.com/item?id=34177331
CaliforniaKarl 2y ago
I know I’ve brought this up before, but I can’t remember exactly where. It would be nice if DHCP could provide the current time as one of the pieces of information it provides with a lease. If you’re already trusting it for your IP address, you might as well trust it with the current UTC time: The current 64-bit timestamp would be enough to get timestamp-sensitive things working.
- citrin_ru 2y ago
  I trust a free wifi hotspot to give me an IP (traffic will be protected by TLS), but not with the time. E. g. time from the past can be used to trick into accepting an expired TLS certificate. But for most wifi hotspots it will be unintentionally wrong because such devices often unmonitored and not maintained until they completely break.
- mapreduce 2y ago
  > If you’re already trusting it for your IP address, you might as well trust it with the current UTC time I don't follow! How does trusting DHCP with IP address automatically mean I should trust it with the current UTC time? Time requires higher degree of trust than IP. I may not care what my IP address looks like but I might care a lot about what the current UTC time is and I might want this to come from a more trustworthy device.
  - Terr_ 2y ago
    To offer an example, if an attacker can manipulate the time then they can make the target accept expired or revoked cryptographic certificates, potentially enabling impersonation or man-in-the-middle attacks. In contrast, a different IP than expected isn't such a big deal... Although it might break the collaboration of two computers as crude denial of service
  - mikepurvis 2y ago
    DHCP servers can command your dns config and hostname too. It's not a total mitm story since the certs are still on the machine itself but it's definitely more than just a local IP address.
cjbillington 2y ago
The wireguard problem is a pain in the neck. Even if you're happy for your system to not have a realtime clock, when it does come online you'd want wireguard to not start until after the clock has synced to a timeserver. The below is what I'm doing at the moment, but I can't say I'm sure it's working - haven't seen wireguard start before the time is synced since doing it, but it could be probabilistic: systemctl enable systemd-time-wait-sync.service mkdir -p /etc/systemd/system/wg-quick@wg0.service.d/ echo "[Unit] After=time-sync.target Wants=time-sync.target " > /etc/systemd/system/wg-quick@wg0.service.d/override.conf I'd be interested if anyone could let me know if they think this is likely to be achieving what I want or not.
- hcfman 2y ago
  Well... I see where you are going with this but is it necessary? What happens if the clock gets it's correct time sync back, will not wireguard come back online by itself ? Then the real solution might involve ensuring good clock sync. Using a GPS synched time source is good for this, such as this one :) https://github.com/hcfman/sbts-aru https://github.com/hcfman/sbts-aru
  - cjbillington 2y ago
    No, I can't say I understand it fully - but if the clock goes backwards, peers will not talk to a client again until I restart the other peers' wireguard services. No matter if the client's clock is subsequently correct. For some people, all their network access goes over wireguard and that includes time syncing. That's not the case for me, but for those people if wireguard isn't working their clock can't sync. That GPS receiver looks cool! However an RTC and a battery will do just fine for me, which is the plan.
- yjftsjthsd-h 2y ago
  That was my thought as well - if we know certain things need time to be in sync, can we use systemd dependencies to enforce that? I'm not sure it would help with OP's problem because that was more or less a circular dependency (time sync needed name lookup which needed time to be in sync), but it could mitigate some of the effects or at worst make the problem more explicit/obvious.
silasdavis 2y ago
Tangential: I have a pi 4 running openwrt and a wireguard interface that is routed over the WiFi access point. If you connect to this network it's as if you are connecting in another country. I have outbound traffic only going over wireguard. If wireguard is down then no internet (to avoid any leakage). However if pi loses power the clock resets, wireguard can't handshake with that much clock skew, NTP can't connect without wireguard. So I need to manually update clock after a power failure. I guess I need requests to the NTP server to go over WAN directly. Always seemed like a hassle to get this to work with openwrt zones and stuff. My eternal gratitude to anyone who shoves me in the right direction...
- pynne 2y ago
  Yeah you got it. Set the listening interface to eth0 or whatever the wan interface is. Might need to get full ntpd instead of the busy box version. The Interface option here: https://www.ntp.org/documentation/4.2.8-series/miscopt/ https://www.ntp.org/documentation/4.2.8-series/miscopt/ I’m actually setting up a router with a rpi4 right now, only using Fedora IoT instead of OpenWrt. It’s a bit more assembly required lol
  - silasdavis 2y ago
    I need to have a play with it, but the problem is i deliberately have a 'kill switch' style setup to stop any requests accidentally going not over wireguard (for reasons) so I assume I'll need to do some other firewall type thing. Maybe something something tagging.
- hcfman 2y ago
  I also use wireguard a lot. I didn't know that clock sensitivity could be an issue here. I've never seen an issue before. Now that I know it could be one I'll make a point of deploying all cellular based systems with a GPS synched time source as well and avoid that sort of problem.
- nix0n 2y ago
  > I guess I need requests to the NTP server to go over WAN directly. ... My eternal gratitude to anyone who shoves me in the right direction... Do you have something on the LAN side that could run an NTP server? I'm thinking you could add your laptop to your Pi's list of NTP servers, then just start that service up temporarily as-needed to give the Pi a time that's good enough, then it can get actual correct time sync from somewhere else once Wireguard is working.
hcfman 2y ago
Install sbts-aru as a base system and it will keep your Pi accuracy to sub-microsecond error. https://github.com/hcfman/sbts-aru https://github.com/hcfman/sbts-aru Problem solved.
- hcfman 2y ago
  Or... Install one of these on a Pi somewhere in your network and use that as a time source for all of your devices. Problem solved for all your machines.
  - hcfman 2y ago
    So you know, sbts-aru is an audio recording program. However, in one command it sets an an overlayFS protected SD card setup and chrony GPS based time synchronization. You can turn off all the recording related stuff and then you have a stratum #1 time server for your network. Real easy, real cheap. No more time problems. There's no reason now for all geeks not to have at least one stratum #1 time server in their networks. Really none.
- mkesper 2y ago
  IF you get a DCF77 (stratum 0) signal, there are receivers for Pi below 10€.
  - ndsipa_pomu 2y ago
    Getting time from DCF77 seems like the best solution for timekeeping on a lot of computers, but there don't seem to be nicely packaged ones over USB etc. I had a quick look for Pi receivers and none of them looked to be nicely packaged, but with an external antenna.
shawabawa3 2y ago
I've personally had so many issues with SD card corruption on raspberry pi's that I've decided it was a mistake to even allow SD card disks I only ever use USB storage on my pi's now and not had a single issue since
- hcfman 2y ago
  Indeed, except if you run an in-memory overlay filesystem this might change dramatically for you. I have several Pi's that run for years and years without SD card corruption this way. Normally, however, as in my sbts-aru project, I don't just do that but also create separate data partitions that are read-write. It appears that even though there are some partitions on the card that are read-write, if they are not the system one it is also much more resilient to fatal corruption.
- yabones 2y ago
  Same with "real" servers. For a while there, it was trendy to boot ESXi from an SD card stuck to the system board and use all of the 2.5" disks for just data storage, no OS. It worked great... until it didn't. It turns out that writing logs and swap to an SD card roasts it after 2-4 years pretty consistently.
  - deleted 2y ago
    [deleted]
  - simoncion 2y ago
    The smart ESXi admins I knew used Compact Flash cards as the system boot disk. (It has been too long since I've been in touch with those guys to remember if they offloaded logs to local disks (or maybe a datastore on a SAN) or if they also wrote them to the CF device. I bet they did the former, but I no longer clearly remember.) Using an SD card is just nuts.
  - yjftsjthsd-h 2y ago
    I wouldn't rule out running the OS off of that kind of storage if you had a read only root file system, but what madness would inspire you to write logs to it? And swap!?
lightswitch05 2y ago
I had this happen before. I used it as an excuse to learn how to setup NTP with a GPS receiver. I made a little blog on it if anyone is interested in the results. Be sure to click the sandwich menu for some real-time data: https://www.developerdan.com/ntp/ https://www.developerdan.com/ntp/
- hcfman 2y ago
  Quick question, because I've followed a number of how tos before that don't, if you disconnected all network connectivity, would it sync to the PPS source? Second, have you tested this, I only ask because 1# I found a lot of time sync was really slow, and #2 some of them reported that they had sub-microsecond time sync but when tested they were several milliseconds out. Anyway, I solved these problems with sbts-aru, but maybe it's interesting for you to test if you haven't already done this.
  - lightswitch05 2y ago
    Good question, it absolutely syncs without any network connectivity. For better or for worse, I have it setup NOT to sync to anything else- being a stratum 1 source is enough. Now… I will admit I’m just a hobbyist with this stuff. Anyways, the way I got around it trying to sync to another source when using the PPS was to define the GPS as two separate sources. One for the PPS signal and another for the NMEA sentences: refclock pps ppspath /dev/gpspps0 prefer time1 0.004 minpoll 3 maxpoll 4 iburst refclock nmea baud 57600 time2 0.068 minpoll 3 maxpoll 4 prefer iburst To get around issues with sync being slow, I have NTPD configured with some extra flags to allow large time jumps at startup: NTPD_OPTS="-ggg -N" I have two raspberry pies setup this way- which I don’t use for much else. My primary computers have these two servers configured as time sources- but also have external sources- so as long as network connections are available, they can determine if they are providing a sane time or are false tickers. As for accuracy, you can view the ntpviz output for each setup in the hamburger menu. For catwoman, its regularly within 4 microseconds.
    - hcfman 2y ago
      Ah okay. Because I found that the default start order of gpsmon and chrony was incorrect meaning that the shared memory communications wasn't working but there were no error messages stating this. So sometimes, it would sync (slowly) and then chrony sources would say that it was having microsecond sync time but when I connected a switch (On/Off type) to the same gpio on two separate Pi's with linked earth connections and had a python program output the system time in nano seconds I found that despite telling you it was keeping time accurate to microseconds it was in fact many milliseconds out. The only thing I can conclude is that despite the setup indicating that it should use the PPS source that use the interrupt, this was in fact not happening and it was somehow getting it's time from the serial line only. Without indicating that this was happening. When I switched the startup of gpsmon and the chrony daemon to the other way around time sync was achieved within 1/2 minute to a minute (Real fast) and the reported time on two separate Pi's with the joined up gpio's (So you can get very close to triggering a time check at the same time) was then sometimes as low as 30 ns's from each other, which is damn good considering that most of the difference would be jitter. And none of the how-to's that I read talked about changing the start order of chronyd and gpsmon so I can only assume they all might be suffering from the same problem. It's important to note that the diagnostic commands from chrony will not indicate this and the only way you will notice if this is happening if you setup a rig like I mentioned above. I noticed because my sound localization calculations were showing errors that didn't make sense.
the-kenny 2y ago
Note that the Pi 5 has a real time clock - you just need to add a small battery and it will keep the time when powered off: https://picockpit.com/raspberry-pi/raspberry-pi-5-has-a-real-time-clock-so-what/ https://picockpit.com/raspberry-pi/raspberry-pi-5-has-a-real...
- billsmithaustin 2y ago
  I guess that's what the article meant here: "This particular Pi doesn't have a real-time clock. The very newest ones (5B) do, but you have to actually buy a battery and connect it. "
doubloon 2y ago
Its weird that people work OK without knowing exactly what time it is. Sure it helps us but if a person lost track of the date for a few days in a city they would still be able to perform basic functions. We only have had timekeeping devices for a tiny part of our history but we invented alot of stuff without precise synchrony.
- hcfman 2y ago
  How did they do sound localization then in those times ???
- avianlyric 2y ago
  Isn’t that just because individual human activities aren’t very precise, and for the most part, the position of the sun in the sky is enough for an individual to broadly synchronise themselves with the rest of society. Unless someone is actually aiming to do a time coordinated task with others, you really need to know the time, or even the day-of-week to do stuff, because most human things in world aren’t actually occurring in small time sensitive windows.
malivvan 2y ago
Unfortunately a Raspberry Pi is a bit ill suited for production environments. Id recommend an RTC module. Otherwise this might be helpful: https://github.com/hcfman/sbts-aru https://github.com/hcfman/sbts-aru
benmmurphy 2y ago
I’m surprised tptacek hasn’t shown up to defend DNSSEC from this libel
- tptacek 2y ago
  DNSSEC is doing a fine job of representing itself here.
smashed 2y ago
> I figured they must be running DNSSEC on that zone (or some part of it), and it must have a "not-before" constraint Since clients will attempt to resolve ntp.org in order to actually sync their clock, there is a good probability that some clients will be way off. Enabling dnssec on that zone was probably not without important drawbacks? I wonder if the operators thought about that potential pitfall. Seems like they might be doing a disservice to their core mission of allowing devices to sync their clock.
- growse 2y ago
  It feels like enabling DNSSEC on any zone is inviting a bunch of fun service risks and unexpected failures. Is my clock more likely to be accurate after NTP.org got signed? It looks like it's less likely, on average.
- cbhl 2y ago
  If you _don't_ enable dnssec on ntp.org, then a mitm can intercept the dns request to redirect to an attacker-owned timeserver with a time in the past. Then the host can have old and expired (without loss of generality) keys/certificates replayed against it. If memory serves though, Raspbian used to not even have `fake-hwclock` by default and even more Pis would end up with this "wildly wrong time near the epoch and can't bootstrap DNS/NTP" failure mode. You'd sometimes also see it with VMs too (esp if they were doing a pure-software clock that ran slower than realtime, instead of patching clock calls to the real hardware clock on the host).
  - Dylan16807 2y ago
    > If you _don't_ enable dnssec on ntp.org, then a mitm can intercept the dns request to redirect to an attacker-owned timeserver with a time in the past. Then the host can have old and expired (without loss of generality) keys/certificates replayed against it. Preventing DNS mitm only matters if you prevent NTP mitm too. What percent of NTP clients talking to these servers are doing it in a secure way? And is that share growing?
the__alchemist 2y ago
It's surprising that it was left out. Ie, the workaround makes sense, but this is a peripheral that is sometimes built into MCUs themselves. For example, most (all?) STM32s include an RTC... with the caveat that if you are using them for canonical use cases, you will need external hardware in the form of a dedicated 32kHz oscillator, and possibly a battery. For a lot of micro-controller timing uses, they aren't required; for example, hardware timers based on the primary clock source, or the Cortex-M systick, but for maintaining accurate dates and times over long periods, the RTC is the right tool. It can also output the dates and times in a convenient format as well, as long as you find named register fields convenient!
- 01HNNWZ0MV43FF 2y ago
  I had to support an IoT device once that lacked an RTC. That was actually the easier hardware platform, because it either had good NTP sync or it didn't reach our C&C servers. The difficult one was the platform that had an RTC. Those would slooowly drift out of sync if NTP was broken. Slowly and silently.
- AdamH12113 2y ago
  An RTC really needs a battery, and even a smaller battery like a 1220 takes up a fair bit of board space -- space that could be used for another connector or more functionality. And even if you have an RTC, you still have to handle the edge case where the battery dies or is removed and the clock resets. Critical low-level internet services probably shouldn't require clients to already have the correct time. I use an external RTC with a Raspberry Pi for some sensor systems I work with. Its purpose is to give a reasonable chance of having the correct time for recording data in remote locations where no internet connection is available at startup. For a Pi that only provides network services (and thus always needs a working internet connection), I probably wouldn't think to bother.
- numpad0 2y ago
  RTC is just an electronically readable Casio watch, it's not even responsible for keeping real time while OS is running. All it does is always incrementing regardless of motherboard power state so to provide time as OS loads.
aimor 2y ago
Two nights ago I rebooted my router and it wasn't able to update its clock. The logs were filled with "ntp: start NTP update" every 5 seconds. Well, it was something related to using AdGuard's (94.140.14.14) DNS and pool.ntp.org. I added Google's (8.8.8.8) to resolv.conf temporarily to get the clock synced. But I'm still not sure what the root cause was.
cbhl 2y ago
I think the last paragraph was my favorite part of this article. For those of you who didn't read the fine article, I've quoted it here: > As usual, this post is not a request for THE ONE to show up. If you are THE ONE, you don't make mistakes. We know. Shut up and go away.
- yjftsjthsd-h 2y ago
  I dunno, I've always found that to be a... less impressive part of the blog. When the author hits problems in her stack, anyone who would have caught it should shut up and stop being THE ONE. Fine, we've all run into people with big skills and bigger egos. But... the other half of her blog posts are her making snide remarks about how stupidly other people have set stuff up. To her (ex-)coworkers, she is THE ONE. So it leaves a sour taste in my mouth.
dclowd9901 2y ago
I’m not sure this is related or not but if there’s one thing I’ve learned about complex objects (engines, computers, software, sewing machines, etc), it’s that every _custom_ thing you do to it has some kind of order of magnitude impact on the overall complexity of the machine, and thus, you should avoid customizing it. At least if your primary concern is usability vs solving specific problems. It’s definitely served me well in a wide variety of disciplines.
- fuzzfactor 2y ago
  There is not much of an allowance any more for devices that are not on the internet 24/7 without fail. Multi-booting PCs with Windows and Linux over the years, I have seen the time sync problem go from nonexistent to show-stopper. For devices that you only need to connect to the internet occasionally or sporadically (so that's what you do), that's where I noticed it most. Linux sets the RTC to UTC, then when you reboot to Windows, Windows uses the RTC as local time. Which for me is 5 or 6 hours different from UTC, depending on Daylight Savings Time. Plus when Daylight Savings comes around, each OS wants to make a 1 hour correction the first time you boot it after that date. With multi-booting this can add up to more than one hour difference too. Didn't used to be so bad, if you were a few hours (days or longer for many PCs) away from the actual time, things did not fail for this reason. Eventually only one hour off was OK for a while, now nope. There are just so many more obstacles to smooth reliable operation, and weak links in a more extensive chain that must remain perfectly strong. Otherwise the chain is broken, you are disconnected, and the weak link is too many sections away from you now to be within reach. Complete perfection is required more so than ever, while at the same time, your efforts to approach perfection are being made more difficult.
  - psz 2y ago
    Set Linux to interpret RTC as local time, or set Windows to interpret RTC as UTC (possibly better, as it won't adjust to DST). https://askubuntu.com/a/1377011 https://askubuntu.com/a/1377011
gorkish 2y ago
The Pi ecosystem is coming apart at the seams. Talk about falling off a pedestal. By the time you buy everything you need to make a Pi 5 even remotely usable and reliable today, you'll have spent as much as a small form factor PC. What do you need? 1. Special power supply - Yes, it's USB-C but it requires a high amperage 5V supply instead of accepting a higher voltage like most USB-C stuff 2. Active cooler - Pi 5 throttles immediately without it; it's not an optional component 3. NVMe Expansion HAT - Seriously why the fiddly little PCIe header? Put an NVMe slot on the bottom and let people adapt *that* connector 4. RTC Battery - At least put a damn capacitor on the board 5. Enclosure - I'm halfway expecting the pi foundation to print some dashed cut lines on the cardboard box so they can say it comes with a case =] At the end of this you get a system that can run ... raspbian Not like the Pi Foundation couldn't have contributed to uboot or the upstream kernel or anything before thier hardware was released. Pretty much nothing released for "Raspberry Pi" runs on a Pi 5 at this point in time. They better try to salvage something before putting out the expected CM5, but I'm not expecting much.
- Dylan16807 2y ago
  The power supply is becoming less special over time. My first result for "usb c power supply" on amazon is a $25 anker 317 that supports PPS and will do 5 amps at any voltage. Not that that's an amazing price, but the feature is spreading. But I agree that the price to get a happy Pi is pretty harsh compared to a tiny PC.
  - pbnjeh 2y ago
    I don't have Pi 5 to check this, but from what I've read, it does not support PPS. And such support is required; otherwise, even with a source that supports PPS, the connection will behave in a non-PPS fashion. Apparently, cable quality is also demonstrating itself to be a significant factor in the Pi 5 power supply environment. Regarding both these points: https://github.com/raspberrypi/rpi-eeprom/issues/497 https://github.com/raspberrypi/rpi-eeprom/issues/497
    - Dylan16807 2y ago
champtar 2y ago
In OpenWrt (Linux distro for routers that often don't have RTC) dnsmasq starts with `--dnssec-no-timecheck` until the ntp client gets a first sync (https://github.com/openwrt/openwrt/commit/5acfe55d7139a5294192bddf10fe3a1de3180e8d https://github.com/openwrt/openwrt/commit/5acfe55d7139a52941...)
raggi 2y ago
If you don't have an RTC I'd recommend having a tlsdate with some bounding heuristics to prevent extreme clock fixation from a mitm. You can relatively cheaply hit a large number of public servers that are likely to have good times available over TLS and trust the common result. You validate the certs without considering the notbefore stamps and then if you're feeling aggressive validate them after you've managed to approximate a date from the cohort. I know there are commercial packages that do this, I'm not sure about OSS ones. Roughtime would be far better, but essentially there's no broad deployment of it yet. Ideally something good would be picked by Raspbian and delivered in the distro as standard.
cyounkins 2y ago
I ran into this before! https://medium.com/@cyounkins/encrypted-dns-ntp-deadlock-9e378940b79f https://medium.com/@cyounkins/encrypted-dns-ntp-deadlock-9e3...

230 points by X-Cubed 2y ago

tl;dr: machine is unable synchronize its clock because it won't trust the NTP server's DNSSEC certificate because its clock isn't synchronized. Oof.

iforgotpassword 2y ago
Yeah in hindsight it might not be the best idea to enable this for ntp.org, especially when the ntp protocol itself isn't secure.
kakkoko 2y ago
https://github.com/systemd/systemd/commit/abf4e5c1d3ad767bc0ed67883e8e4d916af095ec https://github.com/systemd/systemd/commit/abf4e5c1d3ad767bc0... While systemd-timesyncd contains a workaround for this, chrony does not seem to.
jandrese 2y ago
This is the perennial problem with trying to secure NTP. I've also seen people suggest that we just use HTTPS for time sync, not realizing that if your clocks are too far off you can't make a HTTPS connection.
- toast0 2y ago
  It takes more creativity, but you can start to make a narrowing range of times if you're careful. You'll have to ignore DNSSEC time based validation while you start up, of course... Start with the last time you were properly synced: it's most likely less than a month behind then. Then take a sampling of certificates offered from well known sites. If the certificates validate, other than time, no reasonable CA that you trust issues not-before dates in the future, so it's not before the most recent not-before date in a certificate. It's probably not much after the not-after dates either, well know sites tend to replace their certificates (but maybe you're getting MITMed with a compromised, old certificate). Maybe ask for OSCP stapling, which should get you a more narrow range of times. I think OSCP responses are valid for a week? But if you get several from different https servers, chances are good you'll narrow the range. In the case as suggested that the time was only off by about a day, IMHO, something is seriously wrong if it can't get to sync from there, but I guess I'm not really surprised either.
- creeble 2y ago
  You can use plain HTTP for time sync. Almost all HTTP servers respond with a time header.
  - jandrese 2y ago
    The whole point of the exercise was to make it secure though. If you don't care about MITM attackers then NTP works great.
    - numpad0 2y ago
      I don't get the supposed security aspect of getting false time. Hacker gives you 1980-01-01 or 4096-13-32 and mess up CRL and ruin your day...how. Years ago I've tried privilege escalation exploit to play with a phone and it involved rolling back date to unexpire signature, so I know there is exploit potential, but it... it just feels like RTC bootstrap problem should be something solvable.
    - creeble 2y ago
      Isn't that a provably pointless exercise though? Security protocols (at least the ones in common use) require certificates or keys that eventually expire, because of the risk of a permanent key being compromised. If they expire, the protocol needs time. QED.
deleted 2y ago
[deleted]

lifthrasiir 2y ago

> By "everything else", I also mean WireGuard. Did you know that if your machine gets far enough out of sync, that'll stop working, too? I had no idea that it apparently includes time in its crypto stuff, but what other explanation is there? WireGuard only requires a monotonic clock, because it periodically rotates keys to provide the forward secrecy. Peer clocks are otherwise not required to be synchronized [1]. I guess the clock had a higher rate than usual and it couldn't be corrected due to the lack of RTC? [1] https://www.wireguard.com/papers/wireguard.pdf#page=7 https://www.wireguard.com/papers/wireguard.pdf#page=7 "In fact, it does not even have to be an accurate timestamp; it simply must be a per-peer monotonically increasing 96-bit number."

josephg 2y ago
Why does it use the time then? Why not just increment its own 96 bit number whenever you use it?
- lifthrasiir 2y ago
  Because it is required to be monotonic per peer. WireGuard has no intrinsic states, so multiple machines with the same peer key will be seen as a single peer and that is actually a legitimate use of WireGuard. These machines would have to be synchronized to each other (but not necessarily to the external clock), and using a time is a straightforward and reasonable way to ensure this.
  - apexalpha 2y ago
    So could you share a single set of keys on multiple machines and Wireguard will work, as long as all machines use the same ntp server?
    - lifthrasiir 2y ago
      Technically yes, but I don't think (but haven't exactly confirmed) that you can initiate a new session to itself.

wodenokoto 2y ago

What is she referring to by “THE ONE”?

semireg 2y ago
The know it all. This “wouldn’t have happened to them because they are” THE ONE. Also known as, “that guy” as in, “don’t be THAT GUY.”
- k1t 2y ago
  As sibling comment somewhat points out, she is literally referencing The One (2001) https://m.imdb.com/title/tt0267804/ https://m.imdb.com/title/tt0267804/
bjackman 2y ago
It's pre-empting the sibling comment saying "RTC module is $1"
dmvdoug 2y ago
See, e.g., https://rachelbythebay.com/w/2018/04/28/meta/ https://rachelbythebay.com/w/2018/04/28/meta/
- Joker_vD 2y ago
  Amazing. So when you read about some weird (and mostly self-inflicted) problem you've never had in your life, you are not allowed to express your surprise because by doing that you're implying you're better than other people, and that would be horrible (the implying part is what is horrible, I think? Or maybe actually being better is also wrong? Can you even be better―yeah, I guess it's possible to be better than someone else at something).
  - Ensorceled 2y ago
    You're totally allowed to express your surprise, you're totally allowed to be a condescending jerk while doing it, you're totally allowed to call it "self-inflicted". The author is allowed to ask you to shut up and go away if you do.

M95D 2y ago

RTC module for Raspberry is ~ $1.

icehawk 2y ago
and you certaintly get what you pay for, DS3231s fry almost as easily as MAX3232s, (And you can do the same thing were you write the time to the RTC before you have the correct time)
- Joel_Mckay 2y ago
  I have used these in product designs before with zero failures. Where exactly did you buy your samples? We found they drift less than 28s a year in standalone mode. Also, were repeatable within +-18ms when using a NTP client. Microchip RTCs are harder to setup, but also seem to work just as well (ignoring the goofy epoch). Chip shortage caused people to do unspeakable things.... ;-)
  - hcfman 2y ago
    You could buy an rtc and put up with that kind of clock drift, or.... for the same money or less you could be a GPS and have none of that drift.
    - r2_pilot 2y ago
      (a bit more money for the GPS to be fair) not even mentioning the fact that you should have clear-ish access to the sky, the time taken for you to receive the time data, and the possibility of jamming. I could go on but the two methods have differing modes of operation and failure despite ultimately being capable of fulfilling the need for a reasonably synchronized clock
- M95D 2y ago
  I have a DS1307 installed ever since I bought a RPi3B+. It didn't fail and it's accurate enough until NTP updates the time. Are you sure you connected it correctly?

jlubawy 2y ago

My raspberry pi is also currently dead, something it didn't like about updating the OS caused it to be bricked. All updates were done within the OS, so didn't expect anything bad to happen. I could reflash the OS, but I also "don't have the time". I'll just use my laptop until I do have the time someday to resurrect it.

hcfman 2y ago
Quite likely installing the sbts-aru as a base project would have prevented this as well. Loads of SD cards get corrupted historically without apparent good reason. Using a memory-based overlayFS file system solves so many of these problems. sbts-aru, as well as setting up a stratum #1 time server sets up a resilient memory overlayFS as a base.

riobard 2y ago

Speaking of RTC battery, I've recently come to the realization that I have to make sure that BIOS battery is not absolutely dead in always-on PC boxes. Background: I use an x86 box as home router. I've changed the configuration in the BIOS that it should automatically boot up on power. However if the BIOS battery is dead, the config will be lost and it will revert to default settings, which is not to boot on power. But there is no way to know how much juice is left in the BIOS battery, thus there is no way to issue warnings that the battery should be replaced soon. If there is power loss AND the battery is dead, when the power comes back up again, the router will just stay off indefinitely until it's manually turned on. That's when I realized why most consumer routers do not feature RTC nor do they require batteries.

qiqitori 2y ago
Some unsolicited solutions, maybe not for you but somebody in a similar situation: maybe replace it with a supercapacitor (needs some wiring changes, otherwise it won't ever be charged, should last long enough for most power outages), or use a stack of coin cells in parallel (difficult due to physical dimensions, and they'll still go dead at some point). You can also short two wires on the ATX supply to automatically turn on the computer when power is supplied (which may not be good enough if you have to press a key during boot to dismiss the low battery warning). Or hack the BIOS :)
- riobard 2y ago
  Somehow I feel reliability might take a hit… :|
- mkesper 2y ago
  If using supercapacitors, use some known not for leaking else you're back to square one (do a search for capacitor recap).
  - duskwuff 2y ago
    That issue doesn't apply to supercapacitors (nor to any capacitor manufactured in the last 15-20 years, really). It affected some aluminum electrolytic capacitors manufactured with a defective electrolyte in the early 2000s.

Joel_Mckay 2y ago

If you are running Bind or SSL/x.509 certs, than an RTC is pretty much a requirement on the raspberry pi. Network time leaps over NTP even when working (first big leap was allowed etc.) can still bork a lot of services. Local stratum 1 GPS time (even with PPS pin) servers based on Pi's can suffer similar issues when they lose satellite lock (weather etc.) Have a great day =)

hcfman 2y ago
Having said that, I've never personally seen my stratum #1 Pi loose sync due to weather and I run about 4 of em here. I also use a cheap 7 euro ublox 7m. So long as you position it or an antennae with a reasonable view of the sky that shouldn't happen I reckon. Why all the complex solutions when it's so easy and cheap now to install a stratum #1 time server, even a Pi 3 will suffice. It works on a Pi Zero, but then you only have a wifi link. The sbts-au project make's it incredibly easy to setup. git lone and then install. One command, it's that simple.
- teruakohatu 2y ago
  Any advice for a cheap USB GPS for a stratum #1 time server? EDIT: Any any advice on how essential PPS is?
  - Joel_Mckay 2y ago
    The PPS can be good (pi kernel support is stable), but one needs to be careful as some modules PPS signal has a great deal of instability. Some U-Blox modules support the additional ground based navigation signal and cellular beacons. This is recommended in addition to the 3 satellite networks classes, but note some modules will not work with some antennas (get the unified hardware with built in antenna/lna if unsure about RF notes). Best regards, =)
    - hcfman 2y ago
      If you have good sky visibility, then I don't need an antenna. Though this often tends to mean putting a Pi outside. I'm also using just the first Antenna I find actually.

mplaisier 2y ago

I ran into a similar problem on a Raspberry Pi, where my program would ignore data because it was deemed too old. It turns out a Pi uses fake-hwclock. This tool stores a timestamp regularly in a file. On boot the Pi reads from that file to set the time. When the Pi has been turned off for longer then the initial time can be hours or days out of sync. The Pi then takes a couple of seconds to obtain a valid time via NTP, but my program ran quickly enough to work with the old timestamp. After a couple of minutes it would typically work again.

zokier 2y ago

It's unfortunate that using DHCP to get rough time isn't more popular, especially for devices without RTC. Sure DHCP can be spoofed and should be considered more of a last resort, but it sure would be better than nothing and ending up in this kind of a situation where nothing works anymore. Previous discussion on similar thing: "Encrypted DNS + NTP = Deadlock" https://news.ycombinator.com/item?id=34177331 https://news.ycombinator.com/item?id=34177331

CaliforniaKarl 2y ago

I know I’ve brought this up before, but I can’t remember exactly where. It would be nice if DHCP could provide the current time as one of the pieces of information it provides with a lease. If you’re already trusting it for your IP address, you might as well trust it with the current UTC time: The current 64-bit timestamp would be enough to get timestamp-sensitive things working.

citrin_ru 2y ago
I trust a free wifi hotspot to give me an IP (traffic will be protected by TLS), but not with the time. E. g. time from the past can be used to trick into accepting an expired TLS certificate. But for most wifi hotspots it will be unintentionally wrong because such devices often unmonitored and not maintained until they completely break.
mapreduce 2y ago
> If you’re already trusting it for your IP address, you might as well trust it with the current UTC time I don't follow! How does trusting DHCP with IP address automatically mean I should trust it with the current UTC time? Time requires higher degree of trust than IP. I may not care what my IP address looks like but I might care a lot about what the current UTC time is and I might want this to come from a more trustworthy device.
- Terr_ 2y ago
  To offer an example, if an attacker can manipulate the time then they can make the target accept expired or revoked cryptographic certificates, potentially enabling impersonation or man-in-the-middle attacks. In contrast, a different IP than expected isn't such a big deal... Although it might break the collaboration of two computers as crude denial of service
- mikepurvis 2y ago
  DHCP servers can command your dns config and hostname too. It's not a total mitm story since the certs are still on the machine itself but it's definitely more than just a local IP address.

cjbillington 2y ago

The wireguard problem is a pain in the neck. Even if you're happy for your system to not have a realtime clock, when it does come online you'd want wireguard to not start until after the clock has synced to a timeserver. The below is what I'm doing at the moment, but I can't say I'm sure it's working - haven't seen wireguard start before the time is synced since doing it, but it could be probabilistic: systemctl enable systemd-time-wait-sync.service mkdir -p /etc/systemd/system/wg-quick@wg0.service.d/ echo "[Unit] After=time-sync.target Wants=time-sync.target " > /etc/systemd/system/wg-quick@wg0.service.d/override.conf I'd be interested if anyone could let me know if they think this is likely to be achieving what I want or not.

hcfman 2y ago
Well... I see where you are going with this but is it necessary? What happens if the clock gets it's correct time sync back, will not wireguard come back online by itself ? Then the real solution might involve ensuring good clock sync. Using a GPS synched time source is good for this, such as this one :) https://github.com/hcfman/sbts-aru https://github.com/hcfman/sbts-aru
- cjbillington 2y ago
  No, I can't say I understand it fully - but if the clock goes backwards, peers will not talk to a client again until I restart the other peers' wireguard services. No matter if the client's clock is subsequently correct. For some people, all their network access goes over wireguard and that includes time syncing. That's not the case for me, but for those people if wireguard isn't working their clock can't sync. That GPS receiver looks cool! However an RTC and a battery will do just fine for me, which is the plan.
yjftsjthsd-h 2y ago
That was my thought as well - if we know certain things need time to be in sync, can we use systemd dependencies to enforce that? I'm not sure it would help with OP's problem because that was more or less a circular dependency (time sync needed name lookup which needed time to be in sync), but it could mitigate some of the effects or at worst make the problem more explicit/obvious.

silasdavis 2y ago

Tangential: I have a pi 4 running openwrt and a wireguard interface that is routed over the WiFi access point. If you connect to this network it's as if you are connecting in another country. I have outbound traffic only going over wireguard. If wireguard is down then no internet (to avoid any leakage). However if pi loses power the clock resets, wireguard can't handshake with that much clock skew, NTP can't connect without wireguard. So I need to manually update clock after a power failure. I guess I need requests to the NTP server to go over WAN directly. Always seemed like a hassle to get this to work with openwrt zones and stuff. My eternal gratitude to anyone who shoves me in the right direction...

pynne 2y ago
Yeah you got it. Set the listening interface to eth0 or whatever the wan interface is. Might need to get full ntpd instead of the busy box version. The Interface option here: https://www.ntp.org/documentation/4.2.8-series/miscopt/ https://www.ntp.org/documentation/4.2.8-series/miscopt/ I’m actually setting up a router with a rpi4 right now, only using Fedora IoT instead of OpenWrt. It’s a bit more assembly required lol
- silasdavis 2y ago
  I need to have a play with it, but the problem is i deliberately have a 'kill switch' style setup to stop any requests accidentally going not over wireguard (for reasons) so I assume I'll need to do some other firewall type thing. Maybe something something tagging.
hcfman 2y ago
I also use wireguard a lot. I didn't know that clock sensitivity could be an issue here. I've never seen an issue before. Now that I know it could be one I'll make a point of deploying all cellular based systems with a GPS synched time source as well and avoid that sort of problem.
nix0n 2y ago
> I guess I need requests to the NTP server to go over WAN directly. ... My eternal gratitude to anyone who shoves me in the right direction... Do you have something on the LAN side that could run an NTP server? I'm thinking you could add your laptop to your Pi's list of NTP servers, then just start that service up temporarily as-needed to give the Pi a time that's good enough, then it can get actual correct time sync from somewhere else once Wireguard is working.

hcfman 2y ago

Install sbts-aru as a base system and it will keep your Pi accuracy to sub-microsecond error. https://github.com/hcfman/sbts-aru https://github.com/hcfman/sbts-aru Problem solved.

hcfman 2y ago
Or... Install one of these on a Pi somewhere in your network and use that as a time source for all of your devices. Problem solved for all your machines.
- hcfman 2y ago
  So you know, sbts-aru is an audio recording program. However, in one command it sets an an overlayFS protected SD card setup and chrony GPS based time synchronization. You can turn off all the recording related stuff and then you have a stratum #1 time server for your network. Real easy, real cheap. No more time problems. There's no reason now for all geeks not to have at least one stratum #1 time server in their networks. Really none.
mkesper 2y ago
IF you get a DCF77 (stratum 0) signal, there are receivers for Pi below 10€.
- ndsipa_pomu 2y ago
  Getting time from DCF77 seems like the best solution for timekeeping on a lot of computers, but there don't seem to be nicely packaged ones over USB etc. I had a quick look for Pi receivers and none of them looked to be nicely packaged, but with an external antenna.

shawabawa3 2y ago

I've personally had so many issues with SD card corruption on raspberry pi's that I've decided it was a mistake to even allow SD card disks I only ever use USB storage on my pi's now and not had a single issue since

hcfman 2y ago
Indeed, except if you run an in-memory overlay filesystem this might change dramatically for you. I have several Pi's that run for years and years without SD card corruption this way. Normally, however, as in my sbts-aru project, I don't just do that but also create separate data partitions that are read-write. It appears that even though there are some partitions on the card that are read-write, if they are not the system one it is also much more resilient to fatal corruption.
yabones 2y ago
Same with "real" servers. For a while there, it was trendy to boot ESXi from an SD card stuck to the system board and use all of the 2.5" disks for just data storage, no OS. It worked great... until it didn't. It turns out that writing logs and swap to an SD card roasts it after 2-4 years pretty consistently.
- deleted 2y ago
  [deleted]
- simoncion 2y ago
  The smart ESXi admins I knew used Compact Flash cards as the system boot disk. (It has been too long since I've been in touch with those guys to remember if they offloaded logs to local disks (or maybe a datastore on a SAN) or if they also wrote them to the CF device. I bet they did the former, but I no longer clearly remember.) Using an SD card is just nuts.
- yjftsjthsd-h 2y ago
  I wouldn't rule out running the OS off of that kind of storage if you had a read only root file system, but what madness would inspire you to write logs to it? And swap!?

lightswitch05 2y ago

I had this happen before. I used it as an excuse to learn how to setup NTP with a GPS receiver. I made a little blog on it if anyone is interested in the results. Be sure to click the sandwich menu for some real-time data: https://www.developerdan.com/ntp/ https://www.developerdan.com/ntp/

hcfman 2y ago
Quick question, because I've followed a number of how tos before that don't, if you disconnected all network connectivity, would it sync to the PPS source? Second, have you tested this, I only ask because 1# I found a lot of time sync was really slow, and #2 some of them reported that they had sub-microsecond time sync but when tested they were several milliseconds out. Anyway, I solved these problems with sbts-aru, but maybe it's interesting for you to test if you haven't already done this.
- lightswitch05 2y ago
  Good question, it absolutely syncs without any network connectivity. For better or for worse, I have it setup NOT to sync to anything else- being a stratum 1 source is enough. Now… I will admit I’m just a hobbyist with this stuff. Anyways, the way I got around it trying to sync to another source when using the PPS was to define the GPS as two separate sources. One for the PPS signal and another for the NMEA sentences: refclock pps ppspath /dev/gpspps0 prefer time1 0.004 minpoll 3 maxpoll 4 iburst refclock nmea baud 57600 time2 0.068 minpoll 3 maxpoll 4 prefer iburst To get around issues with sync being slow, I have NTPD configured with some extra flags to allow large time jumps at startup: NTPD_OPTS="-ggg -N" I have two raspberry pies setup this way- which I don’t use for much else. My primary computers have these two servers configured as time sources- but also have external sources- so as long as network connections are available, they can determine if they are providing a sane time or are false tickers. As for accuracy, you can view the ntpviz output for each setup in the hamburger menu. For catwoman, its regularly within 4 microseconds.
  - hcfman 2y ago
    Ah okay. Because I found that the default start order of gpsmon and chrony was incorrect meaning that the shared memory communications wasn't working but there were no error messages stating this. So sometimes, it would sync (slowly) and then chrony sources would say that it was having microsecond sync time but when I connected a switch (On/Off type) to the same gpio on two separate Pi's with linked earth connections and had a python program output the system time in nano seconds I found that despite telling you it was keeping time accurate to microseconds it was in fact many milliseconds out. The only thing I can conclude is that despite the setup indicating that it should use the PPS source that use the interrupt, this was in fact not happening and it was somehow getting it's time from the serial line only. Without indicating that this was happening. When I switched the startup of gpsmon and the chrony daemon to the other way around time sync was achieved within 1/2 minute to a minute (Real fast) and the reported time on two separate Pi's with the joined up gpio's (So you can get very close to triggering a time check at the same time) was then sometimes as low as 30 ns's from each other, which is damn good considering that most of the difference would be jitter. And none of the how-to's that I read talked about changing the start order of chronyd and gpsmon so I can only assume they all might be suffering from the same problem. It's important to note that the diagnostic commands from chrony will not indicate this and the only way you will notice if this is happening if you setup a rig like I mentioned above. I noticed because my sound localization calculations were showing errors that didn't make sense.

the-kenny 2y ago

Note that the Pi 5 has a real time clock - you just need to add a small battery and it will keep the time when powered off: https://picockpit.com/raspberry-pi/raspberry-pi-5-has-a-real-time-clock-so-what/ https://picockpit.com/raspberry-pi/raspberry-pi-5-has-a-real...

billsmithaustin 2y ago
I guess that's what the article meant here: "This particular Pi doesn't have a real-time clock. The very newest ones (5B) do, but you have to actually buy a battery and connect it. "

doubloon 2y ago

Its weird that people work OK without knowing exactly what time it is. Sure it helps us but if a person lost track of the date for a few days in a city they would still be able to perform basic functions. We only have had timekeeping devices for a tiny part of our history but we invented alot of stuff without precise synchrony.

hcfman 2y ago
How did they do sound localization then in those times ???
avianlyric 2y ago
Isn’t that just because individual human activities aren’t very precise, and for the most part, the position of the sun in the sky is enough for an individual to broadly synchronise themselves with the rest of society. Unless someone is actually aiming to do a time coordinated task with others, you really need to know the time, or even the day-of-week to do stuff, because most human things in world aren’t actually occurring in small time sensitive windows.

malivvan 2y ago

Unfortunately a Raspberry Pi is a bit ill suited for production environments. Id recommend an RTC module. Otherwise this might be helpful: https://github.com/hcfman/sbts-aru https://github.com/hcfman/sbts-aru

benmmurphy 2y ago

I’m surprised tptacek hasn’t shown up to defend DNSSEC from this libel

tptacek 2y ago
DNSSEC is doing a fine job of representing itself here.

smashed 2y ago

> I figured they must be running DNSSEC on that zone (or some part of it), and it must have a "not-before" constraint Since clients will attempt to resolve ntp.org in order to actually sync their clock, there is a good probability that some clients will be way off. Enabling dnssec on that zone was probably not without important drawbacks? I wonder if the operators thought about that potential pitfall. Seems like they might be doing a disservice to their core mission of allowing devices to sync their clock.

growse 2y ago
It feels like enabling DNSSEC on any zone is inviting a bunch of fun service risks and unexpected failures. Is my clock more likely to be accurate after NTP.org got signed? It looks like it's less likely, on average.
cbhl 2y ago
If you _don't_ enable dnssec on ntp.org, then a mitm can intercept the dns request to redirect to an attacker-owned timeserver with a time in the past. Then the host can have old and expired (without loss of generality) keys/certificates replayed against it. If memory serves though, Raspbian used to not even have `fake-hwclock` by default and even more Pis would end up with this "wildly wrong time near the epoch and can't bootstrap DNS/NTP" failure mode. You'd sometimes also see it with VMs too (esp if they were doing a pure-software clock that ran slower than realtime, instead of patching clock calls to the real hardware clock on the host).
- Dylan16807 2y ago
  > If you _don't_ enable dnssec on ntp.org, then a mitm can intercept the dns request to redirect to an attacker-owned timeserver with a time in the past. Then the host can have old and expired (without loss of generality) keys/certificates replayed against it. Preventing DNS mitm only matters if you prevent NTP mitm too. What percent of NTP clients talking to these servers are doing it in a secure way? And is that share growing?

the__alchemist 2y ago

It's surprising that it was left out. Ie, the workaround makes sense, but this is a peripheral that is sometimes built into MCUs themselves. For example, most (all?) STM32s include an RTC... with the caveat that if you are using them for canonical use cases, you will need external hardware in the form of a dedicated 32kHz oscillator, and possibly a battery. For a lot of micro-controller timing uses, they aren't required; for example, hardware timers based on the primary clock source, or the Cortex-M systick, but for maintaining accurate dates and times over long periods, the RTC is the right tool. It can also output the dates and times in a convenient format as well, as long as you find named register fields convenient!

01HNNWZ0MV43FF 2y ago
I had to support an IoT device once that lacked an RTC. That was actually the easier hardware platform, because it either had good NTP sync or it didn't reach our C&C servers. The difficult one was the platform that had an RTC. Those would slooowly drift out of sync if NTP was broken. Slowly and silently.
AdamH12113 2y ago
An RTC really needs a battery, and even a smaller battery like a 1220 takes up a fair bit of board space -- space that could be used for another connector or more functionality. And even if you have an RTC, you still have to handle the edge case where the battery dies or is removed and the clock resets. Critical low-level internet services probably shouldn't require clients to already have the correct time. I use an external RTC with a Raspberry Pi for some sensor systems I work with. Its purpose is to give a reasonable chance of having the correct time for recording data in remote locations where no internet connection is available at startup. For a Pi that only provides network services (and thus always needs a working internet connection), I probably wouldn't think to bother.
numpad0 2y ago
RTC is just an electronically readable Casio watch, it's not even responsible for keeping real time while OS is running. All it does is always incrementing regardless of motherboard power state so to provide time as OS loads.

aimor 2y ago

Two nights ago I rebooted my router and it wasn't able to update its clock. The logs were filled with "ntp: start NTP update" every 5 seconds. Well, it was something related to using AdGuard's (94.140.14.14) DNS and pool.ntp.org. I added Google's (8.8.8.8) to resolv.conf temporarily to get the clock synced. But I'm still not sure what the root cause was.

cbhl 2y ago

I think the last paragraph was my favorite part of this article. For those of you who didn't read the fine article, I've quoted it here: > As usual, this post is not a request for THE ONE to show up. If you are THE ONE, you don't make mistakes. We know. Shut up and go away.

yjftsjthsd-h 2y ago
I dunno, I've always found that to be a... less impressive part of the blog. When the author hits problems in her stack, anyone who would have caught it should shut up and stop being THE ONE. Fine, we've all run into people with big skills and bigger egos. But... the other half of her blog posts are her making snide remarks about how stupidly other people have set stuff up. To her (ex-)coworkers, she is THE ONE. So it leaves a sour taste in my mouth.

dclowd9901 2y ago

I’m not sure this is related or not but if there’s one thing I’ve learned about complex objects (engines, computers, software, sewing machines, etc), it’s that every _custom_ thing you do to it has some kind of order of magnitude impact on the overall complexity of the machine, and thus, you should avoid customizing it. At least if your primary concern is usability vs solving specific problems. It’s definitely served me well in a wide variety of disciplines.

fuzzfactor 2y ago
There is not much of an allowance any more for devices that are not on the internet 24/7 without fail. Multi-booting PCs with Windows and Linux over the years, I have seen the time sync problem go from nonexistent to show-stopper. For devices that you only need to connect to the internet occasionally or sporadically (so that's what you do), that's where I noticed it most. Linux sets the RTC to UTC, then when you reboot to Windows, Windows uses the RTC as local time. Which for me is 5 or 6 hours different from UTC, depending on Daylight Savings Time. Plus when Daylight Savings comes around, each OS wants to make a 1 hour correction the first time you boot it after that date. With multi-booting this can add up to more than one hour difference too. Didn't used to be so bad, if you were a few hours (days or longer for many PCs) away from the actual time, things did not fail for this reason. Eventually only one hour off was OK for a while, now nope. There are just so many more obstacles to smooth reliable operation, and weak links in a more extensive chain that must remain perfectly strong. Otherwise the chain is broken, you are disconnected, and the weak link is too many sections away from you now to be within reach. Complete perfection is required more so than ever, while at the same time, your efforts to approach perfection are being made more difficult.
- psz 2y ago
  Set Linux to interpret RTC as local time, or set Windows to interpret RTC as UTC (possibly better, as it won't adjust to DST). https://askubuntu.com/a/1377011 https://askubuntu.com/a/1377011

gorkish 2y ago

The Pi ecosystem is coming apart at the seams. Talk about falling off a pedestal. By the time you buy everything you need to make a Pi 5 even remotely usable and reliable today, you'll have spent as much as a small form factor PC. What do you need? 1. Special power supply - Yes, it's USB-C but it requires a high amperage 5V supply instead of accepting a higher voltage like most USB-C stuff 2. Active cooler - Pi 5 throttles immediately without it; it's not an optional component 3. NVMe Expansion HAT - Seriously why the fiddly little PCIe header? Put an NVMe slot on the bottom and let people adapt *that* connector 4. RTC Battery - At least put a damn capacitor on the board 5. Enclosure - I'm halfway expecting the pi foundation to print some dashed cut lines on the cardboard box so they can say it comes with a case =] At the end of this you get a system that can run ... raspbian Not like the Pi Foundation couldn't have contributed to uboot or the upstream kernel or anything before thier hardware was released. Pretty much nothing released for "Raspberry Pi" runs on a Pi 5 at this point in time. They better try to salvage something before putting out the expected CM5, but I'm not expecting much.

Dylan16807 2y ago
The power supply is becoming less special over time. My first result for "usb c power supply" on amazon is a $25 anker 317 that supports PPS and will do 5 amps at any voltage. Not that that's an amazing price, but the feature is spreading. But I agree that the price to get a happy Pi is pretty harsh compared to a tiny PC.
- pbnjeh 2y ago
  I don't have Pi 5 to check this, but from what I've read, it does not support PPS. And such support is required; otherwise, even with a source that supports PPS, the connection will behave in a non-PPS fashion. Apparently, cable quality is also demonstrating itself to be a significant factor in the Pi 5 power supply environment. Regarding both these points: https://github.com/raspberrypi/rpi-eeprom/issues/497 https://github.com/raspberrypi/rpi-eeprom/issues/497
  - Dylan16807 2y ago

champtar 2y ago

In OpenWrt (Linux distro for routers that often don't have RTC) dnsmasq starts with `--dnssec-no-timecheck` until the ntp client gets a first sync (https://github.com/openwrt/openwrt/commit/5acfe55d7139a5294192bddf10fe3a1de3180e8d https://github.com/openwrt/openwrt/commit/5acfe55d7139a52941...)

raggi 2y ago

If you don't have an RTC I'd recommend having a tlsdate with some bounding heuristics to prevent extreme clock fixation from a mitm. You can relatively cheaply hit a large number of public servers that are likely to have good times available over TLS and trust the common result. You validate the certs without considering the notbefore stamps and then if you're feeling aggressive validate them after you've managed to approximate a date from the cohort. I know there are commercial packages that do this, I'm not sure about OSS ones. Roughtime would be far better, but essentially there's no broad deployment of it yet. Ideally something good would be picked by Raspbian and delivered in the distro as standard.

cyounkins 2y ago

I ran into this before! https://medium.com/@cyounkins/encrypted-dns-ntp-deadlock-9e378940b79f https://medium.com/@cyounkins/encrypted-dns-ntp-deadlock-9e3...