4 ms·

Rebuilding a 1995 GPS Time Server so I don't get Telstra'd

39 points by Brajeshwar 1mo ago

itchingsphynx 1mo ago
After the Telstra NTS issue and preparing for the upcoming summer storms and power outages, I set up a chrony mesh on three devices, each with a different upstream stratum 1 NTS. I seriously considered a GPS reference too, but I live on the bottom floor of apartment blocks so reception is probably quite poor. After blocking the firewall ports, the mesh stays linked and drift is well within sensible boundaries for the typical length of internet and power outages of a few hours. Although, a man with a watch knows the time but man with two watches cannot be certain.
- Palomides 1mo ago
  I have a GPS based PTP server, the antenna is indoors 15 feet away from a south facing window, it locks on pretty well (though this will probably depend on the surrounding buildings)
  - geerlingguy 1mo ago
    It's surprising how often I can get an okay-ish GPS signal indoors... it seems to have a lot more to do with building materials than anything else. Multipath reflections means the location (and thus timing accuracy) wanders a lot, but it's still great down to microseconds if you can get 5+ satellites in view.
- anitil 1mo ago
  This sounds pretty interesting - what are you working on where it's necessary to maintain this level of time precision? I've worked on positioning systems where it's obviously important, but it never mattered much at home
  - itchingsphynx 1mo ago
    Nothing special - just a homelab. It's not so much the precision that matters at home as for planning when the home becomes an island, in 'orphan mode' as chrony calls it [1], without a time reference at all. Time really is a 'sovereign function' per Telstra's report [2], especially if you have internal HTTPS, etc. Last summer an 8+ hour mains power outage drained my home CWWK OPNsense router's UPS then internal battery (it must have been losing voltage). When mains power came up the router was unable to connect to the ISP because the internal time was completely wrong and I had to manually set it via console. However, the internal batteries and clocks on other home network infrastructure (HP NAS and Lenovo Tiny) were fine. The chrony.conf orphan mode: server 192.168.1.1 server 192.168.1.2 local stratum X orphan (higher X determines who is master in orphan mode, usually minimum 10) Annoyingly, the OPNsense chronyd plug-in does not (yet) implement the "local orphan" mode [3], so I have not yet confirmed that it does accept the non-reference master orphan time to bootstrap itself... should test before the summer storms start again. 1. <https://chrony-project.org/doc/latest/chrony.conf.html https://chrony-project.org/doc/latest/chrony.conf.html> 2. <https://www.telstra.com.au/exchange/what-we-ve-learned-from-the-external-investigation-of-our-july-o https://www.telstra.com.au/exchange/what-we-ve-learned-from-...> 3. <https://github.com/opnsense/plugins/issues/5115 https://github.com/opnsense/plugins/issues/5115>
    - anitil 29d ago
      Ah I see in my head I expect blackouts to be a few minutes, so I hadn't thought what would happen after several hours. When I was in the embedded world I used to explain that https implies a whole lot of infrastructure that isn't obvious at first for this reason (and also that cert rotation implies the ability to update the firmware or some filesystem which isn't always the case).
  - ErroneousBosh 1mo ago
    If we're going to be stupid with it, I should write up an article on my ultra-precise 16MHz Arduino that runs at cock on 16000000Hz. The crystal is replaced by a GPS-disciplined VCO. Why? Well, I wasn't doing anything else much fun...
throw0101a 1mo ago
On Telstra, they brought in an external auditor and the report was recently release: * Post: https://www.telstra.com.au/exchange/what-we-ve-learned-from-the-external-investigation-of-our-july-o https://www.telstra.com.au/exchange/what-we-ve-learned-from-... * Report: https://www.telstra.com.au/content/dam/tcom/dynamic-media-projects/luke-campbell/TAP-Findings-for-Telstra-Outage.pdf https://www.telstra.com.au/content/dam/tcom/dynamic-media-pr... * https://news.ycombinator.com/item?id=49549656 https://news.ycombinator.com/item?id=49549656
- anitil 1mo ago
  Thanks for sharing that I'm very interested to read the auditor's results Edit: That's it!? It's also not clear on whether it was a configuration or a firmware issue (or both). It sounds like maybe there was some battery backed ram or something that reset maybe? Edit number 2: Ok I re-read it and it is better than I thought, still many questions though.
aeyes 1mo ago
The author is not rebuilding anything, they just stuck a Raspberry Pi in the case.
- ssl-3 1mo ago
  They kept the LCD dot matrix character display, the dual-color status LED, and the chassis. All of the other original guts remain in-place; it's completely reversible and no new holes were made. I agree that rebuilding is an awkward fit here. Repurposing fits better in some ways and worse in others. edit: Maybe retromod works. New stuff in an old box, with continuation of parts of the original vibe.
  - geerlingguy 1mo ago
    In the spirit of a 1990s time server is I also got Timelord + Tardis working (time service over AppleTalk, tested on a Mac SE/30, courtesy of netatalk) and tested multiple NTPv1, NTPv3, and NTPv4 clients through this (and soon an Xserve G5). It's less about getting the 1995 hardware working, and more about rebuilding the actual time server to serve the original purpose, for the next 5-10 years at least. I'm planning on using this chassis to build up an NTP Stratum 1 server for the global NTP pool, hopefully later this year. I thought it would be a lot more fun than letting the box go to e-waste. And leaving the guts completely untouched means I can do a full restoration someday, if I want. The two main things I'd need to do: 1. Modify the RF chain so I can use normal GPS antennas that don't down-convert the frequency. 2. Modify the DC bias voltage so it doesn't fry GPS antennas. After that, I could work on patching firmware, but I'd probably just use the original guts as a frequency reference and leave time to other hardware. I could even run the Pi and the old hardware simultaneously. The old hardware providing IRIG-B, 10 MHz, and 1 PPS references that are a little more accurate (it has a TCXO, haven't measured its timing accuracy yet). And the Pi could provide PTP and NTP, using a more modern GPS chip which even supports dual band and survey-in for more accuracy.
    - anitil 1mo ago
      I thought it was a fun video (the first I've watched of yours, though I read the blog). And I'd honestly forgotten about the Telstra outage even though it was news here
RachelF 1mo ago
This is not a great idea. Pre-2000 GPS time server firmwares do no handle leap seconds properly.

39 points by Brajeshwar 1mo ago

After the Telstra NTS issue and preparing for the upcoming summer storms and power outages, I set up a chrony mesh on three devices, each with a different upstream stratum 1 NTS. I seriously considered a GPS reference too, but I live on the bottom floor of apartment blocks so reception is probably quite poor. After blocking the firewall ports, the mesh stays linked and drift is well within sensible boundaries for the typical length of internet and power outages of a few hours. Although, a man with a watch knows the time but man with two watches cannot be certain.

Palomides 1mo ago
I have a GPS based PTP server, the antenna is indoors 15 feet away from a south facing window, it locks on pretty well (though this will probably depend on the surrounding buildings)
- geerlingguy 1mo ago
  It's surprising how often I can get an okay-ish GPS signal indoors... it seems to have a lot more to do with building materials than anything else. Multipath reflections means the location (and thus timing accuracy) wanders a lot, but it's still great down to microseconds if you can get 5+ satellites in view.
anitil 1mo ago
This sounds pretty interesting - what are you working on where it's necessary to maintain this level of time precision? I've worked on positioning systems where it's obviously important, but it never mattered much at home
- itchingsphynx 1mo ago
  Nothing special - just a homelab. It's not so much the precision that matters at home as for planning when the home becomes an island, in 'orphan mode' as chrony calls it [1], without a time reference at all. Time really is a 'sovereign function' per Telstra's report [2], especially if you have internal HTTPS, etc. Last summer an 8+ hour mains power outage drained my home CWWK OPNsense router's UPS then internal battery (it must have been losing voltage). When mains power came up the router was unable to connect to the ISP because the internal time was completely wrong and I had to manually set it via console. However, the internal batteries and clocks on other home network infrastructure (HP NAS and Lenovo Tiny) were fine. The chrony.conf orphan mode: server 192.168.1.1 server 192.168.1.2 local stratum X orphan (higher X determines who is master in orphan mode, usually minimum 10) Annoyingly, the OPNsense chronyd plug-in does not (yet) implement the "local orphan" mode [3], so I have not yet confirmed that it does accept the non-reference master orphan time to bootstrap itself... should test before the summer storms start again. 1. <https://chrony-project.org/doc/latest/chrony.conf.html https://chrony-project.org/doc/latest/chrony.conf.html> 2. <https://www.telstra.com.au/exchange/what-we-ve-learned-from-the-external-investigation-of-our-july-o https://www.telstra.com.au/exchange/what-we-ve-learned-from-...> 3. <https://github.com/opnsense/plugins/issues/5115 https://github.com/opnsense/plugins/issues/5115>
  - anitil 29d ago
    Ah I see in my head I expect blackouts to be a few minutes, so I hadn't thought what would happen after several hours. When I was in the embedded world I used to explain that https implies a whole lot of infrastructure that isn't obvious at first for this reason (and also that cert rotation implies the ability to update the firmware or some filesystem which isn't always the case).
- ErroneousBosh 1mo ago
  If we're going to be stupid with it, I should write up an article on my ultra-precise 16MHz Arduino that runs at cock on 16000000Hz. The crystal is replaced by a GPS-disciplined VCO. Why? Well, I wasn't doing anything else much fun...

throw0101a 1mo ago

On Telstra, they brought in an external auditor and the report was recently release: * Post: https://www.telstra.com.au/exchange/what-we-ve-learned-from-the-external-investigation-of-our-july-o https://www.telstra.com.au/exchange/what-we-ve-learned-from-... * Report: https://www.telstra.com.au/content/dam/tcom/dynamic-media-projects/luke-campbell/TAP-Findings-for-Telstra-Outage.pdf https://www.telstra.com.au/content/dam/tcom/dynamic-media-pr... * https://news.ycombinator.com/item?id=49549656 https://news.ycombinator.com/item?id=49549656

anitil 1mo ago
Thanks for sharing that I'm very interested to read the auditor's results Edit: That's it!? It's also not clear on whether it was a configuration or a firmware issue (or both). It sounds like maybe there was some battery backed ram or something that reset maybe? Edit number 2: Ok I re-read it and it is better than I thought, still many questions though.

aeyes 1mo ago

The author is not rebuilding anything, they just stuck a Raspberry Pi in the case.

ssl-3 1mo ago
They kept the LCD dot matrix character display, the dual-color status LED, and the chassis. All of the other original guts remain in-place; it's completely reversible and no new holes were made. I agree that rebuilding is an awkward fit here. Repurposing fits better in some ways and worse in others. edit: Maybe retromod works. New stuff in an old box, with continuation of parts of the original vibe.
- geerlingguy 1mo ago
  In the spirit of a 1990s time server is I also got Timelord + Tardis working (time service over AppleTalk, tested on a Mac SE/30, courtesy of netatalk) and tested multiple NTPv1, NTPv3, and NTPv4 clients through this (and soon an Xserve G5). It's less about getting the 1995 hardware working, and more about rebuilding the actual time server to serve the original purpose, for the next 5-10 years at least. I'm planning on using this chassis to build up an NTP Stratum 1 server for the global NTP pool, hopefully later this year. I thought it would be a lot more fun than letting the box go to e-waste. And leaving the guts completely untouched means I can do a full restoration someday, if I want. The two main things I'd need to do: 1. Modify the RF chain so I can use normal GPS antennas that don't down-convert the frequency. 2. Modify the DC bias voltage so it doesn't fry GPS antennas. After that, I could work on patching firmware, but I'd probably just use the original guts as a frequency reference and leave time to other hardware. I could even run the Pi and the old hardware simultaneously. The old hardware providing IRIG-B, 10 MHz, and 1 PPS references that are a little more accurate (it has a TCXO, haven't measured its timing accuracy yet). And the Pi could provide PTP and NTP, using a more modern GPS chip which even supports dual band and survey-in for more accuracy.
  - anitil 1mo ago
    I thought it was a fun video (the first I've watched of yours, though I read the blog). And I'd honestly forgotten about the Telstra outage even though it was news here

RachelF 1mo ago

This is not a great idea. Pre-2000 GPS time server firmwares do no handle leap seconds properly.