25 ms·

Open-sourcing a more precise time appliance

498 points by alexvoica 5y ago

roamingryan 5y ago
Really nice of them to release this. However, unless I'm missing something, the HDL for the FPGA on the Time Card is not part of the release. It looks like it uses a proprietary IP core from NetTimeLogic https://github.com/opencomputeproject/Time-Appliance-Project/tree/master/Time-Card/SOM/FPGA https://github.com/opencomputeproject/Time-Appliance-Project... They do include FPGA bit streams, but it's disingenuous to claim a fully open source release.
tia86 5y ago
I don't see a big deal here. You are using a very expensive free-running time-base (MAC - Rb clock) to implement the holdover. These MACs are really expensive, and basically unobtanium if you are not a big corp. You should have done it with a DOCXO and some clever training software to correct the DOCXO vs temperature and aging when GPS tracking is lost.
nefitty 5y ago
I didn’t want to click this so I assume others here won’t either. Here’s the top points they highlight themselves: - Facebook engineers have built and open-sourced an Open Compute Time Appliance, an important component of the modern timing infrastructure. - To make this possible, we came up with the Time Card — a PCI Express (PCIe) card that can turn almost any commodity server into a time appliance. - With the help of the OCP community, we established the Open Compute Time Appliance Project [1] and open-sourced every aspect of the Open Time Server.[2] 1. https://www.opencompute.org/projects/time-appliances-project-tap-incubation https://www.opencompute.org/projects/time-appliances-project... 2. https://github.com/opencomputeproject/Time-Appliance-Project/tree/master/Open-Time-Server/ https://github.com/opencomputeproject/Time-Appliance-Project...
- the_biot 5y ago
  Whatever you may think of Facebook, they've done awesome things within their Open Compute Project. They also fund a lot of open source work.
  - nefitty 5y ago
    Yes, I appreciate their engineering work. I did have to click the link to post my comment. I posted it for other people’s benefit because I understand the concerns regarding their privacy record.
- efficax 5y ago
  Oh no if you click it Facebook might find out! The horror
  - nefitty 5y ago
    Yes, that’s why I posted that comment.
  - serf 5y ago
    what do you stand to gain by belittling someone's (absolutely sane) privacy considerations regarding facebook?
    - efficax 5y ago
      I don't gain anything, I dunno, just out here on these internets
kd913 5y ago
Why does Facebook need a precision time server?
- kfreds 5y ago
  From the article: "More accurate time keeping enables more advanced infrastructure management across our data centers, as well as faster performance of distributed databases."
  - kd913 5y ago
    That seems rather ambiguous and doesn't really explain why having more time precision results in faster performance or 'advanced infrastructure management'. I can understand why a stock broker or trading firm requires PTP to enable precise date-stamps for auditing/validating trades. I don't see how having a time granularity on the order of picoseconds is needed for a data center.
    - bohemian99 5y ago
      In distributed databases that offer transaction semantics, they need timestamps to order transactions that take place. A tighter synchronization of clocks mean they can execute transactions faster because they can reduce the amount of time they wait (based on the potential clock drift between machines in the data center)
      - williamscales 5y ago
        Wouldn’t it be better to put the effort towards designing a system that doesn’t need precise timekeeping?
    - doublea 5y ago
      See https://cloud.google.com/spanner/docs/true-time-external-consistency https://cloud.google.com/spanner/docs/true-time-external-con... It should explain how clock uncertainty relates to throughput of causally-related transactions.
rkangel 5y ago
I'm still unclear why they need the MAC on top of the GNSS reception. I can understand if it's just to provide holdover in event of GNSS signal loss, but they seem to be saying that it provides better precision when you've got both.
- hoseja 5y ago
  If I read correctly: GNSS PPS is more jittery but does not drift. The MAC has 1000x less jitter but drifts. You can cleverly combine them with statistics and magic to get the best of both worlds.
  - mananaysiempre 5y ago
    From what I remember of textbook discussions of atomic clocks, stabilizing drifting oscillators is also in part how normal, old (50+ years) atomic clocks work as well, isn’t it? You have a fiddly and unreasonably high-frequency reference, you slave a relatively garden-variety standard like an environmentally controlled piezoelectric crystal to it, and then feed your periodic signals, counters, etc. off that.
    - willis936 5y ago
      Yes. Surprisingly, atomic clocks have high jitter compared to ovenized quartz oscillators. Every atomic clock package is outputting its signal from a quartz oscillator that is disciplined by the atomic oscillator.
  - willis936 5y ago
    Phase-locked loops[0] are not magic ;) Also I imagine they are using a bog standard Kalman filter[1]. 0. https://en.wikipedia.org/wiki/Phase-locked_loop https://en.wikipedia.org/wiki/Phase-locked_loop 1. https://en.wikipedia.org/wiki/Kalman_filter https://en.wikipedia.org/wiki/Kalman_filter
    - Unklejoe 5y ago
      I've implemented this (using GPS PPS to discipline a clock) using regular old PI control loops. I just make the loop damped enough to filter out the GPS PPS jitter and you're good.
jedimastert 5y ago
Off topic: I'm reminded of a very small unit of time that was devised to be a least common multiple of various division of time like 24fps, 30fps, 44.1kHz, and a ton of others in video and audio. Does anyone remember what this unit was?
- smnscu 5y ago
  I think you might be referring to this https://en.wikipedia.org/wiki/Flick_(time) https://en.wikipedia.org/wiki/Flick_(time)
  - tyingq 5y ago
    Also, somewhat related, the "TU" https://en.m.wikipedia.org/wiki/TU_(Time_Unit) https://en.m.wikipedia.org/wiki/TU_(Time_Unit)
  - jedimastert 5y ago
    That's it! Thank you very much. I searched around on Wikipedia for it and couldn't find it. I might recommend some edits when I have the time.
- hoseja 5y ago
  Do you mean the facebook/oculus flicks? Wonder why the repo is archived... https://github.com/facebookarchive/Flicks https://github.com/facebookarchive/Flicks ninjaed :(
  - jedimastert 5y ago
    That's exactly it! Thank you. I would imagine it's archived because there's nothing else to really be done. It's basically a blog post. The header can apparently be replaced with three lines.
hoseja 5y ago
Relevant recent HN submission: https://news.ycombinator.com/item?id=28108586 https://news.ycombinator.com/item?id=28108586
michaelt 5y ago
A few years back I was interested in microsecond-precise timekeeping. My experience was GPS-backed network clocks were surprisingly cheap - but PTP-aware network switches were surprisingly expensive. And though I thought I could cut the cost of a grandmaster clock from $2000 down to $200, I figured there'd be no point if I then have to connect it to a $20,000 network switch. I wonder how Facebook are connecting their precise new time servers to their regular servers?
- MisterTea 5y ago
  This gear will get cheaper soon. There is a big push in the Industrial automation world to get TSN/PTP deployed more widely and perhaps universally. The idea is to eliminate the dichotomy of real time vs non real time Ethernet networks in control systems from the plant network down to the machine level. This way you can send EtherCAT frames at equidistant intervals to servo drives in the microsecond range while casually streaming SPC data or Netflix over the same wire. This is also in conjunction with the newer single pair Ethernet tech that the Automotive world is also interested in.
  - throw0101a 5y ago
    > single pair Ethernet For anyone wondering: > In addition to the more computer-oriented two and four-pair variants, the 100BASE-T1 and 1000BASE-T1 single-pair Ethernet PHYs are intended for automotive applications[17] or as optional data channels in other interconnect applications.[18] The single pair operates at full duplex and has a maximum reach of 15 m or 49 ft (100BASE-T1, 1000BASE-T1 link segment type A) or up to 40 m or 130 ft (1000BASE-T1 link segment type B) with up to four in-line connectors. Both PHYs require a balanced twisted pair with an impedance of 100 Ω. The cable must be capable of transmitting 600 MHz for 1000BASE-T1 and 66 MHz for 100BASE-T1. > Similar to PoE, Power over Data Lines (PoDL) can provide up to 50 W to a device.[19] * https://en.wikipedia.org/wiki/Ethernet_over_twisted_pair#Single-pair https://en.wikipedia.org/wiki/Ethernet_over_twisted_pair#Sin... > The IEEE 802.3bu-2016[12] amendment introduced single-pair Power over Data Lines (PoDL) for the single-pair Ethernet standards 100BASE-T1 and 1000BASE-T1 intended for automotive and industrial applications.[13] On the two-pair or four-pair standards, power is transmitted only between pairs, so that within each pair there is no voltage present other than that representing the transmitted data. With single-pair Ethernet, power is transmitted in parallel to the data. PoDL defines 10 power classes, ranging from .5 to 50 W (at PD). * https://en.wikipedia.org/wiki/Power_over_Ethernet#PoDL https://en.wikipedia.org/wiki/Power_over_Ethernet#PoDL 100BASE-T1 is IEEE 802.3bw-2015, 1000BASE-T1 is IEEE 802.3bp-2016. 2.5 Gb/s, 5 Gb/s, and 10 Gb/s over a single pair is 802.3ch-2020: the focus of these is in the embedded automotive space. * https://blog.siemon.com/standards/ieee-p802-3ch-multi-gig-automotive-ethernet-phy https://blog.siemon.com/standards/ieee-p802-3ch-multi-gig-au...
kfreds 5y ago
For anyone else wondering how their clients verify the authenticity of timestamps; It seems they use the NTP server software Chrony, which has support for NTS (Network Time Security, RFC 8915).
MisterTea 5y ago
> While building our Time Appliance, we also invented a Time Card, a PCIe card that can turn any commodity server into a time appliance. Invented? These cards have been around. e.g. Meinberg makes a variety of expansion cards which use various time keeping sources such as WWVB, IRIG-B, GNSS, etc.
- tyingq 5y ago
  Those have the external source but don't seem to have the internal reference (Rubidium, Cesium, etc) for holdover. FB seems to be claiming to have invented a PCI card that makes a commodity server into a Stratum-1 source. Meinburg sells that too, but in proprietary appliance form.
  - abyagowi 5y ago
    Again, invented might be too strong. This is an abstraction of what you need minimally to build a time server on an open source basis. We appreciate and are humble to other products in the market
    - tyingq 5y ago
      The availability of a much cheaper product than the appliances will likely create a bit of stir with competitors. The article mentions better management functions too. So perhaps invented is too strong, but it's more notable than your follow up comments suggest.
- KaiserPro 5y ago
  I read it that they've invented a product called time card, rather than the concept. Its an accurate and stable clock source. which is an important distinction. The Meinberg PCI cards appear not to have a local clock source thats well disciplined (ie a temperature compensated oscillator) Having an accurate time source that is updated once every second (typically) with a 1pps GPS is grand for most things. But if you need a stable clock as well, then you want a tiny atomic clock onboard as well. This is so you can run a clock source for an entire datacenter, or local section.
jrockway 5y ago
What's interesting to me is how the form factor of timing GPS modules has stayed constant over the years. I started my GNSS timing journey with a used Trimble GPS from the 2000s, and it has the same form factor and pinout as the modern multi-constellation timing GPSes from uBlox. I've had a GPS clock going for several years at this point, and without an atomic clock or really any fanciness (just LinuxPPS and Chrony), I see about +/- 380ns, which is pretty good. NTP to the Internet gives me jitter in the range of about 20ms-70ms, about 5 orders of magnitude worse. (The version a few iterations ago looked like this: https://github.com/jrockway/beaglebone-gps-clock https://github.com/jrockway/beaglebone-gps-clock. But I now have a uBlox multi-constellation GPS, which is much more accurate with my limited view of the sky from my Brooklyn apartment. And I 3D printed the case, so it actually looks presentable instead of like some crazed madman that attacked a plastic case with a hacksaw -- which is exactly how I made the first case. As for the DS3231 RTC that I added... that seems to be stable within about 1.5us, which is pretty impressive. I tuned it a little bit with the trim register, though.) My takeaway from this article is that the project seems quite compelling, but a lot of cost is added to support PTP. I see why Facebook needs that, but to sync time precisely throughout my lab, all I need is a coaxial cable with 10MHz on it. I don't mind if my workstation is 20ms off UTC.
- geoffeg 5y ago
  Nice project, I've been working on something similar, but with an Arduino (Adafruit ESP32), an Adafruit GPS featherwing and a RTC. I originally started with a Linux-based solution like yours but no matter what I did I couldn't get the accuracy below about 5ms and my goal is sub-ms (verified with an oscilloscope.)
  - jrockway 5y ago
    Yeah, I think you're on the right track. I like less and less having a Linux box to maintain, and there is less control over the timing of internal operations compared to a microcontroller, of course. On the other hand, Chrony is really, really good, and it would be a lot of code and debugging to write something that good from scratch. With the Beaglebone, it's theoretically possible to drive the oscillator from a calibrated clock source (use the PPS pulse to discipline an oscillator). I was too lazy to build such a circuit and observe the effects. Some uBlox timing GPSes let you output a suitable signal (you can adjust the frequency in software), but the adjustability range when I last checked wasn't enough to make it work for the Beaglebone. Once you have the MCU running at a well-defined and stable frequency, I think it's possible to give Linux extremely accurate data, or at least know which clock cycle was the start of a second. I don't actually know how much any of this will help, and it's honestly not within my capability to measure. For me, the outcome of my GPS clock project is that I never have to set my clock, and it looks really good. Running "chronyc sources" is always pleasurable as well, but I probably wouldn't notice a clock that's 20ms off unless I was comparing it against WWVB or something.
abyagowi 5y ago
I am very thankful to all of you putting all these questions and answers. We have a bi-weekly meeting under the open compute project and you can find previous talks there too. www.ocptap.com
delayclose2 5y ago
> The offset is practically 0 ranging within nanoseconds range. Uhm, no, the graph clearly shows that the offset is varying between 0-2 microseconds, which is 0 - 2,000 nanoseconds. 2,000 nanoseconds is not "practically zero". PTP precision is supposed to be in the 10ns range. Also, how did they get 40us precision on (what I assume to be) a normal kernel on a normal x86 box? I would have assumed that in a datacenter environment, with traffic levels being random, the jitter introduced in the kernel networking stack alone would be in the hundreds of microseconds at least.
- michaelt 5y ago
  PTP is a standard feature on a lot of Intel chipsets so a lot of servers have it 'for free' and aren't using it. It timestamps packets at the network card - independently of any kernel timing.
  - zinekeller 5y ago
    Addenda: Intel networking chips, so even in AMD or ARM systems this actually works (assuming that you have the correct OS and drivers/modules/support software). Also, this isn't exclusive to Intel: most modern (server) NICs at least support TCXO-based timings, and some built for precision uses OCXO for even better precision.
nonameiguess 5y ago
> Now, as long as printing the PCB and soldering tiny components does not sound scary, anyone can build their own Time Card for a fraction of the cost of a regular time appliance. Doesn't sound scary, but not a lot of research labs and data centers are going to want to build their expansion cards from parts, even given free access to instructions. I understand if there isn't enough of a market for this to make it a viable commercial product, but when you have more money than God anyway, what's the reason not to make a few extra and donate or sell them to the small number of users who can use it?
- abyagowi 5y ago
  Of course we are not going to build the card in the lab with assembling SMD parts. This was more a way to explain that building such a card should not be out of access for anyone
  - myself248 5y ago
    I mean, you can buy a syringe of solder paste for thirty bucks and do the reflow in a toaster oven. Anyone not doing it simply isn't trying.
    - abyagowi 5y ago
      You are right. This sentence was not supposed to be a literal description of what is the best way to build the card. It was more referring to the fact the building this card is not that hard since the main goal for this project was to liberate the time appliances from being a closed source box.
- Shish2k 5y ago
  From https://github.com/opencomputeproject/Time-Appliance-Project/tree/master/Time-Card https://github.com/opencomputeproject/Time-Appliance-Project... > Where can I get one? > […] we are currently working with several suppliers and will have their contact info soon available to allow you to puchase an out-of-the-box ready Time Card.
jeffbee 5y ago
Slaving your clocks to GPS seems like it just gives the USAF space command a way to, eventually, wreck your operations. If you don’t have requirements for accuracy referenced to civil time (such as mifid 2) why wouldn’t you just go with the atomic clocks alone, with gps as an advisory-only monitoring reference?
- nullc 5y ago
  Long term drift on these inexpensive secondary atomic clocks isn't that amazing, sadly.
latchkey 5y ago
A very long time ago, HP made a GPS time card just like this, encased in glass and filled with an inert gas such that if the card was tampered with, it was rendered useless. This prevented someone from faking the time signal.
- audiometry 5y ago
  Someone send one of these to CuriousMarc — he is peerless in working on these vintage high precision test pieces.
- BonoboIO 5y ago
  Can you show us a picture. Did not find anything on google beside HP Laptop spam.
  - latchkey 5y ago
    I'm not sure it was ever sold to the public. Buddy of mine owns timestamp.com and we were building a service where you could hash/timestamp emails (we would take over the MX for a domain and become a forwarding service). All of this was long before we had blockchains. Sadly, it never really took off so we abandoned the idea (what is up there now is currently another half attempt at the idea). Others have tried, but it isn't really a huge demand despite the apparent need.
NelsonMinar 5y ago
Man, tough crowd here. And someone who's been following precise timekeeping tech for 25 years now.. this is really cool! I think it's great some smart engineers have built a new reasonably compact and low cost very precise bit of timekeeping hardware. Thank you for sharing it open source!
- toomuchtodo 5y ago
  I am torn. I absolutely love the technical blog post and the effort that went into open sourcing everything needed to build these devices, but, you know, Facebook. Compliments and respect to the engineers involved for really well done engineering.
  - eplanit 5y ago
    They've built and earned their (negative) reputation. This is cool, but it's like Philip Morris Tobacco Company coming out with a useful home appliance. Uhh....thanks?
    - chillingeffect 5y ago
      Even a clock that moves backwards is right four times a day :)
      - ceejayoz 5y ago
        Move the hands fast enough and it's right every second of every day.
    - tourist2d 5y ago
      Except it's a software company releasing an open source software project?
  - judge2020 5y ago
    Amazing people at Facebook make amazing things, but it does seem like only the worst of them (FB, Google) have the cash to burn to hire teams of engineers for projects outside of ones that directly contribute to generating revenue.
adolph 5y ago
Seiko offers a GPS clock in a conventional-home friendly form factor. First introduced in 2013, the Seiko Space Link GPS clock has proven to be a global success. It is the world's first clock for home or office use that receives a time signal from a GPS satellite, thus delivering atomic clock precision. It measures time to within one second every 100,000 years. Equally importantly, it can be used anywhere in the world and can be installed in almost any situation, so long as it is around 15 metres or less from the open sky. The technology is beautifully practical. 4 times every day, the clock receives a time signal from the GPS satellite network and, when conditions are ideal, adjusts automatically in around 10 seconds of receiving the signal. Between these signal receptions, a high accuracy quartz movement keeps the clock on time. Wherever you may need it, Seiko Space Link is there, silently marking the passage of time with an elegance and precision that is redefining the world’s perception of what a clock can be. A new standard of excellence has arrived. https://www.seiko-clock.com/products.html https://www.seiko-clock.com/products.html
- ipsum2 5y ago
  This is a standard "atomic clock", that its not actually uses radioactive decay to accurately measure the passage of time, but receives GPS signals. The innovation in the post is using real "miniature atomic clock" that actually uses caesium 133.
ChuckMcM 5y ago
This is a pretty neat thing and kudos for them open sourcing it. Typically, you'd have to hand over $5K - $25K for this card! This can hopefully make them available at a lower price. I built something similar (although I didn't use a CSAC[1] since it was too expensive but I did have a nice OXCO which is pretty stable. I use a Raspberry Pi to distribute plain old TOD via NTP to my local network, but the 10 MHz output I send to my bench where I can use it for radio projects. It is actually a better clock source (frequency stability wise) than the precision time source in my HP spectrum analyzer but it has worse phase noise (jitter) because it does get corrected by up to +/- 20nS periodically. [1] Chip Scale Atomic Clock - Microsemi -> Microchip https://www.microsemi.com/product-directory/clocks-frequency-references/3824-chip-scale-atomic-clock-csac https://www.microsemi.com/product-directory/clocks-frequency...
jameshart 5y ago
They call it a “Time Appliance”, or a “Time Device”. Missed opportunity to call it a “time machine”. Or, I guess, more prosaically, a “clock”.
isodev 5y ago
Congrats to everyone who made this project possible. It’s certainly a great accomplishment and it solves a real world problem. Now, does it require a Facebook account with my real name in order to use? /s
- yitchelle 5y ago
  What real world problem does this actually solve that has not been solved earlier? Genuinely curious.
  - bobsmooth 5y ago
    Not having a low-cost extremely accurate Stratum 1 in a PCIe form-factor.
triactual 5y ago
What are the pressure sensor and IMU for? You've open-sourced a dev-board version, I'm assuming you'll keep the production version private?
- abyagowi 5y ago
  Production version will come today or tomorrow. Check the github
deleted 5y ago
[deleted]
daddylonglegs 5y ago
I cannot find any source files for the FPGA code in the repository (no .v .vhd .vhdl files and I can't see anything by looking around manually). The FPGA directory [1] contains some binaries and some documents relating to commercial code from a Swiss firm [2]. I've seen plenty of projects described as open source when the firmware is open source and the hardware is closed but this is the first time I've seen one where the schematics and board layout are open but the firmware is closed. Note that the hardware itself is quite simple, the smart stuff happens in three modules: GNSS receiver, precision clock and FPGA. To me, the contents of the FPGA source code are the only interesting part of this project. Additionally, you won't be able to meaningfully modify or reuse this project without editing the source code. The hardware module these FPGA binaries seem to be compiled for is described as an AC7100B, the source given for these is a defunct ebay link [3] [4] :-O This project uses two and half grand's worth of atomic clock and the heart of it runs on a module that fell off the back of a lorry!? [1] https://github.com/opencomputeproject/Time-Appliance-Project/tree/master/Time-Card/SOM/FPGA https://github.com/opencomputeproject/Time-Appliance-Project... [2] https://www.nettimelogic.com/clock-products.php https://www.nettimelogic.com/clock-products.php [3] https://github.com/opencomputeproject/Time-Appliance-Project/tree/master/Time-Card/SOM https://github.com/opencomputeproject/Time-Appliance-Project... [4] https://www.ebay.com/itm/XINLINX-A7-FPGA-Development-board-Artix-7-Core-board-FPGA-Minimum-System/253903903610?ssPageName=STRK%3AMEBIDX%3AIT&var=553209057783&_trksid=p2060353.m2749.l2649 https://www.ebay.com/itm/XINLINX-A7-FPGA-Development-board-A... Edit: I realise this comes across as quite negative. This looks like a neat project, one I would actually use, but only if it were meaningfully open source.
- m-ee 5y ago
  Valve did something similar for the Vive tracker dev kit. There was a time before the standalone tracker was available https://www.vive.com/us/accessory/vive-tracker/ https://www.vive.com/us/accessory/vive-tracker/ where if you wanted a tracker that wasn't a full controller you had to build it yourself. This involved a paid workshop in Seattle to get access to the dev kit and learn their tooling. Hardware was open source, but firmware was closed. If you wanted firmware changes you had to pay Synapse consulting fees for them to do it for you, and the updated firmware would then be made available free of charge to anyone else who paid for the workshop.
neilv 5y ago
Facebook isn't a good fit for me, but their work on open hardware for data centers has occasionally looked appealing. I figure units doing open hardware could be like when the old SNL sketch about AT&T -- having the slogan "We don't care. We don't have to. We're the phone company." -- was of the same company that hosted Bell Labs.
Tepix 5y ago
This seems like a cool project. I didn't see a price estimate for this PCIe card. However, isn't it smarter to try to build an infrastructure/architecture that can deal with somewhat less precise time information (milliseconds instead of nanoseconds)?
jandrese 5y ago
Oh cool, an open source Symmetricom card. These timing cards are low volume products so they have a bad habit of going out of support after a few years and then the kernel level drivers don't get updated and there is hardly any community. Having open hardware that multiple vendors can support makes them easier to get added to the mainline kernel.
- abyagowi 5y ago
  Thats exactly the point for this project. Also, I am hoping once we release the FPGA HDL files, the community will start adding more features like IRIG-B output, etc...
cedricgle 5y ago
If you want the go under the nanosecond there is the White Rabbit [1][2] project. The CERN uses it to control the magnets in the loop of a particule accelator. It is also based on NTP, Ethernet and FPGA. [1]: http://www.ohwr.org/projects/white-rabbit http://www.ohwr.org/projects/white-rabbit [2]: https://en.wikipedia.org/wiki/White_Rabbit_Project https://en.wikipedia.org/wiki/White_Rabbit_Project
- profquail 5y ago
  This was standardized as the L1 Sync feature of IEEE1588:2019, as described here: https://blog.meinbergglobal.com/2020/09/17/whats-new-in-ieee-1588-2019-l1-sync/ https://blog.meinbergglobal.com/2020/09/17/whats-new-in-ieee...
karmicthreat 5y ago
How well does transaction rate increase with more precise time? Seems like it would be a sort of O(log n) progression. Though could very well just be linear as well.
tibbydudeza 5y ago
I wonder what Google uses for Spanner/Firebase.
riobard 5y ago
Do smartphones get their time from GPS?
- kube-system 5y ago
  I believe they usually get it from the network.
  - riobard 5y ago
    That was my impression as well but now I’m puzzled about why. Since most smartphones have GPS builtin why not get time from GPS?
    - becauseiam 5y ago
      Mobile basebands already get the local time and UTC offset when connecting to a tower, there's no need to power up the GPS antenna and receiver separately.
    - jlgaddis 5y ago
      Mostly because 1) your phone is getting time from the tower anyways (they have to be closely sync'd) and 2) GPS doesn't always work very well -- especially indoors.
      - riobard 5y ago
        So every cell tower runs an NTP server?
adolph 5y ago
I don't see ESP8266 mentioned here: Overview: https://hackaday.com/2021/07/25/portable-gps-time-server-powered-by-the-esp8266/ https://hackaday.com/2021/07/25/portable-gps-time-server-pow... Developer story: https://www.linkedin.com/pulse/iot-maker-tale-stratum-1-time-server-built-from-scratch-monteiro/ https://www.linkedin.com/pulse/iot-maker-tale-stratum-1-time... Repo: https://github.com/Montecri/GPSTimeServer https://github.com/Montecri/GPSTimeServer A simpler version from the comments: https://enginemonitor.blogspot.com/2020/09/gps-module.html?m=1 https://enginemonitor.blogspot.com/2020/09/gps-module.html?m...
- jlgaddis 5y ago
  Probably because an ESP8266 isn't anywhere close enough to "real-time" to be of any benefit. Why bother getting a nanosecond-level accurate time signal if you're going to feed it to an ESP8266 to distribute over Wi-Fi (or, in other words, reducing your accuracy to ~10 milliseconds or so)?
johnea 5y ago
Didn't DJB suggest massive improvements to NTP decades ago? https://cr.yp.to/libtai.html https://cr.yp.to/libtai.html No one seemed to care. This system seems to be special only because it's from MyFace. Additionally, everyone has direct access to stratum 1 timing via cheap GPS receivers. TimeNuts have been doing this stuff for decades: http://www.leapsecond.com/time-nuts.htm http://www.leapsecond.com/time-nuts.htm Again, this seems just another piece of bullshit for the zukers and other MyFace lusers...
walrus01 5y ago
> the ConnectX-6 Dx network card I would caution people against using the Mellanox/NVIDIA network interface cards in anything they care about. The driver license is proprietary and you have to build it for your kernel as a DKMS module in a semi manual process. Makes future system updates a real bother. I realize this probably doesn't apply so much to FB internally since they build and maintain their own distro in house. Stuff that's just a couple of years old won't even build the driver for current debian bullseye. Have to run buster? No thanks. As with so many other things NVIDIA the root cause of this is their absurd licensing approach to open source software and drivers for their hardware.
- the_only_law 5y ago
  I’ve been wondering about SmartNICs recently and imagine all sorts of ideas for them. Which brand do you recommended? > Makes future system updates a real bother. Sounds like their GPU drivers too.
  - walrus01 5y ago
    I recommend Intel for 100GbE stuff. Intel's heritage of proper open source drivers (written by their own staff, in house) for FreeBSD and Linux goes back really far. You could look at the man pages for their very expensive 10GbE server NICs in 2003-2004 and see the @intel.com email addresess of the persons who wrote them.
  - kd913 5y ago
    How about the x2 or Alveo series from Xilinx? It is basically the original definer of smartnic. It comes from solarflare who have a long pedigree of low-latency smartnics. They used to supply Cloudflare, and also supply like 50% of fintechs/financial markets. You can also just use openonload to accelerate your programs. In this case just doing straight linux socket programming, which can be accelerated without dpdk. Or just use the open source linux net driver if necessary. https://github.com/Xilinx-CNS/onload https://github.com/Xilinx-CNS/onload
- somebody_amzn 5y ago
  (disclaimer: am an AWS employee) Hello, Where did you get that from? The drivers for NVIDIA networking cards are in the upstream kernel. You have the option of using newer versions from DKMS if the built-in version for your kernel is too old. You can see the release notes for each version as present on the upstream kernel at https://docs.mellanox.com/display/kernelupstreamv512/Linux+Kernel+Upstream+Release+Notes+v5.12 https://docs.mellanox.com/display/kernelupstreamv512/Linux+K... for Linux 5.12 for example.
jlgaddis 5y ago
Well, dammit. Of course they come out with this now, after I've got a few antennas up on the roof, half a dozen GPS-disciplined OCXOs providing 10 MHz and PPS signals throughout the house, which are being fed into SolarFlare PTP NICs, and sync'd throughout the network using PTP. Apparently my workstations' offsets (RMS) are currently 17 and 21 nanoseconds, though, so I suppose there's still room for improvement.
CrazyCatDog 5y ago
Are there further details available about the MAC? Growing up in Boulder, THE atomic clock (NIST) was touted as a modern marvel (circa early 90’s childhood), it’s incredible to think that similar performance is now available on a pci-e card! Any relation to DARPA’s “making progress” announcement in August 2018 [1]? [1] https://www.darpa.mil/news-events/2019-08-20 https://www.darpa.mil/news-events/2019-08-20
pizza 5y ago
> The new service, built in-house and later open-sourced, was more scalable and improved the accuracy of timekeeping in the Facebook infrastructure from 10 milliseconds to 100 microseconds. Interesting.. the infrastructure must therefore have a diameter of at most 30 km, thanks to the speed of light
c_o_n_v_e_x 5y ago
My experience is more on the the navigation side. Wouldn't running an RTK GNSS receiver further improve accuracy and jitter? There's reasonably priced RTK hardware out there now like SwiftNav. Another option is Ublox modules with open source RTKLIB.
Animats 5y ago
How resistant is this to a GPS attack? It has an atomic clock, so it should be producing good data for days while sending alarms in the event the GPS data is bogus. You don't want to create a situation where someone aims a small dish at the roof of your data center and takes down the networking.
- abyagowi 5y ago
  I have been working on a version with two GNSS receivers L1/L2 and L1/L5. The SA53 receives two PPS inputs and that allows you to detect Spoofing and Jamming. Again, I like to emphasize, this is not a done work. We shared whatever we had and we would love to have the community to contribute and help making it better. I love to have as many applications as possible for this Time Card. For example, I am looking for someone willing to port/write a Windows driver for the time card. With regards to the VHDL code; we are working with NetTimeLogic on a version without the need of any of theirs proprietary IPs. Meanwhile, we provided you with whatever we have and we look forward to working with you. Please feel free reach out to me if you like to collaborate on this project.
traceroute66 5y ago
Am I missing something? The FB blog proudly states how they improved accuracy "10 milliseconds to 100 microseconds". Surely that's a lot of effort that anybody with in a datacentres with decent connectivity could replicate with a Chrony instance ? 100 microseconds is 0.0001 seconds, right ? Seems to be something Chrony can do with its eyes closed: System time : 0.000003297 seconds slow of NTP time Last offset : +0.000000549 seconds There are lots of great NTP servers out there hosted by trusted parties such as national time labs. Most people don't need fancy GPS hardware (and the expense of paying someone for roof access to put your antenna !). If you've got good software and good connectivity you can replicate 99.999% of the same thing.

498 points by alexvoica 5y ago

Really nice of them to release this. However, unless I'm missing something, the HDL for the FPGA on the Time Card is not part of the release. It looks like it uses a proprietary IP core from NetTimeLogic https://github.com/opencomputeproject/Time-Appliance-Project/tree/master/Time-Card/SOM/FPGA https://github.com/opencomputeproject/Time-Appliance-Project... They do include FPGA bit streams, but it's disingenuous to claim a fully open source release.

tia86 5y ago

I don't see a big deal here. You are using a very expensive free-running time-base (MAC - Rb clock) to implement the holdover. These MACs are really expensive, and basically unobtanium if you are not a big corp. You should have done it with a DOCXO and some clever training software to correct the DOCXO vs temperature and aging when GPS tracking is lost.

nefitty 5y ago

I didn’t want to click this so I assume others here won’t either. Here’s the top points they highlight themselves: - Facebook engineers have built and open-sourced an Open Compute Time Appliance, an important component of the modern timing infrastructure. - To make this possible, we came up with the Time Card — a PCI Express (PCIe) card that can turn almost any commodity server into a time appliance. - With the help of the OCP community, we established the Open Compute Time Appliance Project [1] and open-sourced every aspect of the Open Time Server.[2] 1. https://www.opencompute.org/projects/time-appliances-project-tap-incubation https://www.opencompute.org/projects/time-appliances-project... 2. https://github.com/opencomputeproject/Time-Appliance-Project/tree/master/Open-Time-Server/ https://github.com/opencomputeproject/Time-Appliance-Project...

the_biot 5y ago
Whatever you may think of Facebook, they've done awesome things within their Open Compute Project. They also fund a lot of open source work.
- nefitty 5y ago
  Yes, I appreciate their engineering work. I did have to click the link to post my comment. I posted it for other people’s benefit because I understand the concerns regarding their privacy record.
efficax 5y ago
Oh no if you click it Facebook might find out! The horror
- nefitty 5y ago
  Yes, that’s why I posted that comment.
- serf 5y ago
  what do you stand to gain by belittling someone's (absolutely sane) privacy considerations regarding facebook?
  - efficax 5y ago
    I don't gain anything, I dunno, just out here on these internets

kd913 5y ago

Why does Facebook need a precision time server?

kfreds 5y ago
From the article: "More accurate time keeping enables more advanced infrastructure management across our data centers, as well as faster performance of distributed databases."
- kd913 5y ago
  That seems rather ambiguous and doesn't really explain why having more time precision results in faster performance or 'advanced infrastructure management'. I can understand why a stock broker or trading firm requires PTP to enable precise date-stamps for auditing/validating trades. I don't see how having a time granularity on the order of picoseconds is needed for a data center.
  - bohemian99 5y ago
    In distributed databases that offer transaction semantics, they need timestamps to order transactions that take place. A tighter synchronization of clocks mean they can execute transactions faster because they can reduce the amount of time they wait (based on the potential clock drift between machines in the data center)
    - williamscales 5y ago
      Wouldn’t it be better to put the effort towards designing a system that doesn’t need precise timekeeping?
  - doublea 5y ago
    See https://cloud.google.com/spanner/docs/true-time-external-consistency https://cloud.google.com/spanner/docs/true-time-external-con... It should explain how clock uncertainty relates to throughput of causally-related transactions.

rkangel 5y ago

I'm still unclear why they need the MAC on top of the GNSS reception. I can understand if it's just to provide holdover in event of GNSS signal loss, but they seem to be saying that it provides better precision when you've got both.

hoseja 5y ago
If I read correctly: GNSS PPS is more jittery but does not drift. The MAC has 1000x less jitter but drifts. You can cleverly combine them with statistics and magic to get the best of both worlds.
- mananaysiempre 5y ago
  From what I remember of textbook discussions of atomic clocks, stabilizing drifting oscillators is also in part how normal, old (50+ years) atomic clocks work as well, isn’t it? You have a fiddly and unreasonably high-frequency reference, you slave a relatively garden-variety standard like an environmentally controlled piezoelectric crystal to it, and then feed your periodic signals, counters, etc. off that.
  - willis936 5y ago
    Yes. Surprisingly, atomic clocks have high jitter compared to ovenized quartz oscillators. Every atomic clock package is outputting its signal from a quartz oscillator that is disciplined by the atomic oscillator.
- willis936 5y ago
  Phase-locked loops[0] are not magic ;) Also I imagine they are using a bog standard Kalman filter[1]. 0. https://en.wikipedia.org/wiki/Phase-locked_loop https://en.wikipedia.org/wiki/Phase-locked_loop 1. https://en.wikipedia.org/wiki/Kalman_filter https://en.wikipedia.org/wiki/Kalman_filter
  - Unklejoe 5y ago
    I've implemented this (using GPS PPS to discipline a clock) using regular old PI control loops. I just make the loop damped enough to filter out the GPS PPS jitter and you're good.

jedimastert 5y ago

Off topic: I'm reminded of a very small unit of time that was devised to be a least common multiple of various division of time like 24fps, 30fps, 44.1kHz, and a ton of others in video and audio. Does anyone remember what this unit was?

smnscu 5y ago
I think you might be referring to this https://en.wikipedia.org/wiki/Flick_(time) https://en.wikipedia.org/wiki/Flick_(time)
- tyingq 5y ago
  Also, somewhat related, the "TU" https://en.m.wikipedia.org/wiki/TU_(Time_Unit) https://en.m.wikipedia.org/wiki/TU_(Time_Unit)
- jedimastert 5y ago
  That's it! Thank you very much. I searched around on Wikipedia for it and couldn't find it. I might recommend some edits when I have the time.
hoseja 5y ago
Do you mean the facebook/oculus flicks? Wonder why the repo is archived... https://github.com/facebookarchive/Flicks https://github.com/facebookarchive/Flicks ninjaed :(
- jedimastert 5y ago
  That's exactly it! Thank you. I would imagine it's archived because there's nothing else to really be done. It's basically a blog post. The header can apparently be replaced with three lines.

hoseja 5y ago

Relevant recent HN submission: https://news.ycombinator.com/item?id=28108586 https://news.ycombinator.com/item?id=28108586

michaelt 5y ago

A few years back I was interested in microsecond-precise timekeeping. My experience was GPS-backed network clocks were surprisingly cheap - but PTP-aware network switches were surprisingly expensive. And though I thought I could cut the cost of a grandmaster clock from $2000 down to $200, I figured there'd be no point if I then have to connect it to a $20,000 network switch. I wonder how Facebook are connecting their precise new time servers to their regular servers?

MisterTea 5y ago
This gear will get cheaper soon. There is a big push in the Industrial automation world to get TSN/PTP deployed more widely and perhaps universally. The idea is to eliminate the dichotomy of real time vs non real time Ethernet networks in control systems from the plant network down to the machine level. This way you can send EtherCAT frames at equidistant intervals to servo drives in the microsecond range while casually streaming SPC data or Netflix over the same wire. This is also in conjunction with the newer single pair Ethernet tech that the Automotive world is also interested in.
- throw0101a 5y ago
  > single pair Ethernet For anyone wondering: > In addition to the more computer-oriented two and four-pair variants, the 100BASE-T1 and 1000BASE-T1 single-pair Ethernet PHYs are intended for automotive applications[17] or as optional data channels in other interconnect applications.[18] The single pair operates at full duplex and has a maximum reach of 15 m or 49 ft (100BASE-T1, 1000BASE-T1 link segment type A) or up to 40 m or 130 ft (1000BASE-T1 link segment type B) with up to four in-line connectors. Both PHYs require a balanced twisted pair with an impedance of 100 Ω. The cable must be capable of transmitting 600 MHz for 1000BASE-T1 and 66 MHz for 100BASE-T1. > Similar to PoE, Power over Data Lines (PoDL) can provide up to 50 W to a device.[19] * https://en.wikipedia.org/wiki/Ethernet_over_twisted_pair#Single-pair https://en.wikipedia.org/wiki/Ethernet_over_twisted_pair#Sin... > The IEEE 802.3bu-2016[12] amendment introduced single-pair Power over Data Lines (PoDL) for the single-pair Ethernet standards 100BASE-T1 and 1000BASE-T1 intended for automotive and industrial applications.[13] On the two-pair or four-pair standards, power is transmitted only between pairs, so that within each pair there is no voltage present other than that representing the transmitted data. With single-pair Ethernet, power is transmitted in parallel to the data. PoDL defines 10 power classes, ranging from .5 to 50 W (at PD). * https://en.wikipedia.org/wiki/Power_over_Ethernet#PoDL https://en.wikipedia.org/wiki/Power_over_Ethernet#PoDL 100BASE-T1 is IEEE 802.3bw-2015, 1000BASE-T1 is IEEE 802.3bp-2016. 2.5 Gb/s, 5 Gb/s, and 10 Gb/s over a single pair is 802.3ch-2020: the focus of these is in the embedded automotive space. * https://blog.siemon.com/standards/ieee-p802-3ch-multi-gig-automotive-ethernet-phy https://blog.siemon.com/standards/ieee-p802-3ch-multi-gig-au...

kfreds 5y ago

For anyone else wondering how their clients verify the authenticity of timestamps; It seems they use the NTP server software Chrony, which has support for NTS (Network Time Security, RFC 8915).

MisterTea 5y ago

> While building our Time Appliance, we also invented a Time Card, a PCIe card that can turn any commodity server into a time appliance. Invented? These cards have been around. e.g. Meinberg makes a variety of expansion cards which use various time keeping sources such as WWVB, IRIG-B, GNSS, etc.

tyingq 5y ago
Those have the external source but don't seem to have the internal reference (Rubidium, Cesium, etc) for holdover. FB seems to be claiming to have invented a PCI card that makes a commodity server into a Stratum-1 source. Meinburg sells that too, but in proprietary appliance form.
- abyagowi 5y ago
  Again, invented might be too strong. This is an abstraction of what you need minimally to build a time server on an open source basis. We appreciate and are humble to other products in the market
  - tyingq 5y ago
    The availability of a much cheaper product than the appliances will likely create a bit of stir with competitors. The article mentions better management functions too. So perhaps invented is too strong, but it's more notable than your follow up comments suggest.
KaiserPro 5y ago
I read it that they've invented a product called time card, rather than the concept. Its an accurate and stable clock source. which is an important distinction. The Meinberg PCI cards appear not to have a local clock source thats well disciplined (ie a temperature compensated oscillator) Having an accurate time source that is updated once every second (typically) with a 1pps GPS is grand for most things. But if you need a stable clock as well, then you want a tiny atomic clock onboard as well. This is so you can run a clock source for an entire datacenter, or local section.

jrockway 5y ago

What's interesting to me is how the form factor of timing GPS modules has stayed constant over the years. I started my GNSS timing journey with a used Trimble GPS from the 2000s, and it has the same form factor and pinout as the modern multi-constellation timing GPSes from uBlox. I've had a GPS clock going for several years at this point, and without an atomic clock or really any fanciness (just LinuxPPS and Chrony), I see about +/- 380ns, which is pretty good. NTP to the Internet gives me jitter in the range of about 20ms-70ms, about 5 orders of magnitude worse. (The version a few iterations ago looked like this: https://github.com/jrockway/beaglebone-gps-clock https://github.com/jrockway/beaglebone-gps-clock. But I now have a uBlox multi-constellation GPS, which is much more accurate with my limited view of the sky from my Brooklyn apartment. And I 3D printed the case, so it actually looks presentable instead of like some crazed madman that attacked a plastic case with a hacksaw -- which is exactly how I made the first case. As for the DS3231 RTC that I added... that seems to be stable within about 1.5us, which is pretty impressive. I tuned it a little bit with the trim register, though.) My takeaway from this article is that the project seems quite compelling, but a lot of cost is added to support PTP. I see why Facebook needs that, but to sync time precisely throughout my lab, all I need is a coaxial cable with 10MHz on it. I don't mind if my workstation is 20ms off UTC.

geoffeg 5y ago
Nice project, I've been working on something similar, but with an Arduino (Adafruit ESP32), an Adafruit GPS featherwing and a RTC. I originally started with a Linux-based solution like yours but no matter what I did I couldn't get the accuracy below about 5ms and my goal is sub-ms (verified with an oscilloscope.)
- jrockway 5y ago
  Yeah, I think you're on the right track. I like less and less having a Linux box to maintain, and there is less control over the timing of internal operations compared to a microcontroller, of course. On the other hand, Chrony is really, really good, and it would be a lot of code and debugging to write something that good from scratch. With the Beaglebone, it's theoretically possible to drive the oscillator from a calibrated clock source (use the PPS pulse to discipline an oscillator). I was too lazy to build such a circuit and observe the effects. Some uBlox timing GPSes let you output a suitable signal (you can adjust the frequency in software), but the adjustability range when I last checked wasn't enough to make it work for the Beaglebone. Once you have the MCU running at a well-defined and stable frequency, I think it's possible to give Linux extremely accurate data, or at least know which clock cycle was the start of a second. I don't actually know how much any of this will help, and it's honestly not within my capability to measure. For me, the outcome of my GPS clock project is that I never have to set my clock, and it looks really good. Running "chronyc sources" is always pleasurable as well, but I probably wouldn't notice a clock that's 20ms off unless I was comparing it against WWVB or something.

abyagowi 5y ago

I am very thankful to all of you putting all these questions and answers. We have a bi-weekly meeting under the open compute project and you can find previous talks there too. www.ocptap.com

delayclose2 5y ago

> The offset is practically 0 ranging within nanoseconds range. Uhm, no, the graph clearly shows that the offset is varying between 0-2 microseconds, which is 0 - 2,000 nanoseconds. 2,000 nanoseconds is not "practically zero". PTP precision is supposed to be in the 10ns range. Also, how did they get 40us precision on (what I assume to be) a normal kernel on a normal x86 box? I would have assumed that in a datacenter environment, with traffic levels being random, the jitter introduced in the kernel networking stack alone would be in the hundreds of microseconds at least.

michaelt 5y ago
PTP is a standard feature on a lot of Intel chipsets so a lot of servers have it 'for free' and aren't using it. It timestamps packets at the network card - independently of any kernel timing.
- zinekeller 5y ago
  Addenda: Intel networking chips, so even in AMD or ARM systems this actually works (assuming that you have the correct OS and drivers/modules/support software). Also, this isn't exclusive to Intel: most modern (server) NICs at least support TCXO-based timings, and some built for precision uses OCXO for even better precision.

nonameiguess 5y ago

> Now, as long as printing the PCB and soldering tiny components does not sound scary, anyone can build their own Time Card for a fraction of the cost of a regular time appliance. Doesn't sound scary, but not a lot of research labs and data centers are going to want to build their expansion cards from parts, even given free access to instructions. I understand if there isn't enough of a market for this to make it a viable commercial product, but when you have more money than God anyway, what's the reason not to make a few extra and donate or sell them to the small number of users who can use it?

abyagowi 5y ago
Of course we are not going to build the card in the lab with assembling SMD parts. This was more a way to explain that building such a card should not be out of access for anyone
- myself248 5y ago
  I mean, you can buy a syringe of solder paste for thirty bucks and do the reflow in a toaster oven. Anyone not doing it simply isn't trying.
  - abyagowi 5y ago
    You are right. This sentence was not supposed to be a literal description of what is the best way to build the card. It was more referring to the fact the building this card is not that hard since the main goal for this project was to liberate the time appliances from being a closed source box.
Shish2k 5y ago
From https://github.com/opencomputeproject/Time-Appliance-Project/tree/master/Time-Card https://github.com/opencomputeproject/Time-Appliance-Project... > Where can I get one? > […] we are currently working with several suppliers and will have their contact info soon available to allow you to puchase an out-of-the-box ready Time Card.

jeffbee 5y ago

Slaving your clocks to GPS seems like it just gives the USAF space command a way to, eventually, wreck your operations. If you don’t have requirements for accuracy referenced to civil time (such as mifid 2) why wouldn’t you just go with the atomic clocks alone, with gps as an advisory-only monitoring reference?

nullc 5y ago
Long term drift on these inexpensive secondary atomic clocks isn't that amazing, sadly.

latchkey 5y ago

A very long time ago, HP made a GPS time card just like this, encased in glass and filled with an inert gas such that if the card was tampered with, it was rendered useless. This prevented someone from faking the time signal.

audiometry 5y ago
Someone send one of these to CuriousMarc — he is peerless in working on these vintage high precision test pieces.
BonoboIO 5y ago
Can you show us a picture. Did not find anything on google beside HP Laptop spam.
- latchkey 5y ago
  I'm not sure it was ever sold to the public. Buddy of mine owns timestamp.com and we were building a service where you could hash/timestamp emails (we would take over the MX for a domain and become a forwarding service). All of this was long before we had blockchains. Sadly, it never really took off so we abandoned the idea (what is up there now is currently another half attempt at the idea). Others have tried, but it isn't really a huge demand despite the apparent need.

NelsonMinar 5y ago

Man, tough crowd here. And someone who's been following precise timekeeping tech for 25 years now.. this is really cool! I think it's great some smart engineers have built a new reasonably compact and low cost very precise bit of timekeeping hardware. Thank you for sharing it open source!

toomuchtodo 5y ago
I am torn. I absolutely love the technical blog post and the effort that went into open sourcing everything needed to build these devices, but, you know, Facebook. Compliments and respect to the engineers involved for really well done engineering.
- eplanit 5y ago
  They've built and earned their (negative) reputation. This is cool, but it's like Philip Morris Tobacco Company coming out with a useful home appliance. Uhh....thanks?
  - chillingeffect 5y ago
    Even a clock that moves backwards is right four times a day :)
    - ceejayoz 5y ago
      Move the hands fast enough and it's right every second of every day.
  - tourist2d 5y ago
    Except it's a software company releasing an open source software project?
- judge2020 5y ago
  Amazing people at Facebook make amazing things, but it does seem like only the worst of them (FB, Google) have the cash to burn to hire teams of engineers for projects outside of ones that directly contribute to generating revenue.

adolph 5y ago

Seiko offers a GPS clock in a conventional-home friendly form factor. First introduced in 2013, the Seiko Space Link GPS clock has proven to be a global success. It is the world's first clock for home or office use that receives a time signal from a GPS satellite, thus delivering atomic clock precision. It measures time to within one second every 100,000 years. Equally importantly, it can be used anywhere in the world and can be installed in almost any situation, so long as it is around 15 metres or less from the open sky. The technology is beautifully practical. 4 times every day, the clock receives a time signal from the GPS satellite network and, when conditions are ideal, adjusts automatically in around 10 seconds of receiving the signal. Between these signal receptions, a high accuracy quartz movement keeps the clock on time. Wherever you may need it, Seiko Space Link is there, silently marking the passage of time with an elegance and precision that is redefining the world’s perception of what a clock can be. A new standard of excellence has arrived. https://www.seiko-clock.com/products.html https://www.seiko-clock.com/products.html

ipsum2 5y ago
This is a standard "atomic clock", that its not actually uses radioactive decay to accurately measure the passage of time, but receives GPS signals. The innovation in the post is using real "miniature atomic clock" that actually uses caesium 133.

ChuckMcM 5y ago

This is a pretty neat thing and kudos for them open sourcing it. Typically, you'd have to hand over $5K - $25K for this card! This can hopefully make them available at a lower price. I built something similar (although I didn't use a CSAC[1] since it was too expensive but I did have a nice OXCO which is pretty stable. I use a Raspberry Pi to distribute plain old TOD via NTP to my local network, but the 10 MHz output I send to my bench where I can use it for radio projects. It is actually a better clock source (frequency stability wise) than the precision time source in my HP spectrum analyzer but it has worse phase noise (jitter) because it does get corrected by up to +/- 20nS periodically. [1] Chip Scale Atomic Clock - Microsemi -> Microchip https://www.microsemi.com/product-directory/clocks-frequency-references/3824-chip-scale-atomic-clock-csac https://www.microsemi.com/product-directory/clocks-frequency...

jameshart 5y ago

They call it a “Time Appliance”, or a “Time Device”. Missed opportunity to call it a “time machine”. Or, I guess, more prosaically, a “clock”.

isodev 5y ago

Congrats to everyone who made this project possible. It’s certainly a great accomplishment and it solves a real world problem. Now, does it require a Facebook account with my real name in order to use? /s

yitchelle 5y ago
What real world problem does this actually solve that has not been solved earlier? Genuinely curious.
- bobsmooth 5y ago
  Not having a low-cost extremely accurate Stratum 1 in a PCIe form-factor.

triactual 5y ago

What are the pressure sensor and IMU for? You've open-sourced a dev-board version, I'm assuming you'll keep the production version private?

abyagowi 5y ago
Production version will come today or tomorrow. Check the github

deleted 5y ago

[deleted]

daddylonglegs 5y ago

I cannot find any source files for the FPGA code in the repository (no .v .vhd .vhdl files and I can't see anything by looking around manually). The FPGA directory [1] contains some binaries and some documents relating to commercial code from a Swiss firm [2]. I've seen plenty of projects described as open source when the firmware is open source and the hardware is closed but this is the first time I've seen one where the schematics and board layout are open but the firmware is closed. Note that the hardware itself is quite simple, the smart stuff happens in three modules: GNSS receiver, precision clock and FPGA. To me, the contents of the FPGA source code are the only interesting part of this project. Additionally, you won't be able to meaningfully modify or reuse this project without editing the source code. The hardware module these FPGA binaries seem to be compiled for is described as an AC7100B, the source given for these is a defunct ebay link [3] [4] :-O This project uses two and half grand's worth of atomic clock and the heart of it runs on a module that fell off the back of a lorry!? [1] https://github.com/opencomputeproject/Time-Appliance-Project/tree/master/Time-Card/SOM/FPGA https://github.com/opencomputeproject/Time-Appliance-Project... [2] https://www.nettimelogic.com/clock-products.php https://www.nettimelogic.com/clock-products.php [3] https://github.com/opencomputeproject/Time-Appliance-Project/tree/master/Time-Card/SOM https://github.com/opencomputeproject/Time-Appliance-Project... [4] https://www.ebay.com/itm/XINLINX-A7-FPGA-Development-board-Artix-7-Core-board-FPGA-Minimum-System/253903903610?ssPageName=STRK%3AMEBIDX%3AIT&var=553209057783&_trksid=p2060353.m2749.l2649 https://www.ebay.com/itm/XINLINX-A7-FPGA-Development-board-A... Edit: I realise this comes across as quite negative. This looks like a neat project, one I would actually use, but only if it were meaningfully open source.

m-ee 5y ago
Valve did something similar for the Vive tracker dev kit. There was a time before the standalone tracker was available https://www.vive.com/us/accessory/vive-tracker/ https://www.vive.com/us/accessory/vive-tracker/ where if you wanted a tracker that wasn't a full controller you had to build it yourself. This involved a paid workshop in Seattle to get access to the dev kit and learn their tooling. Hardware was open source, but firmware was closed. If you wanted firmware changes you had to pay Synapse consulting fees for them to do it for you, and the updated firmware would then be made available free of charge to anyone else who paid for the workshop.

neilv 5y ago

Facebook isn't a good fit for me, but their work on open hardware for data centers has occasionally looked appealing. I figure units doing open hardware could be like when the old SNL sketch about AT&T -- having the slogan "We don't care. We don't have to. We're the phone company." -- was of the same company that hosted Bell Labs.

Tepix 5y ago

This seems like a cool project. I didn't see a price estimate for this PCIe card. However, isn't it smarter to try to build an infrastructure/architecture that can deal with somewhat less precise time information (milliseconds instead of nanoseconds)?

jandrese 5y ago

Oh cool, an open source Symmetricom card. These timing cards are low volume products so they have a bad habit of going out of support after a few years and then the kernel level drivers don't get updated and there is hardly any community. Having open hardware that multiple vendors can support makes them easier to get added to the mainline kernel.

abyagowi 5y ago
Thats exactly the point for this project. Also, I am hoping once we release the FPGA HDL files, the community will start adding more features like IRIG-B output, etc...

cedricgle 5y ago

If you want the go under the nanosecond there is the White Rabbit [1][2] project. The CERN uses it to control the magnets in the loop of a particule accelator. It is also based on NTP, Ethernet and FPGA. [1]: http://www.ohwr.org/projects/white-rabbit http://www.ohwr.org/projects/white-rabbit [2]: https://en.wikipedia.org/wiki/White_Rabbit_Project https://en.wikipedia.org/wiki/White_Rabbit_Project

profquail 5y ago
This was standardized as the L1 Sync feature of IEEE1588:2019, as described here: https://blog.meinbergglobal.com/2020/09/17/whats-new-in-ieee-1588-2019-l1-sync/ https://blog.meinbergglobal.com/2020/09/17/whats-new-in-ieee...

karmicthreat 5y ago

How well does transaction rate increase with more precise time? Seems like it would be a sort of O(log n) progression. Though could very well just be linear as well.

tibbydudeza 5y ago

I wonder what Google uses for Spanner/Firebase.

riobard 5y ago

Do smartphones get their time from GPS?

kube-system 5y ago
I believe they usually get it from the network.
- riobard 5y ago
  That was my impression as well but now I’m puzzled about why. Since most smartphones have GPS builtin why not get time from GPS?
  - becauseiam 5y ago
    Mobile basebands already get the local time and UTC offset when connecting to a tower, there's no need to power up the GPS antenna and receiver separately.
  - jlgaddis 5y ago
    Mostly because 1) your phone is getting time from the tower anyways (they have to be closely sync'd) and 2) GPS doesn't always work very well -- especially indoors.
    - riobard 5y ago
      So every cell tower runs an NTP server?

adolph 5y ago

I don't see ESP8266 mentioned here: Overview: https://hackaday.com/2021/07/25/portable-gps-time-server-powered-by-the-esp8266/ https://hackaday.com/2021/07/25/portable-gps-time-server-pow... Developer story: https://www.linkedin.com/pulse/iot-maker-tale-stratum-1-time-server-built-from-scratch-monteiro/ https://www.linkedin.com/pulse/iot-maker-tale-stratum-1-time... Repo: https://github.com/Montecri/GPSTimeServer https://github.com/Montecri/GPSTimeServer A simpler version from the comments: https://enginemonitor.blogspot.com/2020/09/gps-module.html?m=1 https://enginemonitor.blogspot.com/2020/09/gps-module.html?m...

jlgaddis 5y ago
Probably because an ESP8266 isn't anywhere close enough to "real-time" to be of any benefit. Why bother getting a nanosecond-level accurate time signal if you're going to feed it to an ESP8266 to distribute over Wi-Fi (or, in other words, reducing your accuracy to ~10 milliseconds or so)?

johnea 5y ago

Didn't DJB suggest massive improvements to NTP decades ago? https://cr.yp.to/libtai.html https://cr.yp.to/libtai.html No one seemed to care. This system seems to be special only because it's from MyFace. Additionally, everyone has direct access to stratum 1 timing via cheap GPS receivers. TimeNuts have been doing this stuff for decades: http://www.leapsecond.com/time-nuts.htm http://www.leapsecond.com/time-nuts.htm Again, this seems just another piece of bullshit for the zukers and other MyFace lusers...

walrus01 5y ago

> the ConnectX-6 Dx network card I would caution people against using the Mellanox/NVIDIA network interface cards in anything they care about. The driver license is proprietary and you have to build it for your kernel as a DKMS module in a semi manual process. Makes future system updates a real bother. I realize this probably doesn't apply so much to FB internally since they build and maintain their own distro in house. Stuff that's just a couple of years old won't even build the driver for current debian bullseye. Have to run buster? No thanks. As with so many other things NVIDIA the root cause of this is their absurd licensing approach to open source software and drivers for their hardware.

the_only_law 5y ago
I’ve been wondering about SmartNICs recently and imagine all sorts of ideas for them. Which brand do you recommended? > Makes future system updates a real bother. Sounds like their GPU drivers too.
- walrus01 5y ago
  I recommend Intel for 100GbE stuff. Intel's heritage of proper open source drivers (written by their own staff, in house) for FreeBSD and Linux goes back really far. You could look at the man pages for their very expensive 10GbE server NICs in 2003-2004 and see the @intel.com email addresess of the persons who wrote them.
- kd913 5y ago
  How about the x2 or Alveo series from Xilinx? It is basically the original definer of smartnic. It comes from solarflare who have a long pedigree of low-latency smartnics. They used to supply Cloudflare, and also supply like 50% of fintechs/financial markets. You can also just use openonload to accelerate your programs. In this case just doing straight linux socket programming, which can be accelerated without dpdk. Or just use the open source linux net driver if necessary. https://github.com/Xilinx-CNS/onload https://github.com/Xilinx-CNS/onload
somebody_amzn 5y ago
(disclaimer: am an AWS employee) Hello, Where did you get that from? The drivers for NVIDIA networking cards are in the upstream kernel. You have the option of using newer versions from DKMS if the built-in version for your kernel is too old. You can see the release notes for each version as present on the upstream kernel at https://docs.mellanox.com/display/kernelupstreamv512/Linux+Kernel+Upstream+Release+Notes+v5.12 https://docs.mellanox.com/display/kernelupstreamv512/Linux+K... for Linux 5.12 for example.

jlgaddis 5y ago

Well, dammit. Of course they come out with this now, after I've got a few antennas up on the roof, half a dozen GPS-disciplined OCXOs providing 10 MHz and PPS signals throughout the house, which are being fed into SolarFlare PTP NICs, and sync'd throughout the network using PTP. Apparently my workstations' offsets (RMS) are currently 17 and 21 nanoseconds, though, so I suppose there's still room for improvement.

CrazyCatDog 5y ago

Are there further details available about the MAC? Growing up in Boulder, THE atomic clock (NIST) was touted as a modern marvel (circa early 90’s childhood), it’s incredible to think that similar performance is now available on a pci-e card! Any relation to DARPA’s “making progress” announcement in August 2018 [1]? [1] https://www.darpa.mil/news-events/2019-08-20 https://www.darpa.mil/news-events/2019-08-20

pizza 5y ago

> The new service, built in-house and later open-sourced, was more scalable and improved the accuracy of timekeeping in the Facebook infrastructure from 10 milliseconds to 100 microseconds. Interesting.. the infrastructure must therefore have a diameter of at most 30 km, thanks to the speed of light

c_o_n_v_e_x 5y ago

My experience is more on the the navigation side. Wouldn't running an RTK GNSS receiver further improve accuracy and jitter? There's reasonably priced RTK hardware out there now like SwiftNav. Another option is Ublox modules with open source RTKLIB.

Animats 5y ago

How resistant is this to a GPS attack? It has an atomic clock, so it should be producing good data for days while sending alarms in the event the GPS data is bogus. You don't want to create a situation where someone aims a small dish at the roof of your data center and takes down the networking.

abyagowi 5y ago
I have been working on a version with two GNSS receivers L1/L2 and L1/L5. The SA53 receives two PPS inputs and that allows you to detect Spoofing and Jamming. Again, I like to emphasize, this is not a done work. We shared whatever we had and we would love to have the community to contribute and help making it better. I love to have as many applications as possible for this Time Card. For example, I am looking for someone willing to port/write a Windows driver for the time card. With regards to the VHDL code; we are working with NetTimeLogic on a version without the need of any of theirs proprietary IPs. Meanwhile, we provided you with whatever we have and we look forward to working with you. Please feel free reach out to me if you like to collaborate on this project.

traceroute66 5y ago

Am I missing something? The FB blog proudly states how they improved accuracy "10 milliseconds to 100 microseconds". Surely that's a lot of effort that anybody with in a datacentres with decent connectivity could replicate with a Chrony instance ? 100 microseconds is 0.0001 seconds, right ? Seems to be something Chrony can do with its eyes closed: System time : 0.000003297 seconds slow of NTP time Last offset : +0.000000549 seconds There are lots of great NTP servers out there hosted by trusted parties such as national time labs. Most people don't need fancy GPS hardware (and the expense of paying someone for roof access to put your antenna !). If you've got good software and good connectivity you can replicate 99.999% of the same thing.