9 ms·
How Precision Time Protocol is being deployed at Meta
- epgui 4y agoThis is incredible. If I knew I'd be working on things like this (as opposed to optimizing ad revenue or "user engagement" by A/B testing button copy), I'd possibly consider working at Meta.
- scheme271 4y agoThere's other places that are working on this although they probably don't pay as much as Meta. CERN and a few other european physics projects having been using this to create the white rabbit project over the last decade or so ( https://en.wikipedia.org/wiki/White_Rabbit_Project https://en.wikipedia.org/wiki/White_Rabbit_Project ).
- thatfunkymunki 4y agoIf this is the only thing stopping you, you should apply and tell the recruiter you would like to work in infra. People that don't want to work on product stuff don't get "roped into it", I know people who have worked at Meta/FB for years and have never touched user-facing code or ads stuff.
- capableweb 4y ago> Meta/FB for years and have never touched user-facing code or ads stuff Does that really matter in the end? If you're against writing code for ads/optimizing for quick dopamine hits, is it really so different to be writing the code that shows the ads vs the code that sets up the infrastructure to serve the ads? Or even infrastructure to support the developers who write the code for the ads? If you don't want to support the ad-driven internet, don't work for companies who's entire business model is extracting as much data from users as possible in order to show the "right" ads.
- epgui 4y agoYeah that's exactly my concern! I want to contribute to building cool things, and building a really cool component becomes much less inspiring if it's only used to steer nuclear missiles. (dramatic but not unreasonable example)
- elcomet 4y agoYou could apply to work on the research infrastructure (for instance for AI)
- capableweb 4y agoAnd instead of writing infrastructure for the developers who write code for ads, you now write AI software/do AI research for a company in order to attract enough talent so they can hire developers to write infrastructure for the developers who write code for ads? Or you know, just don't work for companies who tend to collect as much user data as they can?
- elcomet 4y agoBut at least you're contributing to science
- jtsiskin 4y agoAt Meta you choose what team you want to work on during a 3 month “bootcamp” period, very different from Google, Amazon, etc. There’s enough teams solving hard distributed system problems that you’re guaranteed to never need to think about button copy
- loeg 4y agoYou can also be directly hired onto a team, skipping bootcamp (some roles).
- bagels 4y agoIn good times, yes, there are more roles than bootcampers. In bad times (now) there are more bootcampers than roles...
- rawrfml 4y agoHm, the number of teams that work on these distributed systems is smaller than I think you believe exists at Meta, though I suppose it depends on the type of system being built. Even the internal post on this topic links to a note from last year that talks about the 3 teams in the core systems area that needs to work directly with PTP. The vast majority of people will probably end up working on product-related team compared to infrastructure-related teams.
- orangepurple 4y agoThis is the same as joining the Sinaloa cartel as a transportation and logistics expert specializing in detection evasion research and development. So many exciting and hard problems to focus on!
- bradknowles 4y agoWe're doing some interesting things with time sync at AWS, and we are doing so at scale for millions of customers around the world, unlike the small scale stuff that Meta has announced here. Most recently, our team recently announced the Amazon NTP Pools service at time.aws.com. We've got more cool stuff we are working on. If we can get past this hiring freeze problem, we still have a couple of slots we need to fill. If you know a lot about time sync and the NTP and PTP protocols, and you have a passion for building some of the biggest impact systems on the planet, let me know.
- kazinator 4y agoIt is not accurate to say that PTP is a predecessor to NTP. You need both. PTP synchronizes clocks to a ridiculous precision, like down to nanoseconds. To do that, it uses support in the ethernet hardware. Hardware adds precision stamps to the PTP-related ethernet frames, so that the time calculations are free of jitters induced by the higher network stacks. A cool thing is that PTP-aware routers/switches can actually rewrite the timestamps in PTP packets to subtract their own propagation delay accurately. Something in the network has to use NTP, if it is important for the devices to have the correct calendar date and time; you don't use PTP over the Internet to get time from a remote server. The real-time clock being synchronized isn't even the system clock in the first place, but a high resolution RTC that is in the network adapters and switches. As an additional utility on top of the PTP stack, there is the possibility of synchronizing the host system's clock with the help of PTP. PTP is used for things like generating precise signals in certain applications. E.g. Some devices that have PTP support can be programmed to generate periodic pulses which are clocked by the sychronized real-time clock. With PTP, two such signals from separate devices on the network can align very precisely. Such an application doesn't care about the time of day, just that the signals are synced.
- willis936 4y agoJust because high precision tasks don't care about time of day as much, PTP still uses TAI timestamps. The only time of day information missing is leap seconds, and those were a mistake that is being corrected. You don't need NTP for time of day when PTP is used.
- zie 4y agoThe point being you need a source of truth for PTP to figure out what the current date and time are. You won't be pushing PTP over the public internet, that's ridiculous. You can use a GPS/sat sync or atomic clock or NTP, doesn't matter, but something has to tell PTP what time it is. If you don't, then you will be subject to quartz crystal drift. It may or may not matter to your application(s), but for most people that care about time, they want the TAI timestamps to be at least reasonably accurate.
- bombcar 4y agoI assume the reply to the write includes this new super accurate timestamp? So the read server knows it's behind a bit as the read request includes it also?
- CapmCrackaWaka 4y agoWhy do so many tech companies seem to be releasing “secret sauce” for free lately? I see a lot of posts lately detailing how inner production systems work at large companies, and while I’m grateful, I’m curious why the higher ups think it’s worthwhile to release this information.
- jtsiskin 4y agoFacebook in particular is very open about its data center, see https://www.opencompute.org/ https://www.opencompute.org/ See “Commoditize Your Complement” - https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ , https://www.gwern.net/Complement https://www.gwern.net/Complement
- throw0101a 4y ago> Why do so many tech companies seem to be releasing “secret sauce” for free lately? Is it really their "special sauce" though? Do these types of releases actually give away how these companies make money? In this particular case, telling the world how to get to nanosecond levels of timekeeping doesn't really help any competitors take away Metabook's revenues or profits.
- ElevenLathe 4y agoOTOH if there turns out to be some other way of making money on nanosecond timekeeping, then keeping it secret will help them get into that market before competitors can.
- xboxnolifes 4y agoThis is just spending "potential money" for engineering talent advertisements.
- eastbound 4y agoUnless they’re playing 4-D chess and the goal is to make competitors lose time implementing CAP theorem databases, nanosecond precision and React, while they’re pulling ahead.
- throw0101a 4y agoSo they state: > One could argue that we don’t really need PTP for that. NTP will do just fine. Well, we thought that too. But experiments we ran comparing our state-of-the-art NTP implementation and an early version of PTP showed a roughly 100x performance difference: While I'm not necessarily against more accuracy/precision, what problems specifically are experiencing? They do mention some use cases of course: > There are several additional use cases, including event tracing, cache invalidation, privacy violation detection improvements, latency compensation in the metaverse, and simultaneous execution in AI, many of which will greatly reduce hardware capacity requirements. This will keep us busy for years ahead. But given that NTP (either ntpd or chrony) tends to give me an estimated error of around (tens of) 1e-6 seconds, and PTP can get down to 1e-9 seconds, I'm not sure how many data centre applications need that level of accuracy. > We believe PTP will become the standard for keeping time in computer networks in the coming decades. Given the special hardware needed for the grand master clock to get down to nanosecond time scales, I'm doubtful this will be used in most data centres of most corporate networks. Adm. Grace Hopper elegantly illustrates 'how long' a nanosecond is: * https://www.youtube.com/watch?v=9eyFDBPk4Yw https://www.youtube.com/watch?v=9eyFDBPk4Yw How many things need to worry the latency of signal travelling ~300mm?
- BaconPackets 4y agoI would be really curious to drill down into one of these problems. Is super precise really the solution?
- klodolph 4y agoI would also love to see an explanation of “why do we need this much accuracy?” that actually goes through the derivation of how much accuracy you need. Some of the justification for Google’s TrueTime is found in the Spanner docs: https://cloud.google.com/spanner/docs/true-time-external-consistency https://cloud.google.com/spanner/docs/true-time-external-con... Basically, you want to be able to do a “snapshot read” of the database rather than acquiring a lock (for reasons which should be apparent). The snapshot read is based on a monotonic clock. You can get much better performance out of your monotonic clock if all of your machines have very accurate clocks. When you write to the database, you can add a timestamp to the operation, but you may have to introduce a delay to account for the worst-case error in the clock you used to generate the timestamp. More accurate timestamps -> less delay. From my understanding, less delay -> servers have more capacity -> buy fewer servers -> save millions of dollars -> use savings to pay for salaries of people who figured out how to make super precise timestamps and still come out ahead. This kind of engineering effort makes sense at companies like Google and Meta because these companies spend such a large amount of money on computer resources to begin with.
- zzixp 4y agoI've been seeing a lot more of these Meta engineering articles on HN - I'm a fan! If you're writing them, keep up the great work!
- cheriot 4y agoAgreed. I wonder if the motivation is competing for resources with Meta paying more attention to costs now.
- synaesthesisx 4y agoYes! I had no idea Meta actually invested in innovation/engineering like this. I always assumed most of their work was just in ad tech & "metaverse".
- coolestguy 4y agoThat's why they are promoting themselves so much on HN now - to encourage talent to come to Meta
- pcthrowaway 4y agoI mean, they've created a whole bunch of the tech required to support their scale (React, Jest, PyTorch)
- deleted 4y ago[deleted]
- jeffbee 4y agoGreat article, but I have to quibble about the casual ease with which they wave away the complexity of accessing these timestamps. The Linux APIs for accessing them is totally absurd. See for example the gRPC code that associates hardware (possibly) timestamps with messages, for tracing and other reasons. You have to re-arm the timestamp option before every send. And the whole concept of a timestamp for an ethernet frame maps poorly to stream sockets. https://github.com/grpc/grpc/blob/master/src/core/lib/iomgr/tcp_posix.cc#L1242-L1504 https://github.com/grpc/grpc/blob/master/src/core/lib/iomgr/...
- gpderetta 4y agoI don't see the code rearming it at every send nor it matches my experience. The code sets the sock option lazily, and only once . The option is needed to configure the driver and possibly the card. Then the timestamp option need to be passed to the send/recvmsg call to phisically retieve the send/recv timestamp. Far for being absurd it seems a very straightforward extension of BSD socket API.
- waynesonfire 4y agoI remember Google doing this many years ago, does anyone recall the papers that describes distributed storage systems that benefit from highly accurate clocks? Was it spanner or the next generation version of it? I can't remember. I recall finding it fascinating but out or reach since I think they were putting hardware based atomic clocks on their systems, sounded super proprietary.
- anonymousDan 4y agoYeah, Spanner and the Truetime API. Would be interesting to know whether there are many differences between the implementations (or indeed the abstractions offered). I always wonder whether there are rare occasions where such systems fail (e.g. the bounds reported don't actually capture the true time due to some bug), and what the resulting implications are for systems built on top of them.
- RockRobotRock 4y agoAbout a year ago, Linus from LTT collaborated with the Facebook team to make a video about the subject: https://youtu.be/JK3eTGkX6qY https://youtu.be/JK3eTGkX6qY Very cool!
- qwertox 4y agoThanks for sharing! It's really relevant and a nice overview of the problem.
- milleramp 4y agoCompanies have been using PTP to synchronize (frame sync) networked cameras for many years. It is much better than wiring up a separate ttl or differential signals. It is amazing the accuracy you can get with this protocol.
- gh02t 4y agoI've also seen seismologists use it to sync geophones and other instruments. Synchronization is very important for doing time sensitive measurements for things like localization of earthquakes.
- vanshg 4y ago> It’s absolutely essential to ensure an open sky and install a solid stationary antenna What effect, if any, might a major earthquake have on the time calculation?
- matheweis 4y agoThat’s a very interesting question. 1ns of error equates to ~30cm of error (from speed of light), so: You might get a couple of ns of error from the initial s and p waves. (Assuming on the order of 50cm displacement) If you’re on the fault and there is a major displacement (say order of 1-2 meters) there might be several more ns of error. Practically, this seems to suggest that by the time that displacement from an earthquake would cause a significant loss of clock accuracy (say > 10ns), you most likely have bigger problems to worry about..
- s1mon 4y agoIt's interesting that Meta is doing such serious low level R&D like this, but it's funny because they can't even keep their notifications in sync with their web site. I get a notification (from the mobile app, I think) that Joe has a birthday today, I go to the web page to wish them a happy birthday, and despite reloading, the web site still shows the birthdays from the previous day. Then I have to dig around more to find Joe's page to post a HBD. So as user, this level of time synchronization seems to be many orders of magnitude of over kill. I'm sure at some level it is actually important, but it seems like the database sync or whatever is going on with notifications is woefully lagging.
- amluto 4y agoI’m wondering whether the extremely careful GNSS part is really needed. A microsecond of offset between two servers in the same datacenter could easily matter, but I suspect that, if an entire datacenter were off by a microsecond, everything would be fine — communicating from that datacenter to anywhere else will take well over a microsecond, so an offset of this type would be a bit like the datacenter wiggling around in space a bit. On a different note, there’s an Intel feature called the Always Running Timer. In theory one ought to be able to directly determine the TSC <-> NIC clock offset using the ART. I’m not sure anyone has gotten this to work, though.
- walrus01 4y agoHaving rooftop mounted GNSS receive antennas for GPS+GLONASS is extremely common in telecom and ISP infrastructure applications. It's sort of a belt and suspenders approach to obtaining time from low stratum NTP sources and also having a local GNSS timing source to reference from. Or for use in a case where the network has a total absence of connectivity to any internet-based NTP sources (maybe because your management network doesn't talk to the internet at all, for many good reasons), and in the event of loss of transport connectivity to your own somewhere-in-the-region NTP servers in your management system, you want to be absolutely sure the system clocks on your local radio transport equipment, DWDM equipment, metro ethernet stuff are all precise. Using receive-only GPS data as a master time reference source is effectively saying "we think that getting the time as it's set by the USAF/Space Force people at Schriever AFB, who run the GPS system, should be treated as our master point of reference and all other clocks should derive from that". It's not a bad policy, as such things go, because GPS maintaining extremely accurate time is of great interest to the US federal government as a whole. Even a fairly small LTE cellular rooftop site, monopole or tower often has a similar receiver. It doesn't add a lot of cost.
- amluto 4y agoMeta seems to have gone with the expensive option. I think it’s this: https://www.mouser.com/ProductDetail/HUBER%2bSUHNER/Direct-GPSoF-MAC8-1-KIT?qs=sGAEpiMZZMu3sxpa5v1qrt4sXoyvAXXwArVt9FqkVbg%3D https://www.mouser.com/ProductDetail/HUBER%2bSUHNER/Direct-G... Admittedly, at their scale, this is peanuts. But I wouldn’t buy one of these for a scrappy startup :). SparkFun will sell a perfectly serviceable kit for a few hundred dollars. (If you are worried about lightning, then GPSoF looks like cheap insurance.)
- Melatonic 4y agoI have to say I am not a fan of Facebook in general (their primary business that is) but their engineering is always impressive. And equally impressive is that they do everything in house. An interesting movie might be a dystopian future where Facebook itself falls out of use but some type of rogue group takes over the Facebook datacenter infrastructure and uses it for their own purposes.
- nimish 4y agoGiven Meta's resources I'm surprised they didn't go whole-hog with the high accuracy profile (derived from CERN White Rabbit) Why not set up SyncE while you're at it? Not like the hardware cost is any issue and Meta can hire hardware engineers. Might as well get ps level precision with sub-ns accuracy
- nappy-doo 4y agoI worked at FB for a couple of years, in a group that sat near data storage. I used to argue with those folks that without atomic clocks, FB should just shut its doors because there was no way to have reasonable limits on consistency. While I was hyperbolic in my pronunciation, I wasn't wrong in my data. It's been more than 5 years since those conversations. Glad someone got it done. Took 'em far too long.
- dis-sys 4y agoNice tech blog, really love it. From my understanding, basically this new iteration is better then their previous generation chrony based one because the uncertainty in the One Way Delay calculation is largely removed by having those Transparent Clocks capable of reporting their queuing delays. Basically the asymmetry of the delays are gone? Personally, I don't think Chrony should be considered as a NTP implementation as clearly it can utilize those PTP hardware timestamps as well and those hardware timestamps are the "secret sauce" of PTP's high accuracy. With those PTP enabled switches, together with the fact that Chrony can already send NTP packages as PTP packages, surely Chrony can leverage such new capabilities as well with some reasonable updates.
- dis-sys 4y agoA quick question related to this blog - they mentioned that most modern NICs have hardware timestamp capabilities, that is consistent with what I saw. The question is what is the availability of those PTP ready & enabled routers/switches capable of acting as a Transparent Clock? I have two concerns, most of existing routers/switches are not PTP ready, even for those ones with PTP capabilities, its PTP related features may not be enabled in production. Any experience or numbers to share? thanks!
- wmf 4y agoAll recent data center switches I've seen advertise PTP; maybe it doesn't work when you turn it on though. Meta makes their own switches so they can enable and debug the software.