9 ms·
Clockwork raises $21M to keep server clocks in sync
- nimish 5y agoInteresting. CERN has white rabbit for sub-ns timing. Needs special hardware but is now IEEE 1588 (PTP) "High Accuracy" profile. I wonder how this deals with the numerous retimers and DSPs required for really high speed ethernet.
- macroblocks 5y agoI’ve been working on fairly large scale broadcast television back-end systems which have been using 10/25/100G SMPTE 2110 IP for video flows instead of HD-SDI, and those have been timed with PTP for a while (SMPTE 2059-2 standard, which is a PTP profile). The GPS locked master clocks are custom hardware (ex: https://evertz.com/products/5700MSC-IP https://evertz.com/products/5700MSC-IP), but a lot of the edge devices like the video playout servers are standard X86 hardware with Ubuntu Linux, with PTP delivered in-band over the 10G/25G ports to the device (same link as the video/audio flows). (… Which I realize is not as accurate as the newer White Rabbit related PTP update you refer to - but still odd that the original article referenced NTP and not even regular PTP which has been in use for a while and seems pretty close to their claims even before the more recent enhancements)
- nimish 5y agoRegular PTP is old hat as you say. ~5ns though I thought needs some real thought, and picosecond sync needs a combo of PTP + SyncE + link compensation, which is now the "high accuracy" profile. It's all fairly simple in the end if the hw has the tools but good luck buying SyncE or 1588 without $$$.
- zbrozek 5y agoIn a simple 100 mbps network where I needed good synchronization, I used the RXC pin of the MII bus as input to a PLL+VCXO and explicitly configured the PHYs in the link to be master-->slave in the desired clock distribution direction. PTP provided phase. Across a small network (a couple switch layers) I was seeing about ~20 ns of time uncertainty. Clocks were locked within about 2 ppb. Worked well enough for my needs. In hardware that's actually designed to do this, I think it's called "synchronous ethernet" but you can totally duct tape it into pedestrian hardware the way I did.
- Nextgrid 5y agoOut of curiosity, do you have any write-up or some documentation about this? Can't imagine a scenario where I'd need this any time soon but it sounds very interesting.
- nimish 5y agoYeah I think SyncE is what telecom RANs use to get within 5g reqs on timing. White Rabbit operates like 1588 + SyncE + a compensation model for the fiber link. It's pretty slick. It does need L2-L1 coordination but the ethernet MAC usually has no idea of what's happening on the L1 (it speaks "medium independent interface" or *MII). The CERN people warn against using Base-T since the PHY needs to do pretty complex signal processing which would likely destroy the syntonization. They then use basic fiber links, but 100g, 400g+ have a gearbox and DSP and FEC in the way.
- deleted 5y ago[deleted]
- gfd 5y agoI remember googling how to sync time on aws instances before and was surprised that they use atomic clocks: https://aws.amazon.com/blogs/aws/keeping-time-with-amazon-time-sync-service/ https://aws.amazon.com/blogs/aws/keeping-time-with-amazon-ti...
- mc4ndr3 5y agojust make CSAC's cheaper
- hn_throwaway_99 5y agoGoogle Cloud Spanner uses atomic clocks to be able to synchronize using timestamps across distributed DBs. CockroachDB does not require atomic clocks, but I believe there is an "atomic clock mode" available. This approach sounds like it doesn't use atomic clocks, but instead just machine learning algorithms to detect offsets. Would like to understand if the founders consider their approach to be a viable alternative to "atomic clock mode", but without actual atomic clocks.
- dan-robertson 5y agoI thought even spanner only relied on sync in the 100us-1ms range. Maybe I’m out of date or it’s more about having clocks advancing at a very reliable rate? Usually ‘atomic clocks’ means ‘gps appliance gets time from atomic clocks in gps satellites’ but maybe not in this case.
- hn_throwaway_99 5y ago> Usually ‘atomic clocks’ means ‘gps appliance gets time from atomic clocks in gps satellites’ but maybe not in this case. No, Spanner has atomic clocks running in each data center.
- jasonwatkinspdx 5y agoSo Spanner uses TrueTime as an integral part of it's concurrency control algorithm. To preserve external consistency, Spanner sometimes has to wait out an uncertainty interval at transaction commit. The tighter the bounds on timestamp intervals from TrueTime, the less this waiting impacts performance. I've heard through the grape vine that TrueTime currently operates much better than the numbers in the original paper, but can't confirm if that's true. They use GPS for global sync and a local atomic clock as reference. From conversations over the last few years this seems to be the common setup at big DCs, as the cost for a couple atomic clocks is minuscule compared to the scale of everything else.
- evol262 5y ago> “Currently, nobody uses time except for maybe Spanner at Google, CockroachDB or someone doing database things,” Rosenblum said. “We believe that there’s a lot more places, especially as more and more time-critical things came up. We can do time sync, since we figured out how to do that pretty well. And so we asked: is this part of a trend where we’re going to start programming these systems differently? And [researchers] got kind of excited about that possibility of us being able to pull this off.” Who gave this guy who's apparently never heard of kerberos, ceph, SAML, or any other technology the rights to interview? Sure, their windows are larger, but "nobody uses time" is so clueless it blows the mind, and it's hard to imagine who decided that "NTP, but machine learning" is a $21M idea
- nikanj 5y agoTo be fair, their pitch deck did say both Machine Learning and Stanford
- deleted 5y ago[deleted]
- galeaspablo 5y agoYup. Going back to the principles of distributed systems, I’m really wondering how “NTP but machine learning” can be used to minimize the clock skew to hundreds of nanoseconds with software only. The RTT between two servers can vary a lot — and way past the calculated minimum RTT. The figures (eg $21M) and names dropped (eg Stanford) are an appeal to authority, which does make me curious. I’d love to see some papers, I went through all of Balaji Prabhakar‘s publications (titles only), and didn’t see a single paper that sounded like “NTP but machine learning. If anyone else knows more about this, I’d love to hear from you. Surely $21M doesn’t get dropped with at least someone doing due dill on the tech?
- kloading 5y agoI also haven't seen any work of his using ML, that may just be a buzzword thrown in the PR release. But if anyone's interested, I believe this is the paper alluded to in the article: Clock synchronization: https://www.usenix.org/conference/nsdi18/presentation/geng https://www.usenix.org/conference/nsdi18/presentation/geng I think the TechCrunch article doesn't really explain their application of clock synchronization well. Here are the other relevant papers and my attempt at explaining the general idea below. Network edge-based measurement (one-way delay?): https://www.usenix.org/system/files/nsdi19-geng.pdf https://www.usenix.org/system/files/nsdi19-geng.pdf Congestion Control: https://www.usenix.org/system/files/nsdi21-liu.pdf https://www.usenix.org/system/files/nsdi21-liu.pdf Each paper I've listed builds on the last. Their method to synchronize clocks made accurate and efficient measurement of one-way delay possible with commodity hardware. This measurement of one-way delay occurs at the edge, allowing them to "hold" incoming packets at the edge for extremely small periods to reduce congestion (latency) while maintaining throughput. From my understanding, traditional congestion control algorithms require rich telemetry from the entire network, which is likely not accessible in a public cloud environment. Balaji and clockwork's algorithms only need to make these measurements from the edge (which customers in public cloud have access to). I'm curious to see how all this will scale for multi-region deployments. If the latency between VMs from region 1 and region 2 is significant, I wonder if the measurement will actually be useful in deciding to "hold" the packets.
- deleted 5y ago[deleted]
- hansvm 5y agoThere are strong lower bounds in the error of pure-software time synchronization techniques. Is the proposal to incorporate additional hardware, or are they just probabilistically increasing the accuracy? If the latter, what applications can benefit from maybe being better synchronized, not being able to measure how much better the synchronization is, and maybe still having the same worst-case bounds that other algorithms give?
- mateo1 5y agoThat was my fist thought. The second one is, who has any use for such precision. Other than companies that do hft of course.
- pclmulqdq 5y agoThe lower bounds are based on the use of occasional time measurements. In PTP terms, the clockwork algorithm involves using a high volume (~10k/second) of peer delay requests, and discarding almost all of them. Only the ones that are deemed to be "pure" are used to adjust clocks. By using a high volume of requests, they can actually average out a lot of the well-behaved jitter sources.
- hansvm 5y agoThe problem is that a pure software solution can't distinguish between clock skew and forward/backward relative time differences. Consequently, no such solution can guarantee error better than RTT/2. If your RTT is 2 microseconds, it's impossible to guarantee synchronization within hundreds of nanoseconds without incorporating additional information, regardless of jitter. Hence my initial question: are they adding extra information to actually achieve those stated goals, or are there algorithms just "probably" better, and in the latter case what are the use cases? Distributed transactions and whatnot are fundamentally broken if your "better" synchronization might still be wrong.
- pclmulqdq 5y agoYou would enjoy reading the paper. They are making a few assumptions that turn out to be pretty good in a datacenter environment to simplify things. They are also using graph cycles to set clocks, which is a very different approach. My guess is that the precision of their approach comes at an accuracy cost, and the clocks are not particularly accurate.
- CoastalCoder 5y agoI've vaguely wondered what it means for clocks to be in synch, when talking about time differences much smaller than the time it takes for light to travel between the various clocks.
- dharmab 5y agoIf the clocks are programmed to know their distances from each other, you can account for that.
- Dylan16807 5y agoYou can, but is there any practical effect?
- maronato 5y agoImagine that the same message reaches more than one servers. Which one should record it? If they all are in sync you just use the one that received the event at the lowest timestamp
- Dylan16807 5y agoNow imagine the same scenario but one of the servers is eight feet to the left. Does it matter that a different server picked up the message first? Now what if the server just thinks it's eight feet to the left because the clock is out by several nanoseconds? Can't you just ignore that, and still use the server that got the lowest timestamp? What's the practical difference? When the servers can communicate in x amount of time, the question was specifically about clock drift "much smaller than" x.
- radicality 5y agoI did read the article and do see the work FB open sourced linked (100 microsecond accuracy), but will link directly to the FB blog here for anyone interested: https://engineering.fb.com/2021/08/11/open-source/time-appliance/ https://engineering.fb.com/2021/08/11/open-source/time-appli... I’m curious, who is even the target customer here? Places like FB and Google are working on the problems themselves. They talked about some “latency sensei” product or something but what is it gonna tell me? How many nanoseconds it took for my API request to go from a load balancer to the web server, with 5 ns precision? Is that precision really needed?
- zeckalpha 5y agoMy guess is they are aiming to exit with a cloud provider that isn’t doing this stuff themselves yet, or one that doesn’t want their competitors to acquire this team.
- Uptrenda 5y agoI was actually reading about this recently and learned about the time problem. Some interesting stuff I remember: * On LAN: NTP can get up to 1ms accuracy and on WAN about 10ms accuracy. * People's 'system clocks' use NTP for synchronization but can still be completely off. * When you call DataTime.now() in Javascript it returns the unix timestamp in UTC time. * Since it is UTC it will have the same value anywhere in the world - however since it is set based on the the system clock it isn't guaranteed to be very accurate at all. * Browsers now have a high-precision API for doing measurements of elapsed time called 'performance.' Don't use DateTime.now() for this. * HTTP servers seem to return a Date field that has a unix timestamp in it. Some projects have attempted to use this to synchronize time in Javascript. * There appears to be no good, robust Javascript libraries that can synchronize and keep time accurately. Some libraries exist that use a single server + NTP to try calculate clock drift, however. But this isn't as good as the NTP daemon. * Researchers HAVE been able to write software to synchronize a clock with better accuracy than NTP using distributed networks -- this should be closer to what a lot of people are interested in. Here's a relevant paper I found on this: https://scholar.google.com/citations?view_op=view_citation&citation_for_view=QmbeOvsAAAAJ:5nxA0vEk-isC https://scholar.google.com/citations?view_op=view_citation&c... * People have done some pretty cool bench-marking hacks to measure elapsed time in Javascript before the existence of the performance counter API. With performance.now() -- its not a clock but a counter and browsers can intentionally limit its accuracy to make 'fingerprinting' harder. https://stackoverflow.com/questions/6233927/microsecond-timing-in-javascript https://stackoverflow.com/questions/6233927/microsecond-timi...
- paxys 5y agoOne bit of clarification: > it returns the unix timestamp in UTC time Unix timestamp is ALWAYS the number of seconds since January 1st, 1970 00:00:00 UTC. It cannot be in any timezone.
- Dylan16807 5y ago"ALWAYS" is a bit much when we have leap seconds, though.
- dchichkov 5y agoWhy not just synchronize to GPS time? It is pretty straightforward to get hardware timestamp on the 1PPS from the GPS. And then your time is as precise as the GPS clock, which can have a rubidium standard.
- mcshicks 5y agoI wonder the exact same thing. I worked on CDMA infrastructure in the 90s and that's exactly how we did it.
- netizen-936824 5y agoHow long would it take for the sync operation with communication latency? How reliable is the method (how robust)?
- wmf 5y agoThen you need a GPS receiver in each server and a bunch of amps and cables to distribute the signal to every receiver.
- KennyBlanken 5y ago1PPS signal distribution is not a complex or remotely difficult problem.
- dan-robertson 5y agoFor some reason companies that care a lot about clock sync don’t seem to do this so perhaps there is some other reason.
- dchichkov 5y agoA local GPS-disciplined NTP server, distributing time over 1GbE or 10GbE goes a long way. I can't think of any application, besides HFT and maybe some science experiments that need better precision. And even in HFT, it is silly to try to use your x64 system time to get nanosecond precision. Even if you are really good, using bare hardware, all your code and data is in L2, an x64 system would still be giving you 100ns level jitter. No matter what you do. What you want is to have your network card (or FPGA) timestamping the packets accurately. And then releasing these packets at a precise timestamp. Saying that, maybe Clockwork is doing something completely different. And not in nanosecond level, but at a millisecond level. They talk about "measure true one-way delays of a packet or remote procedure call, discover network bottlenecks and "hiccups" (outages lasting a few seconds), and identify underperforming VMs arising from "noisy neighbors." Now, that could indeed be useful.
- assttoasstmgr 5y agoIEEE 1588 (Precision Time Protocol) has already solved this problem for every use case that matters and offers microsecond precision. https://www.eecis.udel.edu/~mills/ptp.html https://www.eecis.udel.edu/~mills/ptp.html PTP takes advantage of special Ethernet switches and other devices which can decode or manipulate the time tags in hardware because yes your switches add latency. This just sounds like more horseshit out of SV. I applaud them for convincing someone to give them money for this snake oil.
- paxys 5y agoApplications like Spanner work with nanosecond-level time difference between datacenters. PTP works in microseconds and can even go up to a few milliseconds, making it useless for actual time sensitive use cases.
- pclmulqdq 5y agoSpanner works fine with millisecond-level timestamps. Spanner needs accuracy, not precision: it relies on knowing the bounds on the the current time, which may be wide.
- welterde 5y agoI think you are confusing PTP and NTP. NTP is in the range of low microseconds to a few milliseconds (with hardware time-stamping can actually do much better than that though), while PTP can synchronize time to better than 100ns.
- jeffwheeler 5y agoTheir website links [1] to a presentation at Stanford [2]. Mendel Rosenblum briefly compares their concept with PTP at about 45:54. He says: > Those people [fintech, etc.] are interested in our stuff, so obviously [that] means that thing [PTP] is not perfect for them. Part of the problem is, you’re trying to measure individual packets that are going through and get times. It means you have some clock there that you’re reading, and when it goes into the switch, and when it comes out, like, what is that clock doing in terms of its varying in frequency, and stuff like that. And then you get one sample and try to make things of it. It sorta pales in comparison to this big data approach we’re looking about. This seems like a mischaracterization of PTP (802.1AS), which does use multiple samples to syntonize. [1] https://www.clockwork.io/resource/ https://www.clockwork.io/resource/ [2] https://www.youtube.com/watch?v=Opf9CBwP5R8 https://www.youtube.com/watch?v=Opf9CBwP5R8
- altantiprocrast 5y agoSorry for the ignorance, but what is the point of getting this accurate in the datacenter (outside of scientific research and measurements I'd imagine)? Wouldn't it be easier to just make distributed servers deal with large 'packets' or large individual tasks on their own?
- paxys 5y agoThere's an entire class of problems where you need to synchronize application state between data centers. For that, if two conflicting requests show up in different data centers at the ~same time, different servers need to agree on which one came first, and for that they need to be on as close to the same time as possible. Even being off by milliseconds doesn't cut it when request volume is high.
- jiangplus 5y agoIs there any public papers about this topic? For high precision clock sync for the datacenter.
- aejnsn 5y agoHow has Chrony not been mentioned here? https://chrony.tuxfamily.org/ https://chrony.tuxfamily.org/
- wmf 5y agoChrony is still NTP. Clockwork is aiming to be far more precise.
- jakegold 5y agoIt's hard to get much more accurate than Chrony with NTP. Even real-world PTP implementations don't often aim for more accuracy than is possible with Chrony. Because it turns out NTP can be much more accurate than most people realize. From the Chrony FAQ[1]: > When combined with local hardware timestamping, good network switches, and even shorter polling intervals, a sub-microsecond accuracy and stability of a few tens of nanoseconds might be possible Good network switches and NICs with hardware timestamping support are commonplace now in server environments. NTP with Chrony is pretty hard to beat in terms of simplicity, reliability, and accuracy. 1. https://chrony.tuxfamily.org/faq.html https://chrony.tuxfamily.org/faq.html
- bitcharmer 5y agoCould you provide examples of NIC models that can achieve that? So far I've only worked with gear that offers HW timestamps but that feature relies entirely on the availability of PTP signals in the first place.
- mlichvar 5y agoSome models that are known to work well with chrony are Intel I210, I350, and X550. Those don't care about the protocol as they can provide hardware timestamps for all received packets. Other popular NICs like the Intel X540 or XXV/XL710 are limited to timestamping of PTP event messages in order to limit the rate of timestamps which needs to be handled by the driver. For those chrony supports an NTP-over-PTP protocol which forces the hardware to trigger the timestamping by wrapping NTP messages in PTP. Sub-microsecond accuracy is certainly possible. Here is an example with 3 network switches: https://chrony.tuxfamily.org/img/client-hwts-3switch-f323.png https://chrony.tuxfamily.org/img/client-hwts-3switch-f323.pn... The accuracy is limited by asymmetries in network switches. In any case, whatever algorithms Clockwork is using with their protocol, I'm sure they could be used with NTP too. If additional information needs to be exchanged between the hosts, extension fields can be specified for that.
- madars 5y agoI can highly recommend Jane Street's Signals & Threads episode on clock synchronization: https://signalsandthreads.com/clock-synchronization/ https://signalsandthreads.com/clock-synchronization/ In fact, the entire podcast is exceptional.
- mprovost 5y agoThis reminds me of djb's clockspeed, but with ML. Not a bad idea. https://cr.yp.to/clockspeed.html https://cr.yp.to/clockspeed.html
- bawolff 5y agoNot trying to be dismissive but honest question: why would machine learning be good in this problem domain? Normally i think of machine learning as a good way to find patterns in high dimensional data, but clock skew doesn't seem like something that is high dimensional.
- hansel_der 5y ago> why would machine learning be good in this problem domain? it doesn't have to be, market will decide.
- bawolff 5y agoI mean yes of course. The implicit underlying question i am asking is does their product make sense or is it a scam/snakeoil?
- daniel-cussen 5y agoThis startup genuinely sounds awesome. I played a lot with an asynchronous chip, meaning it didn't have cores, it had computers, and they didn't have a frequency. Then I connected it to an oscillator (it's easy, just lay one wire) and was getting sub-nanosecond overall measurements with no jitter. And it's like, why can't you hard-code that in? Like in your test suite, test the code and see if it's as fast as it's supposed to be, or so fast it's clearly not doing the work.
- dan-robertson 5y agoAre there any numbers for what accuracy they are hoping to achieve? If you don’t try hard you are likely to end up with something accurate to a few ms. AWS has some service which can get you synced to around 300 mics in normal conditions. But 300mics is pretty bad. Doesn’t this company basically require their customers to not use cloud providers or have they figured out how to get good clock sync despite cloud provider networks? It seems limiting if they don’t work in the cloud. I think the most common problem we have with clock sync (at least the most common problem I see) come from overloaded network cards slowing down timekeeping packets. I wonder if that’s much of a problem with this company’s solution.
- uvdn7 5y agoI am skeptical to say the least. TrueTime, PTP, FB’s time card, all solve one problem that is to provide a bound where “true time” falls in. You need to be able to _guarantee_ the precision. If you say the true time lies between [T - d, T + d], it must be the case. The spanner paper provides data that the probability of TrueTime being wrong is less likely than random hardware failures (bit flip, etc.). Nothing is 100% in computer, but once you have like 20 9’s of reliability, our society has collectively accepted it as good enough (like we assume hash collision would never happen). Now AFAIK, no machine learning model can come even close to 10 9’s of accuracy. Clock, to me, is a piece of foundational infrastructure that should provide a very solid and simple mental model so we can reason about it and build other things on top. I am skeptical that NTP + ML would work.
- mhb 5y agoWhat is the relationship between this service and the advent of cheap (<$2K), small atomic clocks?
- teleforce 5y agoOne of the very few books dealing with timing and synchronization in service providers, servers and enterprise applications. Synchronizing 5G Mobile Networks: https://www.ciscopress.com/store/synchronizing-5g-mobile-networks-9780136836278 https://www.ciscopress.com/store/synchronizing-5g-mobile-net...
- MichaelRazum 5y agoI think the idea is great. Since I had really bad experience with NPT and it was not trivial to sync the clocks. BUT just a bit surprised that they raised so much money.
- ram_rar 5y agoI see this is a problem, but how many companies really need this? In my limited experience I have only seen clock sync issues at scale and simple NTP/Chrony setup solves majority of the problems, unless you're in HFT domain where PTP is needed. Nowadays, many of those atscale systems are getting SaaSified where the companies using them dont have to directly deal with such issues. What am I missing here? Is there really a big market for this?