9 ms·
Amazon Time Sync Service
- babelfish 5y agoUsing atomic clocks too, nice. From an older post: > It uses a fleet of redundant satellite-connected and atomic clocks in each Region to deliver time derived from these highly accurate reference clocks. https://aws.amazon.com/blogs/mt/manage-amazon-ec2-instance-clock-accuracy-using-amazon-time-sync-service-and-amazon-cloudwatch-part-1/ https://aws.amazon.com/blogs/mt/manage-amazon-ec2-instance-c...
- Faaak 5y agoso... GPS ?
- adamfeldman 5y agoGoogle and GCP offer their own NTP endpoints. Does anyone know if GCP also exposes the TrueTime API to customers, or if it's only internal to Spanner? I can't find any instance of the word "bound" in the GCP or Google NTP docs. [1]: https://developers.google.com/time/guides#google_compute_engine https://developers.google.com/time/guides#google_compute_eng... [2]: https://cloud.google.com/compute/docs/instances/managing-instances#configure-ntp https://cloud.google.com/compute/docs/instances/managing-ins...
- londons_explore 5y agoI'm pretty sure if you use the GCP dataflow stuff, the dataflow worker VM's will be given access to an RPC server which has the necessary endpoints for TrueTime. Obviously it's all a matter of reverse engineering rather than documented API's.
- jeffbee 5y agoOne of the key aspects of the TrueTime system is any device with serious clock error is simply murdered. It seems like offering that would significantly benefit users of this AWS API.
- tyingq 5y agoI'm curious about the title on this submission. I thought TrueTime had unusually strong guarantees about accuracy that don't seem to be called out in what I'm reading on the linked article.
- sizediterable 5y agoSorry if it's a bit sensationalized. I wanted to give some color as to why this release might be interesting and took this tweet I saw at face value https://twitter.com/rbranson/status/1455923426359578631 https://twitter.com/rbranson/status/1455923426359578631
- tyingq 5y agoOh, I wasn't criticizing, just genuinely curious. If the AWS service has a similar design, that's interesting to me. AWS doesn't seem to say much about the details.
- dang 5y agoIf you want to say what you think is important about an article, that's great, but please do it by adding a comment to the thread. Then your view will be on a level playing field with everyone else's: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&sort=byDate&type=comment&query=%22level%20playing%20field%22%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so... "Please use the original title, unless it is misleading or linkbait; don't editorialize." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- morei 5y agoNot quite: Truetime has guarantees about ordering, rather than accuracy. If you ask for the time and get A, and then ask for the time again and get B, then Truetime guarantees that A is less than B. Obviously, in a distributed setting this is much easier to do if you have accurately sync'ed clocks, but that accuracy goes to reducing the uncertainty in the time (and hence making truetime faster) rather than providing accuracy.
- londons_explore 5y agoIt's really hard to use these API's correctly. Remember... your CPU can halt at any time for any number of milliseconds. That means simple things like: upperBound, lowerBound = readTime() if (upperBound<deadline) do_stuff(x, y, z) Are incorrect... There is no guarantee that the 'if' statement didn't take many milliseconds, and that the stuff didn't end up happening after the deadline. It's also very easy to write code that works, but is theoretically wrong. You will leave a hidden bug that may only rear its head years down the line.
- karmakaze 5y agoIs that really the API? I'm sure I'd be writing: lowerBound, upperBound = readTime() often without noticing my error.
- dheera 5y agoHow about just syncTime() And they take care of the mess
- Groxx 5y agoYeah, I was thinking about this too. Without a real-time OS on separate hardware, and/or some kind of guaranteed "will not context-switch out of your process for more than X time" + strict coding habits, this seems like at best a practical (>?)99% improvement and at worst snake-oil. Certainly not reliable in any case. edit: well, after reading Spivak's comment[1] a bit, I guess it does provide strict upper bounds. May be useful to reduce how often you need to use fallback behaviors / get more byzantine. Though I'm not yet sure how to turn that into something useful. No doubt there are some though. [1] https://news.ycombinator.com/item?id=29103093 https://news.ycombinator.com/item?id=29103093
- Spivak 5y agoI mean it’s not that hard even on non-rt preemptive schedulers. You can’t avoid false negatives because you could be preempted after func completes but before the time is fetched but if you get a result it will be valid. def must_complete_before(func, deadline): result = func() lower, upper = time.now() if upper < deadline: return result else: throw MissedDeadlineException(“Can’t guarantee func completed before deadline”)
- drewda 5y agoPerhaps the biggest "fake-out" in 21st century computing: Google publicly released its MapReduce paper -- directing most of the rest of the industry toward loosely coupled, overly complex distributed data processing systems like Hadoop for the following decade -- but internally they just bought a bunch of atomic clocks and built a distributed RDBMS. I know this is a somewhat simplified story, but it does make me chuckle.
- skj 5y agoGoogle uses map reduce extensively... where it's appropriate. True time helps with things like spanner transactions. It's just a totally different use case.
- drewda 5y agoYes, which is why it's amusing in hindsight that for a decade everyone* outside Google was forcing all* their distributed data tasks into the MapReduce paradigm, without considering alternative approaches like the one used by Spanner. * slight exaggerations, I know
- mrep 5y agoI'm not sure how you think a distributed data processing technology would "fake-out" other companies when building/choosing database technology. They are totally different problem sets. MapReduce does not have a set in stone data source/sink and can use multiple things like bigtable and spanner so they are complementary technologies.
- deanCommie 5y agoThey released it in 2017. [1] Today they released a new OSS daemon and library for it. [1] https://aws.amazon.com/about-aws/whats-new/2017/11/introducing-the-amazon-time-sync-service/ https://aws.amazon.com/about-aws/whats-new/2017/11/introduci...
- strimp099 5y agoCan someone please explain to me what this is all about like I’m five years old?
- Groxx 5y agoTime is hard. By asking both super-microscopic stuff and stuff way out in space, you can now find out what time it is!
- bogomipz 5y agoOne is cesium? Whats the other stuff, way out in space?
- tadbit 5y agoGPS
- schwede 5y agoThe GPS constellation.
- jfrunyon 5y agoRubidium. Cesium atomic clocks are more expensive.
- baq 5y agoGalileo sats use these https://www.esa.int/Applications/Navigation/Galileo/Galileo_s_clocks https://www.esa.int/Applications/Navigation/Galileo/Galileo_... so, basically magic.
- loxias 5y agoYou think that's magic, check out optical lattice clocks. Blows my mind every time I re-read about them and accept "yes, this actually works".
- tatersolid 5y agoSome Pulsars have extremely stable frequencies https://www.nytimes.com/1983/02/09/us/pulsar-termed-most-accurate-clock-in-sky.html https://www.nytimes.com/1983/02/09/us/pulsar-termed-most-acc...
- loxias 5y agoTwo observations/questions, the first probably naive: * It's not that hard to get your own "world class" time server, for under a thousand. A Rb standard slaved to a GPSDO is gonna be so accurate and stable, and use that to drive a SBC that supports IEEE1588, where you run your NTP and PTP server. Oh, but I guess that box, while inexpensive, isn't in Amazons DC, so doesn't help you. * PTP's absence in the Amazon Time Sync Service article is quite conspicuous!
- Gelob 5y agoThem providing PTP would be really interesting
- loxias 5y agoTo be clear, what I meant was "the fact that they're advertising whiz bang time synchronization, but then not using the correct protocol, makes me question the quality of the whole enterprise". Even on local networks, NTP can only get you so close. If you set up chrony just so, in ideal conditions, I've gotten hundreds of microseconds (more commonly ~500-1000us). But combined with PTP, you can get sub-microsecond accuracy.
- travisgriggs 5y agoWhen Microsoft did that kind of thing we called it “embrace and extend.”
- staticassertion 5y agoThat sounds different. Embrace and extend means implementing a protocol and then adding specific non-spec features. For example, postgres 'embraces' SQL and 'extends' it with specific features - in the future postgres could 'extinguish' SQL by breaking from the spec, forcing an ecosystem split.
- loxias 5y ago
- glenngillen 5y agoBack from my time at Amazon, one of my favourite technical videos on the internal educational/YouTube thing was from a service team explaining how hard accurate clock sync in distributed systems was. One of those problems I’ve just taken for granted over the years. But just layers and layers of complexity where naive assumptions at any point get you the wrong result, but it’s not at all obvious you have the wrong result. I really wish they made more of that stuff publicly available.
- kohlerm 5y agoDoes that mean cockroachdb or others could implement a Spanner like DB?