8 ms·

"NTP isn't accurate enough so we use a hardware clock assist that uses GPS and atomic clocks" - Google I think it is crazy that something as simple as keeping

by davis_m 11y ago

"NTP isn't accurate enough so we use a hardware clock assist that uses GPS and atomic clocks" - Google I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale. Every review I hear about working at Google makes me want to stay away, just like the recent conversations about working at Amazon, but crazy stuff like this always piques my interest.

ska 11y ago
I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale. Time keeping is one of the things that is most often underestimated and screwed up at any scale, in my experience. Many a subtle bug turns out to be a poor assumption or misunderstanding about how clocks work. Or calendars.
- jasonpeacock 11y ago
  i.e. A lot of distributed systems problems can be simplified if you can trust your clocks.
  - heapcity 11y ago
    what time is it anyway?
  - chetanahuja 11y ago
    Or guarantee that two separate coordinates on space-time can have identical values... which would mean breaking some laws of physics.
    - AnimalMuppet 11y ago
      If the two clocks are stationary with respect to each other, that isn't a problem. Most of Google's servers are on the Earth's surface, so... (Edit: Yes, different elevations cause a gravitational time dilation difference. For Earth's gravitational field and the elevation difference between different Google servers, I doubt it's an issue at the time resolution that Google needs to maintain.)
      - dragonwriter 11y ago
        > If the two clocks are stationary with respect to each other, that isn't a problem. Most of Google's servers are on the Earth's surface, so... ...you can't generally guarantee that they are (even approximately) stationary with respect to each other, because points on the earth's surface (in general) are not stationary with respect to each other in an inertial frame of reference.
        AnimalMuppet 11y ago
        > you can't generally guarantee that they are (even approximately) stationary with respect to each other... False. > ... because points on the earth's surface (in general) are not stationary with respect to each other in an inertial frame of reference. True. There is both the earth's rotation, and the relativistic difference due to differing elevations. But given earth's angular velocity and gravitational gradient, points on the surface are still approximately stationary with respect to each other, where "approximately" is defined by the amount of difference it will make compared to the time precision that Google cares about.
      - chetanahuja 11y ago
        Even if the clocks are stationary with respect to each other (within some tolerance), it's impossible to guarantee completely synchronized clocks among the systems. This follows trivially from the impossibility of instantaneous communication due to the second postulate of special relativity.
        AnimalMuppet 11y ago
        If two clocks are stationary with respect to each other, then there is no ambiguity about where the midpoint between them is. Then you just do something like, when your clock hits noon, fire a projectile at a fixed speed toward the other clock. If the projectiles meet at the halfway point, then the clocks are synchronized. And there is no relativistic funny business involved, because the clocks are stationary with respect to each other. There's no difference of viewpoint as to whether the projectiles met at the halfway point, or where the halfway point was, or even how far off from the halfway point they met (and therefore how far off the clocks are from each other). This is the argument used in my relativity class to show that you can synchronize clocks that are stationary with respect to each other. (You have to be able to do that to construct an inertial frame of reference, that is, to be able to determine what time coordinate some event occurs at, no matter what spatial location it occurred at.)
  - chetanahuja 11y ago
    But you simply can't. Not even Spanner (Google's globally distributed db that's using that fancy GPS based clock) doesn't pretend that the clock's are actually synchronized. https://queue.acm.org/detail.cfm?id=2745385 https://queue.acm.org/detail.cfm?id=2745385
- zo1 11y ago
  Can you elaborate on the need? What possible reason, short of a very contrived one, is there for having to keep a large number of machines' clocks in-sync? And for that matter, why would anyone build any process/system/software that requires a distributed system's machines to all have their clocks in-sync. I am baffled.
  - matthewmacleod 11y ago
    Well, it's a reasonable solution to some distributed problems. I imagine that many distributed algorithms can be simplified a lot of you have a reliable and accurate time source. If you can build a reliable clock at less cost (development and/or overhead) than a time-insensitive algorithm would cost, then why not do it?
  - pjungwir 11y ago
    Many security protocols use credentials that are only valid for a brief time. Here is a simple example of a problem caused by my server's time being more than 10 minutes off from the time on S3: http://illuminatedcomputing.com/posts/2015/04/paperclip_expiring_url_and_request_has_expired/ http://illuminatedcomputing.com/posts/2015/04/paperclip_expi...
    - abrookewood 11y ago
      Yep, security is a big one. Authentication & encryption can behave very unpredictably if your times are not in sync. So, you might not be able to login to a server, or connect via HTTPS etc
  - lmorris84 11y ago
    I believe Google Spanner obsesses over timestamps quite a lot to deliver distributed transactions with something called TrueTime [1]. [1] http://research.google.com/archive/spanner.html http://research.google.com/archive/spanner.html
  - amaks 11y ago
    There is a simple reason why it's needed - in distributed transaction you need to establish the causality of changes. That's why truetime is needed.
- mad44 11y ago
  Hybrid Logical Clocks offer a simple and feasible alternative to Google's custom hardware based TrueTime. http://muratbuffalo.blogspot.com/2014/07/hybrid-logical-clocks.html http://muratbuffalo.blogspot.com/2014/07/hybrid-logical-cloc...
  - acqq 11y ago
    It would be nice if the authors would evaluate how the hardware clocks compare to their software method in the case of maintenance regimes. I understand that their method is "good enough" in the "stable" state across the datacenters spread out through the world. The question remains if the existence of the hardware clocks simplifies the "unstable" states (that occur not when everything runs according to your software design, but when you maintain the infrastructure at that scale) and I believe it still does. I don't have proofs, but my belief is that using GPS and atomic clocks at that scale (GPS clocks on different physical locations) isn't an example of doing unnecessary stuff.
MattCruikshank 11y ago
"Every review I hear about working at Google makes me want to stay away" I loved working there. Sure, who your boss is has a huge impact on your happiness. I ended up with a boss I enjoyed working for. Does that review make you want to stay away, too?
- tikhonj 11y ago
  It does for me, at least. I'd absolutely hate having my job satisfaction determined so much by a manager and team that I was assigned. (Ideally I wouldn't want it impacting much at all, but truly flat companies are still hard to find, despite the rhetoric.) It works for some people, but it won't work for me.
  - MattCruikshank 11y ago
    In my hiring process, I interviewed with multiple managers. I picked my favorite, and I ended up working for him. It didn't work out. Within a few weeks I was working for a new manager. And frankly, I've never once worked for a company where I was immune from re-orgs.
  - pauleastlund 11y ago
    I spent about 5 years at Google starting in 2006. When I arrived I was assigned to one manager, but I wasn't _super_ excited about the project. A week in, another manager offered me another opportunity and got me permission to transfer. Problem solved. A couple years later, a new hire got assigned to a team I was on. He was a little bummed because he'd really had his eye on another project. So we talked to our manager about it and he was allowed to transfer to the team he'd been hoping for. I have more anecdotes like this, but the long and short of it is that in my experience Google is a lot less capricious and uncaring an organization than you imagine.
    - hueving 11y ago
      >I have more anecdotes like this, but the long and short of it is that in my experience Google is a lot less capricious and uncaring an organization than you imagine. I think the problem is that their interviewing process is. That's the first thing people encounter and it's the only thing rejected people encounter. That harms people's views of the company, even if they understand the explanation about false positives being so expensive.
robrenaud 11y ago
Working at Google is pretty cool. Personally, I get paid pretty good money and have tons of resources to build models that serve 100s of millions of people.
- drcross 11y ago
  After 9 different interview rounds I've all but giving up on trying to get into there. Any advice on trying to get in?
  - mahyarm 11y ago
    Master Cracking the Coding Interview? Have a positive attitude?
    - hueving 11y ago
      Right, spend all of your free time cramming on algo and data structure questions that you weren't using before and probably wont use at Google.
      - robotresearcher 11y ago
        For 20 years of 6 figure salary at one of the most prestigious engineering jobs in the world? Maybe worth cramming a bit?
        hueving 11y ago
        Google is not one of the most prestigious anymore. It's sought after because it has nice employee perks, but it's fallen a long ways from the early days in terms of prestige. They now have thousands of engineers working on mundane jobs moving around advertising data. I've interviewed a couple of ex-googler's (not fired, still employed there) at a startup and we had to turn them down. They could handle a simple coding exercise fine, but then fell over in the component where they had to debug code and extend it with a feature. That was when I realized a big chunk of Google's employees are going to be a reflection of their interview process (puzzle solvers but not software engineers). This was even reflected in the creation of the Go language. It's stupid simple to deal with Google employees that don't know how to use more advanced features in a language without creating something unmaintainable. You can get 20 years 6 figure salary in most of the locations Google operates offices working for other companies. Especially in the bay.
SEJeff 11y ago
This isn't super surprising tbh, all of the finance industry has PTP timesources in most every datacenter. Different needs for slightly different industries.
munificent 11y ago
> Every review I hear about working at Google makes me want to stay away Google was Fortune Magazine's #1 best place to work for six years in a row. The reason you occasionally hear a story about how Google isn't perfect to work at is because those are news. In general, it's a fantastic workplace.
- untog 11y ago
  Everyone is different - while I don't doubt that Google has fantastic perks, I can't think of many things that I would want to work on that aren't the moonshot crazy experimental projects (and, let's face it, most people won't be working on them). And working on a giant campus that's isolated from the outside world... no thank you. But like I said, we're all very different.
  - cbr 11y ago
    working on a giant campus that's isolated from the outside world You don't have to work in Mountain View! The remote offices are really nice, and are generally well integrated into their cities.
    - Filligree 11y ago
      Right, even the one in San Francisco.
  - DannyBee 11y ago
    "I can't think of many things that I would want to work on that aren't the moonshot crazy experimental projects (and, let's face it, most people won't be working on them)." This is true of almost everyone, of course. Every year i read the incoming intern abstracts, and they all literally say the same thing "I really would like to work on <whatever really popular crazy project was in the news lately>". Literally all of them. That said, often you can work on them if you are good enough at what you do. (But yes, often you have to prove that first, either internally or externally)
    - 11y ago
martincmartin 11y ago
A good article about this is "There is No Now" on ACM: https://queue.acm.org/detail.cfm?id=2745385 https://queue.acm.org/detail.cfm?id=2745385
mmoya 11y ago
> I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale. Check this talk: https://archive.fosdem.org/2015/schedule/event/ntimed_ntpd_replacement/ https://archive.fosdem.org/2015/schedule/event/ntimed_ntpd_r... . It's more difficult than it appear.
mozumder 11y ago
> I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale. Timekeeping is such a basic problem that Albert Einstein invented the theory of relativity because of it.
tajano 11y ago
> "something as simple as keeping time" Keeping time has been a difficult problem for... well, as long as we've tried keeping time. For an interesting historical account on past difficulties, you might check out the book "Longitude" [1]. [1] http://www.amazon.com/Longitude-Genius-Greatest-Scientific-Problem/dp/080271529X http://www.amazon.com/Longitude-Genius-Greatest-Scientific-P...

by davis_m 11y ago

ska 11y ago

I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale. Time keeping is one of the things that is most often underestimated and screwed up at any scale, in my experience. Many a subtle bug turns out to be a poor assumption or misunderstanding about how clocks work. Or calendars.

jasonpeacock 11y ago
i.e. A lot of distributed systems problems can be simplified if you can trust your clocks.
- heapcity 11y ago
  what time is it anyway?
- chetanahuja 11y ago
  Or guarantee that two separate coordinates on space-time can have identical values... which would mean breaking some laws of physics.
  - AnimalMuppet 11y ago
    If the two clocks are stationary with respect to each other, that isn't a problem. Most of Google's servers are on the Earth's surface, so... (Edit: Yes, different elevations cause a gravitational time dilation difference. For Earth's gravitational field and the elevation difference between different Google servers, I doubt it's an issue at the time resolution that Google needs to maintain.)
    - dragonwriter 11y ago
      > If the two clocks are stationary with respect to each other, that isn't a problem. Most of Google's servers are on the Earth's surface, so... ...you can't generally guarantee that they are (even approximately) stationary with respect to each other, because points on the earth's surface (in general) are not stationary with respect to each other in an inertial frame of reference.
      - AnimalMuppet 11y ago
        > you can't generally guarantee that they are (even approximately) stationary with respect to each other... False. > ... because points on the earth's surface (in general) are not stationary with respect to each other in an inertial frame of reference. True. There is both the earth's rotation, and the relativistic difference due to differing elevations. But given earth's angular velocity and gravitational gradient, points on the surface are still approximately stationary with respect to each other, where "approximately" is defined by the amount of difference it will make compared to the time precision that Google cares about.
    - chetanahuja 11y ago
      Even if the clocks are stationary with respect to each other (within some tolerance), it's impossible to guarantee completely synchronized clocks among the systems. This follows trivially from the impossibility of instantaneous communication due to the second postulate of special relativity.
      - AnimalMuppet 11y ago
        If two clocks are stationary with respect to each other, then there is no ambiguity about where the midpoint between them is. Then you just do something like, when your clock hits noon, fire a projectile at a fixed speed toward the other clock. If the projectiles meet at the halfway point, then the clocks are synchronized. And there is no relativistic funny business involved, because the clocks are stationary with respect to each other. There's no difference of viewpoint as to whether the projectiles met at the halfway point, or where the halfway point was, or even how far off from the halfway point they met (and therefore how far off the clocks are from each other). This is the argument used in my relativity class to show that you can synchronize clocks that are stationary with respect to each other. (You have to be able to do that to construct an inertial frame of reference, that is, to be able to determine what time coordinate some event occurs at, no matter what spatial location it occurred at.)
- chetanahuja 11y ago
  But you simply can't. Not even Spanner (Google's globally distributed db that's using that fancy GPS based clock) doesn't pretend that the clock's are actually synchronized. https://queue.acm.org/detail.cfm?id=2745385 https://queue.acm.org/detail.cfm?id=2745385
zo1 11y ago
Can you elaborate on the need? What possible reason, short of a very contrived one, is there for having to keep a large number of machines' clocks in-sync? And for that matter, why would anyone build any process/system/software that requires a distributed system's machines to all have their clocks in-sync. I am baffled.
- matthewmacleod 11y ago
  Well, it's a reasonable solution to some distributed problems. I imagine that many distributed algorithms can be simplified a lot of you have a reliable and accurate time source. If you can build a reliable clock at less cost (development and/or overhead) than a time-insensitive algorithm would cost, then why not do it?
- pjungwir 11y ago
  Many security protocols use credentials that are only valid for a brief time. Here is a simple example of a problem caused by my server's time being more than 10 minutes off from the time on S3: http://illuminatedcomputing.com/posts/2015/04/paperclip_expiring_url_and_request_has_expired/ http://illuminatedcomputing.com/posts/2015/04/paperclip_expi...
  - abrookewood 11y ago
    Yep, security is a big one. Authentication & encryption can behave very unpredictably if your times are not in sync. So, you might not be able to login to a server, or connect via HTTPS etc
- lmorris84 11y ago
  I believe Google Spanner obsesses over timestamps quite a lot to deliver distributed transactions with something called TrueTime [1]. [1] http://research.google.com/archive/spanner.html http://research.google.com/archive/spanner.html
- amaks 11y ago
  There is a simple reason why it's needed - in distributed transaction you need to establish the causality of changes. That's why truetime is needed.
mad44 11y ago
Hybrid Logical Clocks offer a simple and feasible alternative to Google's custom hardware based TrueTime. http://muratbuffalo.blogspot.com/2014/07/hybrid-logical-clocks.html http://muratbuffalo.blogspot.com/2014/07/hybrid-logical-cloc...
- acqq 11y ago
  It would be nice if the authors would evaluate how the hardware clocks compare to their software method in the case of maintenance regimes. I understand that their method is "good enough" in the "stable" state across the datacenters spread out through the world. The question remains if the existence of the hardware clocks simplifies the "unstable" states (that occur not when everything runs according to your software design, but when you maintain the infrastructure at that scale) and I believe it still does. I don't have proofs, but my belief is that using GPS and atomic clocks at that scale (GPS clocks on different physical locations) isn't an example of doing unnecessary stuff.

MattCruikshank 11y ago

"Every review I hear about working at Google makes me want to stay away" I loved working there. Sure, who your boss is has a huge impact on your happiness. I ended up with a boss I enjoyed working for. Does that review make you want to stay away, too?

tikhonj 11y ago
It does for me, at least. I'd absolutely hate having my job satisfaction determined so much by a manager and team that I was assigned. (Ideally I wouldn't want it impacting much at all, but truly flat companies are still hard to find, despite the rhetoric.) It works for some people, but it won't work for me.
- MattCruikshank 11y ago
  In my hiring process, I interviewed with multiple managers. I picked my favorite, and I ended up working for him. It didn't work out. Within a few weeks I was working for a new manager. And frankly, I've never once worked for a company where I was immune from re-orgs.
- pauleastlund 11y ago
  I spent about 5 years at Google starting in 2006. When I arrived I was assigned to one manager, but I wasn't _super_ excited about the project. A week in, another manager offered me another opportunity and got me permission to transfer. Problem solved. A couple years later, a new hire got assigned to a team I was on. He was a little bummed because he'd really had his eye on another project. So we talked to our manager about it and he was allowed to transfer to the team he'd been hoping for. I have more anecdotes like this, but the long and short of it is that in my experience Google is a lot less capricious and uncaring an organization than you imagine.
  - hueving 11y ago
    >I have more anecdotes like this, but the long and short of it is that in my experience Google is a lot less capricious and uncaring an organization than you imagine. I think the problem is that their interviewing process is. That's the first thing people encounter and it's the only thing rejected people encounter. That harms people's views of the company, even if they understand the explanation about false positives being so expensive.

robrenaud 11y ago

Working at Google is pretty cool. Personally, I get paid pretty good money and have tons of resources to build models that serve 100s of millions of people.

drcross 11y ago
After 9 different interview rounds I've all but giving up on trying to get into there. Any advice on trying to get in?
- mahyarm 11y ago
  Master Cracking the Coding Interview? Have a positive attitude?
  - hueving 11y ago
    Right, spend all of your free time cramming on algo and data structure questions that you weren't using before and probably wont use at Google.
    - robotresearcher 11y ago
      For 20 years of 6 figure salary at one of the most prestigious engineering jobs in the world? Maybe worth cramming a bit?
      - hueving 11y ago
        Google is not one of the most prestigious anymore. It's sought after because it has nice employee perks, but it's fallen a long ways from the early days in terms of prestige. They now have thousands of engineers working on mundane jobs moving around advertising data. I've interviewed a couple of ex-googler's (not fired, still employed there) at a startup and we had to turn them down. They could handle a simple coding exercise fine, but then fell over in the component where they had to debug code and extend it with a feature. That was when I realized a big chunk of Google's employees are going to be a reflection of their interview process (puzzle solvers but not software engineers). This was even reflected in the creation of the Go language. It's stupid simple to deal with Google employees that don't know how to use more advanced features in a language without creating something unmaintainable. You can get 20 years 6 figure salary in most of the locations Google operates offices working for other companies. Especially in the bay.

SEJeff 11y ago

This isn't super surprising tbh, all of the finance industry has PTP timesources in most every datacenter. Different needs for slightly different industries.

munificent 11y ago

> Every review I hear about working at Google makes me want to stay away Google was Fortune Magazine's #1 best place to work for six years in a row. The reason you occasionally hear a story about how Google isn't perfect to work at is because those are news. In general, it's a fantastic workplace.

untog 11y ago
Everyone is different - while I don't doubt that Google has fantastic perks, I can't think of many things that I would want to work on that aren't the moonshot crazy experimental projects (and, let's face it, most people won't be working on them). And working on a giant campus that's isolated from the outside world... no thank you. But like I said, we're all very different.
- cbr 11y ago
  working on a giant campus that's isolated from the outside world You don't have to work in Mountain View! The remote offices are really nice, and are generally well integrated into their cities.
  - Filligree 11y ago
    Right, even the one in San Francisco.
- DannyBee 11y ago
  "I can't think of many things that I would want to work on that aren't the moonshot crazy experimental projects (and, let's face it, most people won't be working on them)." This is true of almost everyone, of course. Every year i read the incoming intern abstracts, and they all literally say the same thing "I really would like to work on <whatever really popular crazy project was in the news lately>". Literally all of them. That said, often you can work on them if you are good enough at what you do. (But yes, often you have to prove that first, either internally or externally)
  - 11y ago

martincmartin 11y ago

A good article about this is "There is No Now" on ACM: https://queue.acm.org/detail.cfm?id=2745385 https://queue.acm.org/detail.cfm?id=2745385

mmoya 11y ago

> I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale. Check this talk: https://archive.fosdem.org/2015/schedule/event/ntimed_ntpd_replacement/ https://archive.fosdem.org/2015/schedule/event/ntimed_ntpd_r... . It's more difficult than it appear.

mozumder 11y ago

> I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale. Timekeeping is such a basic problem that Albert Einstein invented the theory of relativity because of it.

tajano 11y ago

> "something as simple as keeping time" Keeping time has been a difficult problem for... well, as long as we've tried keeping time. For an interesting historical account on past difficulties, you might check out the book "Longitude" [1]. [1] http://www.amazon.com/Longitude-Genius-Greatest-Scientific-Problem/dp/080271529X http://www.amazon.com/Longitude-Genius-Greatest-Scientific-P...