4 ms·

> particular well-known machine was unracked are you able to comment a bit further on why this machine was well known?

jeffbee 6y ago
Because of the way engineers habitually run batch jobs with more replicas than there are machines, this one broken computer had crapped up every map/reduce job in that facility for a long time, and it had been sent to repairs many times without benefit. Many people knew instinctively that if their job was stuck it was probably because of the shard on xyz42 (or whatever the node name was).
- the-rc 6y ago
  I still remember the machine name. It starts with an l and ends with a 6. Over the course of a couple of years, pretty much all of its components (CPUs, RAM, drives) were replaced at least once. You could look up its maintenance history and it went on and on. I'm not sure if it was well known across all of engineering; from what I recall, it was in a cluster in Oregon reserved for a specific team. Because it was company property, no matter how doomed, they had to get signoff from upper management, close to Eric Schmidt's level, before they could destroy it.
  - gwillen 6y ago
    My recollection, assuming it's the same machine I'm thinking of, is that it wasn't reserved for our team; rather, we left a do-nothing job permanently allocated to it, in order to prevent some poor other sucker from getting their job scheduled on it. (Because we, through painful experience, were well aware the machine had hardware problems; but we had long since given up on convincing the responsible parties to take it out of the pool, since it passed all their internal tests every time we complained. I don't remember how long this situation existed before someone finally took it out back and shot it.) Could be a different incident and a different machine, though. I'm sure this story happened more than once.
    - the-rc 6y ago
      Maybe a different machine? I meant that it was not in one of the general-purpose clusters: the entire pool was dedicated and a random team couldn't request Borg quota in it. For years, though, half of the Oregon datacenter was special for one reason or another. The infamous machine did go through repairs and part swaps many times, as you could see from its long and troubled hwops history. The worst machines were the zombies with NICs bad enough to break Stubby RPCs, but still passing heartbeat checks. Or breaking connections only when (re)using specific ports. Fun times!
  - exikyut 6y ago
    I wonder if MR could integrate with a fuzzing engine that jumbles random combinations of real inputs into garbage but runnable jobs that cause reproducible crashes above some threshold (eg at least once per day, or if things are bad enough, once per month or something). Regarding this system: the motherboard was never swapped?
- exikyut 6y ago
  In what way had the jobs failed? Very open-ended question :) but just coming from a hardware-diagnosis standpoint. (I guess the canonical answer is "here's the repair history," but yeah, duh.) > engineers habitually run batch jobs with more replicas than there are machines Idly curious, how do I parse parse this? It sounds like the same jobs are replicated to multiple machines as a sort of asynchronous, eventually-consistent lockstep arrangement?

Because of the way engineers habitually run batch jobs with more replicas than there are machines, this one broken computer had crapped up every map/reduce job in that facility for a long time, and it had been sent to repairs many times without benefit. Many people knew instinctively that if their job was stuck it was probably because of the shard on xyz42 (or whatever the node name was).

the-rc 6y ago
I still remember the machine name. It starts with an l and ends with a 6. Over the course of a couple of years, pretty much all of its components (CPUs, RAM, drives) were replaced at least once. You could look up its maintenance history and it went on and on. I'm not sure if it was well known across all of engineering; from what I recall, it was in a cluster in Oregon reserved for a specific team. Because it was company property, no matter how doomed, they had to get signoff from upper management, close to Eric Schmidt's level, before they could destroy it.
- gwillen 6y ago
  My recollection, assuming it's the same machine I'm thinking of, is that it wasn't reserved for our team; rather, we left a do-nothing job permanently allocated to it, in order to prevent some poor other sucker from getting their job scheduled on it. (Because we, through painful experience, were well aware the machine had hardware problems; but we had long since given up on convincing the responsible parties to take it out of the pool, since it passed all their internal tests every time we complained. I don't remember how long this situation existed before someone finally took it out back and shot it.) Could be a different incident and a different machine, though. I'm sure this story happened more than once.
  - the-rc 6y ago
    Maybe a different machine? I meant that it was not in one of the general-purpose clusters: the entire pool was dedicated and a random team couldn't request Borg quota in it. For years, though, half of the Oregon datacenter was special for one reason or another. The infamous machine did go through repairs and part swaps many times, as you could see from its long and troubled hwops history. The worst machines were the zombies with NICs bad enough to break Stubby RPCs, but still passing heartbeat checks. Or breaking connections only when (re)using specific ports. Fun times!
- exikyut 6y ago
  I wonder if MR could integrate with a fuzzing engine that jumbles random combinations of real inputs into garbage but runnable jobs that cause reproducible crashes above some threshold (eg at least once per day, or if things are bad enough, once per month or something). Regarding this system: the motherboard was never swapped?
exikyut 6y ago
In what way had the jobs failed? Very open-ended question :) but just coming from a hardware-diagnosis standpoint. (I guess the canonical answer is "here's the repair history," but yeah, duh.) > engineers habitually run batch jobs with more replicas than there are machines Idly curious, how do I parse parse this? It sounds like the same jobs are replicated to multiple machines as a sort of asynchronous, eventually-consistent lockstep arrangement?