14 ms·

Finding a problem at the bottom of the Google stack

325 points by 9nGQluzmnq3M 7y ago

Nextgrid 7y ago
I wonder why it took as far as having the actual software running on the machine to fail and have user-facing consequences for them to notice that something was wrong. With all that bragging about how good they are, why didn't they have alerts that would let them know the temperature was higher than normal before it got to a level critical enough to affect software operation?
- alephnan 7y ago
  > We consider these types of failures "within error budget" events. They are expected, accepted, and engineered into the design criteria of our systems. However, they still get tracked down to make sure they aren’t forgotten and accumulated into technical debt
  - Nextgrid 7y ago
    Yes, I understand that application-level exceptions can sometimes be "budgeted for" and a certain amount of them is acceptable. Hardware metrics on the other hand should IMO be monitored and alerts should be in place for when things go wrong, as the consequences of it could potentially be way more serious.
    - jefftk 7y ago
      Things look very different when you have as much hardware as Google does. (Disclosure: I work for Google)
      - pas 7y ago
        When the CPU hits the thermal envelope regularly and throttles itself that seems like a serious enough problem. Sure, if it's a normal part of cost optimization, then it won't stand out.
- jpollock 7y ago
  Because the heat increase would have been pretty much immediate after the wheel failed. This wouldn't have been a gradual degradation.
- caconym_ 7y ago
  > Rare, unexpected chains of events happen often. Some have visible impact, but most don't.
- IshKebab 7y ago
  Seems like they don't even have alerts to let them know when they lose emails in your gmail account: https://support.google.com/mail/thread/6187016?hl=en https://support.google.com/mail/thread/6187016?hl=en Still, stuff like this is really hard. It's easy to think of what should have been done in hindsight.
- Forge36 7y ago
  >an abnormally high number of errors I wonder what the initial errors being logged were. Based on the post >These errors indicated CPU throttling It sounds like the first tool is "show error counts". It might have been "production is slower than normal, please investigate" Your suggestion sounds like a good way to flag future issues "1 in 50 racks for X is running a temp", however I'm curious what the false positive rate would be. What if it isn't causing problems? Maybe you've spotted smoke.
  - Nextgrid 7y ago
    My suggestion is that the on-site people should have alerts about hardware events. Unless it's a failing sensor, there shouldn't be any false positives - if a machine is exceeding its recommended operating temperature it's still not good and warrants an investigation even if it isn't yet causing problems (in fact the point of these alerts is to catch and fix these issues before they cause any user-facing issues).
    - Forge36 7y ago
      That I like. From the photos it's also obvious the rack is tipping. I'm curious why someone isn't doing a regular walkthrough. There could also be sounds of some hardware failures. A remote SRE shouldn't need to monitor the hardware health, an on-site person should have caught this sooner. I wonder how the person responsible for fixing/replacing the wheels felt about the follow-up
      - cameronbrown 7y ago
        Sure - the issue is we (the public reading this) don't know when this actually physically failed. It could be that SRE picked up on this before a scheduled walkthrough (of which I'm sure occurs) happened. Disclaimer: Google employee. No knowledge of this event beyond the blog post.
  - jldugger 7y ago
    If BGP is busted, those errors were probably stuff like connection timeouts on services running inside borg/k8s. But like, it's not impossible to measure temp on devices. It's entirely possible to design a temp monitor that tracks the physical grouping of equipment: datacenter -> aisle -> rack -> server The real question is one of prioritization: if a rack's temp has raised but there's no customer error, is it an immediate problem?
- mjw1007 7y ago
  Right. High temperature and thermal throttling are things that can and should be monitored directly. Spotting it by digging down from service-level monitoring instead isn't terrible, but it isn't something to be proud of either.
  - bryanrasmussen 7y ago
    maybe something to be proud of by the person who actually did the digging down.
  - thedance 7y ago
    It’s just the org chart being reflected in their stack architecture. There is a group responsible for machine health. That group doesn’t run any services. There is a group that operates GFEs. They don’t own any machines.
    - fragmede 7y ago
      Does that imply a SaaS-based revision to Conway's law? (GFEs are a reverse proxy that terminates TCP for Google. https://landing.google.com/sre/sre-book/chapters/production-environment/ https://landing.google.com/sre/sre-book/chapters/production-... )
- H8crilA 7y ago
  Probably because it had never happened before. And I bet this is in the "recommendations for the future", or whatever it's called, section of this postmortem.
  - bluedino 7y ago
    They probably had throttling alerts back in the Sandy Bridge days (it was common for Linux machines to hit 100% CPU usage but not actually do anything back then because of some kernel bugs), you'd think they'd have them monitored.
    - thedance 7y ago
      Overheating machines is very common. Even if you only own ten thousand machines it’s pretty much assured you’ve got one machine out there with a cracked heat sink or similar thermal issue.
      - mrep 7y ago
        But at google scale, you'd think they would have something for that. At a previous company I worked at, we had a guy whose entire job was to monitor temperatures throughout all of our data centers. He built a service that collected minute by minute temperature metrics on every server in every datacenter. Once collected it would create automated tickets for data center technicians to investigate when servers and racks got too hot. He even built this cool website where you could view real time maps of each rack and their temperature in every datacenter so that operations teams could visualize the high temperature areas in the data centers.
        thedance 7y ago
        It's not that they don't know, it's that they don't care. Hot machines go into an attention queue and some tech eventually fixes it. Meanwhile, they serve. As I commented elsewhere, I think it's an effective incentive for software to grow more sophisticated. In fleets that are promptly fixed, there may be a tendency to assume that, for example, all backends are equally suited to receive an RPC. Having a lot of ambient brokenness can be helpful, as Netflix advocates with "chaos monkey".
- insomniacity 7y ago
  I've never worked at a FAANG, but there is a school of thought that if an error/alert isn't impacting your SLO, you don't care. As they said, it was within the error budget, which means that they could instead focus on delivering new features. When the errors go out of budget, they stop delivering features and start delivering fixes to bring it back in budget. It's the kind of attitude you can have at that scale.
  - mmckeen 7y ago
    It's the kind of attitude you kinda have to have at that scale, otherwise it's very hard to balance your time between shipping features or fixing issues (you end up biasing towards one side or the other for the detriment of the overall product/team).
- dlandis 7y ago
  > The SRE tracked the problem all the way from an external, front-end system down to the hardware that holds up the machines. I agree, this does seem like an incredibly circuitous and inefficient route to diagnosing the issue, as opposed to the hardware team simply receiving an alert that a server was continuously in an overheated, and then fixing it.
  - jldugger 7y ago
    In my experience the breakfix team has no shortage of work to be done relative to staffing. But they have no insight into the relative priorities of things. That's where your customer metrics and in depth investigation helps. It's entirely possible a server overheating ticket was in the queue so to speak, and the investigation ends up re-prioritizing an alarm that recovered before anyone was on site to fix it.
- cranekam 7y ago
  Because with tens or hundreds of thousands of racks it would be a full-time job to go through such alerts. Some are likely real (hi broken wheels) and some almost certainly aren't (hi someone training a ML model). And users don't care: as a user I don't care if a host or rack is hot. All I care is that when someone (an SRE or automation) notices errors the rack is drained and the errors go away. At somewhere small where an individual rack failing isn't okay it might make sense to monitor physical health more closely; where it doesn't matter it's better to focus on whether the user is having a good time. As someone else said, alert on symptoms (user-facing errors), not causes, because you're never going to be able to enumerate all the potential causes ahead of time and dealing with the alert onslaught from trying is unworkable.
  - dijit 7y ago
    > Because with tens or hundreds of thousands of racks it would be a full-time job to go through such alerts. Sorry, I'm merely a lowly sysadmin and not a godly Google SRE, but environment monitoring is a solved problem, humans need not apply.
    - joshuamorton 7y ago
      He's talking about after you filter them with automation. When you have enough machines, 10 of them are always running hot.
      - popinman322 7y ago
        It's Google; they have sufficient ML capacity to filter cases. It might require additional manpower in the short term, but it also lowers the likelihood of infrastructure failure long-term. What might not make sense, resource-wise, to smaller companies does start to make sense at their scale.
        jldugger 7y ago
        Anomaly detection on timeseries data is surprisingly bad. The data tends to be bad in all the wrong ways -- seasonal, and not normally distributed. And worse, you have no training data, so ML model you train simply learns whatever badness is allowed to persist in production as normal.
- maxwindiff 7y ago
  Waste heat production scales with system load.
dilippkumar 7y ago
This was a fun story. Could've been told in a more engaging way, but still a good read. I can't imagine a better way the phrase "bottom of the Google stack" could have been used. That phrase can now be retired.
rdxm 7y ago
this is awesome. one of the by-products of the public cloud era is a loss of the having to consider the physical side of things when considering operational art.
tgtweak 7y ago
We had a very similar situation recently where the colocation facility had replaced some missing block-out panels in a rack and it caused the top-of-rack switches to recycle hot air... the system temps of both switches were north of 100°C and to their credit (dell/force10 s4820Ts) they ran flawlessly and didn't degrade any traffic, sending appropriate notices to the reporting email. Something as benign as that can take out an entire infra if unchecked. I've seen racking fail in the past (usually someone loading a full 42U with high density disk systems putting the rack over weight capacity by a notable factor) and it is definitely a disaster situation. One datacenter manager recounted a story of a rack falling through the raised flooring back in the days when that was the standard (surprise - it kept running until noticed by a tech on a walkthrough). Good story but comes across as back-patting a bit.
- hinkley 7y ago
  Last time this line of discussion came up someone confessed that they discovered they’d accidentally made an exhaust fan into an intake fan thereby blowing hot air across their cpu into their drives and nearly baking them.
  - dfox 7y ago
    Most rackmount switches are not really designed to be used in datacenter rack as they do not have front-to-back airflow (and even if they would have, networking hardware tends to have all connectors on wrong side of the box). But almost everyone routinely does that which then necessitates knowing that installing all air baffles would lead to exactly this issue.
    - dnautics 7y ago
      > Most rackmount switches are not really designed to be used in datacenter rack as they do not have front-to-back airflow It's my understanding that switchable fans are standard on cisco, arista, and mellanox switches (have no idea about juniper) and besides, front-to-back or back-to-front depends on whether you mount in the back or in the front (both are valid) AND even if you go the "wrong way" patch panels are pretty easy to find and are cheap (at the expense of 1U)
SahAssar 7y ago
> In this event, an SRE on the traffic and load balancing team was alerted that some GFEs (Google front ends) in Google's edge network, which statelessly cache frequently accessed content, were producing an abnormally high number of errors What does "statelessly cache" mean? Stateless means it has no state, and cache means it saves frequently requested operations. How can it save anything without state?
- ivanbakel 7y ago
  Presumably it caches copies of frequently-requested content on nodes closer to the end-user, rather than always fetching that content from an origin node which might be quite far away (especially on a P2P network). It's "stateless" because stateful requests are always sent directly to the origin node, rather than being intercepted and reproduced from the closer cache.
  - SahAssar 7y ago
    I don't get how these two can be true at the same time? > Presumably it caches copies of frequently-requested content on nodes closer to the end-user, rather than always fetching that content from an origin node > It's "stateless" because stateful requests are always sent directly to the origin node What is a stateful request versus a stateless one? Every request has some sort of state.
    - stellar678 7y ago
      My understanding is it's about application state - cookies, mutations, etc... If the request is "give me the profile photo of the logged-in user" - the response depends on the stateful question of who is the logged-in user. If the request is "give me the gmail logo" - the response is identical no matter who's making the request. These can be cached on an edge node without regard to application state.
      - SahAssar 7y ago
        Okay, I understand. I have never heard that being called a stateless cache though, in my experience that is called a public cache. That is also what it is called in the http cache-control standard. I get what you mean, but calling a cache "stateless" seems like a contradiction in terms to me.
bluedino 7y ago
Do people do not visit their datacenters often enough to notice a tilted rack?
- mrep 7y ago
  The big companies have millions of servers. At that scale, they have software teams monitoring every part of the datacenter from temperatures on each server/rack down to where every drive is at and their health. It is also highly automated with systems to direct technicians on what/where/how to repair broken parts and thus they don't have people just patrolling racks looking for problems. They also have special badges to access the data centers so people generally cannot just visit datacenters as only a select few people with business justification can visit them.
- hackmiester 7y ago
  nope. My org could miss this for weeks. We would need alerting to tell us. We run a small data center on premises.
nwallin 7y ago
This is great work, but I feel like the wrong thing triggered the alert. The temperature spike should have been the first alert. The throttling should have been the second alert. The high error count should have been third. If you're thermal throttling, you have many problems that could give give all sorts of puzzling indications.
- bruckie 7y ago
  Google's alerting philosophy biases toward paging people only for user-visible symptoms, not potential causes, but there are still often lower priority ticket-based alerts for stuff like stuff getting too hot. For rationale, see: - https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/preview https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa... - https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/ https://landing.google.com/sre/sre-book/chapters/monitoring-...
  - mjw1007 7y ago
    That way of prioritising things makes sense, but in this case it meant the on-call SRE and someone from the edge networking team wasted their time diagnosing the hardware problem. It seems a shame that the blog post wasn't able to follow « They immediately removed ("drained") the machines from serving, thus eliminating the errors that might result in a degraded state for customers » with something like « and they were shown a notification that there was a low-priority automatic ticket showing a possible hardware problem on those machines, so they subscribed to that ticket and didn't waste any more time ».
    - thethethethe 7y ago
      > on-call SRE and someone from the edge networking team wasted their time diagnosing the hardware problem I highly doubt that the edge oncaller who was paged debugged this issue down to the hardware. The post even said that they worked with other teams to figure this out. Once the SRE figured out it wasn't their problem, the likely moved onto something else
- 7y ago
nn3 7y ago
So they didn't monitor temperature of all systems by default to catch cooling problems? Sounds more embarrassing to me. That's fairly basic mistakes.
- gnarbarian 7y ago
  Hindsight is 20/20. And there is at least some sort of survivership bias. In a huge system with lots of failover you won't notice all the contingencies they identified planed for and successfully mitigated. Then something breaks which; you thought would be covered under another redundancy, was covered until a recent change (firmware change to the HVAC system), you didn't plan for etc... And everyone points at the failure and says "that seems obvious", meanwhile the mountain of tests and monitoring and redundancy goes unnoticed.
  - alexandercrohde 7y ago
    Well, to point, at the very least the SRE post-mortem should have said "And we considered distributing monitoring software to all our datacenters for future alerting" [Not to say that every company out there would do this, nor that this wasn't necessarily considered, but definitely questioning the merit/objectivity of a brag-piece that is trying to rebuild the waning google hype]
  - Dylan16807 7y ago
    Nah, "monitor temperature" isn't a hindsight issue. Temperature is very near the top of the list of things to monitor, and everyone knows it.
- CBLT 7y ago
  Would you share what you have in mind? I (personally) would not page in the middle of the night for a temperature issue. The systems should be resilient enough to handle some cpu throttling. A non-transient issue like this would probably end up in a ticket queue.
CoffeeDregs 7y ago
This is a fun read but should read as a description of a fairly common method of developing products and remediating "deviations". A lot of words are spent on describing what is essentially a root-cause-analysis (see GxP, ISO 9000, fishbone diagram). Hopefully, you thought "check", "check", "check", as you read it. If you thought "couldn't Bob of shimmed it with a block of plywood?", you might want to read up on continuous-improvement. Have Bob put the shim in to fix the problem right quick, then start up the continuous improvement engine...
londons_explore 7y ago
Another way to tell the same story: "Someone at Google bought cheap casters designed to hold up an office table, and put them in the bottom of a server rack that weighed half a tonne. They failed. Tens of thousands of dollars were spent replacing them all"
- ariwilson 7y ago
  Way to take all the fun out of diagnosing a problem. THE most fun part of engineering IMO.
- yellow_lead 7y ago
  At least the story doesn't end with only "And then an RCA was written which is a good practice for any SRE team."
- johnghanks 7y ago
  Why is everyone here so negative all the time?
  - hackmiester 7y ago
    It's hard to say, but I'm feeling somewhat negative about this article too. A more neutral way to tell it might be: "Some of our servers were overheating. We took them out of service, then discovered they were overheating because of an issue where the rack hardware had failed and tiled the rack. We eliminated this problem from this and all other racks." The way it's currently written seems to try to set Google apart, but I'm not sure why it wouldn't work this way at any organization. Maybe I have missed the point.
londons_explore 7y ago
A small tilt like that wouldn't impact most water cooling systems. Most cooling systems would run with a 45 degree tilt, and some are completely sealed and would work any way up. I suspect this isn't a water cooling problem, but instead a heat pipe system, with a phase change material in (often butane). Heat pipes are used to conduct heat from CPU's to heatsinks in most laptops. They have a liquid in which boils, and then condenses at the other end of the pipe, and then flows back again as liquid. Heat pipes usually look like a thick bar of copper, but are in fact way more thermally conductive than copper. The inside of the pipe usually has a felt-like material to 'wick' water from the wet end to the dry end, but wicking is quite slow compared to guaranteeing the pipe is perfectly level and using gravity to just let the liquid flow downhill. I'm 99% sure that's the reason this system doesn't work with a slight slope.
- londons_explore 7y ago
  Interestingly, if you have a heat pipe with a bend in the middle (most laptop heat pipes have a bend), then if you orient the laptop so the bent bit is down, you can make a 'liquid lock' so vapor can't flow from one end of the pipe to the other. If that happens, the heat pipe pretty much stops conducting heat, and you'll see far worse thermal performance.
- hinkley 7y ago
  So that weird laptop stand that holds your laptop up at an angle... more cooling from below, but maybe the heat pipes aren’t as happy?
  - deleted 7y ago
    [deleted]
m0zg 7y ago
The most epic bug I've seen investigated at Google was a CPU bug in the CPUs that Intel made specifically for Google at the time. Only some chips had the bug. And only some cores on those chips had it, and to make matters worse it'd only manifest non-deterministically. I don't recall the full details, nor would I share them even if I did, but what struck me is that this mindset ("at Google one-in-a-million events happen all the time") and software engineering mindset that goes with it (checksumming all over the place, overbuilding for fault tolerance and data integrity) was basically the only reason why this bug was identified and fixed. In a company with lesser quality of engineering staff it would linger forever, silently corrupting customer data, and the company would be totally powerless to do anything about it. In fact they probably wouldn't even notice it until it's far too late. At Google, not only they noticed it right away, they also were able to track it all the way down to hardware (something that software engineers typically consider infallible) in a fairly short amount of time.
perlgeek 7y ago
It's cool to see them track down the errors like this, but I'd like to point out some weird things along the way: * why do the racks have wheels at all? Doesn't seem like a standard build, and turns out the be risky * there should be at least daily checks on the data center, including a visual inspection of cooling systems and the likes. I don't know if daily visual inspection of racks is also a thing, but should find that pretty quickly. * monitoring temperatures in a data center is pretty essential, though I must admit I don't know enough if rack-level temperature monitoring would have caused overheating of CPUs in the rack.
- cranekam 7y ago
  > * why do the racks have wheels at all? Doesn't seem like a standard build, and turns out the be risky They probably aren't "standard" in any kind of common sense. Google and the other big tech firms design their own hardware and likely have it assembled remotely and delivered as a complete rack. Wheels make positioning it easier. It's also not uncommon to need to move racks of machines around (say, during a cluster refit or to change capacity in different DC halls). Not having wheels is likely more of a pain than the potential gains of omitting them. > * there should be at least daily checks on the data center, including a visual inspection of cooling systems and the likes. I don't know if daily visual inspection of racks is also a thing, but should find that pretty quickly. Google has millions of machines and dozens of datacenters. Its techs are likely already 100% busy replacing HDDs and other routine stuff. Noticing a single rack failing (within error budget) isn't going to be a priority so it's unlikely someone is employed to do that. Cooling/power systems are definitely monitored on a more macro level. > * monitoring temperatures in a data center is pretty essential, though I must admit I don't know enough if rack-level temperature monitoring would have caused overheating of CPUs in the rack. There's likely DC-level temperature monitoring. Again, systems are designed to cope with some amount of failure so it's not important to monitor everything and alert someone every time anything changes. When dealing with millions of things you need to look at aggregations.
williesleg 7y ago
I'll tell you what, there's a problem at the top of the Google stack why it takes more than a few days to put up a simple website for Coronavirus. I guess they really aren't that sharp after all.
nitwit005 7y ago
Only hours later did I realize the title of this was a pun.
aeyes 7y ago
> All incidents should be novel. Ah, how nice would it be if every company had infinite engineering capacity.
mroshangara 7y ago
At Google scale, million-to-one chances happen all the time.?!
jeffrallen 7y ago
Wow, I was really NOT expecting that.
thedance 7y ago
The most interesting part of this piece is how it strongly implies that the BGP announcement thing in question is single-threaded. The "overloaded" machine is showing utilization just below 1 CPU, while the normal ones are loafing along at < 10% of 1 CPU.
- Dylan16807 7y ago
  What's interesting about that? Making it parallel is tricky and unnecessary, so they didn't.
williamDafoe 7y ago
Why doesn't Google have a visualization system for temperature of the racks, with monitoring and alarms? It seems the problem went undetected because SREs did a poor job to begin with .... No DC has more than a few tens of thousands of machines which is easily handle by 3 borgmon or 100 monarch machines LOL ...

325 points by 9nGQluzmnq3M 7y ago

I wonder why it took as far as having the actual software running on the machine to fail and have user-facing consequences for them to notice that something was wrong. With all that bragging about how good they are, why didn't they have alerts that would let them know the temperature was higher than normal before it got to a level critical enough to affect software operation?

alephnan 7y ago
> We consider these types of failures "within error budget" events. They are expected, accepted, and engineered into the design criteria of our systems. However, they still get tracked down to make sure they aren’t forgotten and accumulated into technical debt
- Nextgrid 7y ago
  Yes, I understand that application-level exceptions can sometimes be "budgeted for" and a certain amount of them is acceptable. Hardware metrics on the other hand should IMO be monitored and alerts should be in place for when things go wrong, as the consequences of it could potentially be way more serious.
  - jefftk 7y ago
    Things look very different when you have as much hardware as Google does. (Disclosure: I work for Google)
    - pas 7y ago
      When the CPU hits the thermal envelope regularly and throttles itself that seems like a serious enough problem. Sure, if it's a normal part of cost optimization, then it won't stand out.
jpollock 7y ago
Because the heat increase would have been pretty much immediate after the wheel failed. This wouldn't have been a gradual degradation.
caconym_ 7y ago
> Rare, unexpected chains of events happen often. Some have visible impact, but most don't.
IshKebab 7y ago
Seems like they don't even have alerts to let them know when they lose emails in your gmail account: https://support.google.com/mail/thread/6187016?hl=en https://support.google.com/mail/thread/6187016?hl=en Still, stuff like this is really hard. It's easy to think of what should have been done in hindsight.
Forge36 7y ago
>an abnormally high number of errors I wonder what the initial errors being logged were. Based on the post >These errors indicated CPU throttling It sounds like the first tool is "show error counts". It might have been "production is slower than normal, please investigate" Your suggestion sounds like a good way to flag future issues "1 in 50 racks for X is running a temp", however I'm curious what the false positive rate would be. What if it isn't causing problems? Maybe you've spotted smoke.
- Nextgrid 7y ago
  My suggestion is that the on-site people should have alerts about hardware events. Unless it's a failing sensor, there shouldn't be any false positives - if a machine is exceeding its recommended operating temperature it's still not good and warrants an investigation even if it isn't yet causing problems (in fact the point of these alerts is to catch and fix these issues before they cause any user-facing issues).
  - Forge36 7y ago
    That I like. From the photos it's also obvious the rack is tipping. I'm curious why someone isn't doing a regular walkthrough. There could also be sounds of some hardware failures. A remote SRE shouldn't need to monitor the hardware health, an on-site person should have caught this sooner. I wonder how the person responsible for fixing/replacing the wheels felt about the follow-up
    - cameronbrown 7y ago
      Sure - the issue is we (the public reading this) don't know when this actually physically failed. It could be that SRE picked up on this before a scheduled walkthrough (of which I'm sure occurs) happened. Disclaimer: Google employee. No knowledge of this event beyond the blog post.
- jldugger 7y ago
  If BGP is busted, those errors were probably stuff like connection timeouts on services running inside borg/k8s. But like, it's not impossible to measure temp on devices. It's entirely possible to design a temp monitor that tracks the physical grouping of equipment: datacenter -> aisle -> rack -> server The real question is one of prioritization: if a rack's temp has raised but there's no customer error, is it an immediate problem?
mjw1007 7y ago
Right. High temperature and thermal throttling are things that can and should be monitored directly. Spotting it by digging down from service-level monitoring instead isn't terrible, but it isn't something to be proud of either.
- bryanrasmussen 7y ago
  maybe something to be proud of by the person who actually did the digging down.
- thedance 7y ago
  It’s just the org chart being reflected in their stack architecture. There is a group responsible for machine health. That group doesn’t run any services. There is a group that operates GFEs. They don’t own any machines.
  - fragmede 7y ago
    Does that imply a SaaS-based revision to Conway's law? (GFEs are a reverse proxy that terminates TCP for Google. https://landing.google.com/sre/sre-book/chapters/production-environment/ https://landing.google.com/sre/sre-book/chapters/production-... )
H8crilA 7y ago
Probably because it had never happened before. And I bet this is in the "recommendations for the future", or whatever it's called, section of this postmortem.
- bluedino 7y ago
  They probably had throttling alerts back in the Sandy Bridge days (it was common for Linux machines to hit 100% CPU usage but not actually do anything back then because of some kernel bugs), you'd think they'd have them monitored.
  - thedance 7y ago
    Overheating machines is very common. Even if you only own ten thousand machines it’s pretty much assured you’ve got one machine out there with a cracked heat sink or similar thermal issue.
    - mrep 7y ago
      But at google scale, you'd think they would have something for that. At a previous company I worked at, we had a guy whose entire job was to monitor temperatures throughout all of our data centers. He built a service that collected minute by minute temperature metrics on every server in every datacenter. Once collected it would create automated tickets for data center technicians to investigate when servers and racks got too hot. He even built this cool website where you could view real time maps of each rack and their temperature in every datacenter so that operations teams could visualize the high temperature areas in the data centers.
      - thedance 7y ago
        It's not that they don't know, it's that they don't care. Hot machines go into an attention queue and some tech eventually fixes it. Meanwhile, they serve. As I commented elsewhere, I think it's an effective incentive for software to grow more sophisticated. In fleets that are promptly fixed, there may be a tendency to assume that, for example, all backends are equally suited to receive an RPC. Having a lot of ambient brokenness can be helpful, as Netflix advocates with "chaos monkey".
insomniacity 7y ago
I've never worked at a FAANG, but there is a school of thought that if an error/alert isn't impacting your SLO, you don't care. As they said, it was within the error budget, which means that they could instead focus on delivering new features. When the errors go out of budget, they stop delivering features and start delivering fixes to bring it back in budget. It's the kind of attitude you can have at that scale.
- mmckeen 7y ago
  It's the kind of attitude you kinda have to have at that scale, otherwise it's very hard to balance your time between shipping features or fixing issues (you end up biasing towards one side or the other for the detriment of the overall product/team).
dlandis 7y ago
> The SRE tracked the problem all the way from an external, front-end system down to the hardware that holds up the machines. I agree, this does seem like an incredibly circuitous and inefficient route to diagnosing the issue, as opposed to the hardware team simply receiving an alert that a server was continuously in an overheated, and then fixing it.
- jldugger 7y ago
  In my experience the breakfix team has no shortage of work to be done relative to staffing. But they have no insight into the relative priorities of things. That's where your customer metrics and in depth investigation helps. It's entirely possible a server overheating ticket was in the queue so to speak, and the investigation ends up re-prioritizing an alarm that recovered before anyone was on site to fix it.
cranekam 7y ago
Because with tens or hundreds of thousands of racks it would be a full-time job to go through such alerts. Some are likely real (hi broken wheels) and some almost certainly aren't (hi someone training a ML model). And users don't care: as a user I don't care if a host or rack is hot. All I care is that when someone (an SRE or automation) notices errors the rack is drained and the errors go away. At somewhere small where an individual rack failing isn't okay it might make sense to monitor physical health more closely; where it doesn't matter it's better to focus on whether the user is having a good time. As someone else said, alert on symptoms (user-facing errors), not causes, because you're never going to be able to enumerate all the potential causes ahead of time and dealing with the alert onslaught from trying is unworkable.
- dijit 7y ago
  > Because with tens or hundreds of thousands of racks it would be a full-time job to go through such alerts. Sorry, I'm merely a lowly sysadmin and not a godly Google SRE, but environment monitoring is a solved problem, humans need not apply.
  - joshuamorton 7y ago
    He's talking about after you filter them with automation. When you have enough machines, 10 of them are always running hot.
    - popinman322 7y ago
      It's Google; they have sufficient ML capacity to filter cases. It might require additional manpower in the short term, but it also lowers the likelihood of infrastructure failure long-term. What might not make sense, resource-wise, to smaller companies does start to make sense at their scale.
      - jldugger 7y ago
        Anomaly detection on timeseries data is surprisingly bad. The data tends to be bad in all the wrong ways -- seasonal, and not normally distributed. And worse, you have no training data, so ML model you train simply learns whatever badness is allowed to persist in production as normal.
maxwindiff 7y ago
Waste heat production scales with system load.

dilippkumar 7y ago

This was a fun story. Could've been told in a more engaging way, but still a good read. I can't imagine a better way the phrase "bottom of the Google stack" could have been used. That phrase can now be retired.

rdxm 7y ago

this is awesome. one of the by-products of the public cloud era is a loss of the having to consider the physical side of things when considering operational art.

tgtweak 7y ago

We had a very similar situation recently where the colocation facility had replaced some missing block-out panels in a rack and it caused the top-of-rack switches to recycle hot air... the system temps of both switches were north of 100°C and to their credit (dell/force10 s4820Ts) they ran flawlessly and didn't degrade any traffic, sending appropriate notices to the reporting email. Something as benign as that can take out an entire infra if unchecked. I've seen racking fail in the past (usually someone loading a full 42U with high density disk systems putting the rack over weight capacity by a notable factor) and it is definitely a disaster situation. One datacenter manager recounted a story of a rack falling through the raised flooring back in the days when that was the standard (surprise - it kept running until noticed by a tech on a walkthrough). Good story but comes across as back-patting a bit.

hinkley 7y ago
Last time this line of discussion came up someone confessed that they discovered they’d accidentally made an exhaust fan into an intake fan thereby blowing hot air across their cpu into their drives and nearly baking them.
- dfox 7y ago
  Most rackmount switches are not really designed to be used in datacenter rack as they do not have front-to-back airflow (and even if they would have, networking hardware tends to have all connectors on wrong side of the box). But almost everyone routinely does that which then necessitates knowing that installing all air baffles would lead to exactly this issue.
  - dnautics 7y ago
    > Most rackmount switches are not really designed to be used in datacenter rack as they do not have front-to-back airflow It's my understanding that switchable fans are standard on cisco, arista, and mellanox switches (have no idea about juniper) and besides, front-to-back or back-to-front depends on whether you mount in the back or in the front (both are valid) AND even if you go the "wrong way" patch panels are pretty easy to find and are cheap (at the expense of 1U)

SahAssar 7y ago

> In this event, an SRE on the traffic and load balancing team was alerted that some GFEs (Google front ends) in Google's edge network, which statelessly cache frequently accessed content, were producing an abnormally high number of errors What does "statelessly cache" mean? Stateless means it has no state, and cache means it saves frequently requested operations. How can it save anything without state?

ivanbakel 7y ago
Presumably it caches copies of frequently-requested content on nodes closer to the end-user, rather than always fetching that content from an origin node which might be quite far away (especially on a P2P network). It's "stateless" because stateful requests are always sent directly to the origin node, rather than being intercepted and reproduced from the closer cache.
- SahAssar 7y ago
  I don't get how these two can be true at the same time? > Presumably it caches copies of frequently-requested content on nodes closer to the end-user, rather than always fetching that content from an origin node > It's "stateless" because stateful requests are always sent directly to the origin node What is a stateful request versus a stateless one? Every request has some sort of state.
  - stellar678 7y ago
    My understanding is it's about application state - cookies, mutations, etc... If the request is "give me the profile photo of the logged-in user" - the response depends on the stateful question of who is the logged-in user. If the request is "give me the gmail logo" - the response is identical no matter who's making the request. These can be cached on an edge node without regard to application state.
    - SahAssar 7y ago
      Okay, I understand. I have never heard that being called a stateless cache though, in my experience that is called a public cache. That is also what it is called in the http cache-control standard. I get what you mean, but calling a cache "stateless" seems like a contradiction in terms to me.

bluedino 7y ago

Do people do not visit their datacenters often enough to notice a tilted rack?

mrep 7y ago
The big companies have millions of servers. At that scale, they have software teams monitoring every part of the datacenter from temperatures on each server/rack down to where every drive is at and their health. It is also highly automated with systems to direct technicians on what/where/how to repair broken parts and thus they don't have people just patrolling racks looking for problems. They also have special badges to access the data centers so people generally cannot just visit datacenters as only a select few people with business justification can visit them.
hackmiester 7y ago
nope. My org could miss this for weeks. We would need alerting to tell us. We run a small data center on premises.

nwallin 7y ago

This is great work, but I feel like the wrong thing triggered the alert. The temperature spike should have been the first alert. The throttling should have been the second alert. The high error count should have been third. If you're thermal throttling, you have many problems that could give give all sorts of puzzling indications.

bruckie 7y ago
Google's alerting philosophy biases toward paging people only for user-visible symptoms, not potential causes, but there are still often lower priority ticket-based alerts for stuff like stuff getting too hot. For rationale, see: - https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/preview https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa... - https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/ https://landing.google.com/sre/sre-book/chapters/monitoring-...
- mjw1007 7y ago
  That way of prioritising things makes sense, but in this case it meant the on-call SRE and someone from the edge networking team wasted their time diagnosing the hardware problem. It seems a shame that the blog post wasn't able to follow « They immediately removed ("drained") the machines from serving, thus eliminating the errors that might result in a degraded state for customers » with something like « and they were shown a notification that there was a low-priority automatic ticket showing a possible hardware problem on those machines, so they subscribed to that ticket and didn't waste any more time ».
  - thethethethe 7y ago
    > on-call SRE and someone from the edge networking team wasted their time diagnosing the hardware problem I highly doubt that the edge oncaller who was paged debugged this issue down to the hardware. The post even said that they worked with other teams to figure this out. Once the SRE figured out it wasn't their problem, the likely moved onto something else
7y ago

nn3 7y ago

So they didn't monitor temperature of all systems by default to catch cooling problems? Sounds more embarrassing to me. That's fairly basic mistakes.

gnarbarian 7y ago
Hindsight is 20/20. And there is at least some sort of survivership bias. In a huge system with lots of failover you won't notice all the contingencies they identified planed for and successfully mitigated. Then something breaks which; you thought would be covered under another redundancy, was covered until a recent change (firmware change to the HVAC system), you didn't plan for etc... And everyone points at the failure and says "that seems obvious", meanwhile the mountain of tests and monitoring and redundancy goes unnoticed.
- alexandercrohde 7y ago
  Well, to point, at the very least the SRE post-mortem should have said "And we considered distributing monitoring software to all our datacenters for future alerting" [Not to say that every company out there would do this, nor that this wasn't necessarily considered, but definitely questioning the merit/objectivity of a brag-piece that is trying to rebuild the waning google hype]
- Dylan16807 7y ago
  Nah, "monitor temperature" isn't a hindsight issue. Temperature is very near the top of the list of things to monitor, and everyone knows it.
CBLT 7y ago
Would you share what you have in mind? I (personally) would not page in the middle of the night for a temperature issue. The systems should be resilient enough to handle some cpu throttling. A non-transient issue like this would probably end up in a ticket queue.

CoffeeDregs 7y ago

This is a fun read but should read as a description of a fairly common method of developing products and remediating "deviations". A lot of words are spent on describing what is essentially a root-cause-analysis (see GxP, ISO 9000, fishbone diagram). Hopefully, you thought "check", "check", "check", as you read it. If you thought "couldn't Bob of shimmed it with a block of plywood?", you might want to read up on continuous-improvement. Have Bob put the shim in to fix the problem right quick, then start up the continuous improvement engine...

londons_explore 7y ago

Another way to tell the same story: "Someone at Google bought cheap casters designed to hold up an office table, and put them in the bottom of a server rack that weighed half a tonne. They failed. Tens of thousands of dollars were spent replacing them all"

ariwilson 7y ago
Way to take all the fun out of diagnosing a problem. THE most fun part of engineering IMO.
yellow_lead 7y ago
At least the story doesn't end with only "And then an RCA was written which is a good practice for any SRE team."
johnghanks 7y ago
Why is everyone here so negative all the time?
- hackmiester 7y ago
  It's hard to say, but I'm feeling somewhat negative about this article too. A more neutral way to tell it might be: "Some of our servers were overheating. We took them out of service, then discovered they were overheating because of an issue where the rack hardware had failed and tiled the rack. We eliminated this problem from this and all other racks." The way it's currently written seems to try to set Google apart, but I'm not sure why it wouldn't work this way at any organization. Maybe I have missed the point.

londons_explore 7y ago

A small tilt like that wouldn't impact most water cooling systems. Most cooling systems would run with a 45 degree tilt, and some are completely sealed and would work any way up. I suspect this isn't a water cooling problem, but instead a heat pipe system, with a phase change material in (often butane). Heat pipes are used to conduct heat from CPU's to heatsinks in most laptops. They have a liquid in which boils, and then condenses at the other end of the pipe, and then flows back again as liquid. Heat pipes usually look like a thick bar of copper, but are in fact way more thermally conductive than copper. The inside of the pipe usually has a felt-like material to 'wick' water from the wet end to the dry end, but wicking is quite slow compared to guaranteeing the pipe is perfectly level and using gravity to just let the liquid flow downhill. I'm 99% sure that's the reason this system doesn't work with a slight slope.

londons_explore 7y ago
Interestingly, if you have a heat pipe with a bend in the middle (most laptop heat pipes have a bend), then if you orient the laptop so the bent bit is down, you can make a 'liquid lock' so vapor can't flow from one end of the pipe to the other. If that happens, the heat pipe pretty much stops conducting heat, and you'll see far worse thermal performance.
hinkley 7y ago
So that weird laptop stand that holds your laptop up at an angle... more cooling from below, but maybe the heat pipes aren’t as happy?
- deleted 7y ago
  [deleted]

m0zg 7y ago

The most epic bug I've seen investigated at Google was a CPU bug in the CPUs that Intel made specifically for Google at the time. Only some chips had the bug. And only some cores on those chips had it, and to make matters worse it'd only manifest non-deterministically. I don't recall the full details, nor would I share them even if I did, but what struck me is that this mindset ("at Google one-in-a-million events happen all the time") and software engineering mindset that goes with it (checksumming all over the place, overbuilding for fault tolerance and data integrity) was basically the only reason why this bug was identified and fixed. In a company with lesser quality of engineering staff it would linger forever, silently corrupting customer data, and the company would be totally powerless to do anything about it. In fact they probably wouldn't even notice it until it's far too late. At Google, not only they noticed it right away, they also were able to track it all the way down to hardware (something that software engineers typically consider infallible) in a fairly short amount of time.

perlgeek 7y ago

It's cool to see them track down the errors like this, but I'd like to point out some weird things along the way: * why do the racks have wheels at all? Doesn't seem like a standard build, and turns out the be risky * there should be at least daily checks on the data center, including a visual inspection of cooling systems and the likes. I don't know if daily visual inspection of racks is also a thing, but should find that pretty quickly. * monitoring temperatures in a data center is pretty essential, though I must admit I don't know enough if rack-level temperature monitoring would have caused overheating of CPUs in the rack.

cranekam 7y ago
> * why do the racks have wheels at all? Doesn't seem like a standard build, and turns out the be risky They probably aren't "standard" in any kind of common sense. Google and the other big tech firms design their own hardware and likely have it assembled remotely and delivered as a complete rack. Wheels make positioning it easier. It's also not uncommon to need to move racks of machines around (say, during a cluster refit or to change capacity in different DC halls). Not having wheels is likely more of a pain than the potential gains of omitting them. > * there should be at least daily checks on the data center, including a visual inspection of cooling systems and the likes. I don't know if daily visual inspection of racks is also a thing, but should find that pretty quickly. Google has millions of machines and dozens of datacenters. Its techs are likely already 100% busy replacing HDDs and other routine stuff. Noticing a single rack failing (within error budget) isn't going to be a priority so it's unlikely someone is employed to do that. Cooling/power systems are definitely monitored on a more macro level. > * monitoring temperatures in a data center is pretty essential, though I must admit I don't know enough if rack-level temperature monitoring would have caused overheating of CPUs in the rack. There's likely DC-level temperature monitoring. Again, systems are designed to cope with some amount of failure so it's not important to monitor everything and alert someone every time anything changes. When dealing with millions of things you need to look at aggregations.

williesleg 7y ago

I'll tell you what, there's a problem at the top of the Google stack why it takes more than a few days to put up a simple website for Coronavirus. I guess they really aren't that sharp after all.

nitwit005 7y ago

Only hours later did I realize the title of this was a pun.

aeyes 7y ago

> All incidents should be novel. Ah, how nice would it be if every company had infinite engineering capacity.

mroshangara 7y ago

At Google scale, million-to-one chances happen all the time.?!

jeffrallen 7y ago

Wow, I was really NOT expecting that.

thedance 7y ago

The most interesting part of this piece is how it strongly implies that the BGP announcement thing in question is single-threaded. The "overloaded" machine is showing utilization just below 1 CPU, while the normal ones are loafing along at < 10% of 1 CPU.

Dylan16807 7y ago
What's interesting about that? Making it parallel is tricky and unnecessary, so they didn't.

williamDafoe 7y ago

Why doesn't Google have a visualization system for temperature of the racks, with monitoring and alarms? It seems the problem went undetected because SREs did a poor job to begin with .... No DC has more than a few tens of thousands of machines which is easily handle by 3 borgmon or 100 monarch machines LOL ...