6 ms·
I wonder why it took as far as having the actual software running on the machine to fail and have user-facing consequences for them to notice that something was
by Nextgrid 7y ago
I wonder why it took as far as having the actual software running on the machine to fail and have user-facing consequences for them to notice that something was wrong. With all that bragging about how good they are, why didn't they have alerts that would let them know the temperature was higher than normal before it got to a level critical enough to affect software operation?
- alephnan 7y ago> We consider these types of failures "within error budget" events. They are expected, accepted, and engineered into the design criteria of our systems. However, they still get tracked down to make sure they aren’t forgotten and accumulated into technical debt
- Nextgrid 7y agoYes, I understand that application-level exceptions can sometimes be "budgeted for" and a certain amount of them is acceptable. Hardware metrics on the other hand should IMO be monitored and alerts should be in place for when things go wrong, as the consequences of it could potentially be way more serious.
- jefftk 7y agoThings look very different when you have as much hardware as Google does. (Disclosure: I work for Google)
- pas 7y agoWhen the CPU hits the thermal envelope regularly and throttles itself that seems like a serious enough problem. Sure, if it's a normal part of cost optimization, then it won't stand out.
- jpollock 7y agoBecause the heat increase would have been pretty much immediate after the wheel failed. This wouldn't have been a gradual degradation.
- caconym_ 7y ago> Rare, unexpected chains of events happen often. Some have visible impact, but most don't.
- IshKebab 7y agoSeems like they don't even have alerts to let them know when they lose emails in your gmail account: https://support.google.com/mail/thread/6187016?hl=en https://support.google.com/mail/thread/6187016?hl=en Still, stuff like this is really hard. It's easy to think of what should have been done in hindsight.
- Forge36 7y ago>an abnormally high number of errors I wonder what the initial errors being logged were. Based on the post >These errors indicated CPU throttling It sounds like the first tool is "show error counts". It might have been "production is slower than normal, please investigate" Your suggestion sounds like a good way to flag future issues "1 in 50 racks for X is running a temp", however I'm curious what the false positive rate would be. What if it isn't causing problems? Maybe you've spotted smoke.
- Nextgrid 7y agoMy suggestion is that the on-site people should have alerts about hardware events. Unless it's a failing sensor, there shouldn't be any false positives - if a machine is exceeding its recommended operating temperature it's still not good and warrants an investigation even if it isn't yet causing problems (in fact the point of these alerts is to catch and fix these issues before they cause any user-facing issues).
- Forge36 7y agoThat I like. From the photos it's also obvious the rack is tipping. I'm curious why someone isn't doing a regular walkthrough. There could also be sounds of some hardware failures. A remote SRE shouldn't need to monitor the hardware health, an on-site person should have caught this sooner. I wonder how the person responsible for fixing/replacing the wheels felt about the follow-up
- cameronbrown 7y agoSure - the issue is we (the public reading this) don't know when this actually physically failed. It could be that SRE picked up on this before a scheduled walkthrough (of which I'm sure occurs) happened. Disclaimer: Google employee. No knowledge of this event beyond the blog post.
- jldugger 7y agoIf BGP is busted, those errors were probably stuff like connection timeouts on services running inside borg/k8s. But like, it's not impossible to measure temp on devices. It's entirely possible to design a temp monitor that tracks the physical grouping of equipment: datacenter -> aisle -> rack -> server The real question is one of prioritization: if a rack's temp has raised but there's no customer error, is it an immediate problem?
- mjw1007 7y agoRight. High temperature and thermal throttling are things that can and should be monitored directly. Spotting it by digging down from service-level monitoring instead isn't terrible, but it isn't something to be proud of either.
- bryanrasmussen 7y agomaybe something to be proud of by the person who actually did the digging down.
- thedance 7y agoIt’s just the org chart being reflected in their stack architecture. There is a group responsible for machine health. That group doesn’t run any services. There is a group that operates GFEs. They don’t own any machines.
- fragmede 7y agoDoes that imply a SaaS-based revision to Conway's law? (GFEs are a reverse proxy that terminates TCP for Google. https://landing.google.com/sre/sre-book/chapters/production-environment/ https://landing.google.com/sre/sre-book/chapters/production-... )
- H8crilA 7y agoProbably because it had never happened before. And I bet this is in the "recommendations for the future", or whatever it's called, section of this postmortem.
- bluedino 7y agoThey probably had throttling alerts back in the Sandy Bridge days (it was common for Linux machines to hit 100% CPU usage but not actually do anything back then because of some kernel bugs), you'd think they'd have them monitored.
- thedance 7y agoOverheating machines is very common. Even if you only own ten thousand machines it’s pretty much assured you’ve got one machine out there with a cracked heat sink or similar thermal issue.
- mrep 7y agoBut at google scale, you'd think they would have something for that. At a previous company I worked at, we had a guy whose entire job was to monitor temperatures throughout all of our data centers. He built a service that collected minute by minute temperature metrics on every server in every datacenter. Once collected it would create automated tickets for data center technicians to investigate when servers and racks got too hot. He even built this cool website where you could view real time maps of each rack and their temperature in every datacenter so that operations teams could visualize the high temperature areas in the data centers.
- thedance 7y agoIt's not that they don't know, it's that they don't care. Hot machines go into an attention queue and some tech eventually fixes it. Meanwhile, they serve. As I commented elsewhere, I think it's an effective incentive for software to grow more sophisticated. In fleets that are promptly fixed, there may be a tendency to assume that, for example, all backends are equally suited to receive an RPC. Having a lot of ambient brokenness can be helpful, as Netflix advocates with "chaos monkey".
- insomniacity 7y agoI've never worked at a FAANG, but there is a school of thought that if an error/alert isn't impacting your SLO, you don't care. As they said, it was within the error budget, which means that they could instead focus on delivering new features. When the errors go out of budget, they stop delivering features and start delivering fixes to bring it back in budget. It's the kind of attitude you can have at that scale.
- mmckeen 7y agoIt's the kind of attitude you kinda have to have at that scale, otherwise it's very hard to balance your time between shipping features or fixing issues (you end up biasing towards one side or the other for the detriment of the overall product/team).
- dlandis 7y ago> The SRE tracked the problem all the way from an external, front-end system down to the hardware that holds up the machines. I agree, this does seem like an incredibly circuitous and inefficient route to diagnosing the issue, as opposed to the hardware team simply receiving an alert that a server was continuously in an overheated, and then fixing it.
- jldugger 7y agoIn my experience the breakfix team has no shortage of work to be done relative to staffing. But they have no insight into the relative priorities of things. That's where your customer metrics and in depth investigation helps. It's entirely possible a server overheating ticket was in the queue so to speak, and the investigation ends up re-prioritizing an alarm that recovered before anyone was on site to fix it.
- cranekam 7y agoBecause with tens or hundreds of thousands of racks it would be a full-time job to go through such alerts. Some are likely real (hi broken wheels) and some almost certainly aren't (hi someone training a ML model). And users don't care: as a user I don't care if a host or rack is hot. All I care is that when someone (an SRE or automation) notices errors the rack is drained and the errors go away. At somewhere small where an individual rack failing isn't okay it might make sense to monitor physical health more closely; where it doesn't matter it's better to focus on whether the user is having a good time. As someone else said, alert on symptoms (user-facing errors), not causes, because you're never going to be able to enumerate all the potential causes ahead of time and dealing with the alert onslaught from trying is unworkable.
- dijit 7y ago> Because with tens or hundreds of thousands of racks it would be a full-time job to go through such alerts. Sorry, I'm merely a lowly sysadmin and not a godly Google SRE, but environment monitoring is a solved problem, humans need not apply.
- joshuamorton 7y agoHe's talking about after you filter them with automation. When you have enough machines, 10 of them are always running hot.
- popinman322 7y agoIt's Google; they have sufficient ML capacity to filter cases. It might require additional manpower in the short term, but it also lowers the likelihood of infrastructure failure long-term. What might not make sense, resource-wise, to smaller companies does start to make sense at their scale.
- jldugger 7y agoAnomaly detection on timeseries data is surprisingly bad. The data tends to be bad in all the wrong ways -- seasonal, and not normally distributed. And worse, you have no training data, so ML model you train simply learns whatever badness is allowed to persist in production as normal.
- maxwindiff 7y agoWaste heat production scales with system load.