5 ms·

>an abnormally high number of errors I wonder what the initial errors being logged were. Based on the post >These errors indicated CPU throttling It sounds li

>an abnormally high number of errors I wonder what the initial errors being logged were. Based on the post >These errors indicated CPU throttling It sounds like the first tool is "show error counts". It might have been "production is slower than normal, please investigate" Your suggestion sounds like a good way to flag future issues "1 in 50 racks for X is running a temp", however I'm curious what the false positive rate would be. What if it isn't causing problems? Maybe you've spotted smoke.

Nextgrid 7y ago
My suggestion is that the on-site people should have alerts about hardware events. Unless it's a failing sensor, there shouldn't be any false positives - if a machine is exceeding its recommended operating temperature it's still not good and warrants an investigation even if it isn't yet causing problems (in fact the point of these alerts is to catch and fix these issues before they cause any user-facing issues).
- Forge36 7y ago
  That I like. From the photos it's also obvious the rack is tipping. I'm curious why someone isn't doing a regular walkthrough. There could also be sounds of some hardware failures. A remote SRE shouldn't need to monitor the hardware health, an on-site person should have caught this sooner. I wonder how the person responsible for fixing/replacing the wheels felt about the follow-up
  - cameronbrown 7y ago
    Sure - the issue is we (the public reading this) don't know when this actually physically failed. It could be that SRE picked up on this before a scheduled walkthrough (of which I'm sure occurs) happened. Disclaimer: Google employee. No knowledge of this event beyond the blog post.
jldugger 7y ago
If BGP is busted, those errors were probably stuff like connection timeouts on services running inside borg/k8s. But like, it's not impossible to measure temp on devices. It's entirely possible to design a temp monitor that tracks the physical grouping of equipment: datacenter -> aisle -> rack -> server The real question is one of prioritization: if a rack's temp has raised but there's no customer error, is it an immediate problem?
- cowsandmilk 7y ago
  > if a rack's temp has raised but there's no customer error, is it an immediate problem? Here, the problem was on a machine running a google application, so they noticed. But this is a post on the google cloud blog. This just makes me think that Google isn’t monitoring the health of the hardware they provide to customers in the cloud. It is a change you have to make when you change the layer at which you are providing services to customers. If I’m using the google maps web site, I don’t care if they are monitoring cpu temperature if layers above insulate me from impact. If I’m spinning up a virtual machine, I will be directly impacted.
  - jldugger 7y ago
    Netflix keynotes described how entire AWS AZs can and will go offline, and how to induce failures to exercise recovery paths, so why is the evaluation criteria for GCP 'my single point of failure VM cannot ever go down?'
    - AlphaSite 7y ago
      Most companies are not Netflix and I’m not sure I understand why we are discussing AWS? That post is more a statement on how errors which can be handled at the app layer can have catastrophic effects on lower level components. You cannot assume end customers are running thing at the scale or with the fault tolerance of google, or Netflix.
      - joshuamorton 7y ago
pas 7y ago
What's wrong with looking at smoke? Maybe it pays off to put the smokey ones into a separate class, eg batch processing, so they will unlikely impact frontend serving requests.

Nextgrid 7y ago

My suggestion is that the on-site people should have alerts about hardware events. Unless it's a failing sensor, there shouldn't be any false positives - if a machine is exceeding its recommended operating temperature it's still not good and warrants an investigation even if it isn't yet causing problems (in fact the point of these alerts is to catch and fix these issues before they cause any user-facing issues).

Forge36 7y ago
That I like. From the photos it's also obvious the rack is tipping. I'm curious why someone isn't doing a regular walkthrough. There could also be sounds of some hardware failures. A remote SRE shouldn't need to monitor the hardware health, an on-site person should have caught this sooner. I wonder how the person responsible for fixing/replacing the wheels felt about the follow-up
- cameronbrown 7y ago
  Sure - the issue is we (the public reading this) don't know when this actually physically failed. It could be that SRE picked up on this before a scheduled walkthrough (of which I'm sure occurs) happened. Disclaimer: Google employee. No knowledge of this event beyond the blog post.

jldugger 7y ago

If BGP is busted, those errors were probably stuff like connection timeouts on services running inside borg/k8s. But like, it's not impossible to measure temp on devices. It's entirely possible to design a temp monitor that tracks the physical grouping of equipment: datacenter -> aisle -> rack -> server The real question is one of prioritization: if a rack's temp has raised but there's no customer error, is it an immediate problem?

cowsandmilk 7y ago
> if a rack's temp has raised but there's no customer error, is it an immediate problem? Here, the problem was on a machine running a google application, so they noticed. But this is a post on the google cloud blog. This just makes me think that Google isn’t monitoring the health of the hardware they provide to customers in the cloud. It is a change you have to make when you change the layer at which you are providing services to customers. If I’m using the google maps web site, I don’t care if they are monitoring cpu temperature if layers above insulate me from impact. If I’m spinning up a virtual machine, I will be directly impacted.
- jldugger 7y ago
  Netflix keynotes described how entire AWS AZs can and will go offline, and how to induce failures to exercise recovery paths, so why is the evaluation criteria for GCP 'my single point of failure VM cannot ever go down?'
  - AlphaSite 7y ago
    Most companies are not Netflix and I’m not sure I understand why we are discussing AWS? That post is more a statement on how errors which can be handled at the app layer can have catastrophic effects on lower level components. You cannot assume end customers are running thing at the scale or with the fault tolerance of google, or Netflix.
    - joshuamorton 7y ago

pas 7y ago

What's wrong with looking at smoke? Maybe it pays off to put the smokey ones into a separate class, eg batch processing, so they will unlikely impact frontend serving requests.