5 ms·

This is great work, but I feel like the wrong thing triggered the alert. The temperature spike should have been the first alert. The throttling should have bee

This is great work, but I feel like the wrong thing triggered the alert. The temperature spike should have been the first alert. The throttling should have been the second alert. The high error count should have been third. If you're thermal throttling, you have many problems that could give give all sorts of puzzling indications.

bruckie 7y ago
Google's alerting philosophy biases toward paging people only for user-visible symptoms, not potential causes, but there are still often lower priority ticket-based alerts for stuff like stuff getting too hot. For rationale, see: - https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/preview https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa... - https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/ https://landing.google.com/sre/sre-book/chapters/monitoring-...
- mjw1007 7y ago
  That way of prioritising things makes sense, but in this case it meant the on-call SRE and someone from the edge networking team wasted their time diagnosing the hardware problem. It seems a shame that the blog post wasn't able to follow « They immediately removed ("drained") the machines from serving, thus eliminating the errors that might result in a degraded state for customers » with something like « and they were shown a notification that there was a low-priority automatic ticket showing a possible hardware problem on those machines, so they subscribed to that ticket and didn't waste any more time ».
  - thethethethe 7y ago
    > on-call SRE and someone from the edge networking team wasted their time diagnosing the hardware problem I highly doubt that the edge oncaller who was paged debugged this issue down to the hardware. The post even said that they worked with other teams to figure this out. Once the SRE figured out it wasn't their problem, the likely moved onto something else
thethethethe 7y ago
> The temperature spike should have been the first alert. The throttling should have been the second alert. The high error count should have been third. This is a completely backwards. What you are describing is caused based alerting, which is strongly discouraged by SRE. SREs prefer symptom based alerting (e.i are users seeing errors) because you only get alerted when you know that there is a problem that effects your business. If you only have caused based alerts, you will get false alarms and un-actionable pages all of the time. Imagine being an SRE who manages the GFEs in this situation. Now imagine getting a page telling you that the tempurature in a rack where your job is running is hot. What are you supposed to do? Is the issue effecting users? If so, how many? Is it anomalous? Now go poke around the system and try to figure out what is wrong. What would be the first thing you check? Probably error rate/ratio, the thing that you actually care about. If that is your workflow, why not just alert on your error rate in the first place and figure out the rest as you go along? Edit: typos
- fwip 7y ago
  Hardware-based alerts shouldn't go to the SREs, but rather to the people running the datacenter, who are in charge of making sure that broken hardware is replaced.
  - thethethethe 7y ago
    Yes, and I am sure that they were monitoring this, but is a single rack having issues worth paging someone? Do you think they should immediately send someone out to investigate when temperature anomaly is detected? Like I said elsewhere, this was likely add to a queue of lower priority bugs
    - fwip 7y ago
      Yeah, that makes sense. I fixated on the "SRE" part of your post, rather than the "page in the middle of the night" part. I'm working on being less pedantic, so thanks for reminding me that I've still got work to do. <3
- closeparen 7y ago
thedance 7y ago
Historically, google has chosen to leave these problems to the service layer. Their internal systems detect these events but don’t readily remediate them. I think this is great policy, it’s the reason why Google’s stack has excellent, sophisticated client-side load direction and error avoidance.
technofiend 7y ago
VMware purchased a monitoring tool which was previously called Alive, I think it's part of vrops now. The tool modelled measured values and created channels for expected ranges. Nothing earth shaking but the models would adapt to regular deviations so (as an example) Friday night batch runs didn't trigger CPU alerts for 100% CPU but going that high on a Tuesday might. The tool also had the idea of noise built in so it would suggest that in general three monitors could be out of a channel and might not alert unless six were. Still we very rarely turned on alerting out of the tool because so little of the data was actionable. It was still a huge improvement over the Tivoli system it replaced that had one size fits all alerts which generated tens of thousands of unactionable tickets monthly. The most interesting thing it caught for me was when a system deviated on the low side of a channel and I discovered one of three DNS servers was no longer answering queries. So Google seems to have the right of it (at least for their business) by measuring user experience and reacting to those metrics and leaving infrastructure metrics for post mortem.

bruckie 7y ago

Google's alerting philosophy biases toward paging people only for user-visible symptoms, not potential causes, but there are still often lower priority ticket-based alerts for stuff like stuff getting too hot. For rationale, see: - https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/preview https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa... - https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/ https://landing.google.com/sre/sre-book/chapters/monitoring-...

mjw1007 7y ago
That way of prioritising things makes sense, but in this case it meant the on-call SRE and someone from the edge networking team wasted their time diagnosing the hardware problem. It seems a shame that the blog post wasn't able to follow « They immediately removed ("drained") the machines from serving, thus eliminating the errors that might result in a degraded state for customers » with something like « and they were shown a notification that there was a low-priority automatic ticket showing a possible hardware problem on those machines, so they subscribed to that ticket and didn't waste any more time ».
- thethethethe 7y ago
  > on-call SRE and someone from the edge networking team wasted their time diagnosing the hardware problem I highly doubt that the edge oncaller who was paged debugged this issue down to the hardware. The post even said that they worked with other teams to figure this out. Once the SRE figured out it wasn't their problem, the likely moved onto something else

thethethethe 7y ago

> The temperature spike should have been the first alert. The throttling should have been the second alert. The high error count should have been third. This is a completely backwards. What you are describing is caused based alerting, which is strongly discouraged by SRE. SREs prefer symptom based alerting (e.i are users seeing errors) because you only get alerted when you know that there is a problem that effects your business. If you only have caused based alerts, you will get false alarms and un-actionable pages all of the time. Imagine being an SRE who manages the GFEs in this situation. Now imagine getting a page telling you that the tempurature in a rack where your job is running is hot. What are you supposed to do? Is the issue effecting users? If so, how many? Is it anomalous? Now go poke around the system and try to figure out what is wrong. What would be the first thing you check? Probably error rate/ratio, the thing that you actually care about. If that is your workflow, why not just alert on your error rate in the first place and figure out the rest as you go along? Edit: typos

fwip 7y ago
Hardware-based alerts shouldn't go to the SREs, but rather to the people running the datacenter, who are in charge of making sure that broken hardware is replaced.
- thethethethe 7y ago
  Yes, and I am sure that they were monitoring this, but is a single rack having issues worth paging someone? Do you think they should immediately send someone out to investigate when temperature anomaly is detected? Like I said elsewhere, this was likely add to a queue of lower priority bugs
  - fwip 7y ago
    Yeah, that makes sense. I fixated on the "SRE" part of your post, rather than the "page in the middle of the night" part. I'm working on being less pedantic, so thanks for reminding me that I've still got work to do. <3
closeparen 7y ago

thedance 7y ago

Historically, google has chosen to leave these problems to the service layer. Their internal systems detect these events but don’t readily remediate them. I think this is great policy, it’s the reason why Google’s stack has excellent, sophisticated client-side load direction and error avoidance.

technofiend 7y ago

VMware purchased a monitoring tool which was previously called Alive, I think it's part of vrops now. The tool modelled measured values and created channels for expected ranges. Nothing earth shaking but the models would adapt to regular deviations so (as an example) Friday night batch runs didn't trigger CPU alerts for 100% CPU but going that high on a Tuesday might. The tool also had the idea of noise built in so it would suggest that in general three monitors could be out of a channel and might not alert unless six were. Still we very rarely turned on alerting out of the tool because so little of the data was actionable. It was still a huge improvement over the Tivoli system it replaced that had one size fits all alerts which generated tens of thousands of unactionable tickets monthly. The most interesting thing it caught for me was when a system deviated on the low side of a channel and I discovered one of three DNS servers was no longer answering queries. So Google seems to have the right of it (at least for their business) by measuring user experience and reacting to those metrics and leaving infrastructure metrics for post mortem.