6 ms·

I've just been bitten by this too: The broken monitoring hadn’t been noticed partly because this system ‘just worked’ for a long time, and didn’t require any

I've just been bitten by this too: The broken monitoring hadn’t been noticed partly because this system ‘just worked’ for a long time, and didn’t require any change. Any experience on how to deal with it? Who watches the watchers?

nickromano 6y ago
If you use something like CloudWatch, you can set an alarm for “insufficient data” for when a monitor stops reporting.
- EdwardDiego 6y ago
  Yep, and Grafana allows alerting on No Data states.
- harpratap 6y ago
  I don't think it would have been enough for this exact situation. The solution you are providing is for cases where x==nil but in their case there was never an x, so you cannot do x==nil checks unless you deliberately set up x.
  - donavanm 6y ago
    Not quite. Cloudwatch provides for both behaviors. You can treat periods with no datapoints as “breaching” the threshold criteria. You can also independently alert for “insufficient data” where the source is unavailable, doesnt contain enough data to evaluate, etc. In the past I did both. Always emit a 0 datapoint every period + treat missing datapoints as breaching to discover if an application wasnt consistently emitting metrics. In addition a lower severity Insufficient Data alert was used to discover/validate when a meteic stream literally didnt exist (normally through misconfiguration of metric & alarm dimensions). Docs: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...
jacquesm 6y ago
This same sort of thing tends to creep into backup systems that work for a long time without if the backups are never restored. The more reliable the systems they backup the bigger the chance that a restore will fail when you need it most. So test your backups.
- deleted 6y ago
  [deleted]
- thdrdt 6y ago
  Yep. I once worked at a company that had backups of all the data. Until they needed a backup and discovered they were not readable. Months of work were gone.
  - ChrisMarshallNY 6y ago
    This has happened to me. Long story. Incompetent (and dishonest) IT person. In fact, I was attacked for “harassing” the IT person. Nothing was done about it, until the HR DB got borked, and there was no backup. DR and backup are profoundly unpopular topics. They tend to be expensive, and difficult to test. They also presuppose a Very Bad Thing happening, which no one wants to think about. I have scars. I now have multiple layers of backup, and keep an eye on them. I can’t always test every aspect, and have to have faith in cloud providers, but I make sure that, even if we have a tornado, I would still be able to recover the important bits.
    - ripvanwinkle 6y ago
      there's a joke in the data protection industry. Backups always work, it's the restores that have problems
- tinco 6y ago
  Would you consider it evil to use production backups for the QA step in CI/CD? It would catch two birds with one stone, continuously verifying the backups, and ensuring the new code works on real world data. We don't have any personal information in our production database, but even if we did, as long as the QA is thoroughly prevented from interacting with the outside world, it can't hurt to use production data right?
harpratap 6y ago
Chaos Engineering comes into play here. Deliberately break your platform to see if everything works as expected.
- input_sh 6y ago
  To add to that, Netflix has open sourced a tool called Chaos Monkey which randomly terminates production instances so that you can be sure that your overall system is resilient to instance failures.
  - brazzy 6y ago
    I don't think that would have ever provoked this error mode, though.
    - adrianN 6y ago
      It might have prevented this failure by killing stale HAProxy instances...
    - harpratap 6y ago
      It would have. Their haproxy-server-state-management also comes under the scope of Chaos testing. Which was failing but they never set up alerts for it. So they could have seen that their chaos engine killed this process but they never got notified.
- ex_amazon_sde 6y ago
  > Chaos Engineering ...or just "availability testing" or simply "testing". Yet, testing systemic failure modes at scale is way more tricky than shutting down some VMs or some network devices. For example: saturating the uplink bandwidth on a whole datacenter.
- cranekam 6y ago
  But how, exactly? Sure, the chaos monkey could kill haproxy-server-state-management but that wouldn't uncover the bug in question — it'd just demonstrate that without it running HA proxy's view of the world goes stale, which anyone would expect. Triggering the bug would require reducing the number of HA proxy slots below the number of webapps running for many hours. This is clearly something chaos engineering could do but IMO it's highly unlikely anyone would think to do this. If they thought of this they would also have thought about adding tests that caught such as issue long before the code went into production. In my experience chaos engineering is often only as good as the amount of thought put into the things it does. Killing processes here and there can be useful but it often won't expose the kind of when-the-stars-align issues that take down infrastructures. It looks like a classic lack of monitoring, as the article says. Alerting on webapps > slots, early exits, or differing views of the number of webapps up would have likely caught this.
rkagerer 6y ago
There's a reason the military does drills. Do them regularly, and keep your playbook of mock failure scenarios up to date with good coverage of all your systems. It's especially critical for disaster recovery (a DR plan that's never tested isn't worth the paper it's written on). Consider going one step further and randomly injecting artificial failures into production shards, so handling them becomes a regular affair. When you build out monitoring, that might be a good time to think about the kind of stimulus that would be effective at exercising those monitors/metrics/alerts (think unit testing analogy). You can automate an evil adversary bot, or have humans do Red team / Blue team challenges. Yes, you're limited how much you can break without too severely impacting production, but if you've designed enough redundancy you should be able to achieve reasonable coverage. The more you can engineer simulated failures into your regular workflow, the less of a big deal it'll be when real ones occur. Instead of watching the watchers, engage them with stimulus that keeps them sharp and prevents them from getting bored.
- throwaway0a5e 6y ago
  >Do them regularly, and keep your playbook of mock failure scenarios up to date with good coverage of all your systems. All those man hours cost money. Is it worth it? Depends on how much downtime costs you.
  - WJW 6y ago
    Most companies don't know how much downtime costs them and from their behaviour seem to treat it either as $0/hr or $infinite/hr. To be fair, it's difficult to estimate correctly. Apart from the direct loss of income there is also reputation loss, possibly reduced signups and increased churn, etc. It's not at all constant either, one outage per 10 years will give you a radically different reputation than one outage per week.
    - Ntrails 6y ago
      It's also working out which outages matter the most. A 10 day outage is not equivalent to 10 1 day outages, the latter has a multitude of important reputational risks - but the former is quite possibly enough to end the business completely.
huffmsa 6y ago
Chaos engineering. You have to randomly simulate failure cases of all shapes and sizes to be prepared for failure cases of all shapes and sizes. You don't know if your plan will survive contact until it's survived contact. https://medium.com/netflix-techblog/the-netflix-simian-army-16e57fbab116 https://medium.com/netflix-techblog/the-netflix-simian-army-...
waheoo 6y ago
This is what end to end and integration testing is for. You would set up an integration test to trigger the state that results in this alert.
battery423 6y ago
We use Google Cloud Alerting to watch that our alerting works. I would have used some Heartbeatservice but thats not feasible in our environment.
dikei 6y ago
> Who watches the watchers? You would need multiple watchers who watch one another. Of course, if all watchers die at the same time, you're out of luck. This is only part of the solution though, because in order to write and test effective alerting rules, you need to repeatedly and frequently test them with either real or simulated failures, and it can get expensive.
peterwwillis 6y ago
Google goes over this scenario in their SRE books. You literally plan to take down your service so you never have 100% uptime, and literally step through a real failure and recovery. In that process you would go, "Ok, the service is failed and I don't know why. How does the monitoring look?", and then you'd notice it was broken. This is best performed by the person on the team with the least experience, so they ask the most questions, which reveals more. What's really funny is when the recovery you expected to work doesn't work, and then you have bigger problems... But at least you planned for it :)
- cutemonster 6y ago
  What is, according to you, and the SRE book, a good time during the day/week/year to do these real test outages? How much downtime could be ok? For, let's say, a b2b saas They notify the customers first? Like, "we'll a little bit sabotage our, well, your, servers this weekend, to find out if they fail and shutdown completely and cannot start again" :-)
  - peterwwillis 6y ago
    It doesn't have to be a customer impact to bring down a service or parts of a service (if it's designed correctly). But even if it was a customer impact, you can do it at the same time you'd schedule regular maintenance (there's always something that needs maintenance, eventually) and throw up a maintenance page. It's up to the business to define SLAs and SLOs that the customer will be satisfied with.
    - cutemonster 6y ago
      > do it at the same time you'd schedule regular maintenance ... SLA ... SLO I'll do that (some day), thanks
deeteecee 6y ago
haha, curious what the actual context behind that statement is. one thing id ask them is does anyone on that team understand how that system works? also, what's the reason it was broken?
perlgeek 6y ago
One approach is to have two different monitoring checks that could catch the same root cause. In this case, one check could be for max age of actually deployed load balancer config, the other could be for new webapp instances getting traffic. It's unlikely that both monitoring checks fail to trigger, and if only one of them triggers, one should investigate why the other didn't trigger. Of course, that requires some decent amount of awareness from the on-call engineer (and/or an explicit step for that in the playbook); far from perfect.

nickromano 6y ago

If you use something like CloudWatch, you can set an alarm for “insufficient data” for when a monitor stops reporting.

EdwardDiego 6y ago
Yep, and Grafana allows alerting on No Data states.
harpratap 6y ago
I don't think it would have been enough for this exact situation. The solution you are providing is for cases where x==nil but in their case there was never an x, so you cannot do x==nil checks unless you deliberately set up x.
- donavanm 6y ago
  Not quite. Cloudwatch provides for both behaviors. You can treat periods with no datapoints as “breaching” the threshold criteria. You can also independently alert for “insufficient data” where the source is unavailable, doesnt contain enough data to evaluate, etc. In the past I did both. Always emit a 0 datapoint every period + treat missing datapoints as breaching to discover if an application wasnt consistently emitting metrics. In addition a lower severity Insufficient Data alert was used to discover/validate when a meteic stream literally didnt exist (normally through misconfiguration of metric & alarm dimensions). Docs: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

jacquesm 6y ago

This same sort of thing tends to creep into backup systems that work for a long time without if the backups are never restored. The more reliable the systems they backup the bigger the chance that a restore will fail when you need it most. So test your backups.

deleted 6y ago
[deleted]
thdrdt 6y ago
Yep. I once worked at a company that had backups of all the data. Until they needed a backup and discovered they were not readable. Months of work were gone.
- ChrisMarshallNY 6y ago
  This has happened to me. Long story. Incompetent (and dishonest) IT person. In fact, I was attacked for “harassing” the IT person. Nothing was done about it, until the HR DB got borked, and there was no backup. DR and backup are profoundly unpopular topics. They tend to be expensive, and difficult to test. They also presuppose a Very Bad Thing happening, which no one wants to think about. I have scars. I now have multiple layers of backup, and keep an eye on them. I can’t always test every aspect, and have to have faith in cloud providers, but I make sure that, even if we have a tornado, I would still be able to recover the important bits.
  - ripvanwinkle 6y ago
    there's a joke in the data protection industry. Backups always work, it's the restores that have problems
tinco 6y ago
Would you consider it evil to use production backups for the QA step in CI/CD? It would catch two birds with one stone, continuously verifying the backups, and ensuring the new code works on real world data. We don't have any personal information in our production database, but even if we did, as long as the QA is thoroughly prevented from interacting with the outside world, it can't hurt to use production data right?

harpratap 6y ago

Chaos Engineering comes into play here. Deliberately break your platform to see if everything works as expected.

input_sh 6y ago
To add to that, Netflix has open sourced a tool called Chaos Monkey which randomly terminates production instances so that you can be sure that your overall system is resilient to instance failures.
- brazzy 6y ago
  I don't think that would have ever provoked this error mode, though.
  - adrianN 6y ago
    It might have prevented this failure by killing stale HAProxy instances...
  - harpratap 6y ago
    It would have. Their haproxy-server-state-management also comes under the scope of Chaos testing. Which was failing but they never set up alerts for it. So they could have seen that their chaos engine killed this process but they never got notified.
ex_amazon_sde 6y ago
> Chaos Engineering ...or just "availability testing" or simply "testing". Yet, testing systemic failure modes at scale is way more tricky than shutting down some VMs or some network devices. For example: saturating the uplink bandwidth on a whole datacenter.
cranekam 6y ago
But how, exactly? Sure, the chaos monkey could kill haproxy-server-state-management but that wouldn't uncover the bug in question — it'd just demonstrate that without it running HA proxy's view of the world goes stale, which anyone would expect. Triggering the bug would require reducing the number of HA proxy slots below the number of webapps running for many hours. This is clearly something chaos engineering could do but IMO it's highly unlikely anyone would think to do this. If they thought of this they would also have thought about adding tests that caught such as issue long before the code went into production. In my experience chaos engineering is often only as good as the amount of thought put into the things it does. Killing processes here and there can be useful but it often won't expose the kind of when-the-stars-align issues that take down infrastructures. It looks like a classic lack of monitoring, as the article says. Alerting on webapps > slots, early exits, or differing views of the number of webapps up would have likely caught this.

rkagerer 6y ago

There's a reason the military does drills. Do them regularly, and keep your playbook of mock failure scenarios up to date with good coverage of all your systems. It's especially critical for disaster recovery (a DR plan that's never tested isn't worth the paper it's written on). Consider going one step further and randomly injecting artificial failures into production shards, so handling them becomes a regular affair. When you build out monitoring, that might be a good time to think about the kind of stimulus that would be effective at exercising those monitors/metrics/alerts (think unit testing analogy). You can automate an evil adversary bot, or have humans do Red team / Blue team challenges. Yes, you're limited how much you can break without too severely impacting production, but if you've designed enough redundancy you should be able to achieve reasonable coverage. The more you can engineer simulated failures into your regular workflow, the less of a big deal it'll be when real ones occur. Instead of watching the watchers, engage them with stimulus that keeps them sharp and prevents them from getting bored.

throwaway0a5e 6y ago
>Do them regularly, and keep your playbook of mock failure scenarios up to date with good coverage of all your systems. All those man hours cost money. Is it worth it? Depends on how much downtime costs you.
- WJW 6y ago
  Most companies don't know how much downtime costs them and from their behaviour seem to treat it either as $0/hr or $infinite/hr. To be fair, it's difficult to estimate correctly. Apart from the direct loss of income there is also reputation loss, possibly reduced signups and increased churn, etc. It's not at all constant either, one outage per 10 years will give you a radically different reputation than one outage per week.
  - Ntrails 6y ago
    It's also working out which outages matter the most. A 10 day outage is not equivalent to 10 1 day outages, the latter has a multitude of important reputational risks - but the former is quite possibly enough to end the business completely.

huffmsa 6y ago

Chaos engineering. You have to randomly simulate failure cases of all shapes and sizes to be prepared for failure cases of all shapes and sizes. You don't know if your plan will survive contact until it's survived contact. https://medium.com/netflix-techblog/the-netflix-simian-army-16e57fbab116 https://medium.com/netflix-techblog/the-netflix-simian-army-...

waheoo 6y ago

This is what end to end and integration testing is for. You would set up an integration test to trigger the state that results in this alert.

battery423 6y ago

We use Google Cloud Alerting to watch that our alerting works. I would have used some Heartbeatservice but thats not feasible in our environment.

dikei 6y ago

> Who watches the watchers? You would need multiple watchers who watch one another. Of course, if all watchers die at the same time, you're out of luck. This is only part of the solution though, because in order to write and test effective alerting rules, you need to repeatedly and frequently test them with either real or simulated failures, and it can get expensive.

peterwwillis 6y ago

Google goes over this scenario in their SRE books. You literally plan to take down your service so you never have 100% uptime, and literally step through a real failure and recovery. In that process you would go, "Ok, the service is failed and I don't know why. How does the monitoring look?", and then you'd notice it was broken. This is best performed by the person on the team with the least experience, so they ask the most questions, which reveals more. What's really funny is when the recovery you expected to work doesn't work, and then you have bigger problems... But at least you planned for it :)

cutemonster 6y ago
What is, according to you, and the SRE book, a good time during the day/week/year to do these real test outages? How much downtime could be ok? For, let's say, a b2b saas They notify the customers first? Like, "we'll a little bit sabotage our, well, your, servers this weekend, to find out if they fail and shutdown completely and cannot start again" :-)
- peterwwillis 6y ago
  It doesn't have to be a customer impact to bring down a service or parts of a service (if it's designed correctly). But even if it was a customer impact, you can do it at the same time you'd schedule regular maintenance (there's always something that needs maintenance, eventually) and throw up a maintenance page. It's up to the business to define SLAs and SLOs that the customer will be satisfied with.
  - cutemonster 6y ago
    > do it at the same time you'd schedule regular maintenance ... SLA ... SLO I'll do that (some day), thanks

deeteecee 6y ago

haha, curious what the actual context behind that statement is. one thing id ask them is does anyone on that team understand how that system works? also, what's the reason it was broken?

perlgeek 6y ago

One approach is to have two different monitoring checks that could catch the same root cause. In this case, one check could be for max age of actually deployed load balancer config, the other could be for new webapp instances getting traffic. It's unlikely that both monitoring checks fail to trigger, and if only one of them triggers, one should investigate why the other didn't trigger. Of course, that requires some decent amount of awareness from the on-call engineer (and/or an explicit step for that in the playbook); far from perfect.