3 ms·
Google goes over this scenario in their SRE books. You literally plan to take down your service so you never have 100% uptime, and literally step through a real
by peterwwillis 6y ago
Google goes over this scenario in their SRE books. You literally plan to take down your service so you never have 100% uptime, and literally step through a real failure and recovery.
In that process you would go, "Ok, the service is failed and I don't know why. How does the monitoring look?", and then you'd notice it was broken. This is best performed by the person on the team with the least experience, so they ask the most questions, which reveals more.
What's really funny is when the recovery you expected to work doesn't work, and then you have bigger problems... But at least you planned for it :)
- cutemonster 6y agoWhat is, according to you, and the SRE book, a good time during the day/week/year to do these real test outages? How much downtime could be ok? For, let's say, a b2b saas They notify the customers first? Like, "we'll a little bit sabotage our, well, your, servers this weekend, to find out if they fail and shutdown completely and cannot start again" :-)
- peterwwillis 6y agoIt doesn't have to be a customer impact to bring down a service or parts of a service (if it's designed correctly). But even if it was a customer impact, you can do it at the same time you'd schedule regular maintenance (there's always something that needs maintenance, eventually) and throw up a maintenance page. It's up to the business to define SLAs and SLOs that the customer will be satisfied with.
- cutemonster 6y ago> do it at the same time you'd schedule regular maintenance ... SLA ... SLO I'll do that (some day), thanks