4 ms·

You can just do this yourself. Google breaks it's systems intentionally for a week every year, it's called DiRT week. DiRT takes weeks of planning before people

You can just do this yourself. Google breaks it's systems intentionally for a week every year, it's called DiRT week. DiRT takes weeks of planning before people even start debugging. Doing this constantly for a all products in a single region would be absolutely exhausting for SRE teams (Discliamer: I work for ^GOOG and my opinions are my own)

eru 6y ago
That is a good idea, but by yourself you can not go and yank the power cable out of an AWS box. (I also used to work for Google, as an SRE. Fun times. Many of our dirt exercises usually included the stipulation: 'by the way, assume that <scarily over-competent coworker> is on vacation and not to be disturbed'.)
- thethethethe 6y ago
  > That is a good idea, but by yourself you can not go and yank the power cable out of an AWS box. Sure you can. Simulate an Ec2 outage? Turn off your VMs. Lambda down? Turn off your serving version. Etc... I think the only things that difficult to simulate would be control plane outages for testing CI/CD pipelines--though CI/CD pipelines generally are not production critical and probably aren't worth testing directly. If you want to test recovery/repair operations when CI/CD is down you can just prohibit the victim from using it to fix things. For non-automated API calls, you could just say that X API is down for the duration of the test
  - eru 6y ago
    See the other comments. It's not a given that yanking turning off these services gracefully via API is equivalent to yanking a power cable.
Roritharr 6y ago
I remember a time when deployments to large web applications took weeks of planning. I don't work at Google but I imagine just working on making it a dedicated goal to have a continuously faulty region would reduce the burden of everyone involved.
- thethethethe 6y ago
  > I don't work at Google but I imagine just working on making it a dedicated goal to have a continuously faulty region would reduce the burden of everyone involved This sounds nice in theory but I think you are underestimating the vast complexity of large cloud platform internal systems. The amount of work to make this happen reliablely would be immense. You would probably need a few people from each cloud SRE team with some SWE partners working for a year or longer to get this up and running. Then after that we would need to work out all of the bugs caused by these contrived failure modes and build automation to repair damage. It's just too much work to justify when there is so much work to be done for regular prod already

eru 6y ago

That is a good idea, but by yourself you can not go and yank the power cable out of an AWS box. (I also used to work for Google, as an SRE. Fun times. Many of our dirt exercises usually included the stipulation: 'by the way, assume that <scarily over-competent coworker> is on vacation and not to be disturbed'.)

thethethethe 6y ago
> That is a good idea, but by yourself you can not go and yank the power cable out of an AWS box. Sure you can. Simulate an Ec2 outage? Turn off your VMs. Lambda down? Turn off your serving version. Etc... I think the only things that difficult to simulate would be control plane outages for testing CI/CD pipelines--though CI/CD pipelines generally are not production critical and probably aren't worth testing directly. If you want to test recovery/repair operations when CI/CD is down you can just prohibit the victim from using it to fix things. For non-automated API calls, you could just say that X API is down for the duration of the test
- eru 6y ago
  See the other comments. It's not a given that yanking turning off these services gracefully via API is equivalent to yanking a power cable.

Roritharr 6y ago

I remember a time when deployments to large web applications took weeks of planning. I don't work at Google but I imagine just working on making it a dedicated goal to have a continuously faulty region would reduce the burden of everyone involved.

thethethethe 6y ago
> I don't work at Google but I imagine just working on making it a dedicated goal to have a continuously faulty region would reduce the burden of everyone involved This sounds nice in theory but I think you are underestimating the vast complexity of large cloud platform internal systems. The amount of work to make this happen reliablely would be immense. You would probably need a few people from each cloud SRE team with some SWE partners working for a year or longer to get this up and running. Then after that we would need to work out all of the bugs caused by these contrived failure modes and build automation to repair damage. It's just too much work to justify when there is so much work to be done for regular prod already