3 ms·

To me, this depends on the different failure domains and how they interact with each other and based off of that, it can be decided if the teams should coordina

by tetha 3y ago

To me, this depends on the different failure domains and how they interact with each other and based off of that, it can be decided if the teams should coordinate changes, communicate changes or just run. For example we've had situations in which... creative user and amazing code managed to fry both the dedicated elasticsearch cluster for the application as well as the dedicated database, and the application servers were also weird. Here, we'd split into three small teams each responsible for each of these pieces. And looking at these failure domains - ES and postgres don't interact with each other, so both of these teams should just run independently as fast as they can to get their components working again and inform the app team when they are back up. However, the application team should closely coordinate their actions with both of these teams - I've had enough situations in which someone pressed the "hilarious load on ES" button while people were still getting ES up to capacity... and down we go again. Or in a similar way, we had a central database outage and a couple dozen apps got taken out. Database in the lead, sure, but once that's going again, the different application teams can run free and make changes with loose coordination with the database team. However, within the same failure domain, I really don't like to make too many changes in parallel. Getting back up ASAP is a priority, for sure, but what about the outage in 2 hours, or tomorrow because we just hit the system with a couple of wrenches and have no idea why it's back up? Here I strongly prefer deliberate, individual changes so we can get an idea why the system failed. 10 more minutes of downtime / degraded service now can safe us many nerves and downtime hours over the next few days.

BryantD 3y ago
That makes sense. I am slightly envious of the idea of having clearly discernible failure domains but that’s a me problem. The coordination can get hairy at a huge company but that’s why (for example) Amazon has a really challenging program for becoming an incident manager, which creates a pool of people who can understand failure domains quickly for incidents which span a lot of the stack/company.
- tetha 3y ago
  This is a constant architectural struggle, I tell you. Like, inside the infrastructure, I'm constantly updating, poking teams and such to make sure we have clean disaster recovery layers and we have clean documentation about the dependencies between our infrastructural services. And to make sure we don't have cycles - or at least we should have good documentation about these cycles and how to handle them. And to make sure your services don't intermingle too crazily. We should have our postgres bubble, and this should depend on the consul bubble through a clean interface. And similarly, I'm constantly telling our dev-teams that even though we have micro services and everyone is entirely free to do whatever they want within their team (within reason), we should have clean boundaries between these services. Clear interfaces at an HTTP, AMQP and gRPC layer - and dependencies between services maintained by different groups of developers should go through these dependencies. If you want to share something that's currently an internal service of some application or system, don't just depend on it because you can - that will get annoying for /both/ teams. Rather separate it out cleanly into a new service with a defined interface, and share that. Our infrastructure should be a DAG of small-ish, independently fixable, deployable and functional bubbles, not a huge ball of mud with everything going everywhere. It's a constant struggle, but I think it improves both our architecture at a development level, overall dev experience because of better separations of responsiblities and our stability because we can structure incident responses better. Thank you for reading my ted-talk.

by tetha 3y ago

BryantD 3y ago

That makes sense. I am slightly envious of the idea of having clearly discernible failure domains but that’s a me problem. The coordination can get hairy at a huge company but that’s why (for example) Amazon has a really challenging program for becoming an incident manager, which creates a pool of people who can understand failure domains quickly for incidents which span a lot of the stack/company.

tetha 3y ago
This is a constant architectural struggle, I tell you. Like, inside the infrastructure, I'm constantly updating, poking teams and such to make sure we have clean disaster recovery layers and we have clean documentation about the dependencies between our infrastructural services. And to make sure we don't have cycles - or at least we should have good documentation about these cycles and how to handle them. And to make sure your services don't intermingle too crazily. We should have our postgres bubble, and this should depend on the consul bubble through a clean interface. And similarly, I'm constantly telling our dev-teams that even though we have micro services and everyone is entirely free to do whatever they want within their team (within reason), we should have clean boundaries between these services. Clear interfaces at an HTTP, AMQP and gRPC layer - and dependencies between services maintained by different groups of developers should go through these dependencies. If you want to share something that's currently an internal service of some application or system, don't just depend on it because you can - that will get annoying for /both/ teams. Rather separate it out cleanly into a new service with a defined interface, and share that. Our infrastructure should be a DAG of small-ish, independently fixable, deployable and functional bubbles, not a huge ball of mud with everything going everywhere. It's a constant struggle, but I think it improves both our architecture at a development level, overall dev experience because of better separations of responsiblities and our stability because we can structure incident responses better. Thank you for reading my ted-talk.