3 ms·
In my experience this problem arises from a lack of consideration of two related things, and not doing a third thing: 1. Not considering how inevitable develop
by opportune 4y ago
In my experience this problem arises from a lack of consideration of two related things, and not doing a third thing:
1. Not considering how inevitable development issues and bugs will affect the entire system. That is, if a bug gets introduced here, how bad will the systemic effects be, and how do we prevent it? A severe case is if you start writing bad data from one component that then needs to be manually backfilled or reverted (especially if this then generates yet more bad data further down the line).
2. Not considering how entire-system failures will present themselves, and not having a good way to diagnose them.
3. Failure to develop testing (integration or whole-system blackbox) that catches the first case, failure to develop tooling (tracing, logging, synchronizing changes across components) that assists in the second case. Or instead adjusting the system so that the first or second cases are considered.
It’s easy to get stuck in a local optimum where a few old hats who understand the entire system are the only ones capable of predicting and diagnosing failures across components. It’s also easy to say that less skilled or new engineers just need to put in the time to get good, but it’s often the case that the old hats have a potentially-automatible procedure for narrowing down the problem or tracking where the issue came from, or that the benefits of separate components don’t justify the increased rate of bugs and time spent tracking them down to fix them.