3 ms·
> While our systems would automatically scale in response to increased load, the sudden increase in queue depth would outpace our ability to scale up, resulting
by spyspy 8y ago
> While our systems would automatically scale in response to increased load, the sudden increase in queue depth would outpace our ability to scale up, resulting in delays for the newest events.
This strikes me as the core of their problem, and every step taken was a way to bandaid this limitation. Would the cost of moving to faster-scaling infrastructure been so high as rearchitecting the entire system?
> When we wanted to deploy a change, we had to spend time fixing the broken test even if the changes had nothing to do with the initial change.
This seems like a separate and even larger problem. Changes are breaking tests for unrelated code areas? Is the code too tightly coupled? Sounds like it. The unit tests are doing exactly what they're designed to do. Hard to feel sympathy for the person who's breaking them and then trying to figure out a way to sidestep them rather than fix the underlying issues.
- dasil003 8y agoThe job of these services is to transform their internal event format to 140 different output formats. You can imagine that there is a lot of duplication in the functionality that these services need to do. Are you suggesting that they avoid any shared libraries and just rewrite the same code over and over hundreds of times and update them independently?
- spyspy 8y agoIf making a change to a shared library breaks half the services, should it have been shared in the first place? It still smells odd to me there’s so much interdependence amongst the individual services break at will.
- dasil003 8y agoMaybe, but it’s conceivable to me that with 140 endpoints growing organically it would be very hard for an individual engineer to know for sure what should be abstracted or not. I think it’s a fundamentally hard problem even though on the surface it seems like it should be simple. Adding arbitrary new unaffiliated services is exactly the kind of thing that leads to irreducible complexity and a moving target that is very difficult to design for.