4 ms·
This may come across snarky, but it’s unintended. Instead I’m trying to set out a sort of mental framework for answering the question from (slightly less than)
by ineedasername 4y ago
This may come across snarky, but it’s unintended. Instead I’m trying to set out a sort of mental framework for answering the question from (slightly less than) first principles.
That’s a bit like asking “why does any large internet service need infrastructure maintenance?”
If you can’t immediately see the answer in twitter’s case, then you either haven’t yet applied your knowledge of how large cloud services function and need maintenance to keep from cascading failures, or you don’t have that knowledge in which case the twitter case can’t be answered without explaining first the general case of how any large software system needs regular oversight. But then, I think that even if you can’t think of an answer for Twitter, you can probably figure out a little of the answer for the general question.
You can start with asking yourself another question: why would Twitter (or any very large software project) pay $Millions for maintenance that isn’t required? Is there collective delusion on the topic? Or has twitter hit upon the absolute perfectly engineered platform ever? If the answer to either of those is “no” then a twitter system failure is only a matter of time.
How much time is a matter of some debate. The first time something hits a quota or something of that sort that requires a human input to ensure automatic allocations don’t accidentally get insane could be a pebble. Or maybe the first pebble is a minor unpatched security flaw, but twitter is a very sweet high profile target… whatever the first pebble is, it makes a slightly worse issue or multiple additional pebbles much easier, and you enter cascading failure. They generally happen slowly relative to the proportion of a total collapse, and then very very quickly.
So another question to ask is how much of twitter’s infra teams were pure bloat, how much were excess capacity necessary to cover average amounts of employee vacation/sick/leave that goes on at any time, etc, and how elastic the remaining staff can be (putting in extra hours, working in areas that are secondary skills because no one else is available) and for how long they’re willing to do it before burnout or better job offers with less stress and overtime, whether staff losses have already hit a critical mass to make this inevitable…
But I think this gives a general sense of why it can’t tick over forever on its own, or at least some inroads into thinking about the answer.