4 ms·

At the end of the day, most software eats the same stuff and the levers are often rudimentary and grokable, especially if you focus on a specific area. I don't

At the end of the day, most software eats the same stuff and the levers are often rudimentary and grokable, especially if you focus on a specific area. I don't think someone could meaningfully implement a new feature in a night, but could they tackle a problem in a given service and identity a rough remediation to save the next 72h? Yes, this stuff happens all the time. I've jumped on outages like this and figured it out as I went. You're not going to build a new feature but it's probably possible to get oriented enough to avoid catastrophy. People want it all to fail out of spite, and attempt to bring their dreams to life by merely doomsdaying repeatedly and with fervor. We had 4y of Trump and no amount of wishful doomsdaying actually changed anything. The media was full of drama and we told ourselves stories of incredible disfunctions, but at the end of the day the government kept on doing its thing, the country mostly didn't change too much on the fundamentals. Reality is most often more boring than that (save FTX-like events).

gjulianm 4y ago
That’s very naive. We’re talking about a distributed system with thousands of servers, with code from people who might not even work there anymore. Yeah, maybe if a server gets a high disk usage alert because of runaway logs you might fix it… but maybe that fix causes another problem, or maybe the alert is in a system that you don’t even know its purpose. Maybe 72h is just what it takes to you understand the internal tooling and how to do any action there.
- LambdaComplex 4y ago
  > code from people who might not even work there anymore I think that's most of the code, at this point
- AYBABTME 4y ago
  Basic firefighting often revolves around the same set of problems. It's not that naive. Tell me I'm wrong? I work with systems like this. Realistically most problems that show up by themselves (ie not caused by bad code) often have very common mitigations/remedies. There's surely some super complex interaction going on but bugs showing up of this complex nature are usually due to code changes, or scaling events that reach a tipping point. Odds are those aren't the bread and butter of problems faced by Twitter _right now_.
  - gjulianm 4y ago
    > Realistically most problems that show up by themselves (ie not caused by bad code) Well that goes the first assumption. > or scaling events that reach a tipping point Such as having applications running without restarts for a long time due to deploy freezes? > Odds are those aren't the bread and butter of problems faced by Twitter _right now_. Right now, exactly this moment, probably not. But if those were the only worrying events that could happen in the short term, I doubt anyone would need to offer any help to Twitter. > due to code changes Well, Musk has said that he wants to do "hardcore engineering" implying there will be a lot of changes coming up. That surely can't go wrong.
  - brohee 4y ago
    The World Cup is starting Sunday. That usually would be an all hands on decks event at least ops wise, but there are very few hands left and the door to the deck is closed anyway...
- ilyt 4y ago
  In most cases that's enough to keep it running, even if you need to learn 100x of that to start making changes. But if infrastructure wasn't managed like shit it will probably just keep on chugging
koprulusector 4y ago
Dude just getting access / logins you need, client side certs for auth, MFA that’s required by certain systems, SSH keys would take time and be arduous during a regular hire, can you imagine what this would look like trying to show up and have it all in a night for a sev1/p1, when so many (who would manage that access, as others have pointed out in other threads) have been let go or left already? Hell, who even manages IAM or PKI at Twitter at this point?
- AYBABTME 4y ago
  If the problems are "this link is saturated" or "we're out of compute capacity" and other run of the mill stuff, I think getting access, being able to measure, identify and effect change is indeed the tough part. The problems themselves most often have simple short term remedies. I really think people blow out of proportion the actual requirements to keep things running, probably because they wish to see him fail. But everywhere I've worked, systems were most stable when engineers weren't there to muck with them. There's a reason many people refuse to ship on Friday.
vlovich123 4y ago
Are you going to be able to do it under the conditions of massive brain drain? Better hope the engineering and SRE teams had up to date SOPs that work well. Oh, and you’re new. So that means you need to probably start getting access granted to services on the fly first to even see dashboards. But it sounds like that team has also seen massive attrition… Are you sure you can be effective coming completely cold into a company with massive attrition?

gjulianm 4y ago

That’s very naive. We’re talking about a distributed system with thousands of servers, with code from people who might not even work there anymore. Yeah, maybe if a server gets a high disk usage alert because of runaway logs you might fix it… but maybe that fix causes another problem, or maybe the alert is in a system that you don’t even know its purpose. Maybe 72h is just what it takes to you understand the internal tooling and how to do any action there.

LambdaComplex 4y ago
> code from people who might not even work there anymore I think that's most of the code, at this point
AYBABTME 4y ago
Basic firefighting often revolves around the same set of problems. It's not that naive. Tell me I'm wrong? I work with systems like this. Realistically most problems that show up by themselves (ie not caused by bad code) often have very common mitigations/remedies. There's surely some super complex interaction going on but bugs showing up of this complex nature are usually due to code changes, or scaling events that reach a tipping point. Odds are those aren't the bread and butter of problems faced by Twitter _right now_.
- gjulianm 4y ago
  > Realistically most problems that show up by themselves (ie not caused by bad code) Well that goes the first assumption. > or scaling events that reach a tipping point Such as having applications running without restarts for a long time due to deploy freezes? > Odds are those aren't the bread and butter of problems faced by Twitter _right now_. Right now, exactly this moment, probably not. But if those were the only worrying events that could happen in the short term, I doubt anyone would need to offer any help to Twitter. > due to code changes Well, Musk has said that he wants to do "hardcore engineering" implying there will be a lot of changes coming up. That surely can't go wrong.
- brohee 4y ago
  The World Cup is starting Sunday. That usually would be an all hands on decks event at least ops wise, but there are very few hands left and the door to the deck is closed anyway...
ilyt 4y ago
In most cases that's enough to keep it running, even if you need to learn 100x of that to start making changes. But if infrastructure wasn't managed like shit it will probably just keep on chugging

koprulusector 4y ago

Dude just getting access / logins you need, client side certs for auth, MFA that’s required by certain systems, SSH keys would take time and be arduous during a regular hire, can you imagine what this would look like trying to show up and have it all in a night for a sev1/p1, when so many (who would manage that access, as others have pointed out in other threads) have been let go or left already? Hell, who even manages IAM or PKI at Twitter at this point?

AYBABTME 4y ago
If the problems are "this link is saturated" or "we're out of compute capacity" and other run of the mill stuff, I think getting access, being able to measure, identify and effect change is indeed the tough part. The problems themselves most often have simple short term remedies. I really think people blow out of proportion the actual requirements to keep things running, probably because they wish to see him fail. But everywhere I've worked, systems were most stable when engineers weren't there to muck with them. There's a reason many people refuse to ship on Friday.

vlovich123 4y ago

Are you going to be able to do it under the conditions of massive brain drain? Better hope the engineering and SRE teams had up to date SOPs that work well. Oh, and you’re new. So that means you need to probably start getting access granted to services on the fly first to even see dashboards. But it sounds like that team has also seen massive attrition… Are you sure you can be effective coming completely cold into a company with massive attrition?