3 ms·
This might be true of SRE roles, but not really true for most engineers, particularly not where I currently work. I wasn't completely clear but what I'd be aski
by mgraczyk 4y ago
This might be true of SRE roles, but not really true for most engineers, particularly not where I currently work. I wasn't completely clear but what I'd be asking for here is how to predict and design for bottlenecks, not how to mitigate active disasters.
Also making a queue bigger is a bad reflexive response to queues being full. I was very involved in the SEV review (postmortem) process at Facebook and witnessed lots of cases where bad situations were made much worse by misguided hasty responses.
- Wagthesam 4y agodo you work at google? most engineering roles ive been in engineers absolutely own their systems and take on call rotations. I agree though, hasty response is the mark of a junior eng
- mgraczyk 4y agoI do, but I don't interview SREs. "Production down" situations are hopefully rare, so the purpose of the interview is to understand if the engineer can plan ahead by adding monitoring, logging, and knobs, and to know how to design for scale. I don't really ask about what to do in a fire because many people don't have that experience.
- yunwal 4y ago> Also making a queue bigger is a bad reflexive response to queues being full. Depends on the function of the queue. If it's a dead-letter queue, there's literally no downside besides a likely-trivial amount of cost. If the problem is upstream of the queue, and the queue is actively feeding well-functioning consumers, yeah, it could make things way worse. This is where being an experienced engineer comes in. Also having 2-person approval like you would for any code changes. Point still stands that you should take low-risk actions to mitigate if they're available to you before root-causing.