5 ms·

Wow what? If I'm building a system with tight uptime requirements, I make sure I build things that stay up. Product is the front line for pages with whoever is

Wow what? If I'm building a system with tight uptime requirements, I make sure I build things that stay up. Product is the front line for pages with whoever is on the on-call rotation. If a mission critical system I built fails in the middle of the night, I expect to be paged and to hop on and fix it. I'm very well compensated for that, and have historically been more than happy to fire the contract "support" teams in India or wherever that are trained to follow runbooks to manually fix problems that should self-heal if systems were engineered correctly. On-call is just a way for me, an overpaid engineer who cares about craftsmanship, to ensure that problems serious enough to warrant a page (infrequent) are dealt with swiftly, and followed up on with an in depth post mortem to ensure they don't happen again. To be clear I consider getting paged more than two or three times a year unacceptable. Any compamy with on-call should track life interruptions as a KPI and actively work to drive their frequency to zero.

lumost 5y ago
Unless you own the service in isolation, chances are something will break at some point beyond your control. If you own the service in isolation, then you may be on-call 24/7 indefinitely. I build high scale systems for a living, but I also hike, ski, and have a family. I can't guarantee 20 minute response times 24/7 365 days per year. I generally won't join a team with an on-call more frequent than once per 6 weeks. Anecdotally, companies fall into a trap where individual engineer's own services that they must be paged for. These engineers don't get the time to make the service reliable, and they also don't want to admit that the service is a tire fire. This works for the company as long as they can continuously pull in new engineers to feed the on-call fire.
- nicoburns 5y ago
  > Anecdotally, companies fall into a trap where individual engineer's own services that they must be paged for. These engineers don't get the time to make the service reliable I think that’s where I’d draw the line if I was in such a situation. If I’m on the hook for dealing with any outages, then I won’t be working on anything else until the service is reliable.
  - lumost 5y ago
    There really is no end to this though, a service can always be made more robust. The reputational damage of missing a page is extremely high. The risk to the company of missing a page can be high (although hopefully not to high if its a solo engineer project). People need breaks.
    - nicoburns 5y ago
      > a service can always be made more robust It can, but I don't think it's too hard to make it robust enough that pages become a rare occurrence. At my company (a small startup with <5 developers - so we're not exactly flush with development capacity) we've managed to reduce outages in a previously quite buggy app to almost nothing over about 18 months (while also developing new features). I don't think we've had an outage in the last year which wasn't due to either: - An outage at our underling hosting provider (which we can't do anything about anyway) - A code deploy (which we can plan and control the timing of)

lumost 5y ago

Unless you own the service in isolation, chances are something will break at some point beyond your control. If you own the service in isolation, then you may be on-call 24/7 indefinitely. I build high scale systems for a living, but I also hike, ski, and have a family. I can't guarantee 20 minute response times 24/7 365 days per year. I generally won't join a team with an on-call more frequent than once per 6 weeks. Anecdotally, companies fall into a trap where individual engineer's own services that they must be paged for. These engineers don't get the time to make the service reliable, and they also don't want to admit that the service is a tire fire. This works for the company as long as they can continuously pull in new engineers to feed the on-call fire.

nicoburns 5y ago
> Anecdotally, companies fall into a trap where individual engineer's own services that they must be paged for. These engineers don't get the time to make the service reliable I think that’s where I’d draw the line if I was in such a situation. If I’m on the hook for dealing with any outages, then I won’t be working on anything else until the service is reliable.
- lumost 5y ago
  There really is no end to this though, a service can always be made more robust. The reputational damage of missing a page is extremely high. The risk to the company of missing a page can be high (although hopefully not to high if its a solo engineer project). People need breaks.
  - nicoburns 5y ago
    > a service can always be made more robust It can, but I don't think it's too hard to make it robust enough that pages become a rare occurrence. At my company (a small startup with <5 developers - so we're not exactly flush with development capacity) we've managed to reduce outages in a previously quite buggy app to almost nothing over about 18 months (while also developing new features). I don't think we've had an outage in the last year which wasn't due to either: - An outage at our underling hosting provider (which we can't do anything about anyway) - A code deploy (which we can plan and control the timing of)