3 ms·

Unless you own the service in isolation, chances are something will break at some point beyond your control. If you own the service in isolation, then you may

by lumost 5y ago

Unless you own the service in isolation, chances are something will break at some point beyond your control. If you own the service in isolation, then you may be on-call 24/7 indefinitely. I build high scale systems for a living, but I also hike, ski, and have a family. I can't guarantee 20 minute response times 24/7 365 days per year. I generally won't join a team with an on-call more frequent than once per 6 weeks. Anecdotally, companies fall into a trap where individual engineer's own services that they must be paged for. These engineers don't get the time to make the service reliable, and they also don't want to admit that the service is a tire fire. This works for the company as long as they can continuously pull in new engineers to feed the on-call fire.

nicoburns 5y ago
> Anecdotally, companies fall into a trap where individual engineer's own services that they must be paged for. These engineers don't get the time to make the service reliable I think that’s where I’d draw the line if I was in such a situation. If I’m on the hook for dealing with any outages, then I won’t be working on anything else until the service is reliable.
- lumost 5y ago
  There really is no end to this though, a service can always be made more robust. The reputational damage of missing a page is extremely high. The risk to the company of missing a page can be high (although hopefully not to high if its a solo engineer project). People need breaks.
  - nicoburns 5y ago
    > a service can always be made more robust It can, but I don't think it's too hard to make it robust enough that pages become a rare occurrence. At my company (a small startup with <5 developers - so we're not exactly flush with development capacity) we've managed to reduce outages in a previously quite buggy app to almost nothing over about 18 months (while also developing new features). I don't think we've had an outage in the last year which wasn't due to either: - An outage at our underling hosting provider (which we can't do anything about anyway) - A code deploy (which we can plan and control the timing of)

by lumost 5y ago

nicoburns 5y ago

> Anecdotally, companies fall into a trap where individual engineer's own services that they must be paged for. These engineers don't get the time to make the service reliable I think that’s where I’d draw the line if I was in such a situation. If I’m on the hook for dealing with any outages, then I won’t be working on anything else until the service is reliable.

lumost 5y ago
There really is no end to this though, a service can always be made more robust. The reputational damage of missing a page is extremely high. The risk to the company of missing a page can be high (although hopefully not to high if its a solo engineer project). People need breaks.
- nicoburns 5y ago
  > a service can always be made more robust It can, but I don't think it's too hard to make it robust enough that pages become a rare occurrence. At my company (a small startup with <5 developers - so we're not exactly flush with development capacity) we've managed to reduce outages in a previously quite buggy app to almost nothing over about 18 months (while also developing new features). I don't think we've had an outage in the last year which wasn't due to either: - An outage at our underling hosting provider (which we can't do anything about anyway) - A code deploy (which we can plan and control the timing of)