7 ms·

nobody qualified wants the 24/7 SRE job unless it pays an enormous amount of money. i wouldn't do it for less than 500 grand cash. getting woken up at 3am const

nobody qualified wants the 24/7 SRE job unless it pays an enormous amount of money. i wouldn't do it for less than 500 grand cash. getting woken up at 3am constantly or working 3rd shift is the kind of thing you do with a specific monetary goal in mind (i.e., early retirement) or else it's absolute hell. combine that with ludicrous requirements (the same as a senior software engineer) and you get gaps in coverage. ask yourself what senior software engineer on earth would tolerate getting called CONSTANTLY at 3am, or working 3rd shift. the vast majority of computer systems just simply aren't as important as hospitals or nuclear power plants.

nijave 4y ago
Not only that, but you probably need follow the sun if you want <30 minute response time. Given a system that collects minute-based metrics, it generally takes around 5-10 minutes to generate an alert. Another 5-10 minutes for the person to get to their computer unless it's already in their hand (what if you get unlucky and on-call was taking a shower or using the toilet?). After that, another 5-10 minutes to see what's going on with the system. After all that, it usually takes some more minutes to actually fix the problem. Dropbox has a nice article on all the changes they made to streamline incidence response https://dropbox.tech/infrastructure/lessons-learned-in-incident-management https://dropbox.tech/infrastructure/lessons-learned-in-incid...
mnahkies 4y ago
Timezones are a thing - your 3am is someone's 9am and may be a significant part of your customer base. Being paged constantly is a sign of bad alerts or bad systems IMO - either adjust the alert to accept the current reality or improve the system
- inconceivable 4y ago
  spinning up a subsidiary in another country (especially one with very strict labor laws, like in european countries) is not as easy as "find some guy on the internet and pay him to watch your dashboard. and then give him root so he can actually fix stuff without calling your domestic team, which would defeat the whole purpose. also, even getting paged ONCE a month at 3am will fuck up an entire week at a time if you have a family. if it happens twice a month, that person is going to quit unless they're young and need the experience.
  - chatmasta 4y ago
    It's really not that difficult, and there are providers like Deel who can manage it all for you, to the point you just ACH them every month. Source: co-founder of a remote startup with employees in five countries
    - inconceivable 4y ago
      like you said, timezones are a thing. now you're managing a global team.
      - Godel_unicode 4y ago
        That sounds harder than it is, especially if you already allow remote work. It mostly just forces you to have better docs.
  - mnahkies 4y ago
    Sorry to be clear I was replying to this part of your comment > the vast majority of computer systems just simply aren't as important as hospitals or nuclear power plants. I agree that the stakes are lower in terms of harm, but was trying to express that whilst it might not be life and death, it might be hindering someone being able to do their job / use your product - eg: it still impacts customer experience and your (business) reputation. False pages for transient errors are bad - ideally you only get paged if human intervention is required, and this should form a feedback cycle to determine how to avoid it in future. If all the pages are genuine problems requiring human action then this should feed into tickets to improve things
oulu2006 4y ago
I did that for a few years, and wasn't on 500k a year, but I'm also the company co-founder, so you could argue that a "specific monetary goal" was applicable.
hgsgm 4y ago
Constantly? It's one wakeup in 4 months.
okdood64 4y ago
I've worked two SRE (or SRE-adjacent) jobs with oncall duty (Some unicorn and a FAANG). Neither have been remotely as bad to what you're saying. (Only one was actually 24/7 for a week shift.) The whole point is that before you join, the team has done sufficient work to not make it hell, and your work during business hours makes sure it stays that way. Are there a couple bad weeks throughout the year? Sure, but it's far, far from the norm.

nijave 4y ago

Not only that, but you probably need follow the sun if you want <30 minute response time. Given a system that collects minute-based metrics, it generally takes around 5-10 minutes to generate an alert. Another 5-10 minutes for the person to get to their computer unless it's already in their hand (what if you get unlucky and on-call was taking a shower or using the toilet?). After that, another 5-10 minutes to see what's going on with the system. After all that, it usually takes some more minutes to actually fix the problem. Dropbox has a nice article on all the changes they made to streamline incidence response https://dropbox.tech/infrastructure/lessons-learned-in-incident-management https://dropbox.tech/infrastructure/lessons-learned-in-incid...

mnahkies 4y ago

Timezones are a thing - your 3am is someone's 9am and may be a significant part of your customer base. Being paged constantly is a sign of bad alerts or bad systems IMO - either adjust the alert to accept the current reality or improve the system

inconceivable 4y ago
spinning up a subsidiary in another country (especially one with very strict labor laws, like in european countries) is not as easy as "find some guy on the internet and pay him to watch your dashboard. and then give him root so he can actually fix stuff without calling your domestic team, which would defeat the whole purpose. also, even getting paged ONCE a month at 3am will fuck up an entire week at a time if you have a family. if it happens twice a month, that person is going to quit unless they're young and need the experience.
- chatmasta 4y ago
  It's really not that difficult, and there are providers like Deel who can manage it all for you, to the point you just ACH them every month. Source: co-founder of a remote startup with employees in five countries
  - inconceivable 4y ago
    like you said, timezones are a thing. now you're managing a global team.
    - Godel_unicode 4y ago
      That sounds harder than it is, especially if you already allow remote work. It mostly just forces you to have better docs.
- mnahkies 4y ago
  Sorry to be clear I was replying to this part of your comment > the vast majority of computer systems just simply aren't as important as hospitals or nuclear power plants. I agree that the stakes are lower in terms of harm, but was trying to express that whilst it might not be life and death, it might be hindering someone being able to do their job / use your product - eg: it still impacts customer experience and your (business) reputation. False pages for transient errors are bad - ideally you only get paged if human intervention is required, and this should form a feedback cycle to determine how to avoid it in future. If all the pages are genuine problems requiring human action then this should feed into tickets to improve things

oulu2006 4y ago

I did that for a few years, and wasn't on 500k a year, but I'm also the company co-founder, so you could argue that a "specific monetary goal" was applicable.

hgsgm 4y ago

Constantly? It's one wakeup in 4 months.

okdood64 4y ago

I've worked two SRE (or SRE-adjacent) jobs with oncall duty (Some unicorn and a FAANG). Neither have been remotely as bad to what you're saying. (Only one was actually 24/7 for a week shift.) The whole point is that before you join, the team has done sufficient work to not make it hell, and your work during business hours makes sure it stays that way. Are there a couple bad weeks throughout the year? Sure, but it's far, far from the norm.