4 ms·

Burning out from on-call comes from not being able to fix the underlying causes. I'm happy to hold the pager if I've also got the right to block/rollback deplo

Burning out from on-call comes from not being able to fix the underlying causes. I'm happy to hold the pager if I've also got the right to block/rollback deploys until the system is stable - my current job has had two out-of-hours pages in the past year, and we're in the alexa top 10k so it's not like there's no traffic.

taylodl 6y ago
Bingo! There's nothing more motivating to "prioritize" fixing a problem than being woken up at 2 a.m. being affected by it. I've worked in places where one team writes the code and another team supports it and it's always the same: the code is absolute garbage. It ends up being too complex and having way too many moving parts, and is impossible to diagnose. I've worked in places where the team who writes it, supports it - 24x7. Their code is always super simple, easy to diagnose, and easy to maintain. I've been at this for 35 years at many companies and working with many teams and it's always the same: if you want good software then make the team creating it also support it. In every case I've experienced it leads to software requiring little to no support, easy to maintain, and easy to extend. Why? Because nobody wants to get up in the middle of the night or work weekends and moreover, they'd rather be adding features than limping along with existing features.
mjayhn 6y ago
I'm in ops and this is a huge, huge headache for those of us on our end at companies where we're not given Google SRE powers to control release trajectory based on failure budgets of some sort. I've worked at too many places that had no SWEs on-call for the on-call alerts that I get, which the vast majority of the time involves throwing a bandaid (as in redirecting traffic, etc) in front of an internal bug that I hope eventually gets fixed once the RFO/etc has been submitted before it hits my NEXT on-call rotation or my poor coworkers. Without SWEs on my rotation they don't understand the immediacy. They aren't the ones getting their Christmas week interrupted every 4 hours while ops keeps the house running. In Ops having your entire day ruined by various on-call alerts usually feels like you're working without any breaks and nobody even cares. Anyone want a bad golang developer, wannabe ex-ops person who knows a lot about platform reliability and o11y and wants to focus on the golang end finally? I'll make your teams automation and o11y purr no matter where it is (bm, cloud, global pops, serverless..)..

taylodl 6y ago

Bingo! There's nothing more motivating to "prioritize" fixing a problem than being woken up at 2 a.m. being affected by it. I've worked in places where one team writes the code and another team supports it and it's always the same: the code is absolute garbage. It ends up being too complex and having way too many moving parts, and is impossible to diagnose. I've worked in places where the team who writes it, supports it - 24x7. Their code is always super simple, easy to diagnose, and easy to maintain. I've been at this for 35 years at many companies and working with many teams and it's always the same: if you want good software then make the team creating it also support it. In every case I've experienced it leads to software requiring little to no support, easy to maintain, and easy to extend. Why? Because nobody wants to get up in the middle of the night or work weekends and moreover, they'd rather be adding features than limping along with existing features.

mjayhn 6y ago

I'm in ops and this is a huge, huge headache for those of us on our end at companies where we're not given Google SRE powers to control release trajectory based on failure budgets of some sort. I've worked at too many places that had no SWEs on-call for the on-call alerts that I get, which the vast majority of the time involves throwing a bandaid (as in redirecting traffic, etc) in front of an internal bug that I hope eventually gets fixed once the RFO/etc has been submitted before it hits my NEXT on-call rotation or my poor coworkers. Without SWEs on my rotation they don't understand the immediacy. They aren't the ones getting their Christmas week interrupted every 4 hours while ops keeps the house running. In Ops having your entire day ruined by various on-call alerts usually feels like you're working without any breaks and nobody even cares. Anyone want a bad golang developer, wannabe ex-ops person who knows a lot about platform reliability and o11y and wants to focus on the golang end finally? I'll make your teams automation and o11y purr no matter where it is (bm, cloud, global pops, serverless..)..